Re: Balance Line Stability - Development

Do you agree that this one should not show a 99.54 for stabilty?

I completely agree with that.

52 (edited by geektrader 2019-01-03 23:56:35)

Re: Balance Line Stability - Development

DAMN, it seems it was my browsers cache still!! It seems to have cached an old version of the app, although I HAVE cleared the cache. I´ve now gone to it´s /AppData/Roaming/ directory and deleted everything manually there, then reloaded EA Studio and NOW it shows the right results for this one!:

https://i.imgur.com/ca2Ci5M.jpg

Shame on me and sorry for wasting your time! Doing a few more tests just to see if it works all right now.

Re: Balance Line Stability - Development

It seems to be working perfectly fine now! Just did a few optimizations and now it all makes sense and looks correct. Sorry again for this, I was wondering too already why this is so odd although I´ve cleared the cache and refreshed the app, but only deleting the user directory really refreshed EA Studio to the correct version. I will report back if I see anything else, but I think this is perfect now (especially optimizing for the stability looks GREAT and right!). Thank you for all the efforts :-)

Re: Balance Line Stability - Development

I'm not satisfied. It looks like the result depends on the number of deals. The more deals, the higher result it shows. It may be statistically correct, however it makes no sense similar looking IS and OOS parts to have very different Stability result. The previous formula were less depended on the count of deals.

55 (edited by geektrader 2019-01-04 01:03:45)

Re: Balance Line Stability - Development

That might be true, but it gave completely unuseful results (just see the screenshots on page #1, haha) ;-) So you mean that if testing the same period that previously was OOS is not the same IS afterward? I can test that too, just a moment...

Re: Balance Line Stability - Development

I uploaded a new version with fixed OOS Stability. The problem was not in the count of deals but of incorrect index of the OOS transactions. It must be better now.

Please reload and test.

57 (edited by geektrader 2019-01-04 01:13:08)

Re: Balance Line Stability - Development

Ah I see what you mean:

OOS view:

https://i.imgur.com/QVPPseg.jpg

Same data-range IS:

https://i.imgur.com/2UQPDMa.jpg

But that must be a implementation issue, as I´ve just tested that in SQ as well and the values match there. For whatever reason, you are calculating it differently for OOS than if using the same period IS. To make this correct, you´d have to separately calculate the stability for the OOS-view in an extra step, you can´t just take the "remaining values" of the full backtest for the stability in that case, this can´t work as you´ve already noticed.

58 (edited by geektrader 2019-01-04 01:18:46)

Re: Balance Line Stability - Development

Popov wrote:

I uploaded a new version with fixed OOS Stability. The problem was not in the count of deals but of incorrect index of the OOS transactions. It must be better now.

Please reload and test.

Yes, now it matches. In the above examples I´ve posted I am now getting 87.41 IS and 87.46 OOS on the same period tested. Great :-) Almost looks like we can close "the case" now :-) I will test some more during the night while I set up a new build for 28 symbols that will heavily rely on the new balance line stability in many aspects. I will report back tomorrow how the results I am getting look and if everything is all right, but right now it seems like it does - great!! :-) Thanks!

Really hoping for the extended amount of bars supported next, so my history data does stop being cut off at 200.000 bars finally, hehe ;-)

Great work Popov, really appreciate it!

59 (edited by geektrader 2019-01-04 04:33:45)

Re: Balance Line Stability - Development

Popov wrote:

I uploaded a new version with fixed OOS Stability. The problem was not in the count of deals but of incorrect index of the OOS transactions. It must be better now.

Please reload and test.


Ah, bummer, after this fix with OOS / IS, the calculation for the whole backtest now is flawed again. Look at this one, this previously had a Balance line stability of almost 100 and as you can see, this would be right, as it follows almost a straight line. But now it only has a 83, which makes no sense, as you might agree:

https://i.imgur.com/yShjkKg.jpg

Just for fun I´ve then run the above strategy through the Optimizer and let it optimize for the best Balance line stability. This is what it came up with:

https://i.imgur.com/pxzk6mQ.jpg

As you can see, the equity line now looks much worse and less straight after optimizing for best Balance line stability, but EA Studio says it has a higher Balance line stability than the initial strategy. Doesn´t make much sense either, right? :-)

This time I´ve made sure the latest version of the app is loaded and nohting is cached. Hope you can fix this. Thank you :-)

60 (edited by geektrader 2019-01-04 19:29:19)

Re: Balance Line Stability - Development

Good evening :-)

Just checked and refreshed EA Studio, but I still get the above results. Really would like to setup a long EA Studio run for the weekend, any chance you can get the formula fixed this evening? It was already showing the correct results right before the OOS / IS fix you did yesterday evening. But it´s clear that the above strategy #1 does not only have a "83", given that it is almost a straight line. I am here for testing now too, so can respond right away if you do a new version.

Thank you so much.

Re: Balance Line Stability - Development

I rewrote the Balance line stability from scratch. I'm using now a standard r-squared formula.

I'm experimenting with two variants:
- taking into account all traders or only exit (SL, TP, Close) trades.
- scaling the value.

I made many experiments by testing r-squared, Pearson Correlation to a regression line, various combinations of both...
Unfortunately there is no perfect solution.
Finally I decided to implement only r-squared because it is relatively fast for calculation and well documented.

I'll upload the new version of EA Studio within 2 hours.

Re: Balance Line Stability - Development

Hi Popov,

the formula you´ve used yesterday right before the fix with OOS / IS gave great results actually (and accurate ones), I guess that just something got messed up during the OOS / IS fix, but sure, R-Squared is most likely the best idea, every trading platform is using it with some variations.

In terms of variants, surely it would make the most sense to follow the trades as close as possible to get accurate results. But let´s test once you have it uploaded. Waiting for your message here and I will be testing right away all night ;-)

Thanks.

Re: Balance Line Stability - Development

EA Studio is updated.

Balance line stability uses now r-squared (actually r powered by 5 instead of 2).

It must calculate correctly because I compared the values with  some statistical calculators.

r-squared has some drawbacks and may not work as expected in all cases, but it show what is calculated. Later I'll announce this stats  under separate metrics called - "R-Squared" and will continue working on the Balance line Stability as a combination of R-Squared, correlation and better scaling.

You may expect a new version on Tuesday.

Please report only crashes.

The complete code for the calculation is here:https://gist.github.com/PopovMP/9068c8b … dc7e67d8f6

64 (edited by geektrader 2019-01-04 21:53:08)

Re: Balance Line Stability - Development

Looks good at a first test, the problematic strategy I´ve posted a few posts ago that had an almost perfectly straight equity line but just showed "83", now shows "99.26", which is correct, as it is almost a straight line, testing extensively now... If you plan to place this separately as "R-Squared" into the metrics, I would recommend to name it "R-Squared" right now already and re-add "Balance line stability" when you´ve finished it again with your own formula, whenever that is. This way config-files will not get messed up for your users.

Seeing that you´ve placed the code on Github, does this mean we can now somehow code our own metrics in JS and insert them into EA Studio? That would be cool!

Re: Balance Line Stability - Development

Actually you are right.

I'll introduce the new option "R-Squared" within an hour and will return the previous stability formula.

Re: Balance Line Stability - Development

Great! Now the new R-Squared is working right. Look at this.

Initial strategy:

https://i.imgur.com/VEVtFpy.jpg

Optimized for R-Squared:

https://i.imgur.com/Y6yihVV.jpg


As we can see, the values look right (97.X before, 99.X after) and the equity line is more straight too - that all makes sense now. Will run some more tests...

Re: Balance Line Stability - Development

Now we have both: Balance line stability and R - squared in sorting, filtering, Acceptance Criteria, Search best...

https://image-holder.forexsb.com/store/eas-r-squared-thumb.png

Have a great weekend!

Re: Balance Line Stability - Development

Great, just tested that too, but look at how different the values are for this almost perfect equity curve. Why would Balance line stability only show 87? It´s so close to a straight line that it can´t get much better. R-squared shows that correct with 97. But as I understood, you are going to rework the formula for Balance line stability, right?

https://i.imgur.com/McdPR3l.jpg

Have a nice weekend too :-)

Re: Balance Line Stability - Development

But as I understood, you are going to rework the formula for Balance line stability, right?

It is now an adopted version of the code you sent me, however it doesn't work very well as it is now. I'll try to fix and improve it for our case.

Re: Balance Line Stability - Development

Odd, but surely we will see. We have a well working R-Squared for now at least. And as I´ve mentioned the "Balance line stability" showed good values right before you did that OOS / IS fix yesterday (the one where you said the index for the OOS was wrong - right after that Balance line stability began to show these strange values of 83 instead of 97 it showed before for the almost perfect strategy).

I also just had a crash of EA Studio now during optimization after you´ve re-added Balance line stability. Will try to debug the cause...

Re: Balance Line Stability - Development

Seems I can´t reproduce the crash right now, but I´ll let you know if it happens again.

72 (edited by geektrader 2019-01-04 23:18:37)

Re: Balance Line Stability - Development

Just crashed again (Firefox, never crashed before with EA Studio), but I can´t see any output in it´s "Browser Console" that would explain the crash. What´s also odd: I had optimized for R-Squared and it ended up with a value of 99.02. Once going back to the editor (keeping the changes), it jumped down to 98.52. Seems like there is some bug somewhere...

Re: Balance Line Stability - Development

I'm running the latest version with Reactor and R-squared optimization on two machines and no errors or crashes were noted.

If it happens and I can reproduce it I'll be able to fix it immediately. I'm keeping the browser's Console open for any case.

74 (edited by geektrader 2019-01-04 23:28:32)

Re: Balance Line Stability - Development

Yea me too, but nothing in there, the tab just fully crashes, but no logs.

75 (edited by geektrader 2019-01-05 00:59:51)

Re: Balance Line Stability - Development

Hmm, I get the feeling that there is something wrong if optimizing for R-Squared. Could it be that R-Squared in the Optimizer is accidentally linked to the Balance line stability and is really optimizing for that instead of R-Squared? To me, it seems like that is the case. Can you check, please?

E.g.: I select R-Squared as the target for the Optimizer and it optimizes fine, but I somehow get the feeling that it actually optimizes for the highest Balance line stability value instead of the highest R-Squared value, like if there is a wrong link in your code and R-Squared points to Balance line stability during the optimization, so that in reality the optimizer, while correctly calculating R-Squared of course, is really optimizing for the maximum Balance line stability. I think this happened when you´ve re-added / renamed R-Squared *and* Balance line stability this evening, seems like the R-Squared optimizer target is acutally pointing to Balance line stability.