Topic: Recalculating strategies ending with very different results

Hi Popov,

I have had this happen a few times now where I generate a number of strategies using the reactor over a certain data horizon, I then remove the data horizon to see how the startegies performed in an OOS type scenario but using past data rather than future unseen data. I then retick the data horizon box, open collection and click recalculate. On a few straegies I end up with very diffeerent results to what they were previously before touching the data horizon. Here is a few screenshots of some strategies that i had issues with today as well as the opimiser acceptance criteria a strategy had to meet to be generated. As you can see it should be impossible for these strategies to have past validation. It was only after touching the data horizon settings then putting setting the horizon back to its original setting I have these results.

I have checked the data set in question, released the data, reloaded. Everything looks OK no holes in the data and the stats make sence.

Post's attachments

faulty 2.JPG 70.44 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

Re: Recalculating strategies ending with very different results

Another screenshot
https://i.postimg.cc/2LkCG1hG/faulty1.jpg

Re: Recalculating strategies ending with very different results

Does anyone have any ideas?

Re: Recalculating strategies ending with very different results

Hi :-)

I´ve not really experienced this to that degree - as far as I understand some strategies actually have no more trades at all after re-enabling all data but they previously had trades? The only time it was different for me is because of the virtual trading account exchange rate. For example, your account is denominated in AUD, but you trade the EURUSD. So profits must be converted from USD -> AUD. EA Studio does this internally and updates the exchange rates a few times a day (as I´ve noticed as my virtual account is in EUR). Hence net profit may differ if the rates just have been updated. But the amount of trades will stay the same (that always was the case for me so far). To me, this looks like a bug actually, Popov might be able to help :-)

Re: Recalculating strategies ending with very different results

Yes I agree definently has to be a bug. Thanks for the reply!
Also have a look when you get a ch ace at the thread I started about a PC for EA Studio, you have talked me into a Threadripper smile

6 (edited by sleytus 2019-01-22 07:39:24)

Re: Recalculating strategies ending with very different results

These days I only use FSB-Pro -- so take this with a "grain of salt"... 

I don't think this is a bug.  I suspect FSB-Pro and EA Studio use similar or related calibration algorithms.  The algorithms probably implement some form of trial-and-error, repeatedly testing new values to see if the result improves or not.

A trial-and-error algorithm is very much dependent on the starting state.  If the starting states are different each time you press 'Recalculate' then it wouldn't be surprising the final results differ.  I've seen this with FSB-Pro when I fiddle around with re-calibrating strategies.

The calibration algorithm must use some form of trial-and-error since calibration is not something that can be automagically computed using some advanced form of mathematics.  And trial-and-error algorithms depend on the starting state -- it's just the nature of the beast.

Another name for "trial-and-error" is "empirical" (as opposed to theoretical).

Guys -- this should be another clue you are obsessing about the statistics.  Like it or not, Forex isn't a theoretical field -- it's empirical...

7 (edited by geektrader 2019-01-22 07:59:13)

Re: Recalculating strategies ending with very different results

That´s definitely not the case Sleytus, EA Studio does not have this and it would make few sense to just randomly trying to improve strategies without letting the user know at all. And how would it know what exactly is an improvement? Net profit? Ret/dd? R-Squared? An improvement is something else for every trader, so this would be the most stupid feature ever. If I re-calculate MY strategies, I don´t see any changes at all, except if my account exchange rate changes of course (as explained above). The one Michael listed seems to be a bug with index boundaries most likely and Popov will surely look into it.

Re: Recalculating strategies ending with very different results

@geektrader -- okay, fair enough.  I did preface my comment by saying it should be taken with a "grain of salt".

By the way -- I didn't say "randomly" improve, rather "trial-and-error".  And there's a difference.  I guess as a long-time developer of data acquisition and analysis software I assumed "trial-and-error" or empirical would be clear.

During calibration each setting has a 'Start', 'Stop' and 'Step'.  This is also how it works in MT4's Strategy Tester.  The way the algorithm works is beginning with the 'Start' value it steps through possible values until it reaches the 'Stop' value.  Along the way there is a "best" result -- it may be PF, Net, Ret/DD, R-squared, whatever you chose.  But the point is it *steps* through a sequence of possible settings because it doesn't know in advance which one will be the best.  This is what I mean by "trial-and-error".  In contrast, a theoretical approach would be able to compute the best value using some formula or equation.

As an example -- take a cruise missile.  At the time it launches they already know where it will land -- using physics. Yes -- there may be some adjustments along the way -- but thanks to Einstein and those who preceded him we know a lot about mass, acceleration, gravity, friction, etc. and can predict ahead time where a missile will land.

Forex is different -- looking at the equation of your indicator and the current data there is no way to tell what the best settings should be to get the best result for your data horizon.  So, instead, the calibration algorithm steps through a sequence of values looking for the best one.

When Michael performs the calibration the first time the settings have some initial value before the algorithm begins to step through possible values.  Michael then makes some changes to the data horizon, which results in some changes to the settings, and then restores the original data horizon.  When he repeats the calibration his starting point may be different than before -- hence, a different result.  I'm not claiming this is the case -- rather, it's a possible explanation.

Also, when using an empirical approach it turns out there can be thousands or millions of different combinations to try -- perhaps too many to test with each calibration attempt.  And it might be that Popov's algorithm tries something a little bit different each time the calibration gets run.  And this is perfectly fine because no one knows in advance what the optimal setting is -- so, the more and different combinations you try the greater the probability you'll find an optimal one.  This isn't just Popov's software -- this is how calibration algorithms work in general.

Re: Recalculating strategies ending with very different results

Hi Sleytus,
I understand what you are saying however if I am using the a data horizion (box ticked for start date) then untick the box, recalculate and review, retick the start date box and recalculate I should have the same result as I started with as the start date taken from the same data set is identical. I am not reoptimsing, changing dates, timeframes, symbols etc it is literally the exact same backtest.
Does this make sense?

If I create a strategy in one browser. Data set A timeframe A etc etc and reload it again I should be getting the exact same result, if i do not then this defeats the whole purpose of EA Studio which is to build, optimise and stress test EA's. Imagine you had a Excel spreadsheet with a set of formulas and each time you loaded the same values into the spreadsheet you got a different result. It would be useless.

Re: Recalculating strategies ending with very different results

Hi Michael,

Yes -- what you say does make sense, and it very well could be a bug.

However, what I am trying to explain is that "trial-and-error" (i.e. empirical) calibration algorithms could very well have more going on under the covers than you suppose.

I've written calibration-type software that requires comparing billions of combinations of numbers.  If the application were to actually compare the same billions of combinations each time a user pressed "Recalculate" it would take a couple of years.  So, instead, each time "Recalculate" is pressed then a different subset of combinations is tested.

Just like in real life everything comes down to trade-offs.  If you test all billion combinations each time then the upside is you covered all possibilities and each time you get the exact same result, but the downside is each calculation might take two years.  If you test only a subset then the upside is it completes quickly, but the downside is you haven't tested all possibilities.  Those are the trade-offs.

One assumption we used to make is if a user pressed the "Recalculate" button a second time it often meant they weren't satisfied with the initial result, so it made sense to test a different subset of possibilities.

You guys are not programmers.  You don't think like programmers and have no clue as to the amount of data that needs to be processed and the millions of combinations that have to be tested.  The fact that Popov found a clever way to do this that takes only a few seconds is quite amazing.  Have you ever tried back testing using MT4 Strategy Tester -- it's about 10000x slower.  Yet it never dawns on you guys that perhaps there were some optimization speed-ups that Popov implemented.  I don't know about you, but I would take 10000x faster at the expense of 5% less accurate.

Furthermore, back testing statistics are rarely matched in a Real account, so you shouldn't become so dependent on them.  Yes -- it provides a clue that one strategy may be better than another, but it certainly isn't quantifiable.  If back testing statistics were so reliable I should be able to say that Strategy A will perform 23.76% better than Strategy B.  But, of course, no one can claim that.  That's why obsessing over back testing statistics is the booby prize -- i.e. it's not really a prize, but it feels good.  And, of course, spending your time generating, calibrating, back testing, demo'ing, re-optimzing, re-generating, etc. is certainly much safer than trading a Real account.

Since G-d has a sense of humor, this post will probably be followed by one from Popov saying he just found a bug in EA Studio's optimizer, thereby proving I'm an idiot.  That's okay -- it wouldn't be the first time...

11 (edited by sleytus 2019-01-22 15:09:34)

Re: Recalculating strategies ending with very different results

Something else just occurred to me.

It may be that people think the calibration algorithm is similar to a calculation -- i.e. every time I add two numbers the result should always be the same.  EX:  2 + 2 is always equal to 4, no matter how many times I press "Recalculate".

However, calibration is NOT a calculation.  Calibration is more like a "process of elimination" than a calculation.  You start-off with a billion combinations of possible settings and are looking for the best one.  Sort of like searching for a needle in a hay stack.  In fact, there may be more than one needle (i.e. more than one best combination).  So, if you search the hay stack multiple times you might find different needles.  That's why it is not surprising when calibration algorithms yield different results when they are repeated -- there may be more than one best result.

Re: Recalculating strategies ending with very different results

Hello Steve,

The report is regrading the "Recalculate" button (function) in the Collection. This button literally re-calculates the Collection's strategies on the historical data without any changes. The strategies must show the same results on every re-calculation on the same data. The only side factor is the "Account exchange rate" as GeekTrader noticed, but it participate in the profit calculation only when the the trading couple doesn't contain the account currency.

I don't expect a bug, because all strategies are calculated by the same algorithm and an eventual bug would change all results.

There is however something that is actually a feature, however it is not well shown. If the Collection contains strategies from different markets, it loads the particular files before the calculation. However, if some of the files is missing (which, may happen only if we have removed it intentionally) , the Collection must show a note that the particular file is missing and to fall back to the default symbol (the first one for the data-server). Unfortunately, I recently noticed that there is a bug here. The Collection doesn't show the note and recalculates the strategies with the missing files on the first available data silently. It even fails to change the text on the top of the strategy box. Of course the strategy result will be completely different in that case.

It is difficult to experience this result because we normally do not delete the uploaded files (Blahh I usually delete all for various testing).

Anyway, if you have a particular strategy that changes the shown result upon Recalculating, please report. I'll fix it immediately once I can reproduce it.

Re: Recalculating strategies ending with very different results

Thank you, Popov.

I stand corrected and apologize to @Michael1 and @geektrader for going off on a tangent.  This was a report concerning EA Studio which I am no longer familiar with -- I should not have posted responses.  My responses were wrong and only confused the issue.

Re: Recalculating strategies ending with very different results

Hi Popov,

What I am experiencing is a definite problem. I will do some more testing and video the result.

Re: Recalculating strategies ending with very different results

Yes Michael, I just wanted to say, can you make a video of this? Also curious to see it, as I have not experienced it here. If you do not mind, maybe share a bogus (not important) strategy where this happens as JSON? Then one could test even easier.