Another screenshot

Hi :-)

I´ve not really experienced this to that degree - as far as I understand some strategies actually have no more trades at all after re-enabling all data but they previously had trades? The only time it was different for me is because of the virtual trading account exchange rate. For example, your account is denominated in AUD, but you trade the EURUSD. So profits must be converted from USD -> AUD. EA Studio does this internally and updates the exchange rates a few times a day (as I´ve noticed as my virtual account is in EUR). Hence net profit may differ if the rates just have been updated. But the amount of trades will stay the same (that always was the case for me so far). To me, this looks like a bug actually, Popov might be able to help :-)

These days I only use FSB-Pro -- so take this with a "grain of salt"... 

I don't think this is a bug.  I suspect FSB-Pro and EA Studio use similar or related calibration algorithms.  The algorithms probably implement some form of trial-and-error, repeatedly testing new values to see if the result improves or not.

A trial-and-error algorithm is very much dependent on the starting state.  If the starting states are different each time you press 'Recalculate' then it wouldn't be surprising the final results differ.  I've seen this with FSB-Pro when I fiddle around with re-calibrating strategies.

The calibration algorithm must use some form of trial-and-error since calibration is not something that can be automagically computed using some advanced form of mathematics.  And trial-and-error algorithms depend on the starting state -- it's just the nature of the beast.

Another name for "trial-and-error" is "empirical" (as opposed to theoretical).

Guys -- this should be another clue you are obsessing about the statistics.  Like it or not, Forex isn't a theoretical field -- it's empirical...

@geektrader -- okay, fair enough.  I did preface my comment by saying it should be taken with a "grain of salt".

By the way -- I didn't say "randomly" improve, rather "trial-and-error".  And there's a difference.  I guess as a long-time developer of data acquisition and analysis software I assumed "trial-and-error" or empirical would be clear.

During calibration each setting has a 'Start', 'Stop' and 'Step'.  This is also how it works in MT4's Strategy Tester.  The way the algorithm works is beginning with the 'Start' value it steps through possible values until it reaches the 'Stop' value.  Along the way there is a "best" result -- it may be PF, Net, Ret/DD, R-squared, whatever you chose.  But the point is it *steps* through a sequence of possible settings because it doesn't know in advance which one will be the best.  This is what I mean by "trial-and-error".  In contrast, a theoretical approach would be able to compute the best value using some formula or equation.

As an example -- take a cruise missile.  At the time it launches they already know where it will land -- using physics. Yes -- there may be some adjustments along the way -- but thanks to Einstein and those who preceded him we know a lot about mass, acceleration, gravity, friction, etc. and can predict ahead time where a missile will land.

Forex is different -- looking at the equation of your indicator and the current data there is no way to tell what the best settings should be to get the best result for your data horizon.  So, instead, the calibration algorithm steps through a sequence of values looking for the best one.

When Michael performs the calibration the first time the settings have some initial value before the algorithm begins to step through possible values.  Michael then makes some changes to the data horizon, which results in some changes to the settings, and then restores the original data horizon.  When he repeats the calibration his starting point may be different than before -- hence, a different result.  I'm not claiming this is the case -- rather, it's a possible explanation.

Also, when using an empirical approach it turns out there can be thousands or millions of different combinations to try -- perhaps too many to test with each calibration attempt.  And it might be that Popov's algorithm tries something a little bit different each time the calibration gets run.  And this is perfectly fine because no one knows in advance what the optimal setting is -- so, the more and different combinations you try the greater the probability you'll find an optimal one.  This isn't just Popov's software -- this is how calibration algorithms work in general.

Hi Michael,

Yes -- what you say does make sense, and it very well could be a bug.

However, what I am trying to explain is that "trial-and-error" (i.e. empirical) calibration algorithms could very well have more going on under the covers than you suppose.

I've written calibration-type software that requires comparing billions of combinations of numbers.  If the application were to actually compare the same billions of combinations each time a user pressed "Recalculate" it would take a couple of years.  So, instead, each time "Recalculate" is pressed then a different subset of combinations is tested.

Just like in real life everything comes down to trade-offs.  If you test all billion combinations each time then the upside is you covered all possibilities and each time you get the exact same result, but the downside is each calculation might take two years.  If you test only a subset then the upside is it completes quickly, but the downside is you haven't tested all possibilities.  Those are the trade-offs.

One assumption we used to make is if a user pressed the "Recalculate" button a second time it often meant they weren't satisfied with the initial result, so it made sense to test a different subset of possibilities.

You guys are not programmers.  You don't think like programmers and have no clue as to the amount of data that needs to be processed and the millions of combinations that have to be tested.  The fact that Popov found a clever way to do this that takes only a few seconds is quite amazing.  Have you ever tried back testing using MT4 Strategy Tester -- it's about 10000x slower.  Yet it never dawns on you guys that perhaps there were some optimization speed-ups that Popov implemented.  I don't know about you, but I would take 10000x faster at the expense of 5% less accurate.

Furthermore, back testing statistics are rarely matched in a Real account, so you shouldn't become so dependent on them.  Yes -- it provides a clue that one strategy may be better than another, but it certainly isn't quantifiable.  If back testing statistics were so reliable I should be able to say that Strategy A will perform 23.76% better than Strategy B.  But, of course, no one can claim that.  That's why obsessing over back testing statistics is the booby prize -- i.e. it's not really a prize, but it feels good.  And, of course, spending your time generating, calibrating, back testing, demo'ing, re-optimzing, re-generating, etc. is certainly much safer than trading a Real account.

Since G-d has a sense of humor, this post will probably be followed by one from Popov saying he just found a bug in EA Studio's optimizer, thereby proving I'm an idiot.  That's okay -- it wouldn't be the first time...

Hello Steve,

The report is regrading the "Recalculate" button (function) in the Collection. This button literally re-calculates the Collection's strategies on the historical data without any changes. The strategies must show the same results on every re-calculation on the same data. The only side factor is the "Account exchange rate" as GeekTrader noticed, but it participate in the profit calculation only when the the trading couple doesn't contain the account currency.

I don't expect a bug, because all strategies are calculated by the same algorithm and an eventual bug would change all results.

There is however something that is actually a feature, however it is not well shown. If the Collection contains strategies from different markets, it loads the particular files before the calculation. However, if some of the files is missing (which, may happen only if we have removed it intentionally) , the Collection must show a note that the particular file is missing and to fall back to the default symbol (the first one for the data-server). Unfortunately, I recently noticed that there is a bug here. The Collection doesn't show the note and recalculates the strategies with the missing files on the first available data silently. It even fails to change the text on the top of the strategy box. Of course the strategy result will be completely different in that case.

It is difficult to experience this result because we normally do not delete the uploaded files (Blahh I usually delete all for various testing).

Anyway, if you have a particular strategy that changes the shown result upon Recalculating, please report. I'll fix it immediately once I can reproduce it.

Hi Popov,

What I am experiencing is a definite problem. I will do some more testing and video the result.