Topic: Optimize using recent data

https://s8.postimg.cc/fjbl5bxqp/why-use-recent-data.png

I feel it is important to use recent data when optimizing strategies -- I hope the above image helps to explain why.

A strategy combines various indicators to create trading signals.  Indicators are simple formulas -- very similar to the algebraic formula for a straight line:  y = mx + b, where m, b are constants.  Furthermore, most strategies have one or more constants -- similar to 'm' and 'b'.  If you don't believe me, then take a look at the *.mq4 code. 

Let's say you have a bunch of x,y data points and want to calculate the "optimal" line that best matches with the points.  Performing a linear regression will compute the best 'm' and 'b'.  If the data points don't change, then you are done.  On the other hand, if you continue to collect more data points then you will need to perform the linear regression again.  In other words -- the 'm' and 'b' are only good for the data points used when performing the linear regression to calculate the "optimal" line.  If the data changes then 'm' and 'b' that you previously calculated are no longer the optimal values.

Now -- take a look at the image above.  The black line is the "optimal" line that was computed when using all the data points -- i.e. from 2013 through 2018.  The red line is the "optimal" line that was computed when using only the data points from 2018.  They have vastly different settings -- i.e. their 'm' and 'b' values are very different.

Switching over to forex -- when I add a strategy to a Real account all I really care about is how it will perform *now*.  I do not care how it traded 5 years ago, 3 years ago, etc.  That is why I think we should only use recent data when optimizing.

One last point -- I think some people have the impression that strategies "learn" and that by using more historical data they will, in some way, become "smarter".  That's not the way it works.  Optimization does not use Artificial Intelligence (AI) -- i.e. there is no database backing our strategies.  In our case, optimization performs a one-time calculation for 'm' and 'b'.  If we want our strategy's indicators to use the optimal 'm' and 'b' settings for the most recent data, then you will need to optimize using the most recent data.  Using more historical data does not help -- it only skews the computation in favor of the old data.

Re: Optimize using recent data

Another point -- related to manual trading.

There are lots of manual traders, and many are successful.  They do not use tons of historical data.  They may or may not even use indicators -- but if they do, they are only interested in recent data.  They use their brain to look for certain patterns and then decide whether or not to open a trade.  The fact they can be successful proves that tons of historical data is not required to succeed at forex.

I would take it a step further.  At best, tons of historical data doesn't help.  However, at worst, it may actually do harm because all those past years of historical data obscures the most recent data patterns from the optimizer.

Re: Optimize using recent data

Very interesting points Steve. I usually try and stay away from optimizing (due to fear of curve fitting) , but it makes sense what you are saying. So I guess the period to optimize over is depending on the strategy (if it trades seldom - you will have to take a longer time frame to optimize , if it trades a lot - you can have a shorter time frame).. is that how you do it?

Re: Optimize using recent data

Optimizing or curve-fitting is okay -- the danger is in *over* curve-fitting.  And that is the trap I kept falling into.  I think after a while of Real trading and disappointing results using strategies with amazing stats, then you begin to get an eye for optimized versus "over curve-fitted". 

With regards to the period to optimize over -- it depends.  I intentionally refer to "recent data" without defining what I mean by "recent".  In order for Popov's optimization algorithms to work they need a certain amount of data -- is it time, or number of bars or number of executed orders?  For now, I look for a minimum number of executed orders.

As an example -- I exclusively trade EURUSD/H1.  When I specify a data horizon of 4000 bars (approximately 8 months) then I typically end up with between 250-400 executed orders.  And I think that is good enough.  I've experimented with larger numbers -- but without much improvement.  Using smaller numbers often times results in amazing stats -- which I don't trust.

Every couple of weeks I'll refresh FSB-Pro with the latest data from my brokers -- it usually takes less than 5 minutes.  And then I'll re-rerun the optimizer on all my strategies and generate new *.mq4 files.  It takes about 2-3 hours.  With EA Studio you can do the same in less time because the strategies can be saved in collections.