Topic: Optimize using recent data

I feel it is important to use recent data when optimizing strategies -- I hope the above image helps to explain why.

A strategy combines various indicators to create trading signals.  Indicators are simple formulas -- very similar to the algebraic formula for a straight line:  y = mx + b, where m, b are constants.  Furthermore, most strategies have one or more constants -- similar to 'm' and 'b'.  If you don't believe me, then take a look at the *.mq4 code. 

Let's say you have a bunch of x,y data points and want to calculate the "optimal" line that best matches with the points.  Performing a linear regression will compute the best 'm' and 'b'.  If the data points don't change, then you are done.  On the other hand, if you continue to collect more data points then you will need to perform the linear regression again.  In other words -- the 'm' and 'b' are only good for the data points used when performing the linear regression to calculate the "optimal" line.  If the data changes then 'm' and 'b' that you previously calculated are no longer the optimal values.

Now -- take a look at the image above.  The black line is the "optimal" line that was computed when using all the data points -- i.e. from 2013 through 2018.  The red line is the "optimal" line that was computed when using only the data points from 2018.  They have vastly different settings -- i.e. their 'm' and 'b' values are very different.

Switching over to forex -- when I add a strategy to a Real account all I really care about is how it will perform *now*.  I do not care how it traded 5 years ago, 3 years ago, etc.  That is why I think we should only use recent data when optimizing.

One last point -- I think some people have the impression that strategies "learn" and that by using more historical data they will, in some way, become "smarter".  That's not the way it works.  Optimization does not use Artificial Intelligence (AI) -- i.e. there is no database backing our strategies.  In our case, optimization performs a one-time calculation for 'm' and 'b'.  If we want our strategy's indicators to use the optimal 'm' and 'b' settings for the most recent data, then you will need to optimize using the most recent data.  Using more historical data does not help -- it only skews the computation in favor of the old data.

Re: Optimize using recent data

Another point -- related to manual trading.

There are lots of manual traders, and many are successful.  They do not use tons of historical data.  They may or may not even use indicators -- but if they do, they are only interested in recent data.  They use their brain to look for certain patterns and then decide whether or not to open a trade.  The fact they can be successful proves that tons of historical data is not required to succeed at forex.

I would take it a step further.  At best, tons of historical data doesn't help.  However, at worst, it may actually do harm because all those past years of historical data obscures the most recent data patterns from the optimizer.

Re: Optimize using recent data

Very interesting points Steve. I usually try and stay away from optimizing (due to fear of curve fitting) , but it makes sense what you are saying. So I guess the period to optimize over is depending on the strategy (if it trades seldom - you will have to take a longer time frame to optimize , if it trades a lot - you can have a shorter time frame).. is that how you do it?

Re: Optimize using recent data

Optimizing or curve-fitting is okay -- the danger is in *over* curve-fitting.  And that is the trap I kept falling into.  I think after a while of Real trading and disappointing results using strategies with amazing stats, then you begin to get an eye for optimized versus "over curve-fitted". 

With regards to the period to optimize over -- it depends.  I intentionally refer to "recent data" without defining what I mean by "recent".  In order for Popov's optimization algorithms to work they need a certain amount of data -- is it time, or number of bars or number of executed orders?  For now, I look for a minimum number of executed orders.

As an example -- I exclusively trade EURUSD/H1.  When I specify a data horizon of 4000 bars (approximately 8 months) then I typically end up with between 250-400 executed orders.  And I think that is good enough.  I've experimented with larger numbers -- but without much improvement.  Using smaller numbers often times results in amazing stats -- which I don't trust.

Every couple of weeks I'll refresh FSB-Pro with the latest data from my brokers -- it usually takes less than 5 minutes.  And then I'll re-rerun the optimizer on all my strategies and generate new *.mq4 files.  It takes about 2-3 hours.  With EA Studio you can do the same in less time because the strategies can be saved in collections.

Re: Optimize using recent data

Interesting topic, but I think the author has very wrong ideas about the market ant how it works.

In reality exactly such ideas lead to the fact that majority of traders lose money in the markets.

Optimize using recent data means curve fitting to recent data.

Recent market condition (2-3 years) are the worst predictor for next 5 years.

Most manual and and algo traders tend to fit their behavior on recent data, because that is how the human brain works. The markets work in different way going true different cycles. You will get few months good performance from such fitted strategies and then never see them profit again.

6 (edited by D5QM54S7 2019-09-16 17:26:00)

Re: Optimize using recent data

@Sleytus Quantitative and discretionary trading are two different worlds: in the first one you need historical data by its nature, in the second you don't use them because you use sentiment and mental models but these models were made over years of historical data and this does not prove that historical data are useless. The correct question is: why optimize only on "n" data? If for example as a result of quantitative easing the market changes and the latter lasts many years, why not use less recent data for optimization? Every strategy needs its data, there is no general rule;
@FMZ optimization does not create overfitting if the strategy has been built well, it is a phase that has nothing to do with the curve adaptation. If you can no longer make profits it is because the market has changed, not because you have optimized, think about it.

Re: Optimize using recent data

The method that he uses has worked very well for him, so far, and while that method may not suit everyone he has actually made it work.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Optimize using recent data

FMZ wrote:

You will get few months good performance from such fitted strategies and then never see them profit again.

That is the idea Steve had! Few months of good performance, rinse and repeat. Very ignorant to assume that Steve would be such a dimwit to use short-term optimized strats for long-term goals.

Re: Optimize using recent data

I may say that the "long-term goals" are desired by the trial users.
The licensed users want stability and prosperity smile

Re: Optimize using recent data

I follow Steve's ideas and it has helped me to improve.

I had trouble at the start to grasp what he was doing, he had to teach me hahaha

I was quite skeptical until he explained his success knowing nothing about Forex until he somehow discovered FSBPro and started looking into it.

He started off with the robots... not manual trading. and he was making money quickly, I think that is what was the proof in the pudding for me.

His programming background was really helpful to him.

Anyways, I know his approach works well.  It probably does not suit all traders, but for me, it has been most helpful.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....