1 (edited by sleytus 2022-07-31 17:57:01)

Topic: OOS Analysis -- short or long data horizon -- that is the question...

I've recently returned to forexsb -- still the finest software out there...

Stepping up to the podium...

With regards to data horizons -- both in back-testing and OOS analysis -- I believe most traders harbor a misconception.  The misconception is that the longer the data horizon then the smarter the EA.  There are some guys who would go back to dinosaur times tick data if they could get their hands on it.  But here's the deal -- indicators and EAs use algebra and algebra does not learn and can not be trained.  It strictly uses just the input (x,y) data and computes the best settings for that exact input data.

Market conditions constantly change and, so, the input (x,y) data is a moving target.  Unless you have a time machine you can't trade on historical data.  The most important and **relevant** part of your data horizon are the most RECENT time points.  You want the optimizer to compute the best settings for the most RECENT data.  The more historical data you feed to the algebra then the more biased the optimized settings will favor the historical data.  That is -- historical data has a tendency to skew and pollute the results.

So -- I'm claiming that the goal should be the shortest data horizon, with the caveat that you need a minimum number of trades for the computed stats to be statistically significant.

As an example -- suppose I'm trading CADJPY / H1.  I like a data horizon of about one month with an OOS of 30% -- something like that.  Since I trade in real-time then I want the input (x,y) data to favor current market conditions so that the optimizer computes the best settings for current market conditions.

Stepping down from the podium...

Agree?  Disagree?  Don't care?

Best of luck with your trading...

Re: OOS Analysis -- short or long data horizon -- that is the question...

Another  EA developer favors short data, and has impressive returns.

Myself I prefer 3 months

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

3 (edited by sleytus 2022-07-31 18:41:16)

Re: OOS Analysis -- short or long data horizon -- that is the question...

Blaiserboy wrote:

Another  EA developer favors short data, and has impressive returns.

Myself I prefer 3 months


BlaiserBoy -- I defer to you.  Many moons ago you were the one who turned on the light bulb for me and reminded me that indicators and EAs are not a black box -- they're just algrebra...   smile

Re: OOS Analysis -- short or long data horizon -- that is the question...

Hey guys - I'm hoping you can share some advise to me please.

I'm using Reactor to generate strategies, and then Portfolio Export to run them in MT5.

I trade 5m AUDUSD, so I've been using Reactor to generate lots of strategies, but I have to run it quite far back - I use Opt and then Norm and then Monte Carlo.  I've tried running shorter date horizon, because I tend to agree with you about long term optimisation (fitting etc), but Reactor seems to struggle to find strategies that have more than 100 trades in the short period time horizons...

Sleytus - you mentioned 1h over a month horizon...is your criteria 100 trades min as well?

Re: OOS Analysis -- short or long data horizon -- that is the question...

timelleston wrote:

Hey guys - I'm hoping you can share some advise to me please.

I'm using Reactor to generate strategies, and then Portfolio Export to run them in MT5.

I trade 5m AUDUSD, so I've been using Reactor to generate lots of strategies, but I have to run it quite far back - I use Opt and then Norm and then Monte Carlo.  I've tried running shorter date horizon, because I tend to agree with you about long term optimisation (fitting etc), but Reactor seems to struggle to find strategies that have more than 100 trades in the short period time horizons...

Sleytus - you mentioned 1h over a month horizon...is your criteria 100 trades min as well?


This is a good question -- but there is no right / wrong answer.  And that's because the settings you choose can only be determined empirically -- i.e. by testing and trial / error.  And since there are a zillion ways to create a portfolio EA, then no one person will have exhaustively tested all possibilities.

My experience is the shorter the data horizon the better -- with the caveat you need a minimum number of trades for the stats to be statistically significant.  The general rule of thumb in statistics is the sample size should be at least 100.  And if you've been following the law suit between Elon Musk and Twitter you may have read that Twitter estimates the number of fake accounts using a sample size of 100 -- but that's neither here nor there.

After looking at charts I will often go with a smaller sample size because I really, really want the data horizon to be as short as possible.  And that is because I trade in real time.  Sometimes there will be as few as 20-30 trades.  But if the balance chart looks linear then that's good enough for me.

I favor the H1 time frame and a one-month data horizon works for me.  BlaiserBoy prefers a 3-month data horizon, but that may be because he prefers trading longer time frames, in which case he would need a longer data horizon to reach a minimal sample size of trades.  If you are trading M5 then you could probably get by with even a shorter data horizon than me -- perhaps even just 1 week.

The reason you are generating so few strategies is because your conditions are too stringent.  Under 'Optimize Strategies' I only use 'Full Data Optimization'.  Also, though I used to use Monte Carlo -- no more.  Monte Carlo tests for rare events -- but that comes at the expense of ignoring strategies that would otherwise perform well in the short term.   Since EA Studio is so fast and easy to use we don't have to worry about our strategies lasting for months or years -- we can make a new batch of strategies every week, every day, every hour -- whatever you want. 

Keep in mind that our input data is not static -- market conditions are always changing.  The market's data patterns last week may be different than next week.  Monte Carlo is intended to be used with static data sets that don't change and to determine the input data's degree of randomness, which is of questionable relevance when it comes forex price data -- in my opinion.  So -- I think including Monte Carlo does more harm than good -- a statement that will probably get me into trouble.

Re: OOS Analysis -- short or long data horizon -- that is the question...

Thanks Sleytus thats great advise, appreciate it.  I'll try the shorter timeframe as well, but in my testing of this, the number of trades really gets reduced, and, like you, I prefer a minimum of 100 trades for stat-sig.

I wasn't sure about using Monte Carlo - it seemed a good idea, but in hindsight, being easy, I'll give it a try without it.

And no Normalisation then...thats interesting...

And yes, I definitely use the Balance Line Stability.

Much appreciated.

7 (edited by Naya 2022-08-09 08:06:37)

Re: OOS Analysis -- short or long data horizon -- that is the question...

Hey,
Sleytus.
Do you really mean you use less than 3months of H1 data to generate strategies?Or you refer to just OOS analysis? How much data for H1 will you normally use in the generator?

Re: OOS Analysis -- short or long data horizon -- that is the question...

Naya wrote:

Hey,
Sleytus.
Do you really mean you use less than 3months of H1 data to generate strategies?Or you refer to just OOS analysis? How much data for H1 will you normally use in the generator?


Hello...

I think I was pretty clear in my posts above -- read them again if you need to.  I'm not inclined to spend time writing again what I already wrote.  I don't have all the answers and I'm not looking to hold anyone's hand.  There is no right / wrong answer.  The most important advice I can give you is to develop enough confidence you can figure things like this out on your own. 

Good luck with your trading...