• Registered: 2017-03-14
  • Posts: 709

Topic: OOS Analysis and Over Curve-Fitting

I'm a big fan of OOS Analysis, and next to fast strategy generation and backtesting it is one of my favorite features.  Over curve-fitting is the biggest enemy of algorithmic traders and OOS analysis provides a mechanism to mitigate the harm.  However, by including OOS analysis in your work flow can you guarantee your strategies won't be over curve-fitted?  That is the question...

ANSWER: It's a rhetorical question -- and the answer is: It depends.  The curious might ask "What does it depend on?"  It depends on the market conditions in the IS and OOS sections of your data horizon.  If the market conditions are the same in the IS and OOS sections then the strategy will likely be over curve-fitted and OOS analysis offers little to no benefit.  Think about it -- the strategy is optimized against the IS data.  If the market conditions are the same then the OOS section will perform the same as the IS section and does not provide a filter to detect over curve-fitted settings.  On the other hand, if the market conditions are different then the OOS section can provide a filter to detect over curve-fitted settings.

Question #2: How do you know whether market conditions are the same or different in the IS and OOS sections?

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,885

Re: OOS Analysis and Over Curve-Fitting

For me the biggest problem of OOS is that we make the decision when we see the result of the OOS performance. However, it makes it undistinguishable from the full data test.

Let's have a time series from time A to time Band a point M, from where the OOS period starts.
So, if we generate strategies for the period A to M. Then test them from M to B and choose the ones that have good performance. The choose will have obviously good performance for the complete data set from A to B.
How these strategies differ from other then we generate directly from A to B?
Some say that the firsts are battle-scarred.
I assume all the strategies we generate on the complete data seta will pass the OOS test also.

Probably OOS has a point when we use it, but we never check the result. Quantum mystery...

Popov's Website

  • Registered: 2017-03-14
  • Posts: 709

Re: OOS Analysis and Over Curve-Fitting

Yes -- I agree... 

My work flow includes a final "discretionary" step.  I visually inspect the collection and remove strategies that don't pass my visual inspection.  The survivors are then added to a portfolio and I'll generate the corresponding *.mq4 / *.mq5 source.

My visual inspection requires the following:
1. The transition between IS and OOS (i.e. your point M) must be linear and sloping upward
2. The OOS must have a positive profit and almost the same percent of the total profit as the percent OOS. 

Popov -- your software makes it so easy to experiment and test ideas that were impossible before.  With OOS analysis we can clearly see how most strategies begin to breakdown shortly after the transition from IS to OOS.  This shows the harm of over curve-fitting which occurs when EAs don't take OOS analysis into consideration.

Popov wrote:

For me the biggest problem of OOS is that we make the decision when we see the result of the OOS performance. However, it makes it undistinguishable from the full data test.

Let's have a time series from time A to time Band a point M, from where the OOS period starts.
So, if we generate strategies for the period A to M. Then test them from M to B and choose the ones that have good performance. The choose will have obviously good performance for the complete data set from A to B.
How these strategies differ from other then we generate directly from A to B?
Some say that the firsts are battle-scarred.
I assume all the strategies we generate on the complete data seta will pass the OOS test also.

Probably OOS has a point when we use it, but we never check the result. Quantum mystery...

4 Reply by krzysiaczek99 (edited by krzysiaczek99 2022-10-11 14:53:50)

  • Registered: 2022-08-18
  • Posts: 7

Re: OOS Analysis and Over Curve-Fitting

Interesting theories hehehe

Predictive model (build strategy) will have predictive power when fitted properly to some underlying probability distribution (e.g. cycle). When fitted to noise will not have any predictive power so strategy will trade randomly (overfitted). So OOS check makes very much sense as it proves/negates that model is fitted to some real PDF. Unfortunatelly in most of the cases is fitted to noise and it performs well in sample because of multiple testing

https://en.wikipedia.org/wiki/Multiple_comparisons_problem

Prado speaks about this here

https://www.youtube.com/watch?v=QxhxLwNbMMg

Strategy finder to predict properly must use indicators which have predictive power, majority of classic indictors don't have any predictive power, so garbage in garbage out....Another problem is that market signal is very weak if any, and GA during search is not able to differentiate between 'good indicators' and 'bad indicators' which easily fit to the noise. Montecarlo tests and WF tests will not help much either as market conditions are changing so predictive power of indicators also.

Krzysztof

  • Registered: 2022-08-18
  • Posts: 7

Re: OOS Analysis and Over Curve-Fitting

and one more thing. When strategy fitted to the noise it can temporary perform well OS simply by luck but will stop to work fast. By definition noise is random and you can not make money on trading on noise. So its very difficult to differentiate if strategy performed well OS because of luck/noise fit its or market signal fit.

  • Registered: 2017-03-14
  • Posts: 709

Re: OOS Analysis and Over Curve-Fitting

Hello krzysiaczek99...

I enjoyed reading your comments -- and I would agree with them.  Thanks for sharing your insight...

krzysiaczek99 wrote:

and one more thing. When strategy fitted to the noise it can temporary perform well OS simply by luck but will stop to work fast. By definition noise is random and you can not make money on trading on noise. So its very difficult to differentiate if strategy performed well OS because of luck/noise fit its or market signal fit.