1 Topic by sleytus (edited by sleytus 2022-09-11 21:29:51)

  • Registered: 2017-03-14
  • Posts: 709

Topic: Should IS and OOS sections be reversed?

This may be controversial, but here goes...

I'll advocate that for OOS analysis the IS and OOS sections should be reversed.  For example, consider 30% OOS.  In this case the oldest 70% of the data is used to compute the settings and then those settings are applied to the most recent data.  However, since we trade in real-time (i.e. the most recent data) then that should be reversed.  We want the settings to be computed using the most recent 70% of the data and then compared to the oldest 30%.

Market conditions are constantly changing and it seems to me that settings should be "tuned" to the most recent market conditions rather than historical market conditions.

Popov -- would it be possible to add an option to choose between the two possibilities?

  • Registered: 2015-02-22
  • Posts: 430

Re: Should IS and OOS sections be reversed?

This is a good idea, totally make sense

3 Reply by geektrader (edited by geektrader 2022-09-13 17:32:48)

  • From: Germany
  • Registered: 2010-04-05
  • Posts: 842

Re: Should IS and OOS sections be reversed?

I've tested reversed OOS with other platforms many times throughout the last 15 years that I am creating trading systems and it never gave any good results going into live trading if reversed OOS was used. The opposite is true with having OOS at the end like it is now, given that it is long enough (at least 4 years), with less it's simply curve matching and these systems fail like dead flies when going live. Surely having it as an option won't hurt though if someone believes it helps, but I clearly wouldn't use it based on my several statistical live tests with this as it failed everytime.

geektrader's Website

4 Reply by sleytus (edited by sleytus 2022-09-13 20:52:08)

  • Registered: 2017-03-14
  • Posts: 709

Re: Should IS and OOS sections be reversed?

Interesting -- thanks for sharing your experience.

It still seems counter-intuitive to me.  Since the IS section is prone to curve matching regardless if it comes before or after OOS, then it seems it would be preferable to match with more recent market conditions than historical market conditions over 4 years old.  The OOS data has no role in computing settings.  It only comes into play afterwards as a kind of diagnostic that reflects the degree of curve fitting.  If market conditions during OOS are similar to IS and the strategy survives your acceptance criteria and gets included in your collection then OOS has no benefit and may actually result in sub-optimal setting calibrations.  And if market conditions during real-time trading differ from IS and OOS then you are screwed regardless.

However, in the end, testing trumps intuition and if reversed doesn't work then I'm content to let it go.  geektrader -- again -- thanks for your insight.


geektrader wrote:

I've tested reversed OOS with other platforms many times throughout the last 15 years that I am creating trading systems and it never gave any good results going into live trading if reversed OOS was used. The opposite is true with having OOS at the end like it is now, given that it is long enough (at least 4 years), with less it's simply curve matching and these systems fail like dead flies when going live. Surely having it as an option won't hurt though if someone believes it helps, but I clearly wouldn't use it based on my several statistical live tests with this as it failed everytime.

5 Reply by geektrader (edited by geektrader 2022-09-14 00:03:07)

  • From: Germany
  • Registered: 2010-04-05
  • Posts: 842

Re: Should IS and OOS sections be reversed?

I am all for more options to choose from, so I am happy if it would be added. However, many things seem counterintuitive with statistics indeed, that's why such a hypothesis must simply always be tested with cold hard data. Other platforms offer this option already, some even allow you to have several OOS segments spread out through your history data, so I'd say make your tests and see how it does for you. Simply hold out the last 2 to 3 years completely from the development process and develop your systems the exact same way as you do now, then, once you are completely finished with the system, simply backtest it on the hold-out data and see if the ones with OOS at the end or the beginning do better on hold out. Use a significant amount of total data to have statistical relevance. I test everything this way, there is no way around it if we want to stay profitable in this game. Good luck.

geektrader's Website