Topic: What is the point of forward testing?

After several weeks of playing around with FSB/FST, I've come up with some procedures to carry a strategy through it's lifecycle, essentially from inception to death. I like to write and refine procedures for these types of things because I feel that they help me to be more consistent. I would like to believe that a disciplined procedural approach to managing a legion of tradebots will translate to a consistent flow of live success as well.

Part of my approach has always been to 'test drive' the bots on my demo accounts before letting them graduate to their own live accounts. But - I have been thinking about this recently and questioning the purpose of this stage. For example, I would normally generate and optimize a new bot using 30% OOS as part of the process. The OOS visual performance allows me to make a discretionary value judgment between all the rookies, and naturally a great screener for duds. From the cream-of-crop rookies, I would then let them perform in the wild using FST on a demo account. After the demo period I would be able to graduate the best-of-breed to join the legion, each with their own live accounts.

It occurred to me the other day that perhaps the entire stage of demo forward testing is unnecessary. I already trust that FST performs equally well trading demo or live, so there's nothing to prove in that regard. Seems like I could just increase the amount of OOS data to effectively accomplish the same effect I get from real-life forward testing, except that it could be instantaneous.

My understanding about the OOS sample percent is that it is only treated as OOS (ie ignored by FSB) during the Generator stage. That means further manipulations occuring after Generator could be skewed by incorporating the OOS portion. For example I manually set the SL/TP/BE for all rookies by hand. For reasons such as these, it makes sense to physically separate and control OOS data.

So my idea is to use the Generator's OOS% as an initial screen only, but then followup with an instant forward test by swapping the datafile. I could Generate as usual, but using a dataset that is outdated by the same period I would normally forward test with (in my case, 1 full week of M1 data). Then with the best-of-breed bots, I could swap-in the latest dataset and study the further OOS performance curve. Based on this, a bot could be approved to join the legion immediately I think.

Am I missing something here? Why didn't I think of this before? Do you think it's a bad idea to eliminate forward testing on demo acct altogether? if so, why?

Re: What is the point of forward testing?

I would always demo test it first and make sure it agrees with FSB. You are using a different set up for demo trading, i.e. FST & MT4. When testing the strategy you are only using FSB. You cannot know if the slippage is understated, there might be a delay caused by you broker's platform etc. One other thing you may want to do once you are satisfied everything will work is to run a demo platform alongside the real platform to see the difference (if any) in the trading and fills.

Re: What is the point of forward testing?

Forward testing on a demo is hard to justify (I do it minimally). The most important reason for it is significant difference between live and demo accounts - data feed and execution are never the same as in live! That's why I demo just to make sure everything is ok (especially if custom indis are used), it is like a shakedown if you know what I mean. True trading enviroment test can be carried out only on live account.

Forward testing is currently the only way forward for me - simple backtest don't tell you much. I tend to optimize periodically, the system I use is something like this: I have a 70/30 optimizing and testing chart, one unit of time is 100 and I optimize the strategy on 70% of the data and on the rest 30% I test myr strategy in a forward manner, in other words on out of sample data, it's important to emphazise that the last 30% of data is cut off when optimizing. The next round goes just the same except lifting the data horizon -> which means that 30% of the previous 70% is left out, previous 30% OOS is a substitute for it, and there's another 30% on which forward test is carried out.

One final note - you say you trust that FST performs equally well trading demo or live, please bear in mind not to trust the broker. I've learnt it the hard way... Visit this thread from another fx forum to read more about MT brokers - … 8252#p8252

Re: What is the point of forward testing?

Hey thanks very much for the tip on Data Horizon - I had completely forgotten about this feature. It does seem to work great. I'm now doing all the first stage development using a horizon that has the last week clipped. I still use OOS 30% as you say, because I think it's a good pre-filter when running Generator. From the perfected strategy I move to another stage where I reset the Data Horizon to allow the final week (about 7000 M1 bars). On this stage I'm simply observing the continued OOS performance with no tweaks to the original strategy. I'm looking for continued visual progress in the equity line, and no degradation in the drawdown figures. I also like to see improvements in the reward ratio and efficiency metric (either would be a bonus).

I figure I can do this procedure each week and as long as the bot continues to perform, then it can continue trading live.

One oddity I've noticed with my MT4 data -- I keep a live platform running all the time so I can gather the new bars each Friday. But it seems that my broker trims a week from the beginning as well. The odd part is that the number of total bars is still increasing each week. I think I will change my process to start manually appending the latest data to last weeks CSV data, that way I won't continue losing the earliest data.

Re: What is the point of forward testing?

Criminal doesn't trim, MT does that, you just have to set it up correctly - max bars on chart or something like that.