Topic: Backtests over 10 years without crashing & the longer backtest debate

Hi there

I have searched here on this subject of backtests over long time periods and the consensus opinions seem to be that 1 -3 year backtests are good enough. I have seen Metatrader backtests of systems over 10 years with backtest at 99% modelling quality on 'every tick' - that show a smooth equity curve and great results - and perform long-term very well and in line with the backtests expectations in live trading. I know the opinions on here and elsewhere on the quality of Metarader backtests is mixed - but that's a separate issue I have asked about in another post.

It seems to me that it's logical that a system that survives 10 years backtest (of varying market conditions) with a smooth equity curve is much better than one that survived shorter backtests of 1 - 3 years - and if you test a shorter backtest on 'out-of-sample data it usually fails.

I noted the instructions and warning here about manually increasing the Max Bars in a historical data set causing problems with the computer. I have tested this and experienced it. After a few adjustments to the strategy it takes longer and longer (minutes) to calculate and quickly hangs the program and computer in a very nasty way requiring a reboot that goes very slowly. This worries me about the risks to my computer of this repeated occurrence.

This on an HP EliteBook i7 with 4gb Ram and Nvidia 2gb graphics card - not a monster but a decent machine that handles processor intensive tasks like rendering video very well.
The FSB system I'm working on has 3-4 opening and one closing logic slot and uses 2 x indicators and 2-3 timeframes (5 min, 1 hour and 4 hour).

I tried working with 2 x sets of 5 years (setting the data horizon) and the machine handles it better - but the problem is - I get it working for the 1st 5 years then test it on the next 5 years and the results are poor - and vice versa.

Questions:
1. So, my 1st question is how people can logically assert that a backtest over 1 - 3 years will yield a strategy likely to be as robust as one that shows a smooth equity curve over 10 years. Most systems fail on out-of sample data.

2. Is there's any trick or technique I can try to work with a full 10 years price data without hanging my computer in 5 minutes.

Request:
Perhaps you can consider finding a way to make it possible to change FSB so it can handle a 10 year backtest.
I think both the results and the FSB Platform will gain much credibility from this improvement.

Clearly there's some amazing programming and very hard work over many years that's gone into the development of this excellent platform; and it seems that being able to handle large data sets would be a big advance in it's ability and credibility.

Any input much appreciated.

Re: Backtests over 10 years without crashing & the longer backtest debate

I'll make very short notes to your post at that time.
Your machine has a very good CPU but 4GB Ram is below the current standards. I've seen screenshots of users machine with 32GB Ram that manage the program a way better. Probably you have to upgrade your system to at-least 8GB, which should be perfectly enough for most cases (64 windows required).

I have seen Metatrader backtests of systems over 10 years with backtest at 99% modelling quality on 'every tick'

That 'every tick' is just wording smile. There is no such thing as ticks in MT backtester. FSB always uses  'every tick' calculation  as per MT definition even without Tick data.

Perhaps you can consider finding a way to make it possible to change FSB so it can handle a 10 year backtest.

I'm considering to add a separate tool that will be able to calculate a strategy backtest using big data files. However this will be a separate function because I'm afraid making a real-time backtesting with such files is not possible for now.

Re: Backtests over 10 years without crashing & the longer backtest debate

Hi Popov

Thank you for your prompt, to the point and helpful reply as always; and on a Sunday..!

Much appreciated

Re: Backtests over 10 years without crashing & the longer backtest debate

I have a hard time believing the summaries in Metatrader, and of course, their data and real time feeds are suspect as are the wonderful tools they have for cheating the retail trader.

Depending on how a strategy is constructed, 50,000 bars is quite adequate, and in some cases 15,000 will do the job.

If the developer is 'gambling, and just trades whatever the software gives him, then he will need a lot of bars. However, if the developer sets some good 'pre conditions' in the generator.... he can get by with far less.

It is up to the developer to be selective as to what he uses for criteria. and thereby ensure good resuilts.

For example, in one portfolio, I have MA 50 > Ma 100 locked. to try to assure  trend. Of course many currency pairs are not trending every day, I have a strategy on many charts ion order to get trades.

I try to be very selective, thereby reducing the number of bars required for testing.