As a newbie to this I'm wondering how to make a success story out of this because I can't predict the future. And that's the problem of course. I just produced some very good EA's running on data from 2003 to 2012. I made a portfolio EA of 20 of the best ones and it shows a very satisfactory profit and a great equity curve. But when I run that same EA on OOS data from 2013 to 2019 it's a flop, a real loss maker. So what am I doing wrong?
Here is just an idea that I use to make sense of strategies that I data mine using EA studio. It is important to note that I am not after predictive modelling techniques that many favour in using this form of data mining software. If I was, then the workflow process and design logic would be completely different. I am looking for price following techniques as opposed to predictive techniques like mean reverting systems.
I always define a broad design logic within which I then run my workflow process within. This logic is critical as you need to have a good understanding in what market conditions your EA's perform or under-perform in. Remember it is the market condition that determines your profitability in this game....not the performance results of your strategy. If market conditions are favorable then your strategies should also perform. If unfavourable then they should not perform....however most importantly, they should still protect your capital and not lead to excessive drawdowns .
If your EA's design logic performs significantly differently to the market condition, then they are most likely curve fit results which will fall off the cliff when going live.
RESTRICTING THE DESIGN LOGIC
Before I commence my workflow process I first need to establish broad design parameters which restrict the data mining solutions that can be produced.
For example, I prefer momentum/trend following models and therefore ensure that acceptance criteria allow for realistic return/drawdown ratios which I base from industry benchmarks (CTA trend following funds). My return to drawdown ratio over a 10-20 year backtest regime is therefore realistically set to 0.5.
I also accept a small minimum count of trades as I am only after those models which infrequently trade during high volatility regimes. I also set realistic minimum profit factors (>1.1) that must possess at least a very small edge. You will find that many people suggest a higher sample size is essential....but this is not the case for trend following/momentum models. You actually want to decrease the sample size and avoid over-trading.
I restrict my indicator list to a small class of classic momentum/trend following indicators that I fully understand.
I also use a preset indicator using a long range 200 period SMA filter to ensure that my trades are during more exotic volatile market regimes. Furthermore I tend to operate on the longer term timeframes (D1 and H4) where noise and frictional costs such as slippage is less prevalent.
I also apply a tight stop to each strategy (which I visually determine per instrument using MT4) and a trailing stop condition with no profit targets applied. This ensures that all strategies which are generated are trend following in nature and 'cut losses short and let profits run'. It also ensure that all strategies have positive skew which is important for risk management purposes.
Once these broad design conditions have been set, I then data mine on as much data as I can get my hands on.
DATA MINING SETTINGS
I always test using 30% OOS and use optimisation across the IS component only, For example let's assume I am data mining to develop strategies for implementation post 1/1/2016. I will therefore select a data range of say 1/1/2000 to 31/12/2015 as my data horizon and allow for 30% OOS in this time series.
I also use a stringent Monte Carlo validation of 90% on (randomise indicator parameters) to eliminate the potential for curve fit results.
I only use multi-market validation when I have multiple instruments with very similar volatility profiles (in terms of pips). Unfortunately I cannot use this feature greatly until Popov includes volatility based position sizing into the selection criteria.
I run the test on 4 sessions per instrument over about 48 hours to generate a large number of valid strategies. I then pass these into the validator to further compile and eliminate correlated strategies. I then have a check of the equity curves for each strategy and visually eliminate those which appear to have very adverse volatile profiles.
So lets say that I have generated 110 strategies....after validation and a visual check, this might reduce to say 20-30 strats that I am happy with.
I then consolidate these successful strats into a portfolio EA and run this through MT4 on my brokers account and then include local currency treatment and slippage to make things more realistic.
This then generates an equity curve for the portfolio.........and here is the important bit.
You then need to visually map the equity curve result to the market data to see if the design logic of the portfolio makes sense. For example, if your EA's are based on trending/momentum design logic, then you want to ensure that your portfolio is profitable during these periods. If market conditions are not trending, then your portfolio should not be performing well and should be stagnating or incurring drawdowns.
If your portfolio is performing well during non trending market conditions....then it is highly likely to be curve fit.
Here are a few examples of how I map the portfolio equity curve to the market condition. If the mapping process makes sense, then you can be confident that you are not the victim of curve fitting or 'convergent' EA's that are highly likely to fall over in the future.
https://i1.wp.com/atstradingsolutions.c … .png?ssl=1
https://i1.wp.com/atstradingsolutions.c … .png?ssl=1
https://i2.wp.com/atstradingsolutions.c … .png?ssl=1
If you compare performance results post 1/1/2016 against the market data then the performance results make sense and you should be happy with it.
Then you do the same process to a vast collection of other instruments to have portfolios across many asset classes.
Diversification and risk-weighted returns is what this game is about