Two things I've recognized, more acceptance criterias don't make the strategy more robust it seems even to increase the risk of curve-fitting. And somehow strategies for EURUSD seem to have a higher risk of curve-fitting, I don't know why.

I'm reverting back to my earlier approach of backtesting.
I'll generate strategies with a certain data horizon and expand it afterwards to see how the strategies would have performed.
I'll maybe even try OOS I haven't decided it yet.

I think the risk of (over)curve-fitting is the highest risk, and for me the most important topic, cause it's really deceptive and could even be dangerous.

I've seen many great strategies with good or even great stats and their performance dropped like a stone as soon as they leave their inital data horizon.

I have a strategy on my account which looked great in FSB.
Since the live performance was bad, I first thought the timeframe for optimization was wrong, I've modified it and thought; it should work now.

But to my surprise the performance got even worse.
And I think that I've experienced over-fitting.
The problem is all strategies are generated/optimized in a similar way, very often at the beginning of your data horizon it wins many consecutive times to create a buffer to survive the coming drawdowns.
But the starting point of your strategy going live cannot be optimized or simulated cause it's in the present.

With my particular strategy I had a profit factor above 3 and in fact it is my worst performing strategy.

Today I've experimented a bit with the data horizon and set it to just one or two months (eg. September to November).
And to my suprise, even if the original settings were optimized with 12 months of data, the result was totally different.
It was really, really bad, nothing compared to the 12 months range I had before.

Then I compared the better performing strategies and voila their result was much better.

I know, that such a short data horizon is statistically not relevant.
But I will use it as an additional tool, which hopefully will reduce the risk of over-fitting, what's quite hard to detect.

I hope you can understand what I mean

Lagoons -- thank you.  Finally someone else gets it -- at least no one else has bothered to acknowledge this is an important issue.

And I know *exactly* what you mean.  From 30 miles high I've seen what appear to be beautiful balance curves that span several years.  But then you see the Win Ratio is something like 0.51 -- which is lousy.  And you are correct -- from a distance you don't see the draw-downs.  If people would zoom-in then they would see many peaks and troughs. 

Sometimes the troughs last for weeks or months, but you can't see them because the screen resolution of your monitor is 1280x1024 pixels (or worse) and MT4 is attempting to chart thousands of data points.  Since there are more data points than available screen pixels, then the charting software compresses them in order to fit on a chart.

I think many people see a nice shaped equity curve over a huge data horizon but don't realise that the "resolution" (I hope you can understand what I mean) is huge.

So you don't see the drawdowns and (maybe) the lack of performance on a shorter period of time.

If you could scale into it, the picture might be much different.

He showed a couple of charts where his strategy had been back tested against two different data horizons and was wondering why the charts looked so different.

The answer is obvious -- check-out the above link.  His chart comparison also shows why using too much old data when optimizing (that is, computing indicator settings) results in indicator settings that are poorly matched for the current data horizon.

When you look at your own charts it's so obvious.  I'm surprised so few people take notice -- perhaps it's because we only see what we want to see.  I don't know...

Is it more valuable to use most recent data for that as well...?

Good question.  Of course, I can't say for sure, but I can provide some anecdotal observations.  I continue to use strategies that were generated over a year ago (with 2-year-old data).  After refreshing (i.e. re-optimizing) using more recent data then their statistics are as good as before.

So, to answer your question, I suspect it doesn't make much difference what data is used when generating strategies as long as there are around 100 - 200 trades.  After all, these are just simple algebraic formulas whose constants (i.e. indicator settings) are adjusted to suit the current data.  You can continue to use "old" strategies as long as you re-adjust their indicator settings using current data.

But how about the generating of new strategies?
Is it more valuable to use most recent data for that as well, or is it better to use past data for the possibility to have kind of a backtest with the most recent data?

This is most interesting. A lot of great information in these posts.

I have a few strategies that require a lot of data because they are containing  a few indicators and will fail in the short term unless carefully processed with walk forward.

I have another few strategies that are based on patterns that do not require much data.

My good friend, Steve, expects me to disagree with him on the need for data..... hahaha  I am of mixed opinion on this, I suggest that we each have different views as to what we need and those views can vary as time passes.

Because this topic is so important I am changing my folders so that I can conduct some research into different optimization periods ie mark the folders to show the length of the original optimization and then using the handy tools that Steve has developed to measure results as time passes.

On some of my strategies, I have been doing walk forward with another software which demands a lot of data for what it calls 'statistical significance', that has influenced me one way. The results have been, so far, that I get lower win ratios on a few, especially those with more than one or two indicators.

It seems that the lower the number of variables, the shorter time period that is required.

For myself, I have to do some  detailed research to satisfy myself.

I just did small experiment about "less data is better". Before results shown I want first to clear something about my views on using OS (out of sample) checks. There are 2 ways to use it. First one is traditional, if OS is passed then you confirm that strategy is valid and let it trade live. I do not use it anymore, because I did experiment (similar to the below), which showed that OS check do not have any value for future profitability. Also read some information(guess that it was in a paper about "system parameter permutation"), that it is shown/proven that OS pass is the same thing as using little more data, which you exclude for OS in the first place.

And there is 2nd way to use OS, what I use, it is for simulation purposes to find out what really matters in the automated system creation and trading. So if I have idea/question to test out, I do not create bunch of EAs and let it run live trading for 3/6/12 months demo to see if it is working or not. It can take decades to find out what really works or don't this way. I just exclude that wanted data period(6-12 months) and create EAs paying close attention/measure what I want about my idea. After some sample size created (20-100 systems) I then see OS, and make some conclusion if there is some correlation between OS results and my idea, does it improve things or it is just pure random? If no correlation measured/seen then I conclude that my idea is not important in system creation, and if there is, I can incorporate it to my future system creation method/workflow (of course creating systems always with fresh data ).

Like this I found couple things and I can share few examples - 1. Optimized strategies always perform better than not optimized (made maybe 3 experiments about this), also FSB optimizer is reliable compared to other methods, for example like choosing variables from "stable" regions. 2. Over-trading strategies perform very badly in the unseen future, so quality over quantity is important 3. Systems must be re-optimized or thrown out and changed with new ones over time, because there is some point then seemingly winning portfolio turns around (however I still believed that more data is better for a reason that systems can see more different market regimes)

So let's get back to experiment results

I generated 20 system portfolio on 5 different instruments. Then optimized 2 EA versions , one for all 4 year data, and other for 8 months, like Sleytus used in his example. After that I checked what for 2.5 months they would have done in most recent unseen market. 0.01 lot used on H1 timeframe.

My expectation was to get long and short optimization periods both negative portfolio balances on last unseen 2.5 months, because quick random generated strategies with lots of variables and no any robustness testing is usually not a good thing... However both comparable portfolios is with profits, but I consider it to be because of lucky 2.5 months. Also I expected that longer optimization period to outperform shorter period, that was main experiment point. BUT I got both very similar results, almost identical. However with more data it can diverge, and a real winner can pop up. There is a need to reproduce this experiment with more systems, and with different OS months to have bigger sample size to draw conclusions. But my believes are already shaken out, and maybe I will continue this experiment, who knows what results can be found and it is probable that less data will be better, especially it can be true if I start to walk forward on couple OS windows. Also  combined with Steve's sidekick software help you can get very powerful and satisfactory trading methodology I think that Sidekick should be a lot more powerful with less data, than more data, because it easier to get great looking PL curves, so you know very quickly when your system is not performing as expected. Compared with more data, you always have longer and deeper drawdawns, so you can not turn off systems quickly objectively like you can do it with less data. This thing alone can have edge. Hope it make sense what I try to explain here.

It makes me think, maybe what Sleytus already trying to say to us long time is if you generate strategy on long period, and optimize it on short/recent data you can take best things from 2 different views/perspectives - While you generate on long period, you know that systems logic is robust in long term, and while you optimize with only recent data you tune up strategies variables for a most recent market condition. Thank you Steve for trying hard to rise these questions and challenge our believes.

Have a nice day,
Irmantas

Steve wrote - Forex is a different type of beast than other types of investments (which could be a topic for a different thread).  I think the primary reason I can't buy into what you describe is I think the stormy periods would cause draw-downs that would eventually wipe out my account before it had a chance to turn around.

Yes, Steve, without proper weighted risk management, one may not survive the long drawn down and the account can easily wiped out.

I believe Rich, therefore emphasized the importance of Robust EA and proper weighted risk management so that we learn the art of putting together a group of EA that can help keep the account "break even/keep the head floating above water level" during the prolong ranging period (90%) to catch the King hit of 10% windfall.  Without proper weighted risk management, one can blew his account very easily due to over trading, likewise with proper risk management, one may master the art of surviving in the long haul and not just short term success.

Lastly Rich, thanks so much for your delightful contribution and simulated our brains.  Could you kindly write in another separate post about your journey, the path and learning process you took to become where you are now.  I believe a lot of us, who are aspiring to become a fund manager one day, would learn from you and understand the stages of professional growth/milestones, we need to take to be successful.

Can you open another post, something like a personal blog to share whatever you deem is helpful for the aspiring traders.  Can you share the pitfalls to avoid and the road to success that seem so elusive to many of us.

Finally, the pointers I take off from this discussion are:

1) Robust EA - The importance of building more robust EA for long term sustainability (using longer historical data, to expose our EA to be trained in varied market conditions).  Hence, MC and Multi Markets are good tools to use for creating robust EA

2) Divergent EA, though may have terrible short term equity curve, they are by far, more sustainable in the long run compared to convergent (mean reversal EA).

3) Robust Portfolio - for long term survival, it is important to blend different types/diversification of EA to keep a well balance/composite of EA to construct a good, robust portfolio, together with weighted risk management, it is use thereafter to help one maximise the returns/profit.

4) So this is what I would do...

a) In the area of EA development - build more robust EA

b) In the area of Portfolio management - use diversification of EA with uncorrelated strategies/markets to keep a good blend of EA to tide through the storm.  I'm thinking, is it possible, in addition to a robust portfolio, can I still find a sweet spot (by observing my EA's behaviours) to turn on/off my EA (while keeping them running in demo to collect performance data) based on certain metrics?  So instead of having to resign to the fact that 90% of the time, my EA trying to stay a float (of that particular EA, but for a blended portfolio, while some EA are losing, others are winning because the portfolio selects uncorrelated EA/markets), I'm thinking of switching them off so that I don't have to endure the painful experience of bearing with the long drawn DD.

c) In the area of Risk Management - with proper weighted risk management, make my portfolio "unsinkable" so that we can survive over the long haul while many retail traders come and go.

Thanks once again to both Steve and Rich for such wonderful discussion that I certainly look forward for more.

All good mate and thanks for the chance to contribute :-)

Rich

As a research biochemist in a previous life and a developer of wireless diagnostic tools in this one, I've looked at more than my share of data.  And through it all I am the biggest fan of the KISS principle.  I approach forex sort of like a mechanic -- always keeping KISS in mind -- and I think Popov's software gives us the best tools for this type of approach.

The principles and techniques you describe and refer to are way over my head.  That doesn't mean they aren't correct, only I have no means to comment on them.  Also, even if I did understand them, I have zero clue how I would implement and apply them.  Furthermore, at the risk of sounding like a dimwit, I will step out onto the plank and venture to say I don't think they apply here.  Forex is a different type of beast than other types of investments (which could be a topic for a different thread).  I think the primary reason I can't buy into what you describe is I think the stormy periods would cause draw-downs that would eventually wipe out my account before it had a chance to turn around.  Big investors, institutions, banks, etc. can ride out the storms, as you say, but my smallish accounts could not.  Yes -- I know about risk and money management -- and do take care to ensure I live to trade another day.  We trade in the present -- there's no past or future -- which is why current profitability is really all that matters.  And then tomorrow becomes the current and, hopefully, the strategy again exhibits current profitability -- at least that's the plan.

Adapting to current market conditions -- absolutely -- that's why I claim refreshing strategies with recent data is very important.  I mentioned the mechanic analogy -- just like changing your oil every 3000 miles or 3 months (whichever comes first) keeps your car humming, it will do the same for your forex strategies.

Rich -- I thank *you* for the conversation.  One of the best I've had in a long time.

1. Why do EAs require 20 years of historical data,  whereas manual traders can trade profitably using only the last 3 days from a price chart?

The sample size is less important than the different array of market conditions encountered. For example in the High Frequency Trading environment, the sample size is huge (within a single market condition)....and strategies deployed in the very short term environment are very unlikely to still perform if conditions change (some exceptions here like order 'front running' bots). We use sample size more as a proxy to ensure we have a better chance of being exposed to alternating market conditions.

Certain classes of strategy (eg. divergent strategies) such as trend following and momentum methods 'rely' on non-stationery conditions and make their profits with market disruptions. Their performance results are a consequence of their style. They have Low Pwin% but relatively high R:R ratios....but also have a divergent risk feature being their positive skew that makes them robust. This characteristic signature creates the volatility in their equity curves. This doesn't mean they are 'worse strategies' or have 'higher risk exposure'. In fact they are far more long lasting than their convergent cousins and there are a swathe of white papers to demonstrate the efficacy and long lasting nature of these approaches.

The times these class of strategy shine are during volatile less stable market conditions. At other times they churn to simply keep their heads above water while waiting for 'crisis alpha' moments. Statistically these strategies are deployed to catch 'fat tailed' events on either side of the probability distribution.

Have a look at the following chart that demonstrates the 'churn' that is required to generate long term solid performance metrics for a style of divergent strategy (eg. a momentum breakout trader). To achieve these results (which appear easy in hind-site), you have to endure a painful long-winded process of disappointment. Approximately 90% of trades taken were simply to keep the head above water...but the anomaly (10% of trades) were what led to the overall strong performance metrics of this solution. The reason that alpha (arbitrage opportunities) persists for trend following and momentum is that the game is very hard and requires intense discipline and patience. You simply are diversified and follow price again and again and again with the same recipe....and let the market decide when it gives you a windfall.

The flip-side however to this argument is that mean reverting convergent systems may have a hiatus in the sun over a short term interval.....but are very unlikely to last more than a few years duration. They rely on the leptokurtic peaks of the profit distruibution as opposed to the 'fat tails' to make their bread and butter. Under a normal distribution, no strategy wins. Under a Non-Gaussian distribution you get two broad styles of trading strategy that can harvest arbitrage (the tails and the peak of the distribution).

These convergent styles of trading strategy work on the principle of reversion back to an equilibrium whereas divergent styles are forward looking and simply assume that conditions will change. Convergent styles are backward looking and rely on prediction, namely that price has taken an excursion away from a stationary equilibrium....but in the future will revert back to this 'known point'. The assumption typically used by this type of strategy is that the market becomes over-valued or undervalued and will revert. This is great while it lasts but totally disruptive while it doesn't.

For example the chart below represents the Trend Following Index which comprises the equal weighted index of 54 diversified fully systematic trend following FM's. You will note that market conditions post GFC have changed where we get extended periods of mean reversion (that these funds under-perform in) inter-dispersed with short sharp periods of strong volatility and momentum (aka trends). Is trend following dead? Well when viewed from this height you can see it isn't. The overall line of best fit generates a CAGR of 9.3%. The market conditions can in hind-site be attributed to central bank intervention during mean reverting periods, whereas the volatile market disruptions can be attributed to events such as Brexit, the GFC, oil shocks etc. The key to this diagram relates to how different strategies fair under different market conditions.

You can see from this aerial view that a single market conditions such as a mean reverting market context can last for extended periods up to 10 years etc....however this is when conditions change, you get a massive disruption to participants where our mean reverting cousins become extinct :-)

If you only focus on the short term time horizon to derive your strategies, chances are you will skew your portfolio towards 'convergent styles' that bear far more intrinsic risk. They typically have high Pwin%, lower R:R but most importantly have negative skew which is a sure sign that they bear intrinsic risk far greater than what is revealed by their 'closed position' equity curves. If you had access to their floating equity curve....you would see this intrinsic risk in action. 

Other strategic styles that have negative skew are Martingale and Grid Trading variants. The symptoms of this style of strategy is that their equity curves look glorious for a period of time....until they don't....and the account blow up comes without warning. You would think that you could turn off these strategies in time to save yourself from account blowup....but rarely is this the case as a degree of prediction is required to know when to turn them off.

It is not knocked mate....and is a valid and noble objective provided you take the big picture of market uncertainty and the impact of fat tailed events into account. The more you diversify your strategies, the more you tend to come into contact with the fat tails. You can capitalise on this on the principle of catching 'white swans' using divergent tactics....but if you are a convergent trader who is diversified, these 'white swans' turn 'black' very quickly.

Diversification is not just market specific, but also relates to timeframes chosen and different strategies deployed. In fact system diversification is the best way to 'become diversified'. So if you trade a single market and timeframe but do so with a variety of different strategies, you are actually diversified and will come into the realm where fat tails really matter as they are much more frequent than a normal distribution would imply. :-)

Definitely Steve. But it helps to think in terms of 'uncertainty' about the future and your ability to address risk to have a longer livelihood in this game....namely the likelihood and consequence of unfavourable market conditions. Engineers like to think in terms of certainty where there is a reason for why things happen...so they build the perfect bridge.....but the market...being a complex system is not predictive but rather adaptive in nature as principles of emergent relationships drive cause and effect....as with any complex system.....and the market always has the last laugh.

Playing in the sandbox probably would be regarded by the research houses in Quant funds as a bit of a slanted statement. There is much we can learn from the research community that work in the industry that may assist us in our path. Programmers have to learn their craft from theory before they apply it...the same as Fund Managers. Unfortunately such rigor is seldom the case in the retail community.

I would just say that optimised backtests derived from EA studio 'unless stringently treated' will never produce similar results
in the live environment. This is not to say the software is bad....but rather, you need to be aware of it's current limitations.

Cheers guys

I am taking a break for a while as I have mouthed off a bit too much lately. :-)

Thanks for the conversations all.

Rich