Here are a few thoughts to consider.
With data mining software, one of the key issues you need to address is 'curve fitting'. Curve fitting is not only restricted to data mining but is also a feature associated with mean reverting strategies that possess negative skew over the long term.
There are two broad types of strategy within which any strategy can be classified. These are convergent strategies and divergent strategies.
Convergent strategies are 'backwards looking' with a central assumption that price will converge to a past equilibrium. In a sense these strategies are predictive in nature and include all mean reversion strategies, value investing approaches, grid trading and Martingale methods. Given the repetitive nature of price converging towards the current market equilibrium, these strategies are typically quite complex in nature based on the 'expected certainty of this outcome'. The complexity is a symptom of 'fitting' the strategy to the nature of the current market condition.
These types of strategies are what most retail traders prefer given their linear equity curves and high win rates over the periods that equilibrium persists....however when market conditions change to new equilibria, the past stable equilibrium is not respected and you quickly find that your strategy no longer performs as expected and you are left with a sequence of large unfavourable losses that frequently ends up in account blow ups. Because the 'true' equity curve is revealed over the long term, it is advised to use long term data horizons that span a broad array of market conditions to detect them.
If you use short term data horizons to develop your strategy, the chances are great that you are biasing your outcomes to 'convergent styles' that in the long term have negative skew. It is not advised. You will find that this approach means that you need to continuously mine new strategies when existing strategies fail....and you will become a victim of 'strategy hopping'. Be aware that the need to continuously replace your strategies is a form of 'market timing'....and there are swags of white papers available that demonstrate that market timing methods used to 'switch on or switch off' your strategies is a fools errand as the decisions are always lagging in nature.
The other class of strategy type is referred to as 'divergent strategies'. They are the opposite to convergent styles in that they are non-predictive in nature and simply operate off the principle that stable market conditions will not persist and that conditions will move in the future towards new equilibrium. The location of this new equilibrium is unknown given market uncertainty....and the only assumption you use is that it will either be higher or lower than the current equilibrium. As a result, these strategies are said to 'follow' price as opposed to 'predict future price'.
To avoid trading during normal market conditions representative of the current market equilibrium what they do is apply trade filters so that it is only during more exotic times where price moves outside the 'normal', that these strategies participate in the market. The filter is used to significantly reduce your trade frequency and only participate when market conditions become more 'exotic' which provides hints that a new equilibrium may be established.
The way these strategies are developed is through logical design build considerations that enforce an asymmetry into the design through cutting losses short and letting profits run. They all use stop losses and tend to use different forms of trailing stop to manage adverse risk exposure in cutting losses short....but they keep their profit potential open and do not use profit targets. The reason for this open-ended condition statement is simply to be in a position to catch the occasional white swan that according to Ed Seykota 'pays for them all'.
Given the 'cutting losses short' condition applied to these strategies and given the applied filters, these form of strategies are far more long lasting than their convergent cousins. Furthermore these form of strategy only require very simple rules to allow for 'degrees of market freedom' in price movement. They typically have volatile equity curves over their long lifetimes where drawdowns are a result of many small sequential losses but ........ so do convergent strategies if measured over the same time-spans...in fact the vast majority of convergent strategy equity curves over these long periods result in >100% failure. We unfortunately only see the nice linear equity curves when convergent strategies are performing.
So in essence if you are interested in mining over long term data, then consider using the following principles in your workflow:
1. Preserve at least 30% OOS to your long range data set. This ensures that optimisation is only performed on the In Sample 70% component. The OOS component is simply used to ensure that the previous performance metrics are maintained in the future. Discard all strategies where the OOS component falls well below the IS performance metrics. The closer the OOS performance against the IS benchmark the better.
2. Ensure stops and trailing stops are allowed for in the reactor settings and keep profits unlimited in nature.
3. Use a preset filter to avoid trading during the normal 'day to day' conditions.
4. Use the multi-market feature to test your strategy on other unseen data. The greater the number of alternative markets the strategy works on the better.
5. Use stringent settings on your Monte Carlo...but do not set high acceptance levels. Expect variance in your results, but ensure that the MC array is overall generally profitable. This is used to ensure that your strategy requires a 'weak edge'.
Remember that over the long term say 20 years plus...your equity curve is quite deceptive. Despite the 'apparent' fairly linear nature of the curve when viewed from a height, when you drill down, there is considerable volatility and long periods of stagnation. For those looking for 'instant' results, it is unlikely that they will have the patience to tolerate them.
So the way we overcome this volatility and uncertainty associated with the future of a particular strategy, is to diversify into a broad array of different strategies, each with slight positive expectancy that are uncorrelated in nature. The 'uncorrelated' requirement is used to ensure that over 'most times' your portfolio has positive momentum and your 'stagnation period' of your entire portfolio is minimised. The uncorrelated nature of each strategy also ensures that the drawdowns of each component strategy are reduced when compiled into a portfolio.
So the decision to turn off the strategy is a long term one in the divergent space but a short term one in the convergent space. It is very difficult to determine between drawdowns versus total risk of ruin with convergent strategies.....but drawdowns in divergent strategies take a long time to play out plus under diversification where each strategy comprises a small portion of your total equity....you are very unlikely to meet 'total risk of ruin'.
Under a diversified portfolio of divergent strategies you continuously monitor live performance against your long term backtest. You set performance benchmarks for your live trading that uses the backtest performance metrics as a guide. You only drop the strategy when those benchmarks (based on long term data) are exceeded. In this game...you need to rely on the Law of Large numbers as your guide.
I hope this helps.
Diversification and risk-weighted returns is what this game is about