Topic: Proposals to improve the software
I'm really impressed by the quality of FSB!! I like it very much.
I have a few ideas to improve it, so I welcome any comments on them:
- Backtesting with v-fold cross validation. Backtesting is one of the most important stages of a strategy development. So far I haven't found any program capable of performing a so-called v-fold cross-validation. FSB is already good in what it provides (OOS tests), but it's not enough to prevent over-fitting. I think the best way is to divide the entire period under analysis into x periods of equal size (the "folds") - for example, divide 1 year into 12 months - each period then being divided into a training and test samples. A separate optimization would be carried out for each of the folds. The best system can then be selected using different criteria. A good one is calculating a sort of risk factor such as average CAGR / [GSD of each period's equity growth], or average CAGR / average drawdawn. Systems with very poor performance in just one month could also be filtered out. This method is very powerful and is used is many statistical packages (but not those with applications related to the stock market).
- Work works well lately. Another interesting backtesting possibility would be this "WWWL". It is in my view a mistake to optimize systems with long datasets spanning through years and years. This is because structural economic conditions change (eg bull vs bear markets), or more simply because a currency or equity may move from a sideways movement into a trend period. When we optimize a large dataset we are basically optimizing our system for the most common price pattern. So the system will perform well under certain conditions and badly under others. My idea is to optimize the system every month and use the optimized parameters as inputs for the next month, and so on. The backtest would therefore be a combination of several small backtests, but all would be done in one go. This is contradictory with my previous idea; this system would clearly be over-fitted, but with some changes it could be a neat feature.
- Do we need so many indicators? I'm not sure about the need to include so many indicators. Would it be possible to make a pre-selection of the less correlated ones in order to increase the efficiency of strategy generation?