...The Effects of Backtest Overfitting on Out-of-Sample Performance
http://papers.ssrn.com/sol3/papers.cfm? … id=2308659

I would like to draw your attention to this very important paper. Anyone using a program such as FSB must read it, otherwise he/she will surely be disappointed.

This paper corroborates my opinion that avoiding over-fitting is probably the weakest point in FSB. For example: in one wants, in FSB one can filter strategies with disappointing "out-of-sample" results. It happens however that, by definition, such a filter would immediately turn the supposedly "out-of-sample" into "in-sample" data, misleading the strategist into thinking that he/she found a winning strategy. Most likely, as the paper demonstrates, a strategy found with this algorithm would be a disaster when tested with real out-of-sample data.

Programs such as FSB require therefore a considerable knowledge of statistics in order to be rightly used. Read the paper carefully. I hope this paper can enable practitioners of adopting the best practices when devising strategies to maximize profit, avoid big disappointments, and keeping expectations at the right level.