I've had a new experience today. 

I have a strategy on my account which looked great in FSB.
Since the live performance was bad, I first thought the timeframe for optimization was wrong, I've modified it and thought; it should work now.

But to my surprise the performance got even worse.
And I think that I've experienced over-fitting.
The problem is all strategies are generated/optimized in a similar way, very often at the beginning of your data horizon it wins many consecutive times to create a buffer to survive the coming drawdowns.
But the starting point of your strategy going live cannot be optimized or simulated cause it's in the present.

With my particular strategy I had a profit factor above 3 and in fact it is my worst performing strategy.

Today I've experimented a bit with the data horizon and set it to just one or two months (eg. September to November).
And to my suprise, even if the original settings were optimized with 12 months of data, the result was totally different.
It was really, really bad, nothing compared to the 12 months range I had before.

Then I compared the better performing strategies and voila their result was much better.

I know, that such a short data horizon is statistically not relevant.
But I will use it as an additional tool, which hopefully will reduce the risk of over-fitting, what's quite hard to detect.