Topic: Can optimizing in FSB lead to curve fitting?

When too much optimizing lead to curve fitting and how not to optimize too much? Can you share your experiences?

Re: Can optimizing in FSB lead to curve fitting?

In the optimizer select OOS 30, this will exclude 30% of the bars from the optimizer

Re: Can optimizing in FSB lead to curve fitting?

Does OOS 30 removes just random 30% of the bars?
It would be nice to be able to optimize on one part of the data and test the generated/optimized strategy on a different part of the data. Ie. split the whole dataset into disjunct training and testing data.

Is this possible?

Re: Can optimizing in FSB lead to curve fitting?

mcfire wrote:

Does OOS 30 removes just random 30% of the bars?
It would be nice to be able to optimize on one part of the data and test the generated/optimized strategy on a different part of the data. Ie. split the whole dataset into disjunct training and testing data.

Is this possible?

goto menu - ...MARKET/Data Horizon

Select start and end date you want to optimize for......

Hit Optimizer

Then afterwhich, you can go back to Data Horizon menu and turn off start and end dates.

Re: Can optimizing in FSB lead to curve fitting?

soladood wrote:
mcfire wrote:

Does OOS 30 removes just random 30% of the bars?
It would be nice to be able to optimize on one part of the data and test the generated/optimized strategy on a different part of the data. Ie. split the whole dataset into disjunct training and testing data.

Is this possible?

goto menu - ...MARKET/Data Horizon

Select start and end date you want to optimize for......

Hit Optimizer

Then afterwhich, you can go back to Data Horizon menu and turn off start and end dates.

no offence Soladood, but I have a better suggestion - use OOS in the optimizer/generator! Then you can see the result as it develops, and you save time as you change the data horizon only when new data set is needed, otherwise you set data horizon 2x instead of 1x in one optimization cycle.

30% OOS is not on random bars, it's the last 30% part of data, so it provides very good walk-forward testing ability if optimization is in your to-do list.

Re: Can optimizing in FSB lead to curve fitting?

footon wrote:
soladood wrote:
mcfire wrote:

Does OOS 30 removes just random 30% of the bars?
It would be nice to be able to optimize on one part of the data and test the generated/optimized strategy on a different part of the data. Ie. split the whole dataset into disjunct training and testing data.

Is this possible?

goto menu - ...MARKET/Data Horizon

Select start and end date you want to optimize for......

Hit Optimizer

Then afterwhich, you can go back to Data Horizon menu and turn off start and end dates.

no offence Soladood, but I have a better suggestion - use OOS in the optimizer/generator! Then you can see the result as it develops, and you save time as you change the data horizon only when new data set is needed, otherwise you set data horizon 2x instead of 1x in one optimization cycle.

30% OOS is not on random bars, it's the last 30% part of data, so it provides very good walk-forward testing ability if optimization is in your to-do list.

I do actually use the OOS on the 30 setting - but i myself am building lots of profitable strategies in FSB, but when i start to trade them on demo accounts (and a couple of live ones) they've all turned out to be losers.... hmm

So now of course - its time to explore this new feature of the over-optimization report but im having a little confusing time trying to understand how to read the data to be honest.

My brain is starting to fall over Im afraid..... wink

Re: Can optimizing in FSB lead to curve fitting?

I know the feeling, Soladood  lol  I haven't used the report thing either, but I think it should give an insight about which parameters actually work, and which lead to hell. My theory at the moment is that if I optimize very often, it should stay in the green zone, so I've kept things simple by not using the report option.

I've experienced that there are certain key points in data where a strat failure is definite, whatever I do to overcome this issue in a forward test is absolutely useless. I've had well working strategies (confirmed in a forward test), and in live I'm lucky if it works for 3 weeks.

I wish we could define a method or an approach about how to develop a working strategy in a reasonable timeframe.

Re: Can optimizing in FSB lead to curve fitting?

footon wrote:

I've experienced that there are certain key points in data where a strat failure is definite, whatever I do to overcome this issue in a forward test is absolutely useless. I've had well working strategies (confirmed in a forward test), and in live I'm lucky if it works for 3 weeks.

I have experienced the same situation. Forward testing on demo account shows good results put in live trading the strategy will fail after a 3-4 weeks. If I put the same strategy back into demo account I will get good results, but it fails if I put the strategy back into live account.

I don't know how to explain this.

Re: Can optimizing in FSB lead to curve fitting?

zenoni wrote:
footon wrote:

I've experienced that there are certain key points in data where a strat failure is definite, whatever I do to overcome this issue in a forward test is absolutely useless. I've had well working strategies (confirmed in a forward test), and in live I'm lucky if it works for 3 weeks.

I have experienced the same situation. Forward testing on demo account shows good results put in live trading the strategy will fail after a 3-4 weeks. If I put the same strategy back into demo account I will get good results, but it fails if I put the strategy back into live account.

I don't know how to explain this.

This might the case with some brokers having different feed data with demo and live accounts. (perhaps???)

maybe try optimizing the strategy for live data only.

Re: Can optimizing in FSB lead to curve fitting?

soladood wrote:

maybe try optimizing the strategy for live data only.

can you be more specific about it? anyway I found out that my broker's live feed differs a bit from the actual saved data, don't want to raise a conspiracy here, but it's hell-of-a great way to back stab one's carefully optimized strategy.

Re: Can optimizing in FSB lead to curve fitting?

footon wrote:

I know the feeling, Soladood  lol  I haven't used the report thing either, but I think it should give an insight about which parameters actually work, and which lead to hell. My theory at the moment is that if I optimize very often, it should stay in the green zone, so I've kept things simple by not using the report option.

I've experienced that there are certain key points in data where a strat failure is definite, whatever I do to overcome this issue in a forward test is absolutely useless. I've had well working strategies (confirmed in a forward test), and in live I'm lucky if it works for 3 weeks.

I wish we could define a method or an approach about how to develop a working strategy in a reasonable timeframe.

hmmm - this still leaves me none the wiser:)

I tried developing strategies optimizing it often which then produces a very nice looking equity curve - but going forward, thats another story - hence ive now come to learn about the curve fitting phenomena !

Ive tried using different broker feeds - but doesnt fix things.

This curve fitting instance though, has me intrigued - personally for me (logically speaking) it just doesn't make sense.

If a successful trader can trade a certain setup that appears everyday, have it backtested nicely - why cant that be automated successfully going forward?

sad

Re: Can optimizing in FSB lead to curve fitting?

footon wrote:
soladood wrote:

maybe try optimizing the strategy for live data only.

can you be more specific about it? anyway I found out that my broker's live feed differs a bit from the actual saved data, don't want to raise a conspiracy here, but it's hell-of-a great way to back stab one's carefully optimized strategy.

Backstabing????    No not at all smile

All im trying to find out more about is this instance of curve fitting - personally speaking, this is something i need to study more carefully - logic doesn't seem to add up with my knowlege bank - there are obviously other elements at play here - and I would very much like to learn what they may be.

For me - ive optimized a stategy for 1 broker feed, then apply it to another broker feed, the results are different - but this seems to be more noticeable with strategies using 1M and 5M data. When I develop a strategy for 15M and up, the effect of this difference becomes less and less.

Ive also noticed even the same broker feed can have differences if downloaded from different locations - hence the strategy that was optimized for 1 broker at 1 location, can have a different equity curve if I go travelling for example, download the same data from another city - the results are different again.

So now, all i can figure out is that there must be inconsistencies in communication lines at different locations - perhaps some kind of error checking could help with this - i dont know?

I would be keen to know whether your well optimized strategy can trade the same at a different location for example?

And if it does - could you share with us what this 'special sauce' is please smile

Re: Can optimizing in FSB lead to curve fitting?

I think you lost me in previous post, I was trying to say that a broker can alter the behaviour of one's strategy with just tinkering with data.

Example:
http://s1.postimage.org/1boemoamc/Oscar2.jpg

The first is live feed seen from FST, the second one the same data saved into FSB - closing prices and volumes differ! And then look at the indicator values - they don't match because there are data discrepancies, so in the end the result is that trades don't match in FST/FSB comparison. And optimization is affected, and obviously real trading as well.

What you say about data differences if downloaded from different locations... honestly this is shocking, I can't make sense of it. This brings me to a conclusion that optimization will never work, and the sooner we abandon indicators (which calculations are based on prices), the better.

I must correct myself once more - I have a few "well optimized" strategies, but they've all failed in real-time trading, and I haven't tried different trading locations either, so failure is all I can share at the moment unfortunately lol

Re: Can optimizing in FSB lead to curve fitting?

This is a real problem, I can't trust my backtested results because of that. The second problem is related to the differences of the DEMO and LIVE trading. Forward thesting on demo account might be very successful, but the strategy fails in LIVE trading. How to explain this?

In FSB I am using 1-1,5 pip slippage to make backtesting results more reliable.

Re: Can optimizing in FSB lead to curve fitting?

Perhaps there's a way to categorize why they fail:
1) do trades lose by being stopped out by 1 pip (or very small amount)?
2) does FST not give signals due to different live data?
3) does FST give the signals late or early due to different live data?
4) is the signal direction correct in backtesting, then wrong in forward testing or live trading?
5) other
if one pattern stands out, that will lead to more questions to test.

For me, it was 5). I use fixed limit and stop amounts. My backtesting signals entered positions before the price made a move of my limit amount. In live trading over the following week, the signal would enter a position, then the price would go up by a little bit more than my stop amount, then down a little bit more than my stop amount. You can usually see this after NY closes and Tokyo opens, it's flat then goes up-down or down-up about 30-40 pips, about 35% of the time. Long or short didn't matter, either would have lost. This situation also happened in backtesting, but not very often, so the system looked profitable. The overoptimizing found signals in the backtest that would avoid entering in the midpoint, but those conditions and signals did not hold up going forward in live trading.
Solution: still searching for it, but I think it is a system that gives fewer signals, and figuring out if it will be a straight-shot day or an up/down day; or use exit signals instead of fixed stops / limits.

Re: Can optimizing in FSB lead to curve fitting?

I have experienced the same thing with fixed stops/limits, last year I gave up those things and strategies went profitable straight away, but in live the relative drawdowns were not acceptable for me, so I came back to stops and limits.

I've been thinking about optimizing with tighter stops/limits, but opening them up a bit when going live, theoretically if entry is made in the right direction, a wider stop, for example, should save the trade, but only if the direction is right.

All in all it's the entry what counts with stops and limits, and price based indicators just don't deliver, and price differences in data amplifies it even further.

Krog, what do you mean by overoptimizing?

17 (edited by krog 2010-12-21 23:55:33)

Re: Can optimizing in FSB lead to curve fitting?

By overoptimizing, I mean adding a few indicators, then changing the parameters until the equity curve goes mostly up, so probably same as curve fitting.

Sorry to digress from the live data difference, but this was a rough study I did on stops. I compared a system with a stop (of 35 pips) and without a stop, and both with the same closing signal. Allowed only one trade or lot open at a time. I plotted the win/loss amounts as a histogram in Excel. The Stop system lost more money. The win ratio was about the same, but the stop allowed for more trades. Since it was a losing system, more trades resulted in lower ending balance.

The other interesting thing, looking at the profit-loss distribution from each trade, the number of trades resulting in losses slightly higher than the stop (eg, -35 to -20) had a lower count. I think what happens is the stop absorbs some trades that would have gone past the stop but come back to lose a less amount. This would also help lower the ending balance.

I'm still trying to figure out the optimal way to place the stop.

Re: Can optimizing in FSB lead to curve fitting?

I think true curve fitting can be achieved by brute force optimization, but if there are more than 2 parameters, FSB is not able to carry the optimization out in a brute force method, it has its own non-linear method or algorithm, but the results are kind of dwarfed, and I think there's the problem. One chap posted various algorithms the other day, I think it's worth checking and even implementing in FSB.

I thought about the optimal stop placement for a long time myself, and now I'm kind of developing an evaluation procedure. I have a few combinations of SL/TP, and I evaluate their performance on a longer time span, and those, which are more successful, will be forward-tested.