• Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Topic: Over-optimization (curve fitting)

Over-optimization (curve fitting) is a process of adjustment the indicators' parameters to the concrete data.
Computer technology can be easily used to over-optimize a trading system and produce something that looks good. An optimizer tests thousands of parameters value and can find the exact numbers making the optimizing strategy very "profitable". The problem is that "Profitable" strategy is profitable only applied on this data. However, it is highly possible the system to fall apart in real world.

Strategy parameters that work over a range of values are robust.
   If the parameters of an indicator are slightly changed and the performance falls drastically, beware.
For example, if a system works great at MA(20), but does not work at MA(19) or MA(21) you have an over-optimized system.
    On the other hand, if your strategy works with Moving Average with a period from 15 until 25, your system is much more robust (and reliable).

Popov's Website

2 Reply by Melvin Lau

  • Melvin Lau
  • Guest

Re: Over-optimization (curve fitting)

Can we have a Strategy Fitness function check for future version.

What I men is let FSB calculate the profit with random combination of +/- values of the chosen parameters, if one of them is negative it should flag it as a non-robust strategy. If all are profitable it should have a distribution chart of the profits from various parameter  combination deviating from the chosen parameter.

This will give us an ide of how profitable this strategy is going to be in the real world.

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Re: Over-optimization (curve fitting)

Yes, I'm working on it. It will calculate the strategy by deviating the parameters of the indicators with 5 - 10% and will show the result in a profit/deviation chart.

Popov's Website

  • Registered: 2007-04-18
  • Posts: 70

Re: Over-optimization (curve fitting)

An example of an over fitted strategy, with nice number to look at, but is useless for trading in present time as the profit is flat.

Forex Strategy Builder V 2.4.0.3-2 Unstable
Strategy name: New
Exported on: 4/18/2007 4:23:50 PM

Market: USDCHF 1 Day
Spread: 4 pips
Swap Number: 1 pip

Balance: 40563 pips
Minimum account: -270 pips
Maximum drawdown: 1604 pips
Time in position: 71 %

The same direction signal - does Nothing
The opposite direction signal - does Nothing

[Opening point of the position]
Bollinger Bands
     Open long at the Up Band
     Smoothing method  -  Simple
     Base price  -  Close
     MA period  -  25
     Multiplier  -  1.95
     Use previous bar value  -  Yes

[Closing point of the position]
Steady Bands
     Close long at the Up Band
     Smoothing method  -  Simple
     Base price  -  Close
     MA period  -  29
     Margin  -  14
     Use previous bar value  -  Yes


Strategy that you want to come up with must have as straight as possible a positive gradient, making more money then is a matter of using money managing technique and taking bigger risk by buying more lot.

5 Reply by Melvin Lau (edited by Melvin Lau 2007-04-18 10:02:45)

  • Registered: 2007-04-18
  • Posts: 70

Re: Over-optimization (curve fitting)

Can we have an option on what period we want to optimize the parameters on, with the present optimization set up we are discovering what great a strategy this would have been, if we us it at the beginning of the data time.

Please let us select the period when we want to optimize the parameters, may be this will help us discover what strategy works in the present time.

=================================================================
I think this single feature will greatly enhance FSB ability to discover strategy that workable in the present time, not discovering what would have been great years ago.
=================================================================

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Re: Over-optimization (curve fitting)

You are perfectly right. We all need function to set the first and the last bar (or Date) for back-testing (optimization). This will allow as to perform "future" test on history data.

Popov's Website

  • Registered: 2007-04-18
  • Posts: 70

Re: Over-optimization (curve fitting)

I am not sure how  FSB tabulate it profit, but I have experimented with  Altreva ( an  agent-based system ) http://altreva.com/about.htm , my feeling is current optimizing technique places a lot of emphasis on early gain as it get compounded by subsequent trade even when the system get even more mediocre, hence even when the system perform mediocre latter it is pick out as a good system.

In order to get out of the "trap in the past" optimizing due to the compounding effect of subsequent gain. I propose we artificially reverse the calculation of profit just for optimization process, using the last trade by the system as if it is the first, and then investing the capital gain into the next last trade. 

This way we force the optimization to pick the most profitable recent trade as oppose to the earliest trade as is currently seen.

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Re: Over-optimization (curve fitting)

The evaluation criterion is a big question and it needs a deep analysis. I'll open a special topic about it later.  A strategy evaluation function will be included in the future, not only for the optimization. I think to incorporate different evaluation criteria and the user to be able to chose.

Popov's Website

  • Registered: 2007-04-18
  • Posts: 70

Re: Over-optimization (curve fitting)

That's great news, in order to find new profitable strategy, one would have to think outside of the box. The love to experiment helps alot as it preclude old ideas.

As more people makes money, I think the market will evolve in a way that will make it more difficult to make money.

Using FSB I have already shown that in the Fractal Up / Fractal down strategy, it now neutral, you can't make money out ot it or lose much, because too many people are  aware of this strategy and using it in the real market and now it is in balance between the buyer and the seller.

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Re: Over-optimization (curve fitting)

Future test in the past!!!

We can use a little trick.
Exclude last 500 bars from the back-test (use "Data bar Filter").
Refine, and optimize your strategy and when you are ready, just remove "Data bar Filter" slot. This will force the backtester to calculate the strategy using all data bars.
If the pattern of the strategy is not changed it is OK.

Popov's Website

  • Registered: 2008-12-14
  • Posts: 1

Re: Over-optimization (curve fitting)

its wonderful to be on this forum!!! its my firts time here. i stumbled on your website accidentally and am sure glad to have done so.

ive tried several times to download a copy of the free software and each time it keeps telling me the download file is corrupt. is there something am doing wrong?

i will be delighted to be able to download and use the software.

kindly assist.

Tijani

  • Registered: 2013-05-27
  • Posts: 292

Re: Over-optimization (curve fitting)

Would it be possible for someone to post here an overoptimization report of what you consider a good strategy?