• Registered: 2017-05-06
  • Posts: 51

Topic: Walk Forward Optimization advice on how to use it

Hi there,
one of the latest improvements in EA Studio was Walk Forward Analysis/Optimization. We can now use WFA/WFO to check a single strategy, as part of the Validation proccess or as part of the Reactor process. Thanks a lot to the developers for that. But I was wondering how people are using or planning to use this tool and if you could give some advice on how to use the tool.

I only know some basic concepts about WFA. I understand WF analysis is a way to check the robustness of an EA: instead of using a single OOS period we use more OOS periods/a longer OOS period as data scientist typically would do in a cross-validation process. Data in trading is time-series data and a simple way of cross-validation in a time series is WFA. OK with that.

But how are you more experienced traders using the WFA tool in EA Studio or going to use it? The EA Studio tool has different options to choose from: basically the size of the rolling window and the function to optmize (net profit, profit factor, sharpe, SQN, etc) in each in sample. And other options like the variation of the parameters and if we want to optimize SL and TP. What do you think about the options? Could you give us some advice on how to use it?

Thanks a lot in advance
Warmest regards
RJ

2 Reply by rjectweb (edited by rjectweb 2018-07-05 13:16:14)

  • Registered: 2017-05-06
  • Posts: 51

Re: Walk Forward Optimization advice on how to use it

I will start with my views on this subject. Please correct me and contribute with your opinions. Thanks in advance.

I recently read an interview to Robert Pardo, the "inventor" of Walk Forward Analysis in trading, and I got some ideas from that interview:

- About the window size (determining the optimum window size is one of the keys in the method) it is very important to have a significative number of trades in each window (to be statistically significant). So window sizes must be that to have (enough) a number of trades that could be considered significative.

- Window size depends then on how the strategy trades: if it trades quite often, the window sizes could be smaller. Smaller window sizes mean more adaptative strategies, but they must be re-optimized more frequently. On the contrary, if your EA trades less often, then you need bigger window sizes. In both cases, the number of trades both in the IS and the OOS must be significative.

- The number of windows (iterations in WFA) is also important. Concretely, the number of validations must be significative too. The robustness of the EA comes from the EA being able to survive many rolling windows and not only a few that may happen by chance. If the EA passes 10 out of 12 rolling windows it gives more confidence in the reoptimization method than passing 1 out of 3. In the last case, this succesful OOS (1 of 3) might have come by pure luck.

- The windows must be rolling. No anchored windows. The oldest data is less important, so rolling windows are prefered.

- A possible scenario could be 2 years IS - 6 months OOS --> (25% OOS). So, if you want 10 IS/OOS validations of 6 months you will need data from the last 7-8 years. If we don't have those data, we'll get smaller OOS sizes (with the same ratio IS/OOS) and we have to be careful to have enough number of trades on the window. An issue here is we usually don't have so much broker data (we have tickstory/dukascopy data, but don't usually have 8 years of H1 data from our broker, for example).


With these points in mind, I made my first attempts to WFA in EA Studio, so:

- Number of iterations must be high, but at the same time I must have enough OOS to have a number of trades.
What are your window sizes? I can have 10 of 6 months OOS in H4 timeframe, but cannot have that window sizes in H1 for example. I'm testing 2 months OOS right now. Maybe not the best balance. What do you think?

- When optimizing each IIS right now in EA Studio I must use Net balance/profit because otherwise, if I optimize using profit factor, sharpe or SQN for example, the number of trades in IS and OOS are very few. I think there is a problem there or a feature missing because I feel I shouldn't optimize each segment to have the better absolute balance, but the better profit factor or the better sharpe, etc.. However, it should be not the absolute profit factors or sharpes in each segments (because they usually come from taking only a very small number of trades), but these best conditions (best pf,sharpe,SQN) should be met with at least a number of trades. Best profit factor means nothing if there aren't at least a number of trades. On the contrary, best net profit isn't the best parameter to optimize, as the profit might come from a couple of lucky trades only. I think there's a problem in WFA implementation in EA Studio there.

- When using WFA in the reactor right now (and as I use a number of rolling windows) most of the process (processor time) is now consumed in the WFA. ES Studio must carry out n optimizations-validations for each strategy in the WFA stage. I'm not sure but I guess this process could be optimized. For example if I ask the EA to have 8 out of 10 validations, as soon as 3 windows fail the validation it is not needed to continue and finish the WFA. This way it will reduce the time the Reactor is in the WFA section. Another idea could be to continue searching other strategies while one EA is in the WFA section. Why stop generating strategies ehwn one is in the WFA stage? Couldn't we use a different processor / thread to continue generating EAs?

- Expiration dates. Once/If we have built our EAs based on a WFA it is supposed they must be re-optimized on a timely basis. Somehow we should remember that they must be reoptimized (for example) in six monhts from now. Sometimes I think there must be an expiration date in the EA to stop trading/remember us that the EA must be reoptimized before continuing trading.

Please contribute to this topic. I have a lot to learn yet and I'm here to learn from all of you to build better strategies. So far, I'm not taking advantage of using WFA and although I think it is a great improvement in EA Studio I haven't found a reliable way of using it yet.

So what do you think? How are you using the WFA analysis in EA Studio? what do you think I'm not doing well? Is there any idea I must take into account when using WFA?

Thanks a lot in advance!
Best regards.
RJ

  • Registered: 2017-06-16
  • Posts: 109

Re: Walk Forward Optimization advice on how to use it

RJ - Just wanted to jump in and say thanks! Very quality post mate! Thank you for the topics.

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,969

Re: Walk Forward Optimization advice on how to use it

Hello Rj,
Good post!

I feel I shouldn't optimize each segment to have the better absolute balance, but the better profit factor or the better sharpe, etc.. However, it should be not the absolute profit factors or sharpes in each segments (because they usually come from taking only a very small number of trades)

The Acceptance Criteria are designed exactly to solve this problem. Set minimum count of trades for the OOS zone, and the Optimizer will search for best SQ for example but will accept only strategies that comply with the criteria.

Popov's Website

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,969

Re: Walk Forward Optimization advice on how to use it

Another idea could be to continue searching other strategies while one EA is in the WFA section. Why stop generating strategies ehwn one is in the WFA stage?

Because it will not be faster. The browser runs code in a single thread. I know, I know, some "clever" guys will suggest WebWorkers, WebAssy or other stuff... I was such optimist while writing FSB for 15 years, however compare its speed with the online EA Studio.

Better practical than opinionated!

Popov's Website

  • Registered: 2017-04-30
  • Posts: 11

Re: Walk Forward Optimization advice on how to use it

@RJ tq for the post smile

I separate the reactor into 2 process. means I dont use the reactor. use the generator to generate strategies only. then validator to do the rest. that will max out the browser (chrome) resources. I found it to be more productive. previously I ran the reactor with various parameters for 48hrs and resulted in 0 strategies sad

  • Registered: 2021-09-03
  • Posts: 3

Re: Walk Forward Optimization advice on how to use it

ISS period of 6 months is too small. How can you generate 1000s of trades with only 6 months of Optimization Period?

MT4 can give you 10 years of data. It is based on MetaQuotes output but at least you got 10 years there. ( Tools > History Center > Download )

The key to Walk Forward Optimization is statistical significance. If you don't have enough samples ( 1000s ), the "winning" Parameters that graduates has lesser chance of being the best parameter.
         5% edge requires 10000 samples for the best paramter to rise to the top 10.
        10% edge requires  2500 samples for the best paramter to rise to the top 10.
        20% edge requires   500 samples for the best paramter to rise to the top 10.
        Even for a system with a 20% edge requires 1000 samples for the best parameter to rise to the top 2.

Optimize using Profit Factors ( with fix lot size ) or SQN ( with lot size based on % equity ). Profit Factors with lot size based on % equity will skew your results - the first trades will have lower weights vs the middle trades and the middle trades will have lower weights vs the tail-end trades.