Topic: Walk Forward Internals Explained

Hi Popov,

I have been using the walk forward optimisation over the weekend and i have a couple of questions:

1. In the stats it has 3 columns 1. Original 2. Before Op. 3. After Op.

Can you tell me the difference between the original and before optimisation. Arent they the same thing? Obviously they arent because they have different results but I dont understand. we load the starategy in and then it optimises?

2. When using the reactor does EA Studio look at the WFO acceptance criteria? Or does it it only look at common Acceptance Criteria (if the box is ticked). So if we do not have the CAC box ticked are we not looking at any validation criteria?

3. Im still trying to wrap my head around how it deems a strategy 'better'. For example see attached screen shot. In this example ive set the Search Best to R-Squared. The strategy has a better R squared in all OOS periods than Original strategy and Before Optimisation however it tells me it is not better at all?

4. When you are looking at the equity graph as the optimiser is running as each segment is plotted is this the result of the optimisation on just that segement of IS data? If so how come when you look at the parametres tab and view the equity graph 'Full backtest with last parameters' it looks exactly the same as the plotted graph in the Equity Tab page?

5. Just to clear things up does the WFO operate like this:
1: Optimise from first segment of IS data.
2. Test over first segment of OOS data. and carry over the resulting indicator parametres to the next segment of IS to be optimised again.
3. Optimise over second segment of IS data etc etc
Until 4: Optimise on last segment of IS then show the resulting indicator parametres over the entire data range of OOS.

Thanks.

Post's attachments

Capture.PNG 46.46 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

Re: Walk Forward Internals Explained

Thank you for those questions! The answers will be a very good addition to the User Manual.

1. In the stats, it has 3 columns 1. Original 2. Before Op. 3. After Op.

The "Original Strategy" stats show the statistics values for the OOS part of the segments taken from the full backtest of the initial strategy. There is no additional backtest performed, and the strategy parameters are not changed. The stats are just cut out from the complete test performance.

The "Before optimization" values represent the OOS performance before the optimization of the current segment. We show these stats to use them as a basis for evaluating the performance after the IS part optimization. The "Before optimization" stats will differ from the "Original Strategy" stats in the case when the strategy parameters were changed during the optimization of the previous segment. This is how the Walk forward works. We optimize one segment and then go to the next segment with the new parameters.

The "After Optimization" stats represent the extrapolated performance of the strategy. We optimize the In Sample part of the segment and then calculate the stats of the OOS part, which simulate the trading on unknown data.

"Gan Optimization" is the difference between the stats: "After Optimization" - "Before Optimization". We have actual 'gain' when the strategy shows better stats.

Re: Walk Forward Internals Explained

2. When using the reactor does EA Studio look at the WFO acceptance criteria? Or does it it only look at common Acceptance Criteria (if the box is ticked). So if we do not have the CAC box ticked are we not looking at any validation criteria?

The Walk Forward optimization in the reactor works in the same way as the Walk Forward tool under the Strategy tab. During the optimization, the WF uses its internal Validation Criteria from the "Validation" tab.
When all segments are calculated, the WF performs a new backtest of the strategy with the parameters from the last segment and using the full data series.

When the "Use the common Acceptance Criteria" checkbox is checked on, the Walk Forward validates the complete backtest against the Common Acceptance Criteria (from the Tools section of the application).

The purpose of this validation is to provide consistency of the validation. If we have enabled Acceptance Criteria in the Generator, it is a good practice to have them enabled in the Walk forward optimization also. In that case, we will be sure that the optimized strategy will have the same or better quality than the initially generated one.

Re: Walk Forward Internals Explained

3. Im still trying to wrap my head around how it deems a strategy 'better'. For example see attached screen shot. In this example ive set the Search Best to R-Squared. The strategy has a better R squared in all OOS periods than Original strategy and Before Optimisation however, it tells me it is not better at all?

You have to check the full backtest chart in the "Parameters" tab. The full backtest performance with the last segment parameters may not be better than the original strategy. It usually happens when the strategy was first optimised.

Do not use Walk Forward on optimised strategies.

Why?

Because the Optimizer finds the best combinations of the indicator parameters for the complete test. Usually, it does a "good" job to find the higher stats (be aware of possible over-optimisation). If we run the Walk Forward after the Optimizer, it is possible to find a better combination of parameters for a particular segment. However, every deviation of the original parameters will only decrease the performance for the full backtest. In the best case, WF may recover the original parameters in one of the following segments.

In the general case, WF can end up with the same strategy or with one with lower stats.
It is possible but is a rare case WF to find a better strategy from the Optimizer.
I highly recommend you to use either the Optimizer or the Walk Forward optimization.

The Walk Forward accepts a new strategy when it has the selected number of validated OOS segments, it has equal or better stats, and it passes the Common Acceptance Criteria.

WF consider a strategy better when it has a better value of the "Search best" metric.
If "use the common Acceptance Criteria" is not checked on, the WF skips that validation.

Re: Walk Forward Internals Explained

Most helpful!

Thank you

Re: Walk Forward Internals Explained

Very helpful! Thanks Popov for clearing that up! Much appreciated

Re: Walk Forward Internals Explained

The "Walk Forward optimization" in the Reactor is consistent with the method of acquisition of the strategy.

If you generated a strategy with OOS in the Generator, the WF Optimization will use the Generator's OOS to validate the full backtest. It means that it will use the Common Acceptance Criteria to validate the final backtest's IS part, OOS part, and the Complete Test. Then the strategy will be shown with a "green" OOS zone in the collection.

If no OOS is used by the Generator, the WF optimization will validate only the full backtest with the AC. The strategy in the Collection will not have a "green" OOS zone.

The idea is that WF Optimization tries to find a "better" strategy on the same conditions as the acquisition method - Generator, Optimizer, or Validator. It also not changes the initial SL, TP or the Preset Indicators by default.

The consistency and good User Experience is very important. Please share your experience and ideas. It has always room for improvements.

Re: Walk Forward Internals Explained

Thank you for these improvements Popov, I am trying to learn how they work - I think the easiest way to ask my questions is as follows;

Along with the generator, several of the optimization and robustness testing sections now have the option to use various combinations of Acceptance Criteria and OOS, - is there any situation where it would not be recommended to tick the AC box in one section as well as another/several others? And similarly, ticking the OOS box in one section as well as another/several others?

Basically, is it possible to skew the results by ticking all AC and OOS boxes in the Reactor? If so, how could we avoid it?

Thanks,
Simon

Re: Walk Forward Internals Explained

is there any situation where it would not be recommended to tick the AC box in one section

Using Acceptance Criteria for at least basic validation is recommended. However, even without them, the Generator has hardcoded criteria: Minimum profit = 0, and Minimum count of trades = 5.

You may untick the option in cases when the Generator doesn't produce strategies. This will show you if the Acceptance Criteria are too strict. (for example, min profit per day = 20 on a 200 EUR account.)