• Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,995

Topic: What can I do so that the Reactor can generate more EAs?

This is a question I received by email from Shaw Wen:

Dear Mr. Popov,

I took the following screenshot after the Reactor has been running for about 8 hours, and this is for EURUSD M1, I used 200K data bars, what I’m curious about is how come there are only two in the collection…

The Acceptance Criterion I specified are: max ambiguous bars: 10 and min count of trades: 200; everything else is by default.

Monte Carlo Validation: randomize spread + randomize indicator parameter; everything else is unselected.

Now the only feature that I see different this time is the Walk Forward validation, I left all the setting in this validation to default.  Is this new feature the reason why I only have two EAs after running the Reactor for 8 hours?  If so, what can I do so that the Reactor can generate more EAs?


Thank you!
Shaw Wen


https://image-holder.forexsb.com/store/eas-reactor-generated-only-two-strategies.png

Popov's Website

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,995

Re: What can I do so that the Reactor can generate more EAs?

The Reactor is actually a Generator, which passes the generated strategies through additional tools.

You have switched on Walk Forward and Monte Carlo validations in your case.

You can see that the Generator has actually generated 187 strategies that have passed your Acceptance Criteria. This is a good number - not too high and not too low. It means that the Acceptance Criteria are correctly chosen for your data set.

As I said, each strategy must pass all tools in order to be sent to the Collection. The next tool you use is Walk Forward. You can see that it totally 56 strategies have passed it - about 30%. This is also a good value. It also means that the Walk Forward settings are properly set. It is also acceptable the pass-through rate to be even lower, let's say 20%.

If you see the Monte Carlo rate, it is only 3.57%. The Monte Carlo was successfully passed by only 2 strategies. This must be the main reason for the low number of succeeded strategies. If you want more results,  you may reduce the Monte Carlo criteria.  however, I see it absolutely normal. If you say that with these settings the Reactor found 2 strategies for 8 hours, it means that it may find 6 strategies for a day whit all your validations.

Popov's Website

  • Registered: 2017-03-14
  • Posts: 709

Re: What can I do so that the Reactor can generate more EAs?

I have a question -- perhaps it's because I am tired.

If 96% of the strategies that pass Walk Forward fail Monte Carlo, then doesn't that suggest Walk Forward offers little benefit?  Also, Walk Forward is CPU intensive and takes a lot of time.  Wouldn't that time be better spent generating strategies that then get passed directly to Monte Carlo?

4 Reply by moreirajbn (edited by moreirajbn 2018-07-14 01:05:22)

  • Registered: 2018-05-01
  • Posts: 5

Re: What can I do so that the Reactor can generate more EAs?

sleytus wrote:

I have a question -- perhaps it's because I am tired.

If 96% of the strategies that pass Walk Forward fail Monte Carlo, then doesn't that suggest Walk Forward offers little benefit?  Also, Walk Forward is CPU intensive and takes a lot of time.  Wouldn't that time be better spent generating strategies that then get passed directly to Monte Carlo?


Walk Forward and Monte Carlo serve different purposes. Walk Forward is used to avoid overfitting parameters to past data, while Monte Carlo adds random noise to the market data and/or (you choose) trade journal, aiming to test how stable an equity curve would be in the presence of such noise.

In my view, both Walk Forward and Monte Carlo are absolutely necessary. Strategies with low trading frequency and very high winning rates, such as those with very long stops and very small profit targets, will more easily pass Monte Carlo simply because they don't have enough bad trades in the sample. However, it will be much harder to pass Walk Forward. On the other hand, strategies that are already contaminated  with "look ahead" bias will easily pass Walk Forward, but will be hit hard by Monte Carlo. One simple example: everybody knows that stock markets went up like a straight line from 2009 to 2017. So if anyone is developing "long-only" strategies for the S&P 500 in this period, they will have no problem passing walk forward tests, simply because it's already known that stocks went straight up, so long positions have a much higher rate of success than short selling, independently of any parameter in the strategy... will that hold up in an unseen future?

So, it all depends on how unbiased the development process is.

5 Reply by sleytus (edited by sleytus 2018-07-14 02:56:10)

  • Registered: 2017-03-14
  • Posts: 709

Re: What can I do so that the Reactor can generate more EAs?

moreirajbn -- thanks for your response.

I understand the purposes of Walk Forward and Monte Carlo.  And I like the way you explained how these two tests favor different types of data -- I agree.

However, the results are the results.  Shaw Wen's data shows Walk Forward failing 96% of the time.  In hindsight one can come up with explanations as for why -- but it doesn't change the fact that in Shaw Wen's case Walk Forward had zero benefit and, in fact, was harmful because it took away valuable CPU cycles and time that could have been used to generate more (and perhaps) better strategies.  It could very well be that under different conditions he might have gotten different results -- but he doesn't know when those conditions will occur because he can't tell the future.

Walk Forward sounds like a good idea since folks believe it provides a "glimpse" into the future.  But it doesn't -- it is simply a continuation of the IS data.  Walk Forward uses historical data -- keeping good results and throwing out bad results.  This is another form of curve fitting, though perhaps not as obvious.  For Walk Forward to truly be of value it would need to use a chunk of data from a period of time completely unrelated to the IS data -- e.g. a few years prior.  However, when you do this then you will discover every strategy fails.  Strategies only perform well on data they are trained on.  If a strategy happens to perform well on OOS data it is probably due to the OOS data being similar to the IS data.

If you do some Googling you will find lots of articles describing Walk Forward -- but zero evidence it actually works.  It is sort of like an "urban legend" that people like to believe because it gives them hope.

I do not have a solution that is guaranteed to work.  However, as I continue to refine my approach I think it is equally important to be honest about what doesn't work.  In this way you slowly, slowly streamline your approach and improve your results.

And, of course, this is just my opinion.

6 Reply by hannahis (edited by hannahis 2018-07-14 08:45:39)

  • Registered: 2010-11-18
  • Posts: 1,179

Re: What can I do so that the Reactor can generate more EAs?

I agree with Sleytus (Steve)

I question the value of Walk Forward.  I'm not saying it has no value at all but I'm asking myself What's the rationale and assumptions behind it.

Isn't it just a series of training and re training the EA in many windows of time/opportunity.  In another words, many IS training and re-testing it in the next window of time OS and repeat the process several times/continuously.

1st of all, we assuming that if EA is re-trained many times in many windows/duration, it means that EA is robust?  But isn't we re-training our EA to fit into that particular window/segment/duration.  How does that make it robust to fit into other windows?  We are just repeating the process again and again to fit into the latest window/segment.  What make we think it will survive the overall duration?

Isn't that similar to Sleytus method of once a while, optimise his old EA to fit into the most recent window/segment of data?

I think there are two process we need to be mindful of Training and Testing.  But often, they are done together, we lose sight of our intention.

In training, I would like to train my EA to handle a specific window of data (the focus in not the duration of the data, the focus is what type of market conditions within that trading period).  For example, If I develop Trending EA, I would like to optimise my Trending EA in a Ranging data, to train my EA to fine tune their parameters better to avoid false signals.

After training/optimising my EA, I would like to subject it to Testing.  That's where I would like to choose as long as possible the historical data to see how long can this EA survive in various market conditions.

Hence, in my limited understanding of Walk Forward, I personally feel it's just a repetitive training process without any particular intention in mind what kind of EA we are intending to train.  Hence, we get a jack of all trade EA, but a master of none, it's neither a ranging EA nor a trending EA and thus I doubt how good can the results be.  Bear in mind, it's just my opinion.  I would still like to do more Walk Forward testing to see whether the results fits into my assumptions.  And to see how valuable Walk Forward really are.  Opinions are just opinion, until it's tested, it would be just a presumptions.