Topic: Acceptance Criteria for the Optimizer - Development

Optimizer has own Acceptance Criteria

https://image-holder.forexsb.com/store/eas-optimizer-acceptance-criteria-thumb.png

All found strategies by the Optimizer fulfill the selected criteria.
If OOS is "In sample" (which means no OOS is used), the Optimizer validates the complete backtest.
If you use OOS, the  Optimizer validates only the in sample part of the backtest.

If you use optimization with OOS in the Reactor, you can easily validate the complete backtest by switching on some of the other validation tools. For example, you can switch on the Multi Market Validation with the current market only. If you do so, you can even have a different version of the validation criteria for the complete backtest.

2 (edited by geektrader 2019-01-20 01:19:22)

Re: Acceptance Criteria for the Optimizer - Development

Popov, thank you, but it would be VERY important (to me at least) that the extra acceptance criteria for the Optimizer also has 3 parts: Complete Backtest, in sample, out of sample, like the normal acceptance criteria for the generator (and how it was before for the optimizer too). So that if selecting "10% OOS" for the optimizer in the reactor or directly within the optimizer, we can specify for example "at least 95 R-Squared for the COMPLETE backtest, but also at least 80 R-Squared for the 10% OOS part", then begin the optimization. I have always used this to find strategies that are overall stable, but ESPECIALLY stable in the last years (the 10% OOS). This option is not possible anymore now, having just 1 set of acceptance criteria for the whole backtest (or IS, as the optimizer will ignore the OOS part now and we can set now minimum values for the OOS part anymore during an optimization), which is a pitty :-(

I´ve awlays used the optimizer this way: set it to 10% OOS, set the acceptance critera for the 10% OOS to at least a R-Squared of 80, then optimize the IS part for maximum R-Squared value, this isn´t possible anymore now and basically breaks my complete workflow of how I´ve created my good strategies :-(

I think it would be better if you´d simply add a second set of criterias in the settings page and then, in the Reactor / Generator / Optimizer, we can select "criteria set 1" for the generator, and "criteria set 2" for the optimizer, or use "criteria set 1" for both if we don´t need different acceptance criteria. Both "criteria set 1" and "criteria set 2" would have acceptance criteria for the complete backtest, for the IS and for the OOS part. If you can´t do that, can you please bring in the 3 split-parts into the Optimizer directly as well again (complete backtest, IS, OOS). That would be higly important to me as I am stuck now with only having 1 set of criteria for the Optimizer.

On large data-sets that is a real problem, because optimizing for a high R-Squared for the complete backtest can sometimes result in great first 28 years, but bad last 4 years (as I use 32 years of data). When we had the option to optimize the IS part for maximum R-Squared (or whatever) but FORCE at least a R-Squared of 80 in the 10% OOS part, the optimizer found great strategies that would be very stable overall and not just 28 years and then go down sligthly like it often is the case after 2015.

If using the new optimizer now, I have no way to set which minimum criteria to have in the 10% OOS. If I set it to 10% OOS now, the optimizer will only care about the IS and will completely ignore what is going on the in 10% OOS. Previously, the optimizer would also only optimize the IS part, but we could set that in the 10% OOS, it should at least have 100 trades, or at least return/drawdown of 10, or a R-Squared of 80, or whatever. Now it does absolutely not care anymore what happens in the last 10% of the backtest and just optimizes the IS part, regardless of what happens in the 10% OOS part. And that is a real problem at least for me, as I can´t optimize the way I´ve optimized anymore.

So summarized:
- Please bring back the option to use separate critera for the complete backtest, IS and OOS in the Optimizer instead of just 1 set of acceptance criteria.

Thank you so much!!!

Re: Acceptance Criteria for the Optimizer - Development

Now the Optimizer has three sets of Acceptance Criteria: for the complete test, for the Is and for the OOS parts.

https://image-holder.forexsb.com/store/eas-optimizer-acceptance-criteria-is-oos-complete-test-thumb.png

4 (edited by GD 2019-01-20 03:41:22)

Re: Acceptance Criteria for the Optimizer - Development

Thank you Popov.
Thanks.
Great.

5 (edited by GD 2019-01-20 05:01:20)

Re: Acceptance Criteria for the Optimizer - Development

There is a small bug

The lists of In Sample and Out of sample and complete backtest criteria selection in Optimizer go under the black footer of the page.

I cannot select all possible validation criteria!!!


https://i.postimg.cc/yD1XC3G0/2019-01-20-12-46-12.jpg

6 (edited by GD 2019-01-20 07:51:50)

Re: Acceptance Criteria for the Optimizer - Development

Hi Popov

Very nice work

1. Do the Acceptance Criteria of Generator AND Optimizer are common and they are selectively in Optimizer can supersede the validation criteria of Optimizer by changing its settings? Am I correct?

2. I used Reactor with optimization 00S 10% but it did not used Validation criteria from optimizer. Is it Bug or it is expected?

Optimizer alone with validation criteria looks to work.

An explanation of how things work now is needed.

7 (edited by GD 2019-01-20 09:22:29)

Re: Acceptance Criteria for the Optimizer - Development

I think the criteria should be separate for generator and optimizer (2 different pages in settings)
OR
acceptance criteria of generator to supersede optimizer's with the option in Optimizer settings to do not be taken in account (as it is now, I think).

I do not worry if Popov will fix it. He is perfectionist!!!

PS. OOS monitor looks to have a problem also. It is not updated well I think.

Re: Acceptance Criteria for the Optimizer - Development

GD wrote:

I think the criteria should be separate for generator and optimizer (2 different pages in settings)

+1 and as I´ve suggested earlier already. It´s confusing now that some acceptance critera are in the settings page (for the generator only from here on) and others (for the optimizer) are within the Optimizer. Better just make 2 acceptance criteria sets in the Settings page and then let us assign them to the generator / optimizer, etc.

Re: Acceptance Criteria for the Optimizer - Development

GD wrote:

There is a small bug

The lists of In Sample and Out of sample and complete backtest criteria selection in Optimizer go under the black footer of the page.

I cannot select all possible validation criteria!!!


https://i.postimg.cc/yD1XC3G0/2019-01-20-12-46-12.jpg

Use this url https://eas.forexsb.com/

there is no footer

10 (edited by GD 2019-01-20 10:56:41)

Re: Acceptance Criteria for the Optimizer - Development

Roughey

Thanks  a lot. hehehe. Popov has fixed already...

Re: Acceptance Criteria for the Optimizer - Development

Great work Popov! Testing now!

Re: Acceptance Criteria for the Optimizer - Development

Thanks for the quick change Mr. Popov, that is amazing fast in the middle of the night - thank you so much!!! :-)

@GD: Yes, it seems like they are ignored if used through the Reactor and work only in the Optimizer directly, but I have to do more tests to confirm. I am sure Mr. Popov will quickly fix it though as always - what a great guy!

Re: Acceptance Criteria for the Optimizer - Development

Hello Traders,

Thank you for the reports, the feedback and the idea!

Let's summarize:
- to move the Optimizer's criteria under the Tools section.
- make options to control whether the Optimizer will have a separate set of criteria or will use the generic criteria.
- make an option to control whether the Optimization under the Reactor will be finally validated with the generic criteria or only with the Optimizer's criteria.

I'll try to create a prototype this evening and will see how it will work.

14

Re: Acceptance Criteria for the Optimizer - Development

Popov

It looks fine. I remind you also to check OOS report  to be compatible
AND if possible create and 5% OOS.

Thanks in Advance.

To have different criteria for Generator and Reactor creates a new era in the development of EAs.

Re: Acceptance Criteria for the Optimizer - Development

I remind you also to check OOS report  to be compatible

What wrong do you see?

- to move the Optimizer's criteria under the Tools section.

I moved the criteria for a test, but unfortunately it brings more problems than solutions.

It looks like the current version works well. We can add the additional functionality around it.

Regarding the optimization in the Reactor - the Optimizer is after the Generator, so its criteria shape the produced strategy.
You may choose the same, stricter, or looser criteria for the optimizer.

Anyway, it is obvious that introduced separate set of criteria brings problems. Now we need more clicks to do the job.
We may add options in the Reactor optimizations for whether to use the optimizer criteria or the generic ones.

Please note that EA Studio must be understandable for new user. If it is not, it will put an end to the project. From that point of view, the feedback and the experience of the newcomers is equal or even more important than the feedback of the hardcore users.

16 (edited by GD 2019-01-20 19:20:44)

Re: Acceptance Criteria for the Optimizer - Development

Hi Popov

I use as first example Reactor and " in sample" in Generator and in Optimizer and in Editor. I used only Generator and Optimizer clicked.
I made simple acceptance criteria in Generator (clicked) and I accept same acceptance critiria (clicked) in Optimizer (inside editor). In Optimizer I added only minimum r-squared 85% in sample in validation criteria.

Then I run Reactor.   Strategies added to collection. I checked them with editor and I found in OOS monitor and in Stats info r-squared  lower than 85. I expected to see in the collection only the ones with r-sqaured higher than 85. I did not.

Do I have mistake or there is a bug;

Relative to this oppinion

"We may add options in the Reactor optimizations for whether to use the optimizer criteria or the generic ones."

It sounds good. There is a button in editor/ optimization which permits to user to use Acceptance criteria if clicked.
It is simple if it is not cliccked the validation criteria to be used automatically. But it has to be mentiond in that point somehow.

Another way is to keep acceptance criteria in optimizer which do not change in Optimization criteria together with optimisation criteria.

The same manipulation can be done if used In sample and out sample.

This way looks simple for new user

Re: Acceptance Criteria for the Optimizer - Development

Here is an idea for the Reactor / Validator.

We can add two check-boxes:
  - The strategies fulfill the Optimizer's Criteria
  - The strategies fulfill the Acceptance Criteria


All combinations may be possible - to have one option, two options or no options checked.

When both options are checked, the strategies first must fulfill the Optimizer criteria during the optimization, however they will be validated against the generic Acceptance Criteria before to be allowed to proceed further through the workflow.

Such options are necessary only for the optimization task because it is currently the last one that may modify the strategy.

This is a development prototype.

https://image-holder.forexsb.com/store/eas-optimizer-different-accetance-criteria.png

18 (edited by GD 2019-01-20 20:43:27)

Re: Acceptance Criteria for the Optimizer - Development

it looks great.

What about if common selections in criteria Optimization and Acceptance Criteria exist ? Optimizer`s should have priority. Right?

Re: Acceptance Criteria for the Optimizer - Development

The Optimizer is not able to see the common Acceptance Criteria because it has own set now. It will only use his own criteria.
If the "The strategies fulfill the Acceptance Criteria" is set, the strategy will be tested after the Optimizer.

We cannot speak for a "priority" in that case.

20 (edited by geektrader 2019-01-20 21:24:46)

Re: Acceptance Criteria for the Optimizer - Development

I´d still think the easiest solution would be to have:

"Tools" -> "Acceptance Criteria Set 1"
"Tools" -> "Acceptance Criteria Set 2"

In the Generator / Reactor / Optimizer we´d have:

"The strategies fulfill the Acceptance Criteria" as it is right now and to the right side is a dropdown menu to select "Acceptance Criteria Set 1" or "Acceptance Criteria Set 2".

And it would default to "Acceptance Criteria Set 1" for the Generator / Optimizer, so it´s easy for new users and they don´t have to worry about it at all and setup different acceptance criteria in the Tools and additional in the Optimizer. This just complicates things too much for them. And us experienced users can use different acceptance criteria sets for generator / optimizer if needed. This also keeps the whole Acceptance Critera setup in the "Tools" section, not cluttering things up to different places like it is now.

Just my 2 cents :-)

Re: Acceptance Criteria for the Optimizer - Development

GD,
the problem is fixed. Now only validated strategies may reach the Collection.

22

Re: Acceptance Criteria for the Optimizer - Development

I will check. Congratulations.

23 (edited by GD 2019-01-21 04:47:27)

Re: Acceptance Criteria for the Optimizer - Development

Hi Popov

in optimizer/settings  the use of "the strategies fullfill the Optimization's criteria" is not clear.

I will explain.

Suppose i use reactor and I select Acceptance criteria and Optimization's criteria.
Then I go to Optimizer to fix its settings and I found again the same parameter.
Why to click it again?
And if not?

I can imagine BUT It is not clear for a new user or a tired user...

(I will be back in few hours).

24 (edited by geektrader 2019-01-21 05:31:18)

Re: Acceptance Criteria for the Optimizer - Development

I also think all the different naming is confusing.

It´s called "Tools" -> "Acceptance Critera". But under the "Optimizer" it is called "Validation" instead of "Acceptance Criteria", and says "Add validation criteria" instead of "Add acceptance criteria" (which is the same problem under Tools -> Acceptance Criteria too).

I think you should stick to a continuous naming:

Tools -> Acceptance Critera -> Add acceptance criteria
Optimizer -> Acceptance Critera -> Add acceptance criteria

It´s very confusing for new users I´d think, as they do not understand why it is sometimes called this way and other times another way.

Re: Acceptance Criteria for the Optimizer - Development

Thank you for the feedback. then naming should be consistent.
Also when an option is enabled in the Reactor, it must be correspondingly changed in the particular tool.

I'll try to implement it today.