Topic: Strategy Normalizer - development

Hello Traders,

Doue to an enormous burst of inspiration I started working on a Normalizer tool for EA Studio.

It will try to prettify the strategy by improving or at least without ruining its performance.

So, one night later I have a draft of the User Interface and minor part of the functionality implements.

Here are early screenshots:

Normalizer Page

https://image-holder.forexsb.com/store/ea-studio-nromalizer-page-preview-thumb.png


Normalizer Settings

https://image-holder.forexsb.com/store/ea-studio-nromalizer-settings-preview-thumb.png

I'll post daily updates on the progress and hope to have it online very soon.

I'll also post info on the functionality here and later will combine it in a Help article.

Re: Strategy Normalizer - development

Remove Take Profit

The first function of the Normalizer is "Remove Take Profit". (We don't have "Remove Stop Loss" intentionally.)

Imagine a strategy on EURUSD that has Take Profit of 10 000 pips. It is very unlikely it be hit whenever. So, the Normalizer will try to remove it temporarily and will calculate the strategy performance. It works like that:

1. Make a modification step. In that case, the Normalizer will disable the Tak Profit.
2. Backtest the strategy.
3. Validate the strategy against the Common Acceptance Criteria (if the option is on).
4. Calculate strategy fitness according to the "Search best" option.
5. If the strategy is valid and if its fitness is same or better than the original one, it accepts the modification. Restore the previous strategy in other cases.
6. Continue to step 1 or exit. (Remove the Take Profit has only one step, so exit)

The previous post screenshot shows Calculations = 2. It is because the Normalizer first calculates teh original strategy and then it calculates the variant without TP.

The variant without TP shows better stats, so it is accepted.

The balance chart shows the original balance with a grey line and also the final normalized Balance and Equity lines.

(it must show the stats for the Normalized strategy, but I still don't have it wink  )

Re: Strategy Normalizer - development

Wow!

This is most interesting!

Thank you

Re: Strategy Normalizer - development

It looks great, looking forward to it.

Thank you Popov

Re: Strategy Normalizer - development

I'm trying to decide what is the best way to make the normalizer. it must be similar to the Optimizer, with the only difference that it moves the params values only from the current value toward the default indicators values. For the case of SL and TP, it must move the values toward 0. also it will try to round the values to 5 and later to 10 if it is better.

My problem is that I have optimized the Optimizer for more than 10 years. It is now only 288 lines of code, but it so balanced and well finished, that I cannot find a way to make it work differently. The Optimizer's main loop is only 168 lines of code.

I first started making the Normalizer from scratch and it is already 230 lines of code and it can only remove the Take Profit!!! Not good.

Re: Strategy Normalizer - development

I found out that I can use the very same Optimizer for a part of the job.

Currently, we send to the Optimizer the indicators' parameters as a Start value, an End value and Steps. The Start and End value are equal to the current value +- the steps we set (let's say +- 20). I can use it in the Normalizer tool as follows:
- set the Start value equal to the current indicator value
- set the End value equal to the Indicator's default value
- adjust the number of steps depending on the values and the precision.

It may work for both Indicators and protections.

We can have a second step - rounding the values to 5 or 10.

And a third step - trying to remove the indicators to see if strategy becomes better wink  (This can be the first step)

In that way, we will be sure the Normalizer works flawlessly because of the proven Optimizer's algorithm.

This may sound like an easy task and should be. However, please note that the Browser's JavaScript is single-threaded. I make something like virtual multithreading to make it possible all algorithms to work for a long time. It allows us even to have the Generator working and to test or optimise different strategy in the Strategy tab.  Once explained this to a friend, who also has an It company. He only told me that EA Studio must be much more expensive to provide that technology.

7 (edited by ats118765 2019-09-25 17:01:58)

Re: Strategy Normalizer - development

Popov wrote:

I found out that I can use the very same Optimizer for a part of the job.

Great stuff Popov....very clever process. Can't wait to get my hands on the Normalizer module. :-)

Re: Strategy Normalizer - development

Hi Popov,

Firstly this is something that I have thought would be such a good idea for Ea Studio so good work on the development I can see this being a huge advantage for users!
Secondly with the TP could it be possible to have the Normaliser step down the TP rather than have the TP either on or off. Reason I say this is that I generate a lot of strategies that have a huge TP and you may only get a few trades where the TP is reached. Going into the future the chances of it hitting are slim and we end up with strategies that find most of there profits over only a select few trades.

My idea would be to step the TP and SL towards zero to see if the strategy improves this way we can elimate those large losses as well large wins. Kind of like ‘what would the strategy look like if we eliminated the N number of largest wins/losses. This would allow us to have tighter SL and TP towards the mean of all trades.

Re: Strategy Normalizer - development

Secondly with the TP could it be possible to have the Normaliser step down the TP rather than have the TP either on or off.

Yes, of course. This is a planned option. Please see the third option of the Settings screenshot.

Re: Strategy Normalizer - development

I just implement some more functionality and the new Normalizer managed to surprise me smile

I wanted it to remove a dumb Take Profit from a strategy and it did it successfully. However, it removed also one of the indicators !?. Mhm. It appeared that the indicator is needed only if I use this particular TP. Without the TP, the strategy becomes better without that indicator.

..

Update on my progress:

- the Normalizer can remove indicators.
- it correctly updates the Backtest output panel on successful normalization (can be called optimization in that context).
- the Reset button works. It reset all Normalizer settings and also resets the strategy. (I'm planning to add such Reset buttons to the Monte Carlo, the Walk Forward, and the Optimizer tools.)
- the Edit button works. It becomes green when the Normalizer finds a better strategy. It advises to "accept" the change because most probably the normalized strategy is better.


https://image-holder.forexsb.com/store/ea-studio-normalizer-removed-indicators-thumb.png

Re: Strategy Normalizer - development

It already reduces the indicators' parameters towards their default values.

https://image-holder.forexsb.com/store/ea-studio-normalizer-reduce-indicators-parameters-thumb.png

However, I'm a little bit worried regarding the overall effect of the tool. It appears rather powerful because I utilized the full Optimizer's capabilities. It makes several passes through the indicators in search of the best (lower) values.

Now I'm afraid of Overoptimization.

A possible solution is to have enabled by default only the options:
- Remove Take Profit
- Remove Needless Indicators

The other "optimization" options to be off by default:
- Reduce Stop loss
- Reduce Take Profit
- Reduce Indicators' Parameters

This problem doesn't exist in FSB Pro, because its Normalizer only can "remove".

In any case, the Normalizer will stay before Monte Carlo in the Reactor.

Re: Strategy Normalizer - development

Normalizer - Default Settings


https://image-holder.forexsb.com/store/ea-studio-normalizer-settingspng-thumb.png

Re: Strategy Normalizer - development

Awesome work Popov!

Looks very promising, do you have any sort of timeframe as to when this will be available?

Re: Strategy Normalizer - development

EA Studio is updated. You may test the new Normalizer.

Further to the previous notes, it now optimizes the indicator values toward the defaults in both directions. It means it increases the lower values and decreases the higher values toward the default value. The option is renamed to "Normalize indicator parameters" to correspond better to the behaviour.

Please share your opinion and ideas for improvements!

Re: Strategy Normalizer - development

Hi Popov,

Are you able to put the normailizer in the reactor/validator please?

16

Re: Strategy Normalizer - development

Are there any plans to put the normalizer into FSB Pro?

Re: Strategy Normalizer - development

Are you able to put the Normalizer in the reactor/validator, please?

Yes, these are the plans. Let's test it for several days and if there are no major issues, we will integrate it into the Reactor / Validator.

I want to optimize and refactor the Normalizer's code first. Also, I have to figure out how to fix the Progress stats. Because the Normalizer is dynamic, we cannot know the total number of the calculations at the beginning. A difference comes when the Normalizer removes TP or an indicator. Another problem is that the parameters optimization passes several times through all indicators until it finds the best values.

Other improvements I want to consider are to make it visible how useful are the Reactor's tools. For example to have stats how many strategies are actually improved by Optimizer, Walk Forward Optimizer and Normalizer.

Something like that:

https://image-holder.forexsb.com/store/ea-studio-reactor-stats-improved-strategies.png

Re: Strategy Normalizer - development

Are there any plans to put the normalizer into FSB Pro?

FSB Pro has a basic form of Normalizer. It is included in the Generator.
You may switch it on or off from the Generator's Settings. The option is called "Normalize strategies"

You may find some more info in the FSB Pro User Guide: Strategy Generator

My current plans are to add new and experimental features in EA Studio only.

Re: Strategy Normalizer - development

Do you think it is a good idea to place the Normalizer between the Optimizer and the Walk Forward tool?

My motiv is that the Normalizer is nearer to the Optimizer by functionality. It will make the work of WF easier if succeed of removing some indicators.

This arrangement must be the same in the Reactor / Validator also. Please consider wich workflow is more meaningful.

Re: Strategy Normalizer - development

Finally decided that all options of the Normalizer must be on by default. Its job is to normalize after all.

Re: Strategy Normalizer - development

Popov wrote:

I just implement some more functionality and the new Normalizer managed to surprise me smile

I wanted it to remove a dumb Take Profit from a strategy and it did it successfully. However, it removed also one of the indicators !?.

That is actually great to see. So much of a strategy's design complexity is associated with a multi-variate design response particularly when it comes to curve fit solutions. Strategy variables are rarely independent of each other. To see the reduction of multi-variables with the normaliser is a great sign to suggest it will assist in reducing system complexity to improve overall robustness.