forex software

Create and Test Forex Strategies

forex software

Skip to forum content

Forex Software

Create and Test Forex Strategies

You are not logged in. Please login or register.


Forex Software → Forex Strategy Builder Professional → Is it possible to see used param values while doing Monte Carlo?

Pages 1

You must login or register to post a reply

RSS topic feed

Posts: 22

Topic: Is it possible to see used param values while doing Monte Carlo?

Hi,
I want to try to make experiment for choosing more robust indicator values. Idea is to test some set of different strategy indicator values and choose not the best, but take mean/average (not sure what is difference). Now somewhere in the forum discussion is going about "anti-optimization", and it would be my variation to try check it out if this way would show some better(stable) results with unseen data chunk.

Monte Carlo test now randomize indicator values perfectly, but I can not see what values was used and of course get average of them. Maybe there is some temporary file where I can see them?

Re: Is it possible to see used param values while doing Monte Carlo?

Monte Carlo doesn't  remember the tests parameters. It only collects and arranges the results in the Confidence Table.

If we decide to implement Anti Optimizer, it will randomize the parameters and will remember all variants. The traders will be able to select one of the calculated results as a fitness rate. For example 0 % - will be the least profitable and 100 % - the most profitable (or other fitness metric). It will be possible to take the one that is at 80% of the dispersion (with 80% profit).

Re: Is it possible to see used param values while doing Monte Carlo?

I guess we need to make sure if that thing has positive effect. Will make couple experiments "by hand" with this idea few variations on dozens strategies. Will let you know if results will be better than if chosen best variables results.

By the way, I am planning to upgrade my system to AMD Ryzen 7, 8 cores, 16 threads. I hope there is no any limit with FSB about using max cores/threads?

Re: Is it possible to see used param values while doing Monte Carlo?

I'm also planning to take such workstation. FSB will work like a charm on it.

Re: Is it possible to see used param values while doing Monte Carlo?

Hi,
Just finished experiment. Maybe someone will find results useful smile
Planned to do it with at least 20 strategies, but stopped with 5, because it is very time consuming and I can see that strategies results getting worse, so for me it is enough proof not to use it. Used different strategies, pairs and time frames.

Checked 4 variants for selecting parameters. Variants:
1. optimized (FSB full optimization box checked, getting best one from about 200-600 optimization tests)
2. 4th from the top 20 random generated variable sets (SQN sorting, 80% percentile)
3. 10th from the top 20 random generated variable sets (SQN sorting, 50% percentile middle)
4. average indicator values from all 20 random variable sets

Question was if I choose not the best backtest parameters will it trade better than optimized results on unseen data chunk? Does choosing not the best parameter set makes sense? Answer what I found is no. Best sum of unseen data profits is fully optimized strategies. Also it made profits in all 5 strategies, when with other methods some strategies  was in loss, for me it is indication for more robust results.

However it can be that choosing not the best parameter set is effective in other conditions. I mean that FSB chose best from ~300-600 optimization runs, but if I chose best from thousand runs from other backtesting platform, probably there is a lot bigger chance to get overfitted strategies, and that "Anti optimizer" would make sense and better end results smile Also I am aware that using bigger sample size can change things, but I am OK with what I did to stay away from this idea.

Used =RANDBETWEEN() in excel to generate random variable value sets to mimic FSB montecarlo test.

Some pics...

https://s26.postimg.org/l0glwv9p1/image.png

https://s26.postimg.org/tjzzumi1h/example_1.png

https://s26.postimg.org/3pq74ui1h/example_2.png

Re: Is it possible to see used param values while doing Monte Carlo?

Clapping my hands with excitement! smile I reached the same conclusion. But with a little bit of sadness we have to admit it's another way we've found, which doesn't work. What will we do next?

Re: Is it possible to see used param values while doing Monte Carlo?

big_smile I am very low with expectations in trading, so got no sad about this... What's next? Try other things till success maybe? smile Still lots of ideas to test.

What is your craziest idea what you put in a test? For me is "remote viewing" for future market movement big_smile big_smile after 6 losses and 5 wins decided to stop and concluded that it is not working.

Re: Is it possible to see used param values while doing Monte Carlo?

After few months test live EAStudio and FSB, the FSB Strategies are little better then EAStudio...

My Strategie is only using MT4-Tracker from Sleytus, a big big big tool, i am very gladly with that tool.

Thanks again Sleytus smile

Re: Is it possible to see used param values while doing Monte Carlo?

I like this topic -- and have a different perspective...

Consider OOS.  Many people feel that a strategy that performs well for both IS and OOS is more "robust" or less curve-fitted and will, therefore, perform better in a live account on future data.  But isn't there another explanation?  If the OOS portion is similar to the IS portion (i.e. the IS portion did a good job training the strategy for the OOS portion), then it is not surprising the strategy does well in both.  That is, the OOS portion really didn't exercise the strategy in a way that would reveal whether or not it truly is more robust.

footon wrote:

But with a little bit of sadness we have to admit it's another way found, which doesn't work. What will we do next?

In a recent post you mentioned something along the lines we have all the data, we just need to figure-out what the right question is to ask -- which is great insight.  So, in this particular experiment, what exactly was the question?  The experiment yielded interesting results that can help going forward.  I'm curious to hear what people think the right questions are.

For example -- here are a few questions:
 
Does it make a difference which data set is used when generating, optimizing and back testing a strategy?

Should you always use the latest (and greatest data set) when back testing, or is there a particular chunk of historical data that routinely does a good job of training strategies even though it might be "old"?

What is the definition of robust?  And if it is a good thing, then does a metric currently exist that measures "robustness"?  Or could a new metric be computed?

Re: Is it possible to see used param values while doing Monte Carlo?

rantampla -- thanks for the good word.  It is appreciated.

rantampla wrote:

After few months test live EAStudio and FSB, the FSB Strategies are little better then EAStudio...

I also spent a lot of time with EA Studio and recently came to the same conclusion.  It did create some nice strategies that I continue to use, but I've now returned to FSB.  The main thing I miss is the ability to create an EA that contains a portfolio of strategies.  That is a very powerful feature. 

I'm currently working on a tool that will allow one to merge multiple FSB strategies into a single EA -- like EA Studio.  Popov's code is very clean and organized, so I think it is possible.  I hope he doesn't mind...

Re: Is it possible to see used param values while doing Monte Carlo?

sleytus wrote:

rantampla -- thanks for the good word.  It is appreciated.

rantampla wrote:

After few months test live EAStudio and FSB, the FSB Strategies are little better then EAStudio...

I also spent a lot of time with EA Studio and recently came to the same conclusion.  It did create some nice strategies that I continue to use, but I've now returned to FSB.  The main thing I miss is the ability to create an EA that contains a portfolio of strategies.  That is a very powerful feature. 

I'm currently working on a tool that will allow one to merge multiple FSB strategies into a single EA -- like EA Studio.  Popov's code is very clean and organized, so I think it is possible.  I hope he doesn't mind...

No Problem i feel it, and then i must say thanks, it is much big helping for me smile

I have read, that Mr. Popov from 01.10.17 EA-Studio make Pause and he will work on FSB, and i hope he can make Portfolio like EA-Studio.

But when you earlier, i hope you give it free, your work is very good smile

I have no experince with coding smile

Re: Is it possible to see used param values while doing Monte Carlo?

Hi Sleytus,
Your questions is very good ones. And it points me to look for the answer to Regime analysis (maybe even combination with Markov chains). But I know nothing about it, just read some on the net and in Market Wizards book 30 year old (I can look for exact name and paragraph), that one system trader is using Markov chains to decide when to turn off and turn on different systems.
Logically if you train EAs in more trendy data, you expect them to work great in trends live trading. And opposite with more ranging conditions. Maybe there is some indicators pointing out when it is time or there is edge to make these on/offs...
There is some link: https://www.quantstart.com/articles/hid … on-using-r .  I will take it deeper read to this too ....

Robust means working consistently on different pairs and timeframes. It is subjective in my opinion.

Re: Is it possible to see used param values while doing Monte Carlo?

Irmantas wrote:

Maybe there is some indicators pointing out when it is time or there is edge to make these on/offs..

Yes -- an indicator or strategy that is not used for trading, but only to provide a clue as to the trend or pattern of the incoming data.  And this could then be used to turn on / off different EA portfolios -- or something like that.

This would be most interesting...

Re: Is it possible to see used param values while doing Monte Carlo?

Hello, everyone!
I do not know English well, but google is powerful smile
I use double OOS

for example:
1)search strategy 1.1.2016.- 1.5.2017 -20% OOS
2)finding a strategy with a good  first OOS
3)remove data horizon 1.1.2016-present
4)second OOS is good
5)I continue on monte carlo, etc.

it's simple but it works for me

Re: Is it possible to see used param values while doing Monte Carlo?

Beru wrote:

Hello, everyone!
I do not know English well, but google is powerful smile
I use double OOS

for example:
1)search strategy 1.1.2016.- 1.5.2017 -20% OOS
2)finding a strategy with a good  first OOS
3)remove data horizon 1.1.2016-present
4)second OOS is good
5)I continue on monte carlo, etc.

it's simple but it works for me

Interest Idea, have you in live experience with this ea´s then?

How long live?

Re: Is it possible to see used param values while doing Monte Carlo?

now demo but soon .... live:)

it is not a guarantee of good strategies but they work better than others

Re: Is it possible to see used param values while doing Monte Carlo?

https://s26.postimg.org/p25a98p11/oos.jpg

Re: Is it possible to see used param values while doing Monte Carlo?

nice, please write your future tests smile

Re: Is it possible to see used param values while doing Monte Carlo?

Thanks for replays guys! smile I see a need for experiment which will use 3 OOS to test out if double OOS would make money on third OOS big_smile Good thing that trading is not boring, because you will never run out of ideas to test. Until you become profitable sad then you will need to stick to what works...

20 (edited by sleytus 2017-09-12 20:41:02)

Re: Is it possible to see used param values while doing Monte Carlo?

Beru wrote:

it is not a guarantee of good strategies but they work better than others

Thanks for attaching a picture -- it helps a lot.

When I look at the picture then I would not necessarily interpret this as a good strategy.  For example, in the "First OOS" section there are only 3 winning trades over the course of 2-3 months (20% OOS  1.1.2016.- 1.5.2017).  That means winning trades are a very *rare event*.  And that is confirmed in the "Second OOS" -- a winning trade is a very rare event.  If you were to place this in a live account with new incoming data then what is the probability you'll have a winning trade within the next month, or 6 months or next year?  It's hard to say.

I don't know the optimal number of trades when performing back testing, but I've read elsewhere that it should be at least 100 - 200 trades per year.

Re: Is it possible to see used param values while doing Monte Carlo?

it's just an example. This test will tell me whether to leave the strategy or not

Re: Is it possible to see used param values while doing Monte Carlo?

When your strategy gives good results for the OOS section it is not because it has magic properties and can see into the future.  The reason why it does well in the OOS section is because the OOS section is simply a continuation of the IS section.

The best way to take advantage of OOS is to begin with a strategy that *fails* OOS.  In this situation you at least know the IS and OOS sections are different.  You then treat the entire data set as IS and attempt to optimize your strategy until you get a balance chart that is nearly linear throughout the IS and OOS sections.  If you can do that, then you know that strategy is more robust -- since it has been trained against two chunks of different data.

OOS does no harm -- except it does cause people to think they have a strategy that can see into the future.

Posts: 22

Pages 1

You must login or register to post a reply

Forex Software → Forex Strategy Builder Professional → Is it possible to see used param values while doing Monte Carlo?

Similar topics in this forum