• Registered: 2018-09-14
  • Posts: 6

Topic: methodologies - Any empirical tests?

So there is a lot of discussion about how best to trade the generated EAs but so far i havent seen anything with any specific comparisons. Thinking in terms of using OOS data, demo before live or straight to live.

Has anyone tested each method to see which is best or is it all just opinions? From what i can see the idea of generating then going to a demo account is pretty pointless. Its no different than using OOS except you waste real time (like weeks, months) instead of moments using OOS in FSB. That just leaves using OOS or straight to live. For me the idea of going directly to live with a generated EA seems a bit of a risk. From the testing i have done in FSB using 20% OOS its pretty clear that over 80% of the strategies generated are simply artifacts of pure coincidence. This is evidenced by the fact you dont actually need any indication of where the OOS data starts when viewing the strategy. Its pretty clear as the equity line drops of a cliff. But OOS isnt perfect either. FSBs implementation is basic to say the least. Although i havent tried it yet from looking over the docs Strategy Quants implementation of OOS is much more geared to using progressive walk forward analysis and using the OOS as a tool that can actually give at least some indication of robustness.

As others have pointed out in other posts, using OOS (in FSB at least) can be as much pot luck as not using it as it could simply be that you have by luck selected the period where the market conditions for that generation still apply. They may not apply now. Its got to be worth some testing though.

I was thinking 2 demo accounts for test. Use short timeframe charts (so we can get a decent trade count, so perhaps m-5). The basic plan being take a 6 week data horizon. Generate the best 4 strategies on a given pair, one set of 4 using 20% / 25% OOS, the other using all the data in the generation. The best 4 would be ones where the equity line is still at a steep upwards angle at the very end of the 6 week period. Run them for one week then at the weekend bin them and do a new generation of 4 more of each as before. Repeat for a month. Its just an idea based on comments on other threads about other peoples preferred workflows so sort of covers most angles.

It would be interesting to see which approach worked best (if any have a profit at all). Unfortunately a 15 day trail isnt enough by a long way to get any useful data so i cant do this. Just thinking that surely someone must of done some actual testing to evidence that any methodology they use has some robustness.

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,982

Re: methodologies - Any empirical tests?

When I developed the FSB Generator I  found a strategy that uses only one indicator. It was on the far 2006, probably 5 years before the release of the advanced SQ walk-forward. This strategy makes profit up to now - 12 years OOS result:

https://image-holder.forexsb.com/store/fsb-pro-fractal-strategy-eurusd-h4.png

Is this a proof?

Popov's Website

  • Registered: 2018-09-14
  • Posts: 6

Re: methodologies - Any empirical tests?

Not really. Its Proof that long term profitable strategies exist and that FSB can find those profitable strategies. I had no doubt about that. I am sure there are many strategies that are still working. My point is  really when you create a strategy you never know if you just got lucky or if that magic combination is something thats going to be profitable forever. And, you will never know (only in hindsight which is no good).

The idea really is to have some methodology that will increase the probability that when you put any given EA on a live account its going to perform similarly for at least some period of time. I know there are no guarantees or certainties of any sort in this business but this business is built on probabilities and probabilities are quantifiable. 

There are quite a lot of threads dedicated to OOS or not and how people generate then run the strategies but they always seem to come down to it just being a preferred method. Some people are looking for something like the one you highlighted while others are looking for disposable great for now methods that they can get rid of when market conditions change and generate replacements. Then there is a lot of discussion about how best to go about generating these replacements and how often etc.

I was wondering if anyone had spent the time to actually do any tests to see if any particular method was better than another. You could make logical arguments for either case so i would never say one is right or wrong but tests and numbers are useful. They dont guarantee but they do give an idea of the probabilities in a quantifiable way and knowing the probabilities can give confidence in a strategy/system/method of workflow and that is really the only thing that can bring consistent profits. Anything else can only be considered as gambling.

4 Reply by Irmantas (edited by Irmantas 2018-09-19 18:02:04)

  • Registered: 2016-05-16
  • Posts: 190

Re: methodologies - Any empirical tests?

Hi noble-6,
I did some experiments. I rise the question, then make some portfolio and check results on OOS (skipping demoing part). However it was limited to 20-100 system portfolios, and some time horizon. On other data it could be completely different. My few findings what I am willing to share:
* OOS passed portfolio did not showed any better results on 2 OOS. Results on 2 OOS was pretty much scattered evenly with similar results like not passed systems on 1 OOS and optimized on full data. So for me no-brainier, will use all data.
* Optimized systems portfolios always shows better results on OOS. Made maybe 3 experiments about it. So the talk about over-optimization does not hold for me, probably answer hides somewhere else.

I have some other experiments and excels about some isolated questions like isolating some system statistics - does it show correlation with unseen profits etc. I think I have build my workflow with these kind of experiments. Last experiment took me about 8 months maybe, maybe even more, because there was involved mt5 and some excel stuff. From tested 150 systems I found 10 systems which passed hypothetical question/test and 9 of them was with unseen profit smile rest portfolio was total looser. So for me it is good proof that I have something working. However I am not yet millionaire, I expect only 20-50% early growth with something like 20% max drawdawn. I think it is realistic number. However I will not share exactly what I do, unless $ big_smile ... also still no statement is build.

Would be very nice, if you make some empirical test and show something smile

5 Reply by hannahis (edited by hannahis 2018-09-19 18:56:08)

  • Registered: 2010-11-18
  • Posts: 1,179

Re: methodologies - Any empirical tests?

What are you exactly asking for?

1. For someone to test out the various workflow/methodologies and see which is better?

a) In the 1st place, from the various discussion, you would have realised that different people have different methodologies and no one has all the methodologies to do that kind of test you are asking for....the details of each workflow differ from user to user and the combinations are too much to conduct any standardized test, unless everyone contribute their workflow processes, in order to do a well documented comparison (which is an impossibility/who is will to share their trade secret?)

b) Even if someone has done their testing and know which methodologies work, do you think they will share their workflow with you? Or tell you which methodology work better and what's the setting in order to make it work?

c) What you are asking, is to ask people to do your "homework" for you....

Why would any successful trader/user bother to share their workflow?

1. What's there to proof?  Successful trader/user don't bother to proof anything to anyone, what's there to gain?

2.  Successful trader won't bother to share their secret, what for?

3.  Successful trader don't need to proof FSB works, they are not selling the software, so why bother?

Lastly, forex is a tough field, so even if others come forth to tell their successful stories, that doesn't mean you can replicate their success too.

Bottom line, do your own testing and find out for yourself what works and what doesn't.  No one is going to conduct that kind of experiment for you and tell you the findings.

Best of luck

  • Registered: 2018-09-14
  • Posts: 6

Re: methodologies - Any empirical tests?

hannahis:
Was more out of curiosity really. FSB is a powerful tool but it only gets you so far. Getting that bit further is where the work is.  In fact Irmantas has given pretty much a perfect answer.

As for your first point 'a.' that is exactly what i have seen. Lots of discussion but nobody has actually articulated how they come to the conclusions they have or why they pick that workflow. In fact Irmantas is the only person who posted that they have done any reasonable diligence in an effort to gain some form of statistical knowledge on how their particular workflow performs.

Pretty sure i didnt ask for anyone to tell me their workflow. That would be pointless since most of the stuff clearly has no statistical basis. I simply asked if anyone had actually validated their own because it doesnt look like its that common. Also not sure why you are going on about proof. As i said i am already convinced FSB can produce profitable strategies.  Proof though is pretty important. If someone cant prove to themselves that something works and that its consistently repeatable then they are just gambling and the only way to prove it is by standardized testing.

7 Reply by sleytus (edited by sleytus 2018-09-19 21:34:32)

  • Registered: 2017-03-14
  • Posts: 709

Re: methodologies - Any empirical tests?

Noble-6...

Irmantas is great and is generous with his time.  He provided a general overview of how he works -- but he didn't answer in the detail you had requested in your original post.  So -- it is disingenuous to attempt to save face by suggesting his was the "perfect answer".

Hannahis response is the correct one.  FSB is a tool -- like a hammer.  You asked for instructions on how to build a beautiful house.  Irmantas let you know he has been successful using the hammer, but he didn't show you any of his houses or architectural plans.  Hannahis' response was more blunt -- her advice (and mine) is now that you have a hammer, the burden is on you to figure out how to use it.  Also, who are you to lecture the forum about needing to prove something to you? 

Popov has gone to great lengths to provide excellent videos and online documentation.  With minimal searching on this forum you would find numerous threads that discuss different work flows and the pros / cons of a variety of different approaches.  But you didn't bother searching.

Forex is challenging and difficult.  You really have to be motivated and a good detective.  Asking people to hold your hand as you attempt to figure things out might seem like a smart short-cut.  But it is not nice to take advantage of good-natured members who want to help others to succeed.

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,982

Re: methodologies - Any empirical tests?

I think that the market exists just because there are different market players with very different goals and trading plans. There are also central banks, market makers, liquidity providers, high frequency traders, retail brokers, institutional and private traders... All these has own strategies and the moment market condition is a gross sum of the behaviour of all the participants.

If there is even a single strategy that can lead to an expected result, it will be so heavily exploited that will be the end of the market.

Hannah was very correct by saying that everyone has own strategy that works only for him. For example: Baillie Gifford American do well now. However, there is no guarantee - remember Lehman Brothers.

I know that several big names have purchased FSB Pro, but I have no idea if they use it for trading or how they do it.

...

One of the most rising brokers on 2017 took EA Studio. I made a short demo to them. On the next day, one of the owners called me and told me that they created experts and started them on MT, but they did not work. It appeared that "did not work" meas that the experts did not opened positions within 2-3 minutes... Three month later the same guy called me again and told me that these experts still make profit. (the same ones that they generated at the first day) Finally they decided that it is too risky for them to present EA Studio to their traders.

Popov's Website

9 Reply by hannahis (edited by hannahis 2018-09-19 23:44:19)

  • Registered: 2010-11-18
  • Posts: 1,179

Re: methodologies - Any empirical tests?

Yes Popov, EA Studio is really excellent tool.  The rate of success is tremendous.  I'm not surprise that when a person found this gem, it's human nature to hide it to prevent others from enjoying the same success.  Cos imagine more people can trade successfully, that would rob these people of their niche or service or competitive edge, especially if they are fund managers or signal providers etc.

For those who already own EA Studio, exploit it as much as you can and it's a matter of time, you will find your own success workflow. 

Back to Noble-6, you probably is asking "How do you know the EA generated/build is not a matter of statistical coincidence", in another words, "How do you know you have found a workflow that can replicate your success rate of finding robust strategies"

Thus you are therefore wondering whether a) people have found a successful workflow b) and how do they quantify or qualify the effectiveness of their workflow?

First of all, I assume everyone's (users) primary goal is to search for profitable strategies and in the process, hope to develop a workflow/process whereby they can repeat their success and the quality of the search is not due to statistical randomness/curve fitted strategies that failed in live/demo testing.

So your question whether people do their own testing to find out how effective is their workflow, I would assume the answer is "yes".  Why? it's human nature to want to search for excellence and for diligent users, they will keep trying and trying various methods till they can finally find one that produce reasonable results that meet their expectations.

Whether the users are very systematic in documenting these findings, the answer varied according to how well disciplined and systematic the person is. 

So how does one measure which workflow is better?  I think ultimately the answer lies in the performance results of the strategies, i.e. their trade performance (demo/live).

As for me, I "measure" how effective or successful my workflow is via the individual result of each strategy (Fxblue statistics) and also the overall quality of each batches (with different settings) to observe which setting produce a) more profitable strategies, or b) higher quality results/statistics. 

The advantages of EA Studio is that you can save (export) these settings and re-use them again (import) and hence, whenever I found a good setting, I would then either re-use it again to generate more or I would import back the good EA and use it as a Preset indicator to generate better ones, this is one of the easiest and best method to repeat your success.