26 (edited by hannahis 2017-08-31 22:13:24)

Re: Using Synthetic DataHorizons For Generating Strategies

1. Training your EA with Synthetic Data

First of all, I think it's a great idea to use synthetic data to "train" our EA for Specific Market conditions.

It's like Artificial neural networks (ANNs), whereby you "condition" your EA to be good (curve fit) in certain market situations.

If I were to go about with this project, this is what I'll do...

a) Cut and Paste as many ranging/breakout market conditions as I can to create a long series of similar market conditions so that my EA can Identify some Key trading concepts from these similar data. 

b) Next train my EA to Differentiate non similar trading conditions (improving on it's filtering process to avoid false signals).  In order to do this I'll start adding opposing/mix signals.  For example, If I want to develop Breakout Strategies, my 1st synthetic data will have mostly breakout trends cut and pasted together.  And my 2nd training data set will have Ranging data period add in between 2 breakout trends. 

In order to make sure my Breakout strategies are good, I will use "prolong" ranging periods in between 2 breakout trends.  The aim is to filter out poor Breakout Strategies that can't handle ranging periods and the end product will be good Breakout Strategies that can handle prolong ranging in between breakouts.

c) After the Strategies have pass these 2 training (a +b), I'll subject these EA with "unseen" data/OOS data of "a normal real" historical period or put it to demo testing.


2. Subjecting EA with "unseen" data (isn't that also mean OOS data?)

I agree that putting EA to demo testing is rather long process if you look at from 1 EA's point of view.  However, imagine if I've hundreds of EA and I've to keep monitoring and setting up one by one EA for backtesting, I think the demo path seems a much easier task.  I install hundreds of EA and I leave it there for 3mths, in between these 3mths, I develop and create another new batches.  So it's not time consuming because I don't have to do anything during these 3mths for that particular batch of EA in demo testing.  Whereas, if I were to backtest hundreds of EA, I'll be spending lots of hours walking to and fro my computer checking and changing another EA for backtesting, saving the results, putting them together for comparison etc.

So in terms of testing hundreds EA, I would rather just put them in demo and leave them be than to sit in front of the computer, backtesting 1 EA at a time.


3. Backtesting results vs Demo testing (which is more reliable?)

Personally, I trust Demo testing results more than any backtesting tools.  Even when I got an excellent backtesting results, it still leave me with questions whether it can still be profitable in demo/live.  With demo testing, there is no room for doubt, I know I can replicate the success from Demo to live and hence, I trust and rely on my demo statistic to assist me in my portfolio selection.

Lastly, as I mentioned before, even if you have a backtest results, you still won't know what types of EA you have (especially if you depend on FSB to search for you) and you won't know how the EA behave in different market situations.  Thus, these are extra vital information you can get from demo testing.  For example, when I know that my Ea will lose about $3000 during ranging, do I panic and stop the trading.  No, cos since I know how my EA behave, I know that my EA is capable of earning back what it loss 2x or 3x more.  A lot of people panic when their EA lose money and they shut it down prematurely cos they haven't observe their EA enough to have confidence in their EA, all they understand of their EA is just those backtesting statistic. 

It is very important to understand how your EA behave so that you can have confidence whether to keep your EA running during those times when it is losing money.  That's part of the psychology of trading with EA, do you ultimately trust your EA?  And such trust often can only be built over time when you observe your EA's performance in a demo account, not just by staring at the statistical numbers.



4. Does MM really a good filter for robust EA?

I agree with Steve's analogy comparing different markets as different language.  However, for EA, despite different markets, their language is still the same (calculation using those indicators formula to determine a rule is true/false.

So it's not so much about speaking in different language but I think it's more apt to use Dance as the analogy.  Each market has their own unique rhythm/pattern/flow/tempo.  Each has different ranging gaps/channels etc and hence to expect a Jazz dancer to excel well in a tango dance, it would be "off sync", not that the Jazz dancer can't tango, given his natural talent/EA's fundamental theory, he may pass off with a good dance but not an excellent one.

Thus I personally don't agree using MM at all to "train" my EA.  You ended with "Jack of all trades, master of none"

Theoretically speaking, the trading concepts are applicable to all markets.  I use my same trading concept in all different markets, it is sound and hence it is robust.  However to work out the fine details of creating an EA for a particular market, one still need to use that particular market's data set to "understand" what are the reason and rhyme for that EA to click with the market.  Different markets have different ranging duration, different Breakout heights, different government interventions/fundamentals issues...these are times, when technical analysis take on a passenger seat while the fundamental are driving the markets trends (as we have often observe for the past couple months). 

During such fundamental market interventions, a lot of EA will fail especially when it uses laggy indicators/parameters because it would have created many "false" signals (which otherwise are real but because of government interventions, the technical or the normal peak and trough didn't complete it's full cycle, changed "prematurely"). 


In Summary....

Yes, I believe it is good to train our EA with synthetic EA but we don't have to subject it to MM and like what Footon said, ultimately, whatever method it is, the one that helps you yield the best and consistent results is the one most ideal (be it backtesting or demo).  The key is "are you able to repeat your success"?, if you can, you have found an excellent workflow to replicate your success.

Hannah

Re: Using Synthetic DataHorizons For Generating Strategies

Hannah -- thanks for adding some good points...

hannahis wrote:

we don't have to subject it to MM

Regarding MM -- I'm not advocating for or against using it as a filter to select EAs.  I was simply using MM as a *metric* -- i.e. as a way to compare the effect that a synthetic data set could have. 

Again, take the example of starting with a single EA and cloning it -- now you have two *identical* EAs, cloneA and cloneB.  Optimize cloneA using 12 months EURUSD, and optimize cloneB using a hybrid data set of 6 months EURUSD plus 6 months USDCHF.

So what?  I now have two, trained strategies.  How do I compare them?  I don't want to put them in a demo account and wait 3 months.  MM is just a fancy name for performing multiple back tests in parallel using more than one market.  So, I simply used MM as a way to compare how cloneA and cloneB performed against a EURUSD data set and the hybrid EURUSD+USDCHF data set.  The result was cloneA did fine against EURUSD but poorly against the hybrid, whereas cloneB did excellent against both.  The result isn't surprising, but I think it does have important implications -- how one might use it to gain an edge remains to be seen.

Regarding analogies -- language versus dance -- both are fine.  The point is things usually do better when their training takes into account how they will be used.  Other analogies also apply -- how about sports?

Regarding your point (1) -- "Training your EA with Synthetic Data" -- it doesn't have to be that complex.  Well -- if you can tell ahead of time the type of market then, yes,  I see your point.  But I'm assuming one doesn't know ahead of time.  So, my plan is to have a portfolio of EAs that have been trained in a variety of ways and turn them ON and OFF according to a simple policy.  Keep in mind that an EA that was trained for jazz might also do perfectly fine at tango.  Forex is empirical, not theoretical.  Give it a chance to trade and then make your decision to turn it ON / OFF based on how it is trading.  Back to the sports analogy -- the goal isn't to beat your opponent 100-0, but simply score more points than they do.  It's okay to absorb some losses while turning EAs ON / OFF -- as long as you have a reproducible way to come out ahead in the end.

28 (edited by hannahis 2017-09-01 08:19:49)

Re: Using Synthetic DataHorizons For Generating Strategies

Hi Steve,

I'm curious to understand and learn from you what simple "policy" you are going to use to turn off and on.  To know which EA is going to perform/fit better in future market conditions that it is trained for.

By looking at my trades performance statistics, I've been trying to use different trade metrics and see which of these have the "best" predictive element.  Still can't find any statistical metrics that can tell ahead of time where market is ranging or not smile  But over the times, I ended up using EA that has overall best performance statistics, i.e. survivor of the fittest.  Those who have gone through various challenging markets conditions to survive at the top.

Therefore, instead of looking at "predictive" metric, I went back to my drawing board to design better and more robust EA, ultimately its the indicators/EA I'm relying to react to market changes and handle market fluctuation and hopefully they will react better, striking a dedicated balance between speed of reaction vs stability of signals/price changes.  Double edged sword.

So I'm curious, how can one able to come up with a Simple "policy", for such complicated and irregular market conditions.  What is it that you are seeing that I didn't see?  How this simple policy/rule can know ahead of time where the future market conditions is suited with the EA's market condition it is trained for?  In another words, how a simple policy can predict whether the next market condition is ranging or breakout?

Hannah

Re: Using Synthetic DataHorizons For Generating Strategies

Hi,
The disagreement here comes about that MM test should be used to check strategy with unseen/untrained data, that is the point of this test. And your experiment Sleytus used the same data, let it be with some mixing. The point is that it showed nothing for robustness, because you trained on the same data and checked on the same data. I think that you need to check with different MM pairs (what it was not mixed in training) or use different time horizon for OOS.

Re: Using Synthetic DataHorizons For Generating Strategies

hannahis wrote:

what simple "policy" you are going to use to turn off and on.

A "policy" can be as simple or complex as you want -- based on your preferences or trading experience.   Since you prefer complex over simple, then perhaps this wouldn't apply to your trading style.   I've mentioned an example of a simple policy a few times:  "If an EA loses two trades in a row turn it OFF, wait two weeks, turn it back ON".  The idea of using policies is based on the premise there is no such thing as a bad EA -- rather, EAs trade well or poorly based on the current data environment.  Data trends change -- so, it follows that an EA's performance will also change based on the current data environment.  The concept of applying a policy to a dynamic system comes from monitoring communication network performance.  Networks are dynamic and constantly changing.  The goal is to route data in the most efficient way.  So, policies are designed to route data down certain paths depending on the current conditions.  One day a particular path may be inefficient, but you may return to using it a couple days later -- depending on the current conditions.  Network routing, at its most basic level, is about turning paths ON/OFF.

I don't know where you got the idea I'm trying to predict something -- I'm not.  All I'm saying is that an EAs performance is closely tied to the current data environment.  Since I don't know the future, then the only way to know if an EA likes the current data environment is to monitor its current performance and turn it ON / OFF accordingly.  There's no prediction.  If an EA is performing well, then leave it alone.  If an EA is performing poorly then give it a rest.  If you were to ask me how long to rest -- that I don't know.  Forex is empirical -- which means the way you figure things out is by experimenting -- i.e.
trial and error.

Perhaps what I'm describing is too simple or obvious -- I don't know how else to explain it.  Or perhaps it's just nonsense...

Re: Using Synthetic DataHorizons For Generating Strategies

Irmantas wrote:

The point is that it showed nothing for robustness

That was not the point I was trying to make.  The point I was trying to make is the *data set* used during optimization has a huge effect on performance.  Since I didn't want to test cloneA and cloneB for 3 months in a demo account, I simply used MM as a *metric* to compare the two.   The result showed that two *identical* strategies that only differed in the data set used during the optimization could exhibit vastly different trading behavior.

In the short-term, cloneA would probably be more profitable, but cloneB is more "robust".  Some people consider MM a test for robustness.  I am undecided on whether to use MM to select for EAs.  I only used MM as a means for comparing two otherwise identical strategies.

The fact that the data set used during optimizing can have such a big impact on an EAs behavior has some interesting implications.  It also opens the door to using synthetic data sets.

32 (edited by hannahis 2017-09-01 18:58:36)

Re: Using Synthetic DataHorizons For Generating Strategies

sleytus wrote:

Forex is empirical, not theoretical.  .

Forex is both empirical and theoretical

If you are only looking purely at numbers, yes, it is empirical.

But if you are looking for patterns behind those numbers, then these patterns form a "theoretical framework", all our EA rules are about inputting a "theoretical" belief that under certain conditions/rules, the probability of making money is higher.

So if I were to tabulate your sleeping and eating habits into statistical numbers, can I say you are all nothing but empirical?  No, the numbers of time you sleep and how long you sleep...is to determine and find out your sleeping patterns.

Numbers can't tell you what's your motivation behind those number of times you sleep.  But the implications of those numbers that help me understand from your sleep patterns, whether

1) you likely going to have any health problems due to lack or too much sleep
2) your mental state of mind whether you going to be depress due to too much or too little sleep
3) Are you suffering from other issues like anxiety if you having too little sleep/insomnia

Hence by looking at numbers itself, if one know what to look out for, these numbers will help you understand the implications of the person we are measuring.

Likewise, although Forex are all just empirical historical data but under a skillful trained eyes, some can see a "relationship/patterns/theory behinds these numbers.  But to the untrained person, there are all just numbers.

Similarly, if you were to look at a company's financial accounts, if you aren't trained, you won't know whether the company, though making lots of profit is facing the dangerous issue of liquidity issues.  Whether it's liability is greater than it's asset, etc.  To a lay person, they are just numbers. But every number tells a story.

John Maxwell once said, if he can spend a day observing a person, he can tell whether that person is going to be successful or not.  Well I won't have his kind of expertise to do that in a day (and that statement may be over exaggerated .  But what he is trying to bring across is, if you can (whether within a day or weeks) established/find out a person's behavioural patterns, you may then predict the likelihood of his life success.

In another words, the ability to identify patterns would shorten the need to collect extensive and long empirical observational data in order to establish the same finding.  Like a psychologist, in 1 counselling session, he/she may able to identify a person's issues that even the counsellee or his best friends aren't aware of.  Thus this psychologist can "know" this person more in a day then what his best friend could know all these years.

The same is with Forex, if you can identify certain patterns, trading theory, you will then able to establish a certain expected outcomes, that otherwise, would take another person to collect extensive empirical findings to come to the same conclusion.  And if the software is "untrained" and doesn't know how to identify patterns from the numbers, then it would only churn out random combinations that aren't going to be profitable EA because these EA's rules doesn't formed any "reliable, repeatable patterns".

Thus that explain 2 groups of FSB users, one who take the statistical path to find profitable strategies via FSB and another who see those patterns and input their patterns into FSB to narrow the search.

if you can identify patterns and use the statistical approach to narrow down further your search, you are better off then one who purely rely on statistical data to search millions of combination, cos you have shaved off probably a large percentage of "unwanted and random combinations that aren't going to give you any statistical significance.

Hannah

Re: Using Synthetic DataHorizons For Generating Strategies

Sorry -- I guess our understanding of what constitutes empirical versus theoretical differs.

34 (edited by qattack 2017-09-02 13:30:39)

Re: Using Synthetic DataHorizons For Generating Strategies

sleytus wrote:
Irmantas wrote:

The point is that it showed nothing for robustness

The point I was trying to make is the *data set* used during optimization has a huge effect on performance.

Irmantas is correct; using In Sample data creates bias for the specific market(s)/time frame in question. So the EURUSD-optimized clone is curve-fit for EURUSD but has no relation to USDCHF, while the other clone will be mediocre (relative to clone A's performance in EURUSD) for each data set. It will, of course do better with the hybrid, since that's the data is was optimized for.

It may appear "strong," but I wager if you check both clones against OOS data from each market, they will both do poorly. I'd still guess that in the long run clone A does very slightly better with the EURUSD data and very slightly worse with the hybrid data. But not enough to make a huge difference.

Re: Using Synthetic DataHorizons For Generating Strategies

First you say Irmantas is correct, and then you say that a data set creates bias -- which is exactly my point, except I wouldn't use the word 'bias' since it has a negative connotation.  You are aware that *every* strategy that is back tested and optimized is biased, right?   Everyone who creates strategies using FSBPro and EA Studio is creating "biased" strategies. So, why do you choose to use "bias" in this particular case?  "Optimized", "curve-fitted", "trained" would have worked equally well.  But if you want to make me wrong, then use the negative word.

You go on to say the EURUSD clone does better against EURUSD and the other is mediocre against both.  Again, that was exactly the point -- except I left out the "mediocre" part.  The chart I attached showed the hybrid did better than mediocre -- but you choose to say "mediocre" anyways, why do you think that is?

I give up -- you win.  I have nothing more to add to this subject -- it was a mistake creating it in the first place.  Sorry for wasting people's time.

36 (edited by hannahis 2017-09-02 19:33:17)

Re: Using Synthetic DataHorizons For Generating Strategies

Hi Steve,

The purpose of forum is to discuss, brainstorm and get some new, interesting, opposing ideas we would never have thought it out by ourselves.  It is not an examination hall, whereby your answers are either right or wrong.  There are many perspectives to life.  Even a coin has 2 very opposing views (front and back) and yet these views make up the sum of all things, ie. the coin itself.  You can say a coin often "contradict' itself.


The say the least, the topic you opened up gathered lots of discussion, it shows that people are interested with topic you brought up and are chipping in their perspectives, it's not an issue of right or wrong, it's about keeping our minds open to new way of seeing things if there is some creative thinking come out of the discussion.


I thought your intention to open up this topic is for anyone to shoot it down (you welcome any opposing views)

sleytus wrote:

The best we can do is to be aware of the current market trend and which of EAs perform best in the current market.

And this gave me an idea -- please feel free to shoot it down...


And you understand that in when we post our comments, we are all just rambling like you...we may not offer any real and good concrete help...just airing our thoughts....like you (so bear with us, if we don't make sense to you).


sleytus wrote:

This is really interesting to me.  I'm still formulating some thoughts -- so the post below probably is more rambling than usual.


Which do you prefer?

You opened a topic and get lots of discussion.  You get different views, from different "schools of thoughts", you get interesting ideas that you never would have considered in the past and you sometimes get weird ones or "lousy" ones and at the end of the day, you sieve through these ideas and keep those that makes more sense to you and put aside the rest. And you are grateful for these people are come along the way to give their 2 cents worth of thoughts...etc

or

You opened a topic and nobody commented on your post (in the future, too) because either they aren't interested to share their views or they are "afraid" of "offending" you or worried they are not clear of their statement (becos they are rumbling too) to be misunderstood by you, afraid that if I don't think clearly, and some how came up with "contradictions" and if I used the "wrong" choice of words, you think I'm insinuate some hidden underlying intentions?


Personally, I don't think it's a waste of time at all for your topic and there is so much possibility out of your "original" goal of this project, if you were interested to explore further. 

Take it easy Steve, we are all here to learn from one and another and I must say, your presence has added much value to this forum's discussion and it's community and it's a pity if you "quit" posting your topics that present very interesting discussion.

Hannah

Re: Using Synthetic DataHorizons For Generating Strategies

Hannah,

I opened a topic about a new and interesting idea.  I even did an experiment to show there is something to it.  And you know what?  No one asked a single question.  Furthermore, most of the respondents lectured me where my thinking was wrong or to push their approach -- even though they never really grasped the simple concept I was trying to explain.  For example, your most recent post lectures me about the purpose of the forum.  Really?  I introduced a new topic, wrote a few thousand words and shared my data.

Yes -- I was a bit taken aback.  I had assumed members of a forum would be interested to learn more -- but they aren't.   Also, I got tired of explaining it only to have people make up things I didn't say.

There are a couple experienced traders who support this forum and I've learned a lot from them.  Introducing this topic and sharing my results was a way for me to partially give back.  From now on I'll just stick to asking questions.

38 (edited by qattack 2017-09-02 22:34:51)

Re: Using Synthetic DataHorizons For Generating Strategies

sleytus wrote:

First you say Irmantas is correct, and then you say that a data set creates bias -- which is exactly my point, except I wouldn't use the word 'bias' since it has a negative connotation.  You are aware that *every* strategy that is back tested and optimized is biased, right?   Everyone who creates strategies using FSBPro and EA Studio is creating "biased" strategies. So, why do you choose to use "bias" in this particular case?  "Optimized", "curve-fitted", "trained" would have worked equally well.  But if you want to make me wrong, then use the negative word.

You go on to say the EURUSD clone does better against EURUSD and the other is mediocre against both.  Again, that was exactly the point -- except I left out the "mediocre" part.  The chart I attached showed the hybrid did better than mediocre -- but you choose to say "mediocre" anyways, why do you think that is?

I give up -- you win.  I have nothing more to add to this subject -- it was a mistake creating it in the first place.  Sorry for wasting people's time.

Steve, I'm sorry if I offended you; please let me explain myself a little further.

First, your ideas are great--and when I say "great", I mean "great"--I'm not belittling you or your ideas one bit.

I'm just attempting to explain in words that I know how things are not quite working correctly so that we can (as a group) get the ideas that you originate on a track that will ultimately benefit us.

You misunderstand my use of the word "bias". This is one of the most commonly used words in the field of Statistics to indicate an area that may be problematic in use of a data analysis result. In fact, there is a "Statistical Bias" attached to nearly EVERY statistical result presented.

The Internet is sometimes not the greatest place to glean the meaning of someone's tone. It's almost impossible to write your tone in the message. I think you would agree that my posts are long enough (and yes, windy at times) without qualifying all my statements with, "I don't mean to offend you" or "This is only my opinion" or "Don't take this the wrong way"...

And you are correct when you say everyone is creating "biased" strategies. We want to reduce that bias as much as possible with the tools at our disposal. In fact, that is our primary goal. The more we can reduce that bias, the less "noise" remains and the closer these strategies will perform in real trading versus the generated statistics.

And I'm sorry, I didn't see where you had posted the results. I've been working 60 hours the last two weeks and haven't had time to read everything carefully, but I'm following your posts with interests...they are heading in the right direction, for sure. And that doesn't often happen.

Don't take what I say personally; unless I make a direct attack on you (which I'm not doing at all. Really.) I'm just trying to bring what experience I have to the table--and that experience is NOT trading. It's from a mathematical/statistical viewpoint. I'm sorry to say that I cannot contribute much beyond that at the moment. But I believe a mathematical approach is very important to what we are doing. So it was my attempt to help.

Got to get to work...

Re: Using Synthetic DataHorizons For Generating Strategies

qattack wrote:

So the EURUSD-optimized clone is curve-fit for EURUSD but has no relation to USDCHF, while the other clone will be mediocre (relative to clone A's performance in EURUSD) for each data set. It will, of course do better with the hybrid, since that's the data is was optimized for.

No apologies are needed.  The issue is not about whether or not I feel offended.  The issue is you effectively kill the thread with your statements -- since as the last post then yours is the most commonly read.  Future readers won't be interested because who is interested in "mediocre" results?  They don't know you hadn't bothered to check the results I had attached (and referred to numerous times).  For the record, the results were *striking* -- not mediocre.

Sharing different perspectives? Sure, that's great.  But how about some "due diligence" before posting statements that mislead everyone else.

Look it -- it's not a big deal.  What gets posted here has no effect on which direction I take.  I'll do my own experimentation and proceed accordingly.  My only intention was to see if I could pique anyone else's interest.