I was reading your post and it makes me think about things.

You mention my approach includes OOS -- but, actually, in a different thread I was making an argument why I do *not* use OOS.   Remember -- when the software performs its optimization what it is really doing is optimizing indicator settings using the input data specified by your Data Horizon.  Since I'm trading in 2017 then I want the indicator settings optimized using 2017 data and not data from 2011 or 2013 or whatever.  To be sure, there is no guarantee that a 2017-optimized EA will perform better than a 2015-optimized EA -- and no doubt there will be periods in the future when a 2015-optimized EA will perform better -- but if I'm going to go with the odds, then I'll go with 2017.

As for methodically optimizing Acceptance Criteria -- that seems like a noble endeavor, but I wonder what it gains in the end.  Unless you can correlate your results with performance in a live account, then it doesn't really count for much.  I mean, most of us have at one time created EAs with amazing statistics and demo account performance -- only to watch it stumble when placed in a live account.  I'm not sure -- but I think it has to do with the nature and pattern of forex data -- it really is true that past data patterns do not predict future data patterns.  So, even EAs with the greatest stats aren't guaranteed to be winners -- though they probably have a better chance than EAs with poor stats.

Several months ago when I first started with Popov's software one of the more experienced traders on the forum suggested I learn to take what the software gives me -- and a light bulb went on in my head.  Popov has not only made it easy for us, he has also introduced a paradigm shift -- we can now trade *portfolios* rather than EAs.  There is no longer a need to effort at creating the first holy grail EA.  Instead we can now create a nice collection of 100 EAs with good stats, add them to a live, micro account and monitor their initial trades.  You will probably find that approximately 50% perform well and 50% don't.  And then you prune the poor-performers -- leaving behind a portfolio that trades profitably.  It really is that simple.

qattack -- I apologize.  In re-reading my previous post in the light of day it came across harsher than it should have.  I applaud you for making the effort to do the additional testing.  I think it was because I've been there, done it, I was suggesting a different direction.  There is no right or wrong way to experiment with Popov's software -- I'll be curious to learn your results -- and I will probably incorporate them into my approach and have to admit the error of my ways.  As for 'my method' -- I'm not comfortable taking credit for anything.  I'm simply following guidance I've gleaned from other's posts on this forum.  There are a number of very experienced and wise traders who support this forum -- I am not one of them.  I'm here as someone who wants to learn and can't help himself from playing devil's advocate.

Your last sentence about going easy with the Acceptance Criteria and then generating and sorting over SQN sounds more in line with what I have found that works well and is easy to do.  The only thing I would add is you may be pleasantly surprised there will be more than two or three good strategies in that collection of 100.

The Magic Numbers assigned to strategies in the Portfolio Expert created by EA Studio are sequential -- from 0 through 99.  So, strategy 100000 is the first strategy and 100099 is the 100th strategy (assuming '100' is the Magic Number's base).  You would think that most of the winners come from strategies with the smaller Magic Numbers and the poor performers would be further down the list (since I sorted using SQN).  But that's not my experience -- winners and losers can appear anywhere within the list of strategies.  I wonder if others have also seen that or have different results.  EA Studio is still very new, so it may take awhile to accumulate observations from different users.

My observation is similar, the best in backtesting is never the best in OOS or real trading. Furthermore, it is quite often that the first 10 strats in the collection never make it to the green side... The biggest problem in my experience.

But I second to Sleytus and will be waiting results!

I've studied strat distribution a bit... Yes, page number 2 is better than first, nevertheless it's all about if's and but's to my observations, pardon statisticians if I use exact terms too loosely, but winning strat distributions are still more or less random.

It seems to be that skipping first strats in collection is not really the same as skipping best optimization versions, mind you - every strat in the collection is optimized to a T.

No offense taken; I usually don't judge the tone of Internet posts. It's very easy to type something in manner that can be misinterpreted and different people also have different tones and different command of the language (speaking in general terms.). Your posts are always very thoughtful and I welcome them.

I the one that often tends to put "too much thought" into things...and I realize that. But I'm not going to change, I'm 49 years old. And I'm always curious about the math behind things (and if it's quantifiable).

Going into this project, I did realize that it could mostly turn out uneventful, but I do think I can't help but to find at least a couple helpful discoveries. And any little bit helps, as there is not really a clear-cut way to proceed at the moment.

And maybe you haven't come up with many of the original ideas on your own, but there really are not too many truly new ideas; and you've pieced together a string of related ideas and re-share them in ways that make more sense.

I haven't yet put too much thought into this type of observation, but there are a number of reasons this can happen.

I believe that most reasons depend to some extent on small sample size, either directly or indirectly.

My current guess is that the primary reason this occurs is that when a strategy is put through the Optimizer, it finds the "best" values for each parameter. Those "best" values are based upon...a small sample size. Many of the values that it decides upon are actually anomalies. If you took one step away from those values in either direction, the strategies SQN would drop by varying degrees. The more values that are optimized, the more the effect. It's  probably an exponential effect to some degree.

The market has many random fluctuations. Our various strategies' indicators (alone and in combination) will run into varying degrees of this randomness. Look at this hypothetical (and yes, simple) range of profit vs. parameter values. Pretend you are optimizing over only this parameter.

Parameter value/Profit$
20   100
21   130
22   120
23   210
24   240
25   370
26   140
27   150
28   210
29   200
30   250
31   300
32   330
33   340
34   290
35   340
36   330
37   310
38   290
39   250
40   220

Looking through this list, which parameter value would you prefer? The Optimizer would choose (I assume) a value of 25, the highest in the list. However, The best value may be 34...not even one of the top values. What's more, if #34 had a profit value of 400 and you are stepping by 5 (...and landing on 0's and 5's...centered around 30, for example), it would miss Parameter value #34 completely.

I'm making some assumptions in the above example. I'm not 100% certain this is how the optimizer works. I also don't know enough about the math of market fluctuation and trends to know if these numbers are logical...but intuitively I think they could very easily fall like this. It comes back to small sample size...a relatively small number of possible trades affecting each Parameter point. And if one of those trades is larger than the rest, this trade would abnormally affect the average.

Again, I may be wrong about how the Optimizer works. It would be helpful if the Optimizer looked at all closely surrounding values and was affected by them to some specified degree (this degree would be affected differently by different indicators, so that's probably not feasible). Or perhaps just a simple smoothing of adjacent values.

My overall point is that a number of seemingly small differences in a strategy can result in a huge difference in SQN.

SQN also takes into account number of trades and Standard Deviation. It great that it does this, but both of these stats can throw kinks into our plans.

As number of trades increases, SQN will grow disproportionately from over-optimization, greatly increasing our error.

If I'm not mistaken, Win Rate of a strategy indirectly depends upon Standard Deviation. Let's say, for example, you found a "winning" strategy with a Win Rate of 18%. Assuming the strategy performs live exactly as in tests (no, of course it doesn't, but just assume...), your short-term results with this strategy will be all over the place compared to a strategy with Win Rate of 75%.

(As a side note, when I was using StrategyQuant to generate over SQN, I found TONS of strategies with less than 20% Win Rate. And VERY few over 42% or so. I'm lovin' the higher win rates found by EA Studio.)

Be very careful, though, of "observational bias." This is a primary reason I've made most of my living for the last 20 years playing poker. Bad players constantly make horrible plays and end up winning. They can get very lucky in the short-term and take down a few pots--even make a lot of money quickly--but they are guaranteed to repeat those same mistakes and lose it all back and then some.

I played for years with a lady who had won a $60,000 holdem pot in Vegas back in the sixties with a "63 suited" (both cards spades). It was her "lucky hand." Every time she was dealt that hand, she raised and played it come Hell or high water. I even saw her make a straight flush with it. But I guarantee she was a lifetime loser with that hand.

That's an extreme example, yet other poker players make similarly bad plays (though not as glaringly obvious) based upon "observational bias"--relying on a relatively small number of observations to formulate strategy in a game with millions of possibilities.

The same applies to trading. There is one major difference in this respect between trading and poker. In poker, you can come up with a pretty mathematical estimation of any action (given enough time). But with trading, there are so many factors and so much noise, that estimations become much less accurate. But if anything, that only increases the need for mathematical analysis, because observational bias will be inherently greater.

Yes, small sample size. There is likely never a big enough sample size when it comes to trading. That's why everyone on the Internet cause pose as the trading guru.

What I hope to gain from my Acceptance Criteria analysis is probably much less than most people are thinking. I will happy to glean any useful information I can from it. I am attempting to define what are the greatest Acceptance Criteria values I can use without possibly starting to over-filter. My guess is that I will end up with very low values set for most criteria.

And I do agree with you that Generation Metrics, Optimization Metrics, and Validation Methods are much more important than Acceptance Criteria. I'm just tackling the easiest one first.

qattack -- just to be clear, you understand in the previous post when I mentioned leaving the live testing to you, I was joking.  It is no fun to lose money and I am not prepared to perform the live tests myself.  I was just making a point and attempting to inject a little bit of humor that I hoped made it clear I was just blowing smoke and *not* ready to risk my own money on my own theories.

It's early to report any findings yet, and this is jumping the gun, but I am running eight instances against "Maximum Balance Deviation" Acceptance Criteria. The range is from 6 to 13 in steps of 1.

Strategies that "Passed Validation" (in the Generator section) decline as expected with smaller MBD values. However, what is surprising is that values of 9 and lower, the "Passed Validation" (in Monte Carlo) DECLINES. This could be just due to small sample size, but the four larger values are in line with the baseline (with only minimum Acceptance Criteria). The four smaller values are at 7.49% to 10.93%, compared to 12.73% for the baseline. I expected all MC validation values to rise.

This doesn't make sense to me right now; I'm a bit too tired to think about it. Maybe a "too smooth" Equity Curve is a good flag of an over-optimized strategy? I'm going to leave this generation running overnight to get a more accurate picture.

Yes, that is the point I try to convey. We are dealing with so many strats, it is impossible to approach a collection with the same mindset and methods we use on individual strats. With the current level of automation lets let the machine do the work, it is only about having the "right" way of letting it do its thing, but first we have to find it.

I've sampled things quite a bit and I've achieved 80% greens at best, the worst result was around 30%. Can't give you other reason than data. It might as well be that we have to choose carefully the right data horizon window upon which generation and optimization takes place. I've been using numbers mainly - 75% train, 25% test and so on, but apparently this not the right way to do this.
My line of thinking - there's a method to produce a collection of strats, which when combined have a positive expectancy. I define this expectancy as having at least 65% of strats in the green zone. The problem is that I haven't been able to be consistent with those methods I've tested, having a drop of 50% in profitability is frankly a disaster

Another observation - Sleytus, you are assuming that those strats, which survive, retain their edge, correct? My modelling shows that this is not the case, there are still too many bad apples which go bad after initial pruning, thus rendering the overall profitability to breakeven minus trading costs in the best case scenario...

---

I woke up this morning and realized that the optimizer has an option - Add the optimized strategies to the Collection
It means that all optimization cycles, which are better than the last one, get pushed into Collection, therefore I can get a glimpse of this "anti-optimization" idea by testing a handful of strats (equal number of good and bad strats) and evaluate their effectiveness.

Lastly, John, forgive me for going off-topic, if you want I can move my ramblings to a different thread.

Small sample size...How much is "large enough" sample size. I fear it's in the tens of thousands of strategies. For comparison, run a Monte Carlo test with 20 runs and examine the Confidence Table, taking note of every Confidence Level. Then repeat the exercise for 100 runs and 1000 runs and compare the results. You will find that they vary wildly.

The various strategies we generate are MUCH more volatile than this compared to one another. That, with the combination of not having the luxury of 1000 test runs, will cause each group of 100 EAs to have drastically different performance levels. Only by finding the "best" selection methods can we reduce this problem significantly (as detailed below).

This is exactly what I'm trying to say in many other posts...it just takes me 490 to say it instead of 49. How long is "long enough" before you decree that these strategies are "profitable"? The "profitability test" is over the lifetime of deployment of the strategies. A group of 100 EAs may perform impressively during the first week, with 60% winning strategies. In fact, you may be able to retain that 60% figure through pruning for a month or longer, yet over the lifetime of that group, it may still very well be an overall loser.

I cannot locate this option; I opened up a new browser to search.

What would the effect of this option be? Am I correct in saying you wouldn't want to use this in normal generation, as the Collection would quickly become full of the same strategy Optimized in different ways? So it's just for observation of how the Optimizer works?

Your comments relate well enough to the topic at hand. It's more that the Topic doesn't relate to the topic. Maybe we can have Popov change the topic to something that will better depict our mission.

ThanX!
John