Great topic for a post.  Not that it helps -- but rest assured you are not alone.  This is all part of the journey.  There is no single, correct recipe.  Your work flow sounds detailed and methodical -- which is good.  Your mind is also thinking like a detective -- also good.  And you've identified the issue as over curve-fitting -- which I would agree with.

I have a few comments now and depending on how this topic evolves I may add a few more.  Your rut sounds very similar to mine until I returned to Forexsb a couple of months ago.  Popov's software is a game changer and it allows me to do things I never could before -- in particular, to generate high quality portfolio EAs with the press of a button.  Why is that important to me?  I can now generate portfolio EAs in ***near real-time***.  Do you understand what that means?  The market is constantly changing and that is why there will always be a disconnect between an OHLC simulator and real-time trading.  Loading-up on historical data is a fool's errand because you can't train algebra -- it's not a database.  In the context of data analysis, optimization only makes sense when a data set is ***static*** -- that is, optimization algorithms find the best fit for static data.  But forex data is not static, so optimizing a best fit for historical data is doomed to fail -- the market condtions and data patterns are constantly changing.  And the proof is staring you in the face -- even though performance is poor you'll keep repeating the same steps.  We have all been conditioned to think that way.

In an ideal world the perfect generator would be able to generate new strategies in real-time that align with current market conditions.  And with the upcoming Express Generator we'll be able to approach that.  I like CJ / M15 with a max of 300 bars (a little less than 3 days).  I only use the generator -- no OOS and no optimization.  I'll let it run a few hours and take the top 50 strats out of about 200.  The uploaded data from my broker is only 15 minutes delayed.  Since I let the generation run for about 4 hours then by the time the portfolio EA begins trading it is only 4 hours behind current market conditions -- which is pretty damn good.  I repeat this every day.  The portfolio EA trades well for a day or so and then noticeably begins to taper off.  No problem -- press a button and create a new one based on current market conditions.

My invdividual strategies average about 10 trades per 300 bars (a little less than 3 days).  10 trades is not statistically significant.  However, when you accumulate a bunch of statistically insignificant measurements into one larger measurement (i.e. a portfolio of strategies) then the overall measurement does become statistically significant.  That's the power of averaging...

Hello timelleston...

This is great -- let's continue -- I'd like to address some of your points...

1. "if you’re training on a very short period of time then it seems it’s optimised for that short period so it would make sense to me that it starts to go awry after a couple of days."
Yes -- exactly -- it's curve-fitted for the most recent 2-3 days.  That's what I mean by ***near real-time***.  Real-time would be every tick.  In this case ***near*** real-time means the most recent 2-3 days.  Since I don't have a crystal ball, then I'll take that.  Though market conditions change, my experience is that probability is on my side that current market conditions will persist that long or longer.

2. "With a longer time frame I would expect it to last longer,"
No -- it's just the opposite.  And you have the proof -- it's the reason you started this topic.  The longer the data horizon, the more historical (and irrelevant) data is used to compute settings.  Since we trade **now**, in real-time, we don't want to pollute our input data with irrelevant data.  Again -- algebra can't be trained and irrelevant data only serves to dilute the importance of the most recent data when computing the best settings.

3. "That does seem quite onerous though, doing that everyday."
Not really -- how long does it take to press a button?  I've found that I need and enjoy being engaged -- trading a custom EA for a couple of weeks with zero intervention doesn't work for me since I'll then look for others ways to get into trouble.  Now I spend 15 minutes per day -- that includes pressing a button to start generation, returning 4 hours later to move the top 50 to a portfolio, press another button to generate source code, press another button to compile, drag and drop the compiled portfolio EA onto a CJ/M15 chart.  It takes me longer to type-out the work flow than to actually perform it.  And, again, it's enough to keep me engaged and involved.

4. "I assume you have a bunch of open trades each time that you have to either manually close, or let the EA close out at some point in the future, so you’d leave it running with zero as the max trades? "
You're a smart fellow -- yes, after a couple of days I set Max Open Trades to zero and let the trades close gracefully.  As the week progresses I add a new portfolio EA each day and gracefully retire aging ones.  If I were hand-coding custom EAs this would be extremely tedious and time-consuming -- but with Popov's software I'm just a button-press away from a new portfolio EA so it's no longer an issue.

It’s interesting talking about algebra etc; as my background was a spot CAD interbank trader for Citibank for about 8 years, then as a founder of a data analytics company for 10+ years.

From a trading standpoint, I would be trading hundreds of millions of dollars a day, multi-million dollar positions each time. Technical trading was part of it, but also very mindful of market conditions. Banks move markets. Period. Retail traders get closed out easily/on purpose.

After trading, and from an analytics business standpoint, we used predictive modelling to help our clients market to individuals that were likely to buy something. You know, propensity modelling etc. The more data, the more accurate the models. We consumed data.

So, this is like a reverse universe, an alternate universe.

Lesser timeframe (like 3-5 days) has the propensity to be more accurate because the market has moved on so quickly - I get that. And it’s a short window of opportunity. And that makes sense too.

So, I liken you to bank trader - one that is not trend trading (no time length to really do that), more like scalping or short term swing trader?  And that’s how you’re using EAS to generate strategies and open positions…?

I read somewhere else that you're a manual trader … do you mean you let the EA open a trade and you manually close it when you believe the time is right? Or, do you let it run its course?

I went through a stage where I closed manually, but I found the EA was actually better at doing it so I forced myself to let it run its course…

Thanks for sharing a bit about your background -- it helps create a context for your comments.  You bring a lot to the table and are more experienced in this field than me.  In a previous life I was a research biochemist (15+) and, subsequently, software engineer for troubleshooting wireless networks (30+).  I cut my teeth on FSB Pro, left for a few years, and returned a couple of months ago to EA Studio.

> "so you’ve basically accepted that overfitting can’t be avoided, and you’ve embraced it, and developed a methodology that enables you to leverage the overfitting for a short period of time."
Yes -- that's an accurate and concise way of summarizing in one sentence what took me 50 sentences to describe in my previous post... 

>"I read one of your posts about OOS and you were doing it, but I recall it was on 1H timeframe, and checking slopes etc before letting the strategy run … do you use OOS at all, such as for longer timeframes, or are you ok focusing purely on regenerating strategies near-real-time instead?"
I was using OOS on H1, but no more.  It continues to amaze me how feature-rich Popov's software is and OOS is an interesting area that few traders outside of Forexsb take advantage of or are even aware of.  But, for now, I'm focusing on near-real-time portfolio EAs.

>"It’s interesting talking about algebra etc"
It was on this forum about 5 years ago that Blaiserboy made a comment that "it's just algebra".  It was an "ah ha" moment for me.  Until then Forex was kind of a black box to me.  Since then I've coded many indicators and EAs and understand how things work under the covers -- and, as Blaiserboy stated, it really is just algebra.  In fact, it's "baby" algebra.

>"After trading, and from an analytics business standpoint, we used predictive modelling"
I'm not familiar with that type of modeling but I am with other types.  It has occurred to me recently that Forex data is different and doesn't necessarily lend itself to traditional data optimization techniques.  Almost all the data analysis I've done in the past assumed **static** input data sets.  But Forex data isn't static since market conditions are always changing.  I've now become less enamored of optimization.  The Generator suits me fine -- since only the best strategies ascend into the collection and I migrate only the top 50 into a portfolio, then that serves as a kind of optimization.

>"So, I liken you to bank trader - one that is not trend trading (no time length to really do that), more like scalping or short term swing trader?  And that’s how you’re using EAS to generate strategies and open positions…?"
I'll go with swing trading -- that's fair to say.  It's not that that was my intention -- rather, swing trading strategies is what the Generator is creating.  I've read elsewhere that swings occur approximately every 3-8 bars -- regardless of timeframe.  And from MT5 statistics I can see the average position holding time for the strategies in my portfolio EAs is about 4 bars.  So, that fits with swing trading.

>"I read somewhere else that you're a manual trader … do you mean you let the EA open a trade and you manually close it when you believe the time is right? Or, do you let it run its course?"
I refer to manual trading as "discretionary" trading -- and I'll answer your question with a "yes"...   Since I develop software then I've done all combinations.  I definitely prefer algorithmic trading because staring at a screen looking for trades doesn't suit me.  But I also like some form of engagement and total automation is too boring.  I like what EA Studio provides for me.  I trust the code and usually let trades run their course.  If it is the end of the week and I'm in the green and the DD (draw down) isn't too bad then I'll manually close so that it doesn't weigh on my mind over the weekend -- something like that...

With regards to swing trading -- I don't think the Generator has any bias to a particular trading pattern, be it swing, trend, ranging, breakout, reversal, whatever.  However, when the data horizon is short (e.g. 300 bars) then the only patterns visible to the generation software are the short swings.  When the data horizon is large then the swings are masked because they get hidden behind the gobs of historical data the generation software has to deal with and take into account during its optimization.

The disconnect between Backtesting / Strategy Tester and real-time trading can be attributed to the following reasons (and probably more):

1. OHLC bar simulation is not the same as streaming ticks -- it's like comparing apples to oranges.

2. Optimizing leads to computing the best settings for historical market conditions that never existed and will never occur in the future.  Market conditions are reflected by the most ***recent*** bars.  A data horizon of 1 year that includes 1000's of bars represents an abstract market condition that has never existed.

3. Large data horizons mask swings -- whereas the simple algebraic rules (used by our strategies) lend themselves best for detecting swings.  Large data horizons include trends, ranges, breakouts, reversals, swings and the optimization software has to deal with all of those -- and there is no way simple algebraic rules can generate a strategy optimal for all situations.  However, when you use a short data horizon the generator can focus on computing the best settings for swing trades.

EDIT: Come to think of it, all there are are swings.  Swings are the "building blocks" of the other data patterns.  Consider what looks like a trend on an MT chart and then look closer -- you'll see that the trend is actually made up of swings...

I'd be interested to learn more as your test results come in.  I'm always fiddling and trying new things.  It would be nice to continue the conversation and perhaps others could join in... 

Best practices?  I must have slept through that class...   

Hi GD - I used both fixed stop and fixed profit, but I normalised over the period, which I set to reduce the SL and TP.

GD - What are your thoughts on the topic as a whole - OOS or no OOS?

Here's the results from the Generator, In Sample only (no OOS), for the same periods - likely different strategies though - it's the curve we're looking at though.

As you can see, the balance lines immediately start to decay at the end of the period they were generated for.  This I would expect...but not the previous one if it was using OOS.

One could argue that the OOS is 2 days where it starts to decline.  If that was the case, then generating these without the OOS over the same period with the same settings, should in theory, allow them to survive at similar performance rates for at least 2 days.

If the strategies were generated on 80% of the data (the In Sample) and then "checked" against the 20%, then I would expect the In Sample strategies over the 1000 bars, with no OOS, to demonstrate similar performance - lasting at least 2 days.

Agreed. 

Agreed, thats completely what I'd expect...except that didn't happen either - none of the cases began to taper off during OOS.  They all went immediately after OOS.

Agreed - logically yes, but that suggests that market conditions just happened to change, each and every time, coinciding with the end of the OOS period.   5 times in a row. Consistently.

The curve during OOS was similar to the IS curve.  That suggests that the market conditions were exactly the same during each of the 2 week periods.  Over 5 different periods.  And then to suddenly change on the same day each time?  Hmmm...doesn't seem plausible to me.

Without the line I put on the chart, you'd not know where OOS started.  Market conditions have definitely changed yes, which is why (we actually both agree) OOS shouldn't be exactly the same as IS.  But it hasn't happened that way.

AudUsd 15m, 1000 bars, so approx 10 days.  OOS was approx 2 days.

My point is that it made no difference using OOS vs not using OOS.  The decay happens almost instantly at the end of the period irrespective of OOS or not.  That makes no sense.

If the strategy is generated on 80% of data (800 bars), and then 20% is used to validate the strategy (OOS, 200 bars), and it performs as well during that OOS period, that strategy "survived" for two days during the OOS period.  It had the same curve, same performance - but on unseen data.  To have such a massive decay, instantaneous almost, at the end of OOS, seems unlikely.  Even if 50% of them started to decay immediately during OOS, you'd see that in the charts, as they'd start to taper off...but they don't.  They perform just as well as they did during IS.

Everything you'd said is correct and I agree with it in principal...however, the data is suggesting something else is going on here.

I'm going to re-run some data, with 20% OOS, but move the date forward by 2 days.  I know, descending into madness...tie a rope on to pull me out later - hahaha.

Okay -- I agree I didn't give a lot of thought to your results and may have spoken too soon.  I did a quick view from 30000 feet and it appeared consistent with what I've seen and experienced, as well.

Your point is that regardless of the data horizon and OOS, performance tapers-off at around the same distance after IS.  Is that right?

The point at which it tapers off isn't so important to me -- since that would depend on how similar or different market conditions were between IS and OOS.  However, what is really clear is that IS looks good and shortly later things go south.  When I was using OOS I saw many strategies that seemed to survive OOS, but I also saw that their OOS statistics were not as good as IS stats.  That is, it was clear things begin to decay once you leave IS but it wasn't necessarily fatal.  There were other strategies that were clearly over-fitted and failed OOS.

I'm not prepared to get too down in the details and mud with you -- not because I think you are wrong, but because I don't think it will lead anywhere.  It is too complex and there are too many interdenpencies for us to understand it all.  My inclination is to start from 30000 feet and gradually work my way down. 

From what I've seen of your data so far I'd say it was still over-fitted since it always breaks down as it enters real-time.  That has to be clear, right?  This is simple algebra.  In linear regression the slope and y-intercept look great based on the original 50 x,y inputs.  But when more data is added then the y-distance from the original slope begins to deviate.

Though we use the same tools and have a good understanding of forex, our perspectives lead us in different paths.  You take the path of computing a strategy that performed well in the **past** and hope it will continue to perform well into the future.  I take the path of using minimal recent data to compute settings that work for **current** market conditions and hope today's market conditions will continue tomorrow.  Your approach relies on past market conditions.  My approach relies on current market conditions -- i.e. I don't care about the past.

Topic for a different thread:  Define "market conditions" and how they might affect EA performance...

You know -- we throw around the term "market conditions" but I've never seen a definition.  I think most traders believe it refers to the direction of price, but it doesn't.  It is much more complex.  I think it takes into account ADR, bar size, number of bars per swing, influence from other pairs, and more.  Furthermore, market conditons will vary from pair-to-pair.  I don't think market conditions can be defined or understood.  However, I do know that they affect EA performance.  So, from my perch 30000 feet high I'm not interested in the details of "market conditions" -- I'm only interested in computing an algorithm that works for today's "market conditions" and trade it tomorrow.  I also think that is the best we can do, and your excellent and graphical test results further confirm that for me.

By the way -- I hope you don't take my bluntness the wrong way.  In a post we have limited time and space to share complex ideas and it is too inefficient to worry whether someone will take something the wrong way -- so, I tend to err on the side of being direct.  And I understand we are paddling the same boat hoping to make it to shore...

RE: Market conditions -- lately they have been crazy.  Not only the war in Ukraine but also last week Japan propping-up the Yen (for the first time in 24 years) and this week UK doing the same for the pound.  There have been events this year that never occurred before -- and there are people who believe that using last year's data will make their algorithm trade smarter tomorrow?  I don't know...