1 (edited by timelleston 2022-09-27 06:45:27)

Topic: Definition of insanity...looking for some guidance

They say the definition of insanity is doing the same thing over and over again and expecting a different result. I'm getting there...

So, I'm asking forum members to try to point out what I must be doing wrong when generating a portfolio of strategies, because as soon as I put them into market, they start to lose money.

The Portfolios I use are all focused on AUDUSD, 15m, and generally have between 50-80 strategies in them.  Let me also add that I just love this software - it makes it all so quick to generate and test things.  But...

Here's my current workflow:
1) Download latest data on M15 AUDUSD from my broker, which generally has a Zero spread, although I set the spread manually to 2 (I've also tested 10).
2) Ensure the commission is set to 3.5, per lot, per side.
3) Set the data horizon to anywhere between Sept 2021 & June 2022 - I've used different start dates so that I can backtest properly and get a decent OOS sample as well. Shorter time horizons doesn't give enough time (IMHO) for decent OOS.
3) Set up Reactor with the following:
a) Strategy: Long and Short, Ignore or Reverse, Always use stop loss, fixed or trailing (35/250), take profit, may use (10/1000)
b) Generator: Net Balance, 20% OOS, Entry=4, Exit=2
c) Full Data Opt: Balance Line Stability, 20% OOS, Opt SL/TP
d) Normalisation: everything ticked, Net Balance, 20% OOS

Once it's then generated 200-300 strategies, I then filter the collection with:
a) Min Profit Factor: 2
b) Min Balance Stability: 85
c) Min Win/Loss: 0.65
d) SQN: 3

- I'll play around with those numbers initially to get a list of approx 70/80 acceptable strategies.

Then, taking a leaf from @sleytus approach, I look through each strategy to look at the slope of balance curve, to ensure it continues into the OOS as a similar slope as the IS data.  I'll also check that during the OOS, the balance curve doesn't "top out".  I'll remove those strategies that I don't like the look of.

Once I'm happy with them, I'll add them to the Portfolio, recheck the metrics for the portfolio, and then export it to MT5.  So it's all good up to this point.

I'll then open in MT5, and run a backtest to confirm its got similar results, which I usually do.

I'll then export the deals into a spreadsheet I have, and run a number of other tests across it to look at portfolio metrics from IS to OOS, looking at daily metrics, OOS vs IS.

Once again, if all checks out, and it usually does, I'll push the portfolio (of maybe 50-80 strategies) to my VPS and run it on a demo account for a while.  I have compared back tests from my VPS to my desktop and they are reasonably similar...so the issue isn't there.

But, this is generally when things start to fall over.  Typically what happens is the balance curve starts to flatten off pretty quickly (within a couple of days) and then deteriorates, and I'm trying to figure out what I'm doing wrong.  It's easy to look at the curve and pick the point where the OOS date actually ended as well, because from that point on, it starts to decline. 

Clearly my process is overfitting, but I'm not sure where.  I've tried using different data horizons, but I get similar results.  I've also tried running the data horizon to a date about 4 weeks back (so I might use 5 months of IS, then 1 month OOS, ending a month ago, which is about 20% OOS) and then running it in back test on MT5 to see what happens over the last month...and I get the same flattening of the curve to the point where it loses money.  I've also tried using different OOS percentages.

Things that I've checked:
1) make sure broker data is the same across Live and Demo, and that the spreads are the same
2) make sure that the broker data is loaded consistently into EAS
3) Tried different approaches to the Opt/Norm stage, using different "search best..." settings
4) Tried really short data horizons, but the OOS doesn't tend to generate enough results for me to be comfortable with them (only a few days in OOS)
5) Ensured my latency is good - around 70ms.
6) checked in MQL signals for slippage, doesn't appear to be anything worth worrying about

I'm curious how others are developing strategies and putting them into market, whereby the strategies last at least a few weeks before they need to change...  I fully understand that no strategy is perfect or will last forever, but by my thinking, if they've been generated on a reasonable length of time in sample (seen a few different market conditions) and tested on out of sample data (again a few different market conditions) (which is why I like the 12 months of data, 20/30% OOS), then they should at least last a little while, as they've seen a few conditions and been optimised for them...? 

I know the market conditions are always changing, but this is why we test to generate robust strategies...although clearly I'm doing something wrong.  Please help fix my insanity.

So I'm wondering if anyone has any pearls of wisdom that they'd be comfortable sharing with me - so I can get off the merry-go-round of insanity.

Many thanks.

2 (edited by sleytus 2022-09-27 08:06:03)

Re: Definition of insanity...looking for some guidance

Great topic for a post.  Not that it helps -- but rest assured you are not alone.  This is all part of the journey.  There is no single, correct recipe.  Your work flow sounds detailed and methodical -- which is good.  Your mind is also thinking like a detective -- also good.  And you've identified the issue as over curve-fitting -- which I would agree with.

I have a few comments now and depending on how this topic evolves I may add a few more.  Your rut sounds very similar to mine until I returned to Forexsb a couple of months ago.  Popov's software is a game changer and it allows me to do things I never could before -- in particular, to generate high quality portfolio EAs with the press of a button.  Why is that important to me?  I can now generate portfolio EAs in ***near real-time***.  Do you understand what that means?  The market is constantly changing and that is why there will always be a disconnect between an OHLC simulator and real-time trading.  Loading-up on historical data is a fool's errand because you can't train algebra -- it's not a database.  In the context of data analysis, optimization only makes sense when a data set is ***static*** -- that is, optimization algorithms find the best fit for static data.  But forex data is not static, so optimizing a best fit for historical data is doomed to fail -- the market condtions and data patterns are constantly changing.  And the proof is staring you in the face -- even though performance is poor you'll keep repeating the same steps.  We have all been conditioned to think that way.

In an ideal world the perfect generator would be able to generate new strategies in real-time that align with current market conditions.  And with the upcoming Express Generator we'll be able to approach that.  I like CJ / M15 with a max of 300 bars (a little less than 3 days).  I only use the generator -- no OOS and no optimization.  I'll let it run a few hours and take the top 50 strats out of about 200.  The uploaded data from my broker is only 15 minutes delayed.  Since I let the generation run for about 4 hours then by the time the portfolio EA begins trading it is only 4 hours behind current market conditions -- which is pretty damn good.  I repeat this every day.  The portfolio EA trades well for a day or so and then noticeably begins to taper off.  No problem -- press a button and create a new one based on current market conditions.

My invdividual strategies average about 10 trades per 300 bars (a little less than 3 days).  10 trades is not statistically significant.  However, when you accumulate a bunch of statistically insignificant measurements into one larger measurement (i.e. a portfolio of strategies) then the overall measurement does become statistically significant.  That's the power of averaging...

Re: Definition of insanity...looking for some guidance

sleytus wrote:

I only use the generator -- no OOS and no optimization.  I'll let it run a few hours and take the top 50 strats out of about 200.  The uploaded data from my broker is only 15 minutes delayed.  Since I let the generation run for about 4 hours then by the time the portfolio EA begins trading it is only 4 hours behind current market conditions -- which is pretty damn good.  I repeat this every day.  The portfolio EA trades well for a day or so and then noticeably begins to taper off.  No problem -- press a button and create a new one based on current market conditions.
...

Thanks Sleytus. Interesting approach as always from you smile. That does seem quite onerous though, doing that everyday. Although I do agree with your comment about the volume of historical data. However if you’re training on a very short period of time then it seems it’s optimised for that short period so it would make sense to me that it starts to go awry after a couple of days.

With a longer time frame I would expect it to last longer, albeit not as high-performing as it could be, because it’s, if you like, averaged more of the performance over the timeframe. Your approach would be highly optimised for a short period of time for sure.

While I’m a fan of OOS, I’m not a fan of walk forward though, because to me, that seems just like a bunch of OOS and the only version to use is the latest version over the most recent data - seems little point in looking at the first few sets of WF because they were never used in the real-time trading anyway.

I really like the portfolio approach because with that many strategies, you’re going to average out good/bad performers. If you pre-prune, as we do, the selection, theoretically you’ll have an edge.

In you approach, with launching a new portfolio every day (or on a high frequency at least) I assume you have a bunch of open trades each time that you have to either manually close, or let the EA close out at some point in the future, so you’d leave it running with zero as the max trades?  Am I correct in thinking then you’ll be having quite a few running portfolios, waiting for them to close out old positions?

Overfitting is obviously a problem that we’re all trying to avoid. I wholeheartedly agree Mr Popovs software allows us to generate and test in super-quick time, but if we’re using OOS or walk forward or multi-market testing to try to avoid/reduce overfitting, what else can we be doing to further mitigate it?

Otherwise we’re all on this merry go round…

Re: Definition of insanity...looking for some guidance

Hello timelleston...

This is great -- let's continue -- I'd like to address some of your points...

1. "if you’re training on a very short period of time then it seems it’s optimised for that short period so it would make sense to me that it starts to go awry after a couple of days."
Yes -- exactly -- it's curve-fitted for the most recent 2-3 days.  That's what I mean by ***near real-time***.  Real-time would be every tick.  In this case ***near*** real-time means the most recent 2-3 days.  Since I don't have a crystal ball, then I'll take that.  Though market conditions change, my experience is that probability is on my side that current market conditions will persist that long or longer.

2. "With a longer time frame I would expect it to last longer,"
No -- it's just the opposite.  And you have the proof -- it's the reason you started this topic.  The longer the data horizon, the more historical (and irrelevant) data is used to compute settings.  Since we trade **now**, in real-time, we don't want to pollute our input data with irrelevant data.  Again -- algebra can't be trained and irrelevant data only serves to dilute the importance of the most recent data when computing the best settings.

3. "That does seem quite onerous though, doing that everyday."
Not really -- how long does it take to press a button?  I've found that I need and enjoy being engaged -- trading a custom EA for a couple of weeks with zero intervention doesn't work for me since I'll then look for others ways to get into trouble.  Now I spend 15 minutes per day -- that includes pressing a button to start generation, returning 4 hours later to move the top 50 to a portfolio, press another button to generate source code, press another button to compile, drag and drop the compiled portfolio EA onto a CJ/M15 chart.  It takes me longer to type-out the work flow than to actually perform it.  And, again, it's enough to keep me engaged and involved.

4. "I assume you have a bunch of open trades each time that you have to either manually close, or let the EA close out at some point in the future, so you’d leave it running with zero as the max trades? "
You're a smart fellow -- yes, after a couple of days I set Max Open Trades to zero and let the trades close gracefully.  As the week progresses I add a new portfolio EA each day and gracefully retire aging ones.  If I were hand-coding custom EAs this would be extremely tedious and time-consuming -- but with Popov's software I'm just a button-press away from a new portfolio EA so it's no longer an issue.

Re: Definition of insanity...looking for some guidance

Ah wow that’s awesome. So you’ve basically accepted that overfitting can’t be avoided, and you’ve embraced it, and developed a methodology that enables you to leverage the overfitting for a short period of time.

Makes total sense. “If you can’t beat them, join them”. Love it.

And your thoughts about having more data not helping makes sense too; definitely experiencing that outcome I believe.

I’ve watched many videos on WF etc, and while OOS makes sense, I can see how your approach works - your OOS is actually in-market.

I read one of your posts about OOS and you were doing it, but I recall it was on 1H timeframe, and checking slopes etc before letting the strategy run … do you use OOS at all, such as for longer timeframes, or are you ok focusing purely on regenerating strategies near-real-time instead?

BTW - this is a great conversation, thank you.

Re: Definition of insanity...looking for some guidance

It’s interesting talking about algebra etc; as my background was a spot CAD interbank trader for Citibank for about 8 years, then as a founder of a data analytics company for 10+ years.

From a trading standpoint, I would be trading hundreds of millions of dollars a day, multi-million dollar positions each time. Technical trading was part of it, but also very mindful of market conditions. Banks move markets. Period. Retail traders get closed out easily/on purpose.

After trading, and from an analytics business standpoint, we used predictive modelling to help our clients market to individuals that were likely to buy something. You know, propensity modelling etc. The more data, the more accurate the models. We consumed data.

So, this is like a reverse universe, an alternate universe.

Lesser timeframe (like 3-5 days) has the propensity to be more accurate because the market has moved on so quickly - I get that. And it’s a short window of opportunity. And that makes sense too.

So, I liken you to bank trader - one that is not trend trading (no time length to really do that), more like scalping or short term swing trader?  And that’s how you’re using EAS to generate strategies and open positions…?

I read somewhere else that you're a manual trader … do you mean you let the EA open a trade and you manually close it when you believe the time is right? Or, do you let it run its course?

I went through a stage where I closed manually, but I found the EA was actually better at doing it so I forced myself to let it run its course…

7 (edited by timelleston 2022-09-27 13:04:57)

Re: Definition of insanity...looking for some guidance

It’s interesting talking about algebra etc; as my background was a spot CAD interbank trader for Citibank for about 8 years (late 80s early 90s), then as a founder of a data analytics company for 10+ years.

From a trading standpoint, I would be trading hundreds of millions of dollars a day, multi-million dollar positions each time. Technical trading was part of it, but also very mindful of market conditions. Banks move markets. Period. Retail traders get closed out easily/on purpose.

After trading, and from an analytics business standpoint, we used predictive modelling to help our clients market to individuals that were likely to buy something. You know, propensity modelling etc. The more data, the more accurate the models. We consumed data. Predictive modelling works well on non-fast-moving data (to your point).

So, this is like a reverse universe, an alternate universe.

Lesser timeframe (like 3-5 days) has the propensity to be more accurate because the market has moved on so quickly - I get that. And it’s a short window of opportunity. And that makes sense too.

So, I liken you to bank trader - one that is not trend trading (no time length to really do that), more like scalping or short term swing trader?  And that’s how you’re using EAS to generate strategies and open positions…?

I read somewhere else that you're a manual trader … do you mean you let the EA open a trade and you manually close it when you believe the time is right? Or, do you let it run its course?

I went through a stage where I closed manually, but I found the EA was actually better at doing it so I forced myself to let it run its course…

8 (edited by sleytus 2022-09-27 15:31:03)

Re: Definition of insanity...looking for some guidance

Thanks for sharing a bit about your background -- it helps create a context for your comments.  You bring a lot to the table and are more experienced in this field than me.  In a previous life I was a research biochemist (15+) and, subsequently, software engineer for troubleshooting wireless networks (30+).  I cut my teeth on FSB Pro, left for a few years, and returned a couple of months ago to EA Studio.


> "so you’ve basically accepted that overfitting can’t be avoided, and you’ve embraced it, and developed a methodology that enables you to leverage the overfitting for a short period of time."
Yes -- that's an accurate and concise way of summarizing in one sentence what took me 50 sentences to describe in my previous post...  smile


>"I read one of your posts about OOS and you were doing it, but I recall it was on 1H timeframe, and checking slopes etc before letting the strategy run … do you use OOS at all, such as for longer timeframes, or are you ok focusing purely on regenerating strategies near-real-time instead?"
I was using OOS on H1, but no more.  It continues to amaze me how feature-rich Popov's software is and OOS is an interesting area that few traders outside of Forexsb take advantage of or are even aware of.  But, for now, I'm focusing on near-real-time portfolio EAs.


>"It’s interesting talking about algebra etc"
It was on this forum about 5 years ago that Blaiserboy made a comment that "it's just algebra".  It was an "ah ha" moment for me.  Until then Forex was kind of a black box to me.  Since then I've coded many indicators and EAs and understand how things work under the covers -- and, as Blaiserboy stated, it really is just algebra.  In fact, it's "baby" algebra.


>"After trading, and from an analytics business standpoint, we used predictive modelling"
I'm not familiar with that type of modeling but I am with other types.  It has occurred to me recently that Forex data is different and doesn't necessarily lend itself to traditional data optimization techniques.  Almost all the data analysis I've done in the past assumed **static** input data sets.  But Forex data isn't static since market conditions are always changing.  I've now become less enamored of optimization.  The Generator suits me fine -- since only the best strategies ascend into the collection and I migrate only the top 50 into a portfolio, then that serves as a kind of optimization.


>"So, I liken you to bank trader - one that is not trend trading (no time length to really do that), more like scalping or short term swing trader?  And that’s how you’re using EAS to generate strategies and open positions…?"
I'll go with swing trading -- that's fair to say.  It's not that that was my intention -- rather, swing trading strategies is what the Generator is creating.  I've read elsewhere that swings occur approximately every 3-8 bars -- regardless of timeframe.  And from MT5 statistics I can see the average position holding time for the strategies in my portfolio EAs is about 4 bars.  So, that fits with swing trading.


>"I read somewhere else that you're a manual trader … do you mean you let the EA open a trade and you manually close it when you believe the time is right? Or, do you let it run its course?"
I refer to manual trading as "discretionary" trading -- and I'll answer your question with a "yes"... smile  Since I develop software then I've done all combinations.  I definitely prefer algorithmic trading because staring at a screen looking for trades doesn't suit me.  But I also like some form of engagement and total automation is too boring.  I like what EA Studio provides for me.  I trust the code and usually let trades run their course.  If it is the end of the week and I'm in the green and the DD (draw down) isn't too bad then I'll manually close so that it doesn't weigh on my mind over the weekend -- something like that...

Re: Definition of insanity...looking for some guidance

Thanks for sharing that too Sleytus, much appreciated.

It's an awesome piece of software - and I can appreciate the time/effort Popov has put into it.  In fact, I'm a bit awestruck that he's managed to do it all in the browser, at incredible speed.  I can't wait to get the Express Generator, but I can't find a link to it anywhere.  I've been following some of the forum discussions there.

>In fact, it's "baby" algebra
Yes, at the end of the day, it's a bunch of deterministic rules that decide whether to trade or not.  (Without wishing to simplify the complexity in the generation of those rules). 

>static data sets
Yes I would totally agree - when doing modelling, scoring, etc, it's on static data...or at least data sets that are generally not high frequency.  Bank anti-fraud-modelling is possibly the highest frequency, but certainly nothing to the same frequency as forex data.  So it's not surprising that it loses its optimised accuracy after a few additional bits of data, not to mention hundreds of bits (bars).

>swing trading strategies is what the Generator is creating
Yes I was thinking it was doing that, but, optimising it was maybe masking the effect of it over a longer timeframe.  I was noticing that many positions were opened in seemingly the wrong direction - it was shorting a rising market, and vice versa, and I'd sit there and wonder why, when other indicators are showing the short-term trend wasn't over yet. 

I started looking at MyFXBook data analysis on the trades taken and I noticed that quite a few entered the market too soon.  So I started then looking for additional entry indicators, but haven't managed to crack that nut yet.     

Correlations between MAE and MFE in some cases were negative, which indicates its HODL until it comes back into favour.  Average position hold time is definitely short, but in some cases really long (comparative).  They may come back into profit, but in the case of AUDUSD, it's generally a trending (downward) market, so longs are staying offside for a long time - and more likely to hit SL.  So that all makes sense now. 

I wasn't sure whether it was trying to catch early trend moves or not.  I'd looked at the strategy report, and the balance chart, and seen in some cases that it was much better at either long or short positions, not both - so I started experimenting with generating only Long or Short strategies... juries out on that one at the moment.

Understanding its generating swing strategies makes it clearer as to whats happening.

And I like the really-short term/near real time approach you've taken.  I think I might give that a go.

I just fired up a new generator and I had got some fairly (what I thought) decent acceptance criteria in it... stability 85, win/loss 0.6, etc.  Removing those generates a large amount of strategies quickly, and then I can filter afterwards.

Ive watched many videos on OOS and Walk Forward.  I'm a believer in OOS, but to be honest, I don't understand why WF works any better.  To me, it's breaking down the overall period into smaller samples and optimising what would have happened in that timeframe, and then repeating it to a timeframe that is still not in market.  I watched a video where they were doing WF and then they got to the end of the period, and it was basically ready to go live into market, and they said that there should next be a pre-live optimisation period again - taking the next would-be In Sample period and optimising it, and calling it the pre-live optimised period.  I'm thinking...why bother with WF, if all you're actually doing is optimising the last segment/date/period and launching with that...

Anyway, enough rambling, thanks very much for your input Sleytus, much appreciated - I'll try a different approach and maybe some other pairs too.

10 (edited by sleytus 2022-09-28 05:42:51)

Re: Definition of insanity...looking for some guidance

With regards to swing trading -- I don't think the Generator has any bias to a particular trading pattern, be it swing, trend, ranging, breakout, reversal, whatever.  However, when the data horizon is short (e.g. 300 bars) then the only patterns visible to the generation software are the short swings.  When the data horizon is large then the swings are masked because they get hidden behind the gobs of historical data the generation software has to deal with and take into account during its optimization.

The disconnect between Backtesting / Strategy Tester and real-time trading can be attributed to the following reasons (and probably more):

1. OHLC bar simulation is not the same as streaming ticks -- it's like comparing apples to oranges.

2. Optimizing leads to computing the best settings for historical market conditions that never existed and will never occur in the future.  Market conditions are reflected by the most ***recent*** bars.  A data horizon of 1 year that includes 1000's of bars represents an abstract market condition that has never existed.

3. Large data horizons mask swings -- whereas the simple algebraic rules (used by our strategies) lend themselves best for detecting swings.  Large data horizons include trends, ranges, breakouts, reversals, swings and the optimization software has to deal with all of those -- and there is no way simple algebraic rules can generate a strategy optimal for all situations.  However, when you use a short data horizon the generator can focus on computing the best settings for swing trades.


EDIT: Come to think of it, all there are are swings.  Swings are the "building blocks" of the other data patterns.  Consider what looks like a trend on an MT chart and then look closer -- you'll see that the trend is actually made up of swings...

Re: Definition of insanity...looking for some guidance

> I don't think the Generator has any bias to a particular trading pattern...However, when the data horizon is short (e.g. 300 bars) then the only patterns visible to the generation software are the short swings. When the data horizon is large then the swings are masked ...

Totally agree with you there in terms of the masking of swings due to massive amounts of random noise.

> OHLC bar simulation is not the same as streaming ticks -- it's like comparing apples to oranges.

Yes indeed - I always backtest in MT5 using the setting "Every tick based on real ticks".  You get more realistic price on a trade, whether it be open, stop loss, take profit or close.  The OHLC method just uses your SL or TP setting on the individual EA, which doesn't reflect true market conditions.

> Large data horizons mask swings
Yes (an ah-ha moment) I hadn't thought about it like that before.

> Swings are the "building blocks" of the other data patterns.
Put that on a T-Shirt smile

So, I'm now running a test on a short term generation - I actually used 1000 bars of 15m - basically 2 weeks - and I might narrow that down to 500 bars depending on the results.  I didn't optimise though, but I did normalise it.  And I used very tight and fixed stop-loss and take profit - much much lower than I would normally.  In the mindset of many small wins aggregates up quickly.  Not really worried that I'm exiting too soon possibly.  I also had an acceptance rule of Win/Loss of 60% and based my SL and TP accordingly. 

It quickly generated over 700 strategies, and I took the top 50 based on r-squared.  It had some interesting results in the MAE/MFE correlation too...first time I've seen a slightly positive correlation between the two.

Pushed it to the demo account and now will watch it for the next couple of days to see if/when it starts to decay in performance.  I'll try that a few times and then I'll be able to hopefully gauge when to re-generate strategies again - and then have the confidence to put them into my live account.

By the sounds of it, Express Generator will be really suited to this methodology, automating much of the that process, as we can script the timing and generation of strategies, filtering into portfolios, and then push them to MT5.  Can't wait to get my hands on it smile

Re: Definition of insanity...looking for some guidance

I'd be interested to learn more as your test results come in.  I'm always fiddling and trying new things.  It would be nice to continue the conversation and perhaps others could join in...  smile

Re: Definition of insanity...looking for some guidance

More than happy to share the results/findings, and certainly anyone else feel free to contribute too. It’s an interesting topic - kind of bucks the best-practices.

Re: Definition of insanity...looking for some guidance

Best practices?  I must have slept through that class...   smile

timelleston wrote:

More than happy to share the results/findings, and certainly anyone else feel free to contribute too. It’s an interesting topic - kind of bucks the best-practices.

15

Re: Definition of insanity...looking for some guidance

Hi timelleston
I tried but not happy with results. I will leave it for 2 days.
Do you use fixed stop or profit?

16 (edited by timelleston 2022-09-29 07:38:20)

Re: Definition of insanity...looking for some guidance

GD wrote:

Hi timelleston
I tried but not happy with results. I will leave it for 2 days.
Do you use fixed stop or profit?

Hi GD - I used both fixed stop and fixed profit, but I normalised over the period, which I set to reduce the SL and TP.

GD - What are your thoughts on the topic as a whole - OOS or no OOS?

Re: Definition of insanity...looking for some guidance

Ok, I've had some very curious results.

I ran Reactor 5 times, using 1000 bars, and shifted the start date by 1 week each time, starting from 5 Aug, through to 2 Sept.  So I ended up with 5 portfolios, separately generated, over 5 different two-week periods.

I let the Reactor for 60 minutes each time.

I had Net Balance and a 20% OOS set for Generator.
I had Normaliser set for Net Balance and 20% OOS.

I then took the top 50 r-squared and moved to a Portfolio, noted all the results for each seperate portfolio.

I downloaded each portfolio and ran in backtester on MT5, using ticks based on real ticks.

I started each back test on the same date the EAS started it, and let it run to the last day of data in MT5 (today).

I then exported the graphs and the deals, loaded them into Excel and looked at the daily balance curve for each.

My hypothesis was that the portfolio would survive for at least a couple of days after the end of the OOS period before it started to decline... but...

None of them did.  On the 11th day, without fail, as soon as the OOS period ended, the portfolio performance began to decline significantly.

See chart below - which is time-shifted to align all of the start/end/OOS dates.  You can see where the OOS date ended.

I thought, ok, so perhaps the strategies lasted the 2 days through OOS and then declined.  So I regenerated a few with no OOS - just in sample.  Same result, on the 11th day they started to decline.

It's almost like the generator, generated the strategies on the whole in-sample time period (1000 bars) and didn't take into account 20% of OOS.

Does anyone know if thats whats happening?  Is there a cut-off period internally where it doesn't use OOS anymore?

Post's attachments

20% OOS and Sample.png 154.13 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

18 (edited by timelleston 2022-09-29 04:54:49)

Re: Definition of insanity...looking for some guidance

Here's the results from the Generator, In Sample only (no OOS), for the same periods - likely different strategies though - it's the curve we're looking at though.

As you can see, the balance lines immediately start to decay at the end of the period they were generated for.  This I would expect...but not the previous one if it was using OOS.

One could argue that the OOS is 2 days where it starts to decline.  If that was the case, then generating these without the OOS over the same period with the same settings, should in theory, allow them to survive at similar performance rates for at least 2 days.

If the strategies were generated on 80% of the data (the In Sample) and then "checked" against the 20%, then I would expect the In Sample strategies over the 1000 bars, with no OOS, to demonstrate similar performance - lasting at least 2 days.

Post's attachments

Screen Shot 2022-09-29 at 1.41.03 pm.png 148.49 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

19 (edited by sleytus 2022-09-29 05:15:11)

Re: Definition of insanity...looking for some guidance

>"None of them did.  On the 11th day, without fail, as soon as the OOS period ended, the portfolio performance began to decline significantly."

Yep -- this is to be expected.  In EA Studio the OOS period represents the most recent data, yet it is not included in the computations.  Why would you expect settings computed using past market conditions to be compatible with current market conditions?

This is the "insanity" you referred to in your initial post -- we've all succumbed to it.  Algebra can not be trained.  Performance looks good up until OOS (because that is the IS data that was used to compute the best settings).  And, then, as you begin to enter OOS it begins to taper off -- which would be expected.  How quickly it tapers off depends on how similar market conditions in OOS compare with IS.  If market conditions in OOS are similar to IS then your strat looks good and you are fooled into thinking your strat is a winner and is not overly curve-fitted.  But that is not necessarily the case if OOS market conditions are simply a continuation of IS market conditions -- in which case your strat will fail as soon as market conditions deviate from your IS section.

Also -- what time frame are you using?


EDIT: Your tests are great and the results show exactly what is happening -- and the interpretation seems obvious (at least to me).  In a previous post you mentioned "best practices" -- to me that's a problem.  There are no "best practices".  In retail forex trading there are only bad habits.  Sorry for being blunt.  It's not important that you agree with me, but your test results can be explained if you're willing to contemplate that "algebra can not be trained" -- it can only be fitted to the input data points...

20 (edited by timelleston 2022-09-29 07:37:05)

Re: Definition of insanity...looking for some guidance

sleytus wrote:

>Performance looks good up until OOS (because that is the IS data that was used to compute the best settings).

Agreed. 

sleytus wrote:

>And, then, as you begin to enter OOS it begins to taper off -- which would be expected.  How quickly it tapers off depends on how similar market conditions in OOS compare with IS.

Agreed, thats completely what I'd expect...except that didn't happen either - none of the cases began to taper off during OOS.  They all went immediately after OOS.

sleytus wrote:

>But that is not necessarily the case if OOS market conditions are simply a continuation of IS market conditions -- in which case your strat will fail as soon as market conditions deviate from your IS section.

Agreed - logically yes, but that suggests that market conditions just happened to change, each and every time, coinciding with the end of the OOS period.   5 times in a row. Consistently.

The curve during OOS was similar to the IS curve.  That suggests that the market conditions were exactly the same during each of the 2 week periods.  Over 5 different periods.  And then to suddenly change on the same day each time?  Hmmm...doesn't seem plausible to me.

Without the line I put on the chart, you'd not know where OOS started.  Market conditions have definitely changed yes, which is why (we actually both agree) OOS shouldn't be exactly the same as IS.  But it hasn't happened that way.

sleytus wrote:

>Also -- what time frame are you using?

AudUsd 15m, 1000 bars, so approx 10 days.  OOS was approx 2 days.

My point is that it made no difference using OOS vs not using OOS.  The decay happens almost instantly at the end of the period irrespective of OOS or not.  That makes no sense.

If the strategy is generated on 80% of data (800 bars), and then 20% is used to validate the strategy (OOS, 200 bars), and it performs as well during that OOS period, that strategy "survived" for two days during the OOS period.  It had the same curve, same performance - but on unseen data.  To have such a massive decay, instantaneous almost, at the end of OOS, seems unlikely.  Even if 50% of them started to decay immediately during OOS, you'd see that in the charts, as they'd start to taper off...but they don't.  They perform just as well as they did during IS.

Everything you'd said is correct and I agree with it in principal...however, the data is suggesting something else is going on here.

I'm going to re-run some data, with 20% OOS, but move the date forward by 2 days.  I know, descending into madness...tie a rope on to pull me out later - hahaha.

21 (edited by timelleston 2022-09-29 08:18:58)

Re: Definition of insanity...looking for some guidance

Ok, so re-ran a generation, with 20% OOS on 1000 bars, from 30th Aug to 13 Sept.

Same process as above - plotted it in Excel.

Identical results as before.

The OOS slope matches the IS slope.  As soon as OOS finishes, it crashes.

I'm sorry, but that cannot be market conditions.  It was 2 days apart.  That would mean that the market conditions has coincidentally significantly changed every time at the end of OOS. Thats not likely.

If the strategy is generated on 80% data, and the generator never sees the 20% OOS data, there's no way these can continuously repeat themselves in their results i.e. OOS slope consistent until end of OOS, then it caves in. 

And, I re-ran this on 300 bars, and on 500 bars, it did exactly the same.  OOS fine, end of OOS performance crashed.  I'm sorry, but that doesn't make sense.  It's too consistent.


Mr Popov, can you please confirm that the Generator and Reactor do not see/use OOS data in the generation of strategies, when you select an OOS percentage?

Re: Definition of insanity...looking for some guidance

Okay -- I agree I didn't give a lot of thought to your results and may have spoken too soon.  I did a quick view from 30000 feet and it appeared consistent with what I've seen and experienced, as well.

Your point is that regardless of the data horizon and OOS, performance tapers-off at around the same distance after IS.  Is that right?

The point at which it tapers off isn't so important to me -- since that would depend on how similar or different market conditions were between IS and OOS.  However, what is really clear is that IS looks good and shortly later things go south.  When I was using OOS I saw many strategies that seemed to survive OOS, but I also saw that their OOS statistics were not as good as IS stats.  That is, it was clear things begin to decay once you leave IS but it wasn't necessarily fatal.  There were other strategies that were clearly over-fitted and failed OOS.

I'm not prepared to get too down in the details and mud with you -- not because I think you are wrong, but because I don't think it will lead anywhere.  It is too complex and there are too many interdenpencies for us to understand it all.  My inclination is to start from 30000 feet and gradually work my way down. 

From what I've seen of your data so far I'd say it was still over-fitted since it always breaks down as it enters real-time.  That has to be clear, right?  This is simple algebra.  In linear regression the slope and y-intercept look great based on the original 50 x,y inputs.  But when more data is added then the y-distance from the original slope begins to deviate.

Though we use the same tools and have a good understanding of forex, our perspectives lead us in different paths.  You take the path of computing a strategy that performed well in the **past** and hope it will continue to perform well into the future.  I take the path of using minimal recent data to compute settings that work for **current** market conditions and hope today's market conditions will continue tomorrow.  Your approach relies on past market conditions.  My approach relies on current market conditions -- i.e. I don't care about the past.

Topic for a different thread:  Define "market conditions" and how they might affect EA performance...

You know -- we throw around the term "market conditions" but I've never seen a definition.  I think most traders believe it refers to the direction of price, but it doesn't.  It is much more complex.  I think it takes into account ADR, bar size, number of bars per swing, influence from other pairs, and more.  Furthermore, market conditons will vary from pair-to-pair.  I don't think market conditions can be defined or understood.  However, I do know that they affect EA performance.  So, from my perch 30000 feet high I'm not interested in the details of "market conditions" -- I'm only interested in computing an algorithm that works for today's "market conditions" and trade it tomorrow.  I also think that is the best we can do, and your excellent and graphical test results further confirm that for me.

By the way -- I hope you don't take my bluntness the wrong way.  In a post we have limited time and space to share complex ideas and it is too inefficient to worry whether someone will take something the wrong way -- so, I tend to err on the side of being direct.  And I understand we are paddling the same boat hoping to make it to shore...

Re: Definition of insanity...looking for some guidance

Here's a test -- I hope I'm not burdening you.  It's been many years since I last used Excel...

Repeat your tests using CADJPY / M15 with a maximum of 300 bars.  300 bars is the fewest bars allowed by EA Studio.  No Reactor, no optimization, no OOS -- just the Generator.  Let it run for a couple of hours and sort on SQN, move the top 50 strats to a portfolio...

Re: Definition of insanity...looking for some guidance

RE: Market conditions -- lately they have been crazy.  Not only the war in Ukraine but also last week Japan propping-up the Yen (for the first time in 24 years) and this week UK doing the same for the pound.  There have been events this year that never occurred before -- and there are people who believe that using last year's data will make their algorithm trade smarter tomorrow?  I don't know...

25 (edited by timelleston 2022-09-29 10:12:30)

Re: Definition of insanity...looking for some guidance

sleytus wrote:

From what I've seen of your data so far I'd say it was still over-fitted since it always breaks down as it enters real-time.  That has to be clear, right?

You've nailed my point entirely..."real time" thats the point.  These strategies were generated with an OOS.  Thats the equivalent of "real time" data back then. Thats data that the algebra should never have seen, or based it calculation on. Thats when it should start to break down.  But it didn't.  And that's my point.  It breaks down at the end of the OOS - not the beginning, or during OOS. 

And thats what leads me to question whether it's possible, that the OOS data period is somehow being included in the strategy generation as "In Sample", not "Out of Sample".  If it was truly OOS, the results would break down earlier (during the OOS or "real-time"), not specifically when the OOS finishes.  It consistently happens, more than probability would suggest. 100% of the time.

sleytus wrote:

By the way -- I hope you don't take my bluntness the wrong way.  In a post we have limited time and space to share complex ideas and it is too inefficient to worry whether someone will take something the wrong way -- so, I tend to err on the side of being direct.  And I understand we are paddling the same boat hoping to make it to shore...

Of course not, and the feeling is mutual.  It's a great discussion.  I'm just happy to have someone to discuss it with smile