It happens when a strategy trades continuously.

I'm trying to detect such cases by adjusting the Data Horizon in EA Studio to start testing on the same day as MetaTrader tester.

...

You may tray splitting your Portfolio in parts and testing them separately.

You can  change the Symbol Settings in the MetaTrader 5 Tester

Hello, After testing the same Expert Advisor on several MT5 platforms, I found that the quality of the backtest history was very low unlike the MT4 platform, this may be due to brokers because they can make changes in the data of the history they import into their demo account.

I tested the same EA over the same timeframes with the same settings on 3 different brokers and got vastly different results.

Yes, I always use the "live" account to export data from (into EAS), rather than the demo account, and then I'll use the "live" account to run any backtest from -despite then possibly running the EA in the demo account.   Your point is correct though, not all demo data is the same!

I tested the same EA over the same timeframes with the same settings on 3 different brokers and got vastly different results.

If you've found conditions where EAS backtesting and MT Strategy Tester results align (e.g. shorter data horizons) then I would go with that.  Maybe it has to do with spreads or maybe there is something else going on with the simulated OHLC data -- regardless, our time is better spent trading and using this amazing tool to create new and different portfolio EAs...

Completely agree, and thanks for spending some time looking into this with me.  I'll concentrate on closer timeframes for sure, and yes, it's an amazing tool for sure.

Thanks again Sleytus.

As for whether your explanation makes sense -- perhaps.  Right now I'm being forced to think harder than what I'm up to.  In MT4's Strategy Tester it was possible to specify the spread, but that feature seems to be missing in MT5 -- or, at least I haven't yet figured out how to set it.  I have no idea what spread MT5 is using.  My rule of thumb is to use 40 points in EAS.

My view from 30000 feet...
Strategy Tester is unreliable -- that is, sometimes it turns out to be accurate and others times not, you just never know which it will be.  So, I wouldn't beat my head over this.  If you've found conditions where EAS backtesting and MT Strategy Tester results align (e.g. shorter data horizons) then I would go with that.  Maybe it has to do with spreads or maybe there is something else going on with the simulated OHLC data -- regardless, our time is better spent trading and using this amazing tool to create new and different portfolio EAs...   

I think it may have something to do with differences in the spread.

I fixed the 'array out of range' error so I could test in Strategy Tester.  The results were still quite different between EAS and Strategy Tester -- with EAS slightly profitable and ST unprofitable.  But then I fiddled with AUDUSD's spread in EAS and changed it from the default of 10 points to 40 points and the overall perfermance of the portfolio EA became negative in EAS, as well.

Yes that error shows up on the un-edited version.  Once you change the size of the array it should work as you found.

So my spread with ICMarkets currently is on average 2 points. 

Maybe whats happened is that ICMarkets has recently brought in Zero spread...but back further, spreads existed...that might explain it.  That could be why the MT5 data, back in time, doesn't have a spread of zero (it might have 10-40 points) and EAS, because I over-rode the data setting with 2 points, based everything on that...

Hmmm.  Interesting...

That would/could also explain why more recent timeframe backtests work (3m, 9m, 12m) - because both platforms use the same spreads to generate/optimise/validate (i.e. 9 months in MT5 and in EAS match, because the broker data using the backtest matches what EAS used to build it).  Does that make sense?

I fixed the 'array out of range' error so I could test in Strategy Tester.  The results were still quite different between EAS and Strategy Tester -- with EAS slightly profitable and ST unprofitable.  But then I fiddled with AUDUSD's spread in EAS and changed it from the default of 10 points to 40 points and the overall perfermance of the portfolio EA became negative in EAS, as well.

2022.09.19 01:23:20.333    Core 1    2022.07.17 00:00:00   array out of range in 'Portfolio Expert AUDUSD#a M15 138 4 years.mq5' (1133,16)

This must be due to your editing.  Have you tried testing an unedited version of your portfolio EA?

I'm beginning to think that somehow I generated and optimised a bunch of strategies on bad data, which somehow passed all of the tests within EAS...   not sure how.  Maybe something went wrong with the data export from MT5/Broker, or something...not sure.

But, one lesson here, always test within MT5 to "confirm" things.

for it to run in MT5.

So, I just added some filters in EAS to get the number less than 100 strategies (got to 96) and re-ran it in MT5.
Still no joy.  Curve is entirely different.  This was on a data horizon from 3/Jan/22 to 19/Sept/22.  The only place the two platforms agreed with was the number of bars

And another thought -- what about testing on MT4's Strategy Tester...

Oh that’s interesting… hadn’t tried that…

Okay -- then I'm stumped, as well...

But you did say that data horizons of less than 9 months always seemed to work -- what about in this case?  What if you recalculated your current portfolio EA with a data horizon of 3 months and tested that on MT5?  Actually, if you are trading M15 then you could shorten that to 1 month.

Actually I did that try that too. I took the 138 Portfolio and reran it over the last 9 months - pretty much failed entirely in EAS - I was validating it only not optimising.

But you did say that data horizons of less than 9 months always seemed to work -- what about in this case?  What if you recalculated your current portfolio EA with a data horizon of 3 months and tested that on MT5?  Actually, if you are trading M15 then you could shorten that to 1 month.