forex software

Create and Test Forex Strategies

forex software

Skip to forum content

Forex Forum

Forex Software, Forex Strategies, Expert Advisors Generator

You are not logged in. Please login or register.


Forex Forum → Portfolio Expert → Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Pages 1

You must login or register to post a reply

RSS topic feed

Posts: 25

1 (edited by timelleston 2022-09-18 23:44:10)

Topic: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

First time this has happened to me and wanted to check in with others...

Loaded 100,000 bars of M15 data across 5 currency pairs - primary was AUDUSD, with a spread of 2, and a commission of 3.5 per lot per side. ICMarkets has a spread of 0, but use 2 generally.

Started a Validator on previously generated strategies (about 1300 loaded) and had dates set from 2018/Sept/10 to 2022/Sept/16.

Optimisation was 20% OOS, balance line stability.
Normalizer was 20% OOS, net balance.
Multi market across 5 pairs, with 4 out of 5 to pass.

It validated 138 strategies, with a fairly decent curve over the 4 years, $301,201 profit (0.1 lots), stability of 93.3, 16469 trades, 0.63 win loss, 0.14 sharpe etc.

I then exported to MT5, and re-ran the back test over the same period...with lots of 0.01, but it totally failed.  Negative growth, lost about $500, over 22500 trades..almost like running a totally different portfolio.  I've re-confirmed all settings in MT5 have matched what I normally do.

First time I've ever seen such a monumental difference.

I've reloaded the currency pair data (re-exported and re-imported), re-ran the validator, but it made no difference to the results in MT5.

When I re-run validator on a shorter data horizon, virtually all of the same strategies fail in EAS.

When I've done this in the past, the portfolio has closely matched the MT5 backtest, so this is a first...

Anyway, wanted to see if anyone had any ideas (other than run it on shorter timeframes smile)

Cheers
Tim

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Interestingly as well, the collection said that each of the strategies ended up long at the end of the test - never seen that either...seems unlikely to me.

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Just noticed that in MT5, the backtest history quality was very low ... its likely thats the cause.  Trying to fix that, and then I'll update this post again.

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Ok, fixed the quality issue - reloaded the data to EAS, and reran, then reran in MT5 - same result...
Perplexed...anyone else had a similar issue?

5 (edited by sleytus 2022-09-18 16:44:45)

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

A couple of thoughts...

1. Are you using your broker's data in EAS?  If not, then the cause could be due to different data.  Small differences over 22500 trades can add up.

2. 22500 trades is a lot and way more than you need for accurate analysis.  On M15 the P/L value of each trade is small and, so, spread can make a difference.  22500 trades where the spread differs between EAS and MT5 could easily account for $500.  I would suggest repeating your test on a higher timeframe (e.g. H1) to see if the problem persists.  And more than 100 trades is overkill -- you can't "train" algebra by feeding it more historical data.  More historical data only serves to skew optimal settings toward historical data and away from the most recent data.

timelleston wrote:

Ok, fixed the quality issue - reloaded the data to EAS, and reran, then reran in MT5 - same result...
Perplexed...anyone else had a similar issue?

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Another thing -- in MT5 look at the timestamps of the actual trades.  Even though you specified the same data horizon in Strategy Tester as you used in EAS, I seriously doubt that Strategy Tester loaded 100,000 OHLC bars -- that would be a broker limitation.  Under the 'Backtest' tab right-click and select 'Orders' -- you can then verify whether the trades go back to the data horizon you specified.

timelleston wrote:

Ok, fixed the quality issue - reloaded the data to EAS, and reran, then reran in MT5 - same result...
Perplexed...anyone else had a similar issue?

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

sleytus wrote:

1. Are you using your broker's data in EAS?  If not, then the cause could be due to different data.  Small differences over 22500 trades can add up.

Thanks Sleytus.  Yes I'm using my broker data - same process as I've always done.  Extracted from MT5, loaded into EAS.  Checked the spread (2) and changed the commission values.

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

sleytus wrote:

Another thing -- in MT5 look at the timestamps of the actual trades.  Even though you specified the same data horizon in Strategy Tester as you used in EAS, I seriously doubt that Strategy Tester loaded 100,000 OHLC bars -- that would be a broker limitation.  Under the 'Backtest' tab right-click and select 'Orders' -- you can then verify whether the trades go back to the data horizon you specified.

Looking at the orders, first order was 12th Sept 2018.  In the backtest report, it says bars 100,079, history quality 100%, ticks 67,438,337.

9 (edited by timelleston 2022-09-18 23:40:09)

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

sleytus wrote:

2. 22500 trades is a lot and way more than you need for accurate analysis.  On M15 the P/L value of each trade is small and, so, spread can make a difference.  22500 trades where the spread differs between EAS and MT5 could easily account for $500.  I would suggest repeating your test on a higher timeframe (e.g. H1) to see if the problem persists.  And more than 100 trades is overkill -- you can't "train" algebra by feeding it more historical data.  More historical data only serves to skew optimal settings toward historical data and away from the most recent data.

$500 yes for sure, but, not $300,000 wink  Also, bear in mind that this portfolio is made up from 130 strategies, so expecting a lot of trades in total; each underlying strategy only produces on average 500 trades over the 4 year period.

The two equity curves are attached below.  And yes, totally agree with you re your last comment about length of backtest.

I've previously used 12 months of data in both EAS and MT5, and it works perfectly - aligns mostly and the trades are pretty similar.  This though, is completely different - like I'm using a different pair altogether haha.

10 (edited by timelleston 2022-09-18 23:21:40)

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Equity curve:

Post's attachments

Screen Shot 2022-09-19 at 8.14.42 am.png 954.86 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Yes -- I agree -- the screenshot looks odd...

I have seen strange things with Strategy Tester -- both MT4 and MT5 -- similar to this but not necessarily associated with EAS.  For example, after running many different simulations one day on different pairs I'll revisit them the next day and get completely different results -- as you say, it looks like a different pair.

Strategy Tester caches a lot of settings under <DataFolder>\Tester.  What I've done in the past is clean things up by:
1. Deleting the 'Tester' folder.
2. Exiting and then relaunching Strategy Tester.
      or
Test the portfolio EA on a virgin installation of MT5 from a different broker -- it's okay that the data is different.

12

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Hi
Did you check Number of Trades taken by Portofolio EA in Strategy Tester?
Is it the same?

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

GD wrote:

Hi
Did you check Number of Trades taken by Portofolio EA in Strategy Tester?
Is it the same?

Hi - no its not the same...approx 16400 in EAS, and approx 22400 in MT5.

14 (edited by timelleston 2022-09-19 08:30:14)

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

sleytus wrote:

Yes -- I agree -- the screenshot looks odd...

Strategy Tester caches a lot of settings under <DataFolder>\Tester.  What I've done in the past is clean things up by:
1. Deleting the 'Tester' folder.
2. Exiting and then relaunching Strategy Tester.
      or
Test the portfolio EA on a virgin installation of MT5 from a different broker -- it's okay that the data is different.

#1 & #2 - tried deleting all the cache files for all brokers - restarted MT5, reran but had same result for ICMarkets.

Logged into my other broker (Markets.com) and tried on another account - similar results.  29,500 trades, lost about $2200 over the 4 years.  Different spreads, swaps and commissions, but it certainly wasn't $300k in profit.

I then tried uploading Markets.com data into EAS - and re-calcing the portfolio using that data as a test/guide.  See below - it still has a positive curve, much different to MT5.  Obviously expecting differences here due to different spreads, comms, swaps etc.  But still very different to MT5.

I also tried testing on another instance of MT5 I have running on a VPS - mostly got the same results.

When I run other portfolios of strategies (up to around 100 in each portfolio) in MT5 over shorter timeframes (say 9 months) the results have always been failry similar between EAS and MT5.  So, still stumped sad

Post's attachments

Screen Shot 2022-09-19 at 2.01.36 pm.png 519.72 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Okay -- then I'm stumped, as well...

But you did say that data horizons of less than 9 months always seemed to work -- what about in this case?  What if you recalculated your current portfolio EA with a data horizon of 3 months and tested that on MT5?  Actually, if you are trading M15 then you could shorten that to 1 month.


timelleston wrote:
sleytus wrote:

Yes -- I agree -- the screenshot looks odd...

Strategy Tester caches a lot of settings under <DataFolder>\Tester.  What I've done in the past is clean things up by:
1. Deleting the 'Tester' folder.
2. Exiting and then relaunching Strategy Tester.
      or
Test the portfolio EA on a virgin installation of MT5 from a different broker -- it's okay that the data is different.

#1 & #2 - tried deleting all the cache files for all brokers - restarted MT5, reran but had same result for ICMarkets.

Logged into my other broker (Markets.com) and tried on another account - similar results.  29,500 trades, lost about $2200 over the 4 years.  Different spreads, swaps and commissions, but it certainly wasn't $300k in profit.

I then tried uploading Markets.com data into EAS - and re-calcing the portfolio using that data as a test/guide.  See below - it still has a positive curve, much different to MT5.  Obviously expecting differences here due to different spreads, comms, swaps etc.  But still very different to MT5.

When I run other portfolios of strategies (up to around 100 in each portfolio) in MT5 over shorter timeframes (say 9 months) the results have always been failry similar between EAS and MT5.  So, still stumped sad

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

And another thought -- what about testing on MT4's Strategy Tester...

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

sleytus wrote:

Okay -- then I'm stumped, as well...

But you did say that data horizons of less than 9 months always seemed to work -- what about in this case?  What if you recalculated your current portfolio EA with a data horizon of 3 months and tested that on MT5?  Actually, if you are trading M15 then you could shorten that to 1 month.

Actually I did that try that too. I took the 138 Portfolio and reran it over the last 9 months - pretty much failed entirely in EAS - I was validating it only not optimising.

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

sleytus wrote:

And another thought -- what about testing on MT4's Strategy Tester...

Oh that’s interesting… hadn’t tried that…

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

So I thought it might also be something to do with being above 100 strategies in the portfolio - because that appears to not work anyway, and I've had to change line 56:
from...
int indHandlers[100][12][2];
to...
int indHandlers[138][12][2];

for it to run in MT5.

So, I just added some filters in EAS to get the number less than 100 strategies (got to 96) and re-ran it in MT5.
Still no joy.  Curve is entirely different.  This was on a data horizon from 3/Jan/22 to 19/Sept/22.  The only place the two platforms agreed with was the number of bars smile

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Well, I re-validated the 138 portfolio, on a 3 month timeframe, using re-opt, re-norm, and re-multi-market, and only 1 passed the full re-validation...

I'm beginning to think that somehow I generated and optimised a bunch of strategies on bad data, which somehow passed all of the tests within EAS...   not sure how.  Maybe something went wrong with the data export from MT5/Broker, or something...not sure.

But, one lesson here, always test within MT5 to "confirm" things.

Post's attachments

Portfolio Expert AUDUSD#a M15 138 4 years.mq5.zip 21.49 kb, 2 downloads since 2022-09-19 

You don't have the permssions to download the attachments of this post.

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

I tried testing it -- got an 'array out of range' error.

2022.09.19 01:23:20.333    Core 1    2022.07.17 00:00:00   array out of range in 'Portfolio Expert AUDUSD#a M15 138 4 years.mq5' (1133,16)

This must be due to your editing.  Have you tried testing an unedited version of your portfolio EA?



timelleston wrote:

Well, I re-validated the 138 portfolio, on a 3 month timeframe, using re-opt, re-norm, and re-multi-market, and only 1 passed the full re-validation...

I'm beginning to think that somehow I generated and optimised a bunch of strategies on bad data, which somehow passed all of the tests within EAS...   not sure how.  Maybe something went wrong with the data export from MT5/Broker, or something...not sure.

But, one lesson here, always test within MT5 to "confirm" things.

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

I think it may have something to do with differences in the spread.

I fixed the 'array out of range' error so I could test in Strategy Tester.  The results were still quite different between EAS and Strategy Tester -- with EAS slightly profitable and ST unprofitable.  But then I fiddled with AUDUSD's spread in EAS and changed it from the default of 10 points to 40 points and the overall perfermance of the portfolio EA became negative in EAS, as well.

23 (edited by timelleston 2022-09-19 09:54:10)

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

sleytus wrote:

I think it may have something to do with differences in the spread.

I fixed the 'array out of range' error so I could test in Strategy Tester.  The results were still quite different between EAS and Strategy Tester -- with EAS slightly profitable and ST unprofitable.  But then I fiddled with AUDUSD's spread in EAS and changed it from the default of 10 points to 40 points and the overall perfermance of the portfolio EA became negative in EAS, as well.

Yes that error shows up on the un-edited version.  Once you change the size of the array it should work as you found.

So my spread with ICMarkets currently is on average 2 points. 

Maybe whats happened is that ICMarkets has recently brought in Zero spread...but back further, spreads existed...that might explain it.  That could be why the MT5 data, back in time, doesn't have a spread of zero (it might have 10-40 points) and EAS, because I over-rode the data setting with 2 points, based everything on that...

Hmmm.  Interesting...

That would/could also explain why more recent timeframe backtests work (3m, 9m, 12m) - because both platforms use the same spreads to generate/optimise/validate (i.e. 9 months in MT5 and in EAS match, because the broker data using the backtest matches what EAS used to build it).  Does that make sense?

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

For EAS I only use the Premium Data from DukasCopy.  I trade prop firm accounts and I'm disinclined to complicate things by having to remember to repeatedly import data from a variety of different brokers.  Though broker data differs, I figure if my strategies are that sensitive then they will likely fail in real-time.  So, before trading them I will do a sanity check using MT4 / MT5 Strategy Tester for each broker I'm about to trade with.  So far the Premium Data has been good enough for my purposes.

As for whether your explanation makes sense -- perhaps.  Right now I'm being forced to think harder than what I'm up to.  In MT4's Strategy Tester it was possible to specify the spread, but that feature seems to be missing in MT5 -- or, at least I haven't yet figured out how to set it.  I have no idea what spread MT5 is using.  My rule of thumb is to use 40 points in EAS.

My view from 30000 feet...
Strategy Tester is unreliable -- that is, sometimes it turns out to be accurate and others times not, you just never know which it will be.  So, I wouldn't beat my head over this.  If you've found conditions where EAS backtesting and MT Strategy Tester results align (e.g. shorter data horizons) then I would go with that.  Maybe it has to do with spreads or maybe there is something else going on with the simulated OHLC data -- regardless, our time is better spent trading and using this amazing tool to create new and different portfolio EAs...   





timelleston wrote:
sleytus wrote:

I think it may have something to do with differences in the spread.

I fixed the 'array out of range' error so I could test in Strategy Tester.  The results were still quite different between EAS and Strategy Tester -- with EAS slightly profitable and ST unprofitable.  But then I fiddled with AUDUSD's spread in EAS and changed it from the default of 10 points to 40 points and the overall perfermance of the portfolio EA became negative in EAS, as well.

Yes that error shows up on the un-edited version.  Once you change the size of the array it should work as you found.

So my spread with ICMarkets currently is on average 2 points. 

Maybe whats happened is that ICMarkets has recently brought in Zero spread...but back further, spreads existed...that might explain it.  That could be why the MT5 data, back in time, doesn't have a spread of zero (it might have 10-40 points) and EAS, because I over-rode the data setting with 2 points, based everything on that...

Hmmm.  Interesting...

That would/could also explain why more recent timeframe backtests work (3m, 9m, 12m) - because both platforms use the same spreads to generate/optimise/validate (i.e. 9 months in MT5 and in EAS match, because the broker data using the backtest matches what EAS used to build it).  Does that make sense?

Re: Advice needed - Portfolio Expert vs MT5 completely opposite backtest

sleytus wrote:

If you've found conditions where EAS backtesting and MT Strategy Tester results align (e.g. shorter data horizons) then I would go with that.  Maybe it has to do with spreads or maybe there is something else going on with the simulated OHLC data -- regardless, our time is better spent trading and using this amazing tool to create new and different portfolio EAs...

Completely agree, and thanks for spending some time looking into this with me.  I'll concentrate on closer timeframes for sure, and yes, it's an amazing tool for sure.

Thanks again Sleytus.

Posts: 25

Pages 1

You must login or register to post a reply

Forex Forum → Portfolio Expert → Advice needed - Portfolio Expert vs MT5 completely opposite backtest

Similar topics in this forum