Create and Test Forex Strategies
You are not logged in. Please login or register.
Forex Software → Expert Advisor Studio → backtesting result is different from the real trading
I have the same problem, using MT4 backtester and EA studio I get the same result however if I check the history of trades it is very different . No idea why this happens but it is very frustrating and it means the validatir is useless
I can´t confirm this problem. My live trades do match the backtest in MT4. Of course, you can *only* backtest your live trades after the live trades have been done already. Then you´d take the brokers data that your live trades were done on and load it into MT4 and backtest the EA over the exact same time-range and trades will match. Been doing that since 10 years just to always make sure everything is working live as intended. And it´s no different with EA Studio, everything is fine and my live trades, once I backtest them *afterward* with the EXACT same data of my broker, match up 100% apart from small slippage differences.
If you get differences, you are doing something wrong in terms of either not using 100% the same data that the live trades were exposed too (e.g. you are using data from IC Markets for the backtest, but traded on Pepperstone), the spread used might be incorrect, there might be holes in the downloaded data that you use for the backtest, wrong spikes because of manipulated broker data, etc. etc.
If you want, record a video and I´ll take a look and let you know what the issue is.
I have a PC that runs MT4 24/7 which I take my data from. I have only had it running 3 weeks now however at the end of each week I output the data from MT4 into EA studio which what I’ve been told will be correct data from the broker as the PC is running full time. (Correct me if I’m wrong) I then backtest in both MT4 and EA Studio and compare the results to my Journal on FX blue and have never had the same results?
Any ideas? How can I be sure I am getting correct broker data? I’m half tempted to open a trading account with Dukascopy due to the fact that they seem to have the best available data
OK, this sounds all good. The only thing I am doing different is to only use M1 data, then create the higher timeframes via a script, then export this to EA Studio and compare the backtests with the live trades. Everything always matches there for me.
But, could it have to do with your other issue? Because if you trade strategies that are affected by that issue that they trade completely different as soon as you change the starting / ending date already within EA Studio, it would be no wonder that you don´t get matching live trades either, as the starting date is basically another one again. And of course, you also need to look at a longer time-range. At the very beginning of attaching your EA to a chart, it might well be the case that the live trades are different, because in the backtest the EA already might be within a trade that you´ve missed live as you´ve just attached the EA to the chart, but the indicators still give a signal which was already entered earlier in the backtest, makes sense? If you use indicators only for the exits, then this will get in sync faster than if using TP or trailing, as this depends on the trades current profit, while an indicator exit depends just on the market and should put you in-sync with the backtest after the first (indicator based) exit. I get this too, but it should be in sync within a week (depending on your timeframe too and how often the EA trades). But for me it´s usually in sync within 1 week, trading H1.
Hi,
I´m new here so sorry if i´m missing something obvious..
I compared my result from my live mt4 to EA Studio backtester and the trades does not match..?
1. I imported live history data from the same MT4 terminal as I trade in as a .json
2. Imported the EA from the live MT4 terminal using the validator
But the result does not match at all.
For eg, shown live trades from MT4 is not in the backtest result in EA Studio, how come?
And some others have completely different time stamps/result
Such differences usually happen if the strategy is extremely unstable and optimized "on the edge" (so that just a few pips difference in price which happens during live trading naturally will make a huge difference). Can you please test this strategy with the Monte Carlo validator in EA Studio? Use skip entries, skip exits and slippage, then set the skipping to 5% for both and the slippage to "20" (points). Does it pass this test or do the resulting equity curves look completely bad? If it does not pass this, it´s a strategy you should not trade live, as it will be too weak/unstable for live trading and such differences will happen then.
At least I can tell you that my strategies (which have passed the mentioned MC tests) match within the backtests between EA Studio <-> MT4 and with the live trades too. Nothing to complain there and it´s even more accurate than with most other trading platforms out there - EA Studio is almost always a 100% match for everything (EA Studio backtest, MT4 backtest, live trades), thanks to Popov´s good work.
Such differences usually happen if the strategy is extremely unstable and optimized "on the edge" (so that just a few pips difference in price which happens during live trading naturally will make a huge difference). Can you please test this strategy with the Monte Carlo validator in EA Studio? Use skip entries, skip exits and slippage, then set the skipping to 5% for both and the slippage to "20" (points). Does it pass this test or do the resulting equity curves look completely bad? If it does not pass this, it´s a strategy you should not trade live, as it will be too weak/unstable for live trading and such differences will happen then.
At least I can tell you that my strategies (which have passed the mentioned MC tests) match within the backtests between EA Studio <-> MT4 and with the live trades too. Nothing to complain there and it´s even more accurate than with most other trading platforms out there - EA Studio is almost always a 100% match for everything (EA Studio backtest, MT4 backtest, live trades), thanks to Popov´s good work.
Hi
I understand the power of monte carlo and I really think popovs work is truly fantastic, but what I dont understand is why EA Studio backtest does not show the same trades to what really happened in live MT4.
If there is a mismatch between them and EA Studio not showing the same result as live with the same history data, how can I then know that the generated strategies will perform live?
The EA has performed well in demo and also live but why the different result? So its performing well live but i´m puzzled with the mismatch in the backtest.
I´m not by any means well experienced in Popovs world, but It should show the same result without monte carlo?
The data is the same and the execution should be the same?
But now I made a mt4 backtest and it also did not show the live trades 30 november, same as EA Studio..?
It is a M5 strategy, is it to "fast"?
I have the same issue however my broker Pepperstone has quite bad data to load into EA Studio.
So I now use Dukascopy data which I load into my MT4 then into ea studio.
I run a backtest on MT4 which matches EA Studio perfectly then I put the strategies on a demo account.
This is really the only way I can see with a broker that has very limited data.
If there is a mismatch between them and EA Studio not showing the same result as live with the same history data, how can I then know that the generated strategies will perform live?
The EA has performed well in the demo and also live but why the different result? So it's performing well live but I´m puzzled with the mismatch in the backtest.I´m not by any means well experienced in Popov's world, but It should show the same result without Monte Carlo?
The data is the same and the execution should be the same?
But now I made an mt4 backtest and it also did not show the live trades 30 November, same as EA Studio..?
It is an M5 strategy, is it to "fast"?
It´s all fine if it passed the Monte Carlo tests I´ve mentioned. The reason is very simple: all your backtests use a FIXED spread, but all your live trades use a dynamic spread. That mainly affects strategies that are built "on the edge" - for example, in your backtest at the open of the new bar, the price might have been 1.23001 and 1.23006 (so a fixed spread of 0.5 pips (or whatever you´ve set) that is used throughout the complete backtest) and the indicators will use these both prices to calculate their values to determine a possible entry. But, on your live trade at that exact bar open time, the prices might have been 1.23001 and 1.23010 (so a spread of 0.9 pips at that exact same time as in the backtest) and that little difference CAN cause another entry because the indicators gave different values live because of the different spread than in the backtest - so no trade or A trade was triggered that does not match the backtest. You can´t avoid that at all, it´s absolutely normal - I have it day in and out since 10 years of EA trading with all my strategies and it´s nothing you need to worry about, because, in the long run, these small differences will not matter for a STABLE strategy. And that is the important point: you need to make sure that your strategy is STABLE via Monte Carlo by randomizing spread and slippage by meaningful values - if the strategy is still profitable with randomized spread + slippage, randomly skipped trades and a randomized starting bar (to avoid trade chain dependency), you can go ahead and trade it live - at least you can rest assured it will not fail because it is highly spread, slippage or trade-chain sensitive. Do the same for the entries and exits, skip 5% of them in the Monte Carlo simulation - this basically simulates these missed / wrong trades. If the strategy is stable with all 4 of those (spread, slippage, randomized starting bar and skipped entries / exits by at least 5%), you can be sure that you can now trade it live, even if trades will sometimes differ.
Good luck :-)
P.S.: The backtests between EA Studio / Metatrader 4 (haven´t tested 5) should match almost 100%, at least for the amount of trades, as net profit can be a little different based on your account currency used in EA Studio / MT4 and the current base value if the pair you trade is not denominated in the currency of your account (both EA Studio and MT4 might use different exchange rates for your account currency in that case). Also Metatrader takes into account commission (pulled from your broker from when you´ve last connected to their server), so profit will not be an exact match. But the amount of trades + equity curve "by eye" should be very close between both (at leats it is for all my *stable* strategies).
The data is the same and the execution should be the same?
The data and execution are not the same. There is a "disconnect" between back testing and Demo accounts, and another "disconnect" between Demo and Real accounts.
Back testing gives you a clue whether a strategy might or might not perform well in a Real account.
The best advice I received from this forum a couple of years ago was to jump into a Real, micro-account. Yes -- you'll lose a few bucks -- so what. In return you'll gain some "street smarts" and become a better trader sooner. Sort of like learning to swim -- I mean, you can spend a lot of time in the class room with someone teaching you the mechanics and practicing swimming exercises, but you won't have a clue what it's really like until you're actually in the water. Trading is similar in that way.
Sure thing, this will help too, but if he does the Monte Carlo in the way mentioned, he can be sure his strategies "do not care" about different slippage, spreads, randomly skipped trades, etc. - so such additional or skipped trades in a live account won´t hurt the longtime profitability of the strategy (if it has an edge of course, which is the hardest part, hehe). I am doing it like this since 10 years in the Forex market already (automated strategies only) and wouldn´t be here anymore after that long time if it wouldn´t work to some degree :-) This really is the best way to make sure the strategies are at least stable to any technical issues they might face. Again, this has nothing to do with finding PROFITABLE strategies, that is still the hard part :-)
Personally I´ve generated billions (no joke) of strategies with EA Studio in 2018 (apart from other platforms), having a Threadripper 1950X running 24/7 with a great electricity bill, haha, and yet I´ve just added 10 strategies during 2018 that EA Studio came up with - so strict are the stress-tests each strategy has to fullfil (apart from passing the mentioned Monte Carlo tests, each strategy has to pass 10 additional markets with 32 years of history data each with a R-Squared of at least 85 without ANY optimization right "out of the box"). Meaning those are strategies that just have been generated on one symbol, but work on at least 10 unseen markets x 32 years (320 years of history data in total) with an almost straight equity curve on any market without any optimization right away (still doing that afterward though to adjust somewhat for each symbol). So, it pays off, as you can find extremely good strategies this way (almost holy grails). Just don´t lose the faith in EA Studio, it does come up with extremely good strategies and they DO work live.
Yes, the EA that I mentioned does perform good in live mt4 trading.
I was merely troubled with the mismatch with the EA Studio backtest and actual trade results. But I suppose it can be as you say Geektrader, the spread can affect..
An open question here to you members, what do u reckon is the minimum requirements for the Monte Carlo validation?
A course that I bought on Udemy recommended these parameter as minimum;
Randomize indicator parameters
Randomize backtest starting bar
Whats the opinion? Is it enough?
Another thing that I wonder is,
if I have a batch of EAs (say 20) and lift them in in EA Studio via Validator only a few EA gets through even if I have unticked everything that has to do with validation in the whole EA Studio?
Is it because they show a negative result?
I still would like to put ALL strategies back unregarding if the show + or - result in order to tweak them..
@Timpa:
I don´t think much about randomizing indicator parameters, because if you use this for validation during strategy generation, EA Studio will simply start to "prefer" strategies with indicators that have fewer input parameters or where the parameters are hard coded, which still can be a curve-match. Instead I prefer to do multi-symbol validation as explained in detail above. If a strategy does well on completely unseen symbols with acceptable good results (R-Squared -> ~82), you can be absolutely sure it is not curve-matched.
However, as mentioned, here are the tests I always do in Monte Carlo:
- Randomize slippage by up to 20 points
- Randomize spread
- Randomize starting bar (no trade chain dependency is very important!)
- Randomly skip entries by 5% chance
- Randomly skip exits by 5% chance
These 5 guarantee that your strategy is prone to any execution issues (as you´ve already experienced them) and is technically stable at least.
Just letting you know that I´ve done some tests today between EA Studio <-> MT4 backtests and I am getting PERFECT matches between the backtests, down to the EXACT amount of trades, even with 10000+ trades (ignore the net profit, it is a little in higher in MT4 as I am always using double the real commission in EA Studio just to worsen the results a bit for extra stability). See:
I´ve done this with 10 strategies, 32 years of history data each, many different indicators and I´ve always got a match down to the trade. So at least nothing to worry about the backtests between EA Studio <-> MT4 and great work Mr. Popov!
@geektrader
Thank you so much for the tips, I´ll try them. But I dont have so much history data, only 200k bars, will your settings be to harsh in criterias then?
Also, what settings for the "Acceptance criteria" is recommended?
I´m struggling to get relevant broker data, since not been using mt4 for long. Maybe its best just to use JFD as broker, as I understand EA Studio demo data is from..?
It´s good to see that your mt4 backtest match EA Studio
I have found the solution for you! Pepperstone only allows up to 50 orders to be open at one time on a demo account. I have asked them to lift it on my account however they say they can’t.
I’ll be opening an account with another broker unfortunately
Hi Michael, do you mean timpa? If so, his statements don´t look like he ever had 50 trades at once in parallel so that he´d even hit that limit, it rathers comes down to the reasons I´ve explained (spread).
Sorry I do mean timpa, I’m not in front of the computer at the moment but if he has not had more than 50 open trades at one time then that would throw that theory out the window. Just something to keep in mind however
Hi
@Michael1
I think my problem is as geektrader mention, either spread or maybe out of "sync" because of the starting bar sensitivity and possibly the following chain of events thereafter
@geektrader
I would be happy if u could share your view
what settings for the "Acceptance criteria" is recommended when starting in reactor mode?
I´ve just compared some of my last live trades today with the EA Studio backtest (feeding it the same data of course), I´ve got a perfect match down to a few points on the entry and exit prices (spread differences and slippage - just normal). If your strategies pass the mentioned MC tests, you should be fine (also in terms of starting bar, don´t forget that).
My Acceptance Criteria in Reactor mode totally depends on what I am looking for and how many years of data I am using. In general here is what I use *most* of the time on my 32 years H1 data set (all pairs):
Minimum R-Squared: 80 (want at least to have a *somewhat* acceptable curve in the initial strategy before handing it over to the Optimizer, as I don´t want to waste CPU power on the Optimizer for strategies that anyway will fail even after the Optimizer as their initial stage is too bad right from the start)
Minimum Trades: 1500 (now that could be much too much if using < 32 years of data, but for my huge set it´s the minimum I want to see because anything lower is not really statistically relevant)
Minimum Backtest Quality: 99 (obvious, right?)
That´s it already, nothing special.
did you ever get good strategies with such amount of trades?
I played with eastudio and i find out that.
You use e.g 10 years backtest data and want 1000 trades than i never find good strategies when use Stop loss.
So i find out higher minimum count of trades need high stop loss to get good strategies.
I only get good strategies when i use minimum 200 trades. Maybe i do something wrong. But i never found strategie for example stop loss 50-100 pips. Only with small number of trades.
@Roughey: https://forexsb.com/forum/post/54133/#p54133
And yes I do, even on 32 years of H1 data - having ~4000+ trades there, so /3, you should be able to get 1000 as a minimum for 10 years too. All my strategies (you can see them in that post) have a stop loss of max 60 pips. Most are in the 3X pips range. I don´t like strategies with a big stop loss. Shows that the entry (or exit) is too inaccurate and the exploited edge too weak. Wouldn´t trade those personally.
Just a short update: I now also had one case where the backtests between EA Studio and MT4 do not match, even though the exact same data was used and all other strategies backtest with the exact same results as shown above. I´ve already reported it to Popov, but haven´t heard back yet. It seems to be a special constellation of indicator combinations when this happens, but it still must be extremely seldom as I never had that before.
Forex Software → Expert Advisor Studio → backtesting result is different from the real trading
Powered by PunBB, supported by Informer Technologies, Inc.