1 (edited by vidagig_13 2015-04-30 15:16:01)

Topic: Big difference in backtest on FSB vs. MT4 using identical price data

Hi there

I see this subject has been discussed a lot but I don't see any clear answer to the simple question:
Which backtest platform is more rigorous, accurate and reliable - FSB or MT4?

I ran backtests using the identical 16 months of Alpari historical price data and start and end dates on a strategy developed on FSB and saved as an MT4 EA.

I would expect very similar results but this was not the case at all:
The strategy had no losing trades on FSB (299 total trades) but there were 5 losing trades on MT4 and some different trades were taken. (the strategy requires an extremely high win/loss ratio - taking small profits and big losses)

On FSB with 'pessimistic' setting I got virtually identical results with the Alparia data vs. the FSB demo data.
The Alpari MT4 backtest used the 'every tick' model - and showed 90% modelling quality in the report (with 278 trades).

Is the MT4 backtest more strict and/or accurate than the FSB backtest and therefore the more rigorous test result to evaluate a strategy backtest?

I started a live 'forward' test and the 3rd trade was a loser.

Any guidance would be much appreciated.

Thank you.

Re: Big difference in backtest on FSB vs. MT4 using identical price data

MT4 is full of problems, one of those being slippage.

If you feel that FSBPro is causing a difficulty, please email  to Popov a description of the problem, a copy of your EA and pics of the problem area, he will deal with it as soon as he gets a chance.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Big difference in backtest on FSB vs. MT4 using identical price data

There is no known bugs in the FSB Pro backtester. However, exporting EAs is a new feature and everyone has to be very careful with testing an EA before trading on a real account.

I have noticed, from my tests, that sometimes MT backtester doesn't set the tick time correctly and occasionally misses closing on Bar Closing indicator. Also the setting of "Use previous bar value" parameter when an indicator uses Longer Time Frame (LTF) has to be monitor carefully.

If you are experiencing differences between FSB Pro tester and the MT4 tester, you have to post your strategy and to point the differences. If a discrepancy is a result of a bug, it must be fixed as soon as possible.

It is possible to have a differences because of the different interpolation methods used by FSB and MT. Here FSB is a clear winner.

There are two checks anyone has to make before start trading with experts:
  1. Indicator values in the EA and in the FSB tester must be equal on equal data
  2. A full cycle of opening (adding, reducing) and closing signals must follow the strategy rules and executed at the defined prices and time.

Your signals may differ if you start the tests or trading on different state of the account for that market. The best is to start trading or testing on MT4 when the backtester of FSB has no open position.

Re: Big difference in backtest on FSB vs. MT4 using identical price data

Thank you for the quick and very helpful replies.

I will send you the Strategy and MT4 files privately if it will be helpful.
I've included some necessary detail below and attached a spreadsheet with backtest reports and trades on FSB and MT4 including the starkly different Balance Charts.

The strategy uses multiple chart timeframes including daily with a +-130 day moving average, so I understand a few months prior price data is required to construct the m.a. before trades start.
The FSB backtest report shows it starts on 1 July 2013 with the first trade on 23 December 2013.
The MT4 backtest report shows it starts on 2 December 2013 with the first trade on 9 December 2013.

In my experience of profitably investing substantial amounts in EA's - I've received MT4 backtests over 10 years price data over multiple time-periods; Monte Carlo analyses; and verifiable forward tests over 2 years.
I believe that testing over long time periods is essential to verify with some reassurance a strategy's robustness in multiple market conditions - and even then it's high risk as future market conditions unexpectedly differ - as I've experienced too...

I get the impression from my small experience and some research - that successful long-term backtests on MT4 appear to be the most rigorous test.

(The memory problems I'm having - making it virtually impossible to run backtests on FSB over long time periods - is a separate subject.)

FSB has proved invaluable to allow me entry - as a non-programmer - into the world of EA development.
So, your guidance and input to learn to use FSB correctly and effectively is greatly appreciated.

Post's attachments

150430 FSB & MT4 Reports, Balance Charts and Trades.xlsx 433.77 kb, 11 downloads since 2015-04-30 

You don't have the permssions to download the attachments of this post.

Re: Big difference in backtest on FSB vs. MT4 using identical price data

I agree re long backtests....

I think we need to have about 15 gig Ram, I know I get into trouble when running two backtests, I just do not have enough Ram.

I am positive that Popov will detect whatever problem there is with your EA comparison. very quickly.

The more bug reports he gets, the better for all of us.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

6 (edited by vidagig_13 2015-05-01 08:49:30)

Re: Big difference in backtest on FSB vs. MT4 using identical price data

To try survive the MT4 backtest from December 2013 (posted above) I set a suicidally large Stop of -280 points considering the risk per trade - supposedly 30% but in Alpari it seems to risk more - why I don't know.

On 5 June 2014 with a Balance of $56976 it closes a position before the Stop at -85 points and then halts the backtest - wish I understood why - if it's a Margin Call it didn't say so.
I'd expect it to just continue trading at small size using the remaining account balance of $377 - maybe this is too little for the smallest trade size available.

This is the kind of inexplicable technical problem I constantly encounter that makes me question if I have sufficient technical understanding to even try to develop EAs.

Post's attachments

150501 MT4 Strange Backtests.JPG 236.73 kb, 2 downloads since 2015-05-01 

You don't have the permssions to download the attachments of this post.

Re: Big difference in backtest on FSB vs. MT4 using identical price data

Did you load same data into each Mt4 as you have in FSBPro....? ie  using Tickstory?

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Big difference in backtest on FSB vs. MT4 using identical price data

1. I used Alpari MT4 with data going back to 1999 and imported it into FSB.

2.a. Re. my previous post on 1 May - about the backtest halting in MT4:
I now understand that it was a Margin Call.

2.b. Which brings me to another issues I've noticed but can't explain:
In the FSB strategy I specify a % of the account to risk per trade.
I notice in MT4 backtests it risks a considerably higher % of the account balance than the % of account risk specified in the FSB strategy and displayed in the MT4 Expert Properties.

4. As per my original posts:
I still don't understand the apparent significant discrepancy in results between an FSB backtest and an MT4 backtest using the same historical data and time horizon.

Re: Big difference in backtest on FSB vs. MT4 using identical price data

The history centre in FSBPro will help you to determine the quality of your data. There is often a number of errors in MT4 data,

Myself, I use the data available in FSBPro, for a couple reasons, one is that it is easy and simple to access and of relatively good quality. The second reason is that I know that Popov is using the same data. in case I need some help.

Did you check the journal of both programs to check for any errors..

I wonder if the settings of FSBPro are the same as MT4. Account risk may be calculated in a  different manner, I am not sure.

So far I have not had the difficulty that you are experiencing, so I  have not had to be a detective in that area.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Big difference in backtest on FSB vs. MT4 using identical price data

Thank you for the helpful input.

1. Data errors
I see no errors re. the Alpari data anywhere in FSB. When the backtest starts in MT4 I see the 'error:
"2015.05.03 07:31:13.966    TestGenerator: unmatched data error (volume limit 639 at 2015.04.28 15:00 exceeded)
Considering it's at the end of the test period and one entry it doesn't appear serious though I don't know what it means.

2. Time to run Mt4 backtests
I made a new version of this Strategy in FSB with 4 x opening logic conditions and using 5 Minute and Hourly chart timeframes.
In FSB it only takes 32 trades from 1 January 2013 and the results are encouraging.
The backtest on 'every tick' in MT4 has been running since 07h30 this morning and is only half complete - 12 hours and counting.
This seems very slow indicating the code may be very complex.
- I note the EA produced by FSB has 6883 lines of code and the MQL4 source file is 271kb compared to the  admittedly very simple sample MT4 MACD EA being 7kb and 154 lines of code.

3. I see already that the trade dates differ from those in the FSB backtest bringing me back to my original concern raised about differences between FSB and MT4 backtest results.

4. My experience in the treacherous and beguiling world of EAs has taught me to question and verify everything I can - because even the very rare seemingly brilliant EAs with vast backtests and spectacular multi-year forward tests have the inconvenient habit of blowing up unexpectedly.

So, this is why I have attempted to verify the results shown by FSB backtests by comparing them with MT4 backtests, raising the discrepancies I inquired about.

Re: Big difference in backtest on FSB vs. MT4 using identical price data

P.S. and I consider any Martingale or Grid EAs a fool's errand.

Re: Big difference in backtest on FSB vs. MT4 using identical price data

I don't want to sound harsh or arrogant, but how effective is it to perform another round of backtests with MT? It would be meaningful if MT backtest would be superior to FSB, otherwise it seems like a waste of time and resource. I don't see any reassurance that the strat wouldn't blow up in the future if it was double-tested, depending on the nature it will still lose performance at an unknown point in time, and lets not forget that some variables in the market are out of our control. One black swan event and any results backtests gave are absolutely useless. If trading behaviour in forward-tests is satisfactory, it's time to live trade.

Re: Big difference in backtest on FSB vs. MT4 using identical price data

I have said more than once that MT4 is less than accurate.

I wonder if you could make pics of the different dates so as to show the indicator values.

I suspect the data and MT4.

I know how aggravating this can be.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Big difference in backtest on FSB vs. MT4 using identical price data

Thank you again for all the helpful input from your experience. I appreciate it.
I note your reservations about MT4 backtests with interest.
I'm not sure how I will proceed but I will post anything useful if I find it.

Re: Big difference in backtest on FSB vs. MT4 using identical price data

The only reason to run MT test is to check if the indicator and expert work correctly. However, you don't need a full time series test 1 week or 1 month is enough.