1 Topic by Irmantas (edited by Irmantas 2016-05-22 19:00:53)

  • Registered: 2016-05-16
  • Posts: 190

Topic: Very different result in builder/MT4 backtest

Hi,

I got a strategy and it looks nice in builder. But in mt4 it's a shit ... I think that's something wrong with pivot indicator or my data. It uses weekend data, I can see in the indicator chart, few stupid time bars (Saturday), and Monday pivots are very near and EA makes a fortune around them smile Again, it somehow makes unrealistic profits. Is there any way how to remove weekend bars from metaquotes demo data? Or some settings on that pivot indicator or other work around to prevent future time wasting? 

http://s32.postimg.org/tk161hfg1/fsb_test.jpg
http://s32.postimg.org/rnioli9v5/mt_test.jpg
http://s32.postimg.org/oz24k2pgx/indicator_window.jpg

Just wasted all the day to find out something not true with backtests again sad Already did few WFA test manually with some statistics and WFA profit loss curve... big_smile Little bit frustrated. Maybe you are planning near future adding WFA? For me it would be golden thing, not tu run it manually.

Thanks

Post's attachments

EU m15 pivots frama bolinger.xml 14.36 kb, 11 downloads since 2016-05-22 

You don't have the permssions to download the attachments of this post.
  • Blaiserboy
  • Blaiserboy
  • Junior Part-Time Aspiring Space Cadet
  • Offline
  • From: Toronto
  • Registered: 2010-09-06
  • Posts: 2,849

Re: Very different result in builder/MT4 backtest

Which data source please?

Blaiserboy's Website

  • footon
  • footon
  • ◄≡≡≡►
  • Offline
  • Registered: 2009-08-17
  • Posts: 2,969

Re: Very different result in builder/MT4 backtest

Irmantas, which platform do you trust more in backtesting, FSB or MT?

4 Reply by yonkuro (edited by yonkuro 2016-05-23 00:29:08)

  • Registered: 2015-02-22
  • Posts: 430

Re: Very different result in builder/MT4 backtest

Hi Irmantas, did you use same data between FSB Pro and MT4?

and about the WFA, have you tried OOS tools? it's explained here http://forexsb.com/wiki/fsbpro_guide/strategy_generator

  • Registered: 2016-05-16
  • Posts: 190

Re: Very different result in builder/MT4 backtest

Blaiserboy wrote:

Which data source please?

Both FSB and mt4 uses metaquotes demo data, downloaded from mt4 history center. Same time zone.

Watched few entries in both platforms. Few seems same, but others not, sometimes it's made in mt4, sometimes in FSB. But it's clear that EA is not producing same entries/resutls in both platforms. Used every tick in mt4 too see differences. For longer period only bar open, to compare end results.

Makes the same:
http://s33.postimg.org/5er8l0vuz/1_1.jpg
http://s33.postimg.org/arg2z5jrf/1_2.jpg

Mt4 makes, FSB not:
http://s33.postimg.org/u6i4k5f7v/2_1.jpg
http://s33.postimg.org/ukjgjqzbf/2_2.jpg

FSB makes, mt4 not:
http://s33.postimg.org/5lstnklsb/3_1.jpg
http://s33.postimg.org/4xjz4mn2j/3_2.jpg

So I don't get it. Clearly something wrong.

  • Registered: 2016-05-16
  • Posts: 190

Re: Very different result in builder/MT4 backtest

footon wrote:

Irmantas, which platform do you trust more in backtesting, FSB or MT?

After finding in few days some unrealistic/bugus profit/loss curves, witch can't be replicated with mt4, I don't trust FSB. Practice to look suspicious to very good system profits is healthy.

So you suggest ignore mt4 results and run it demo/live? Where is the proof that it will generate results like FSB showed, and not what mt4 showed. It clearly produces different entries and it must be looked why and fixed. Don't you agree?

  • Registered: 2016-05-16
  • Posts: 190

Re: Very different result in builder/MT4 backtest

yonkuro wrote:

Hi Irmantas, did you use same data between FSB Pro and MT4?

and about the WFA, have you tried OOS tools? it's explained here http://forexsb.com/wiki/fsbpro_guide/strategy_generator

Hi,
Yes, data is the same. From mt4 metaquotes demo, downloaded from history center. I use data "horizon tool" to produce WFA test like this.

http://s33.postimg.org/j5nybd9e3/wfa1.jpg

For me is needed more sophisticated WFA test than simple FSB OOS, with chart/stats produced with unseen data like these what I made in excel. I want to make decisions looking in these pics. And copying, optimizing and producing it manually takes a lot of time (not even near how slow with mt4) and I make errors, it would be great to have this WFA tool in FSB smile

http://s33.postimg.org/jwpv8mqiz/wfa2.jpg
http://s33.postimg.org/fov2zvp3f/wfa3.jpg

  • Registered: 2016-05-16
  • Posts: 190

Re: Very different result in builder/MT4 backtest

Maybe someone downloaded and tested this strategy attached in the first post and tested with your data? Does it show good profits like for me?

Thanks

9 Reply by yonkuro (edited by yonkuro 2016-05-23 15:16:28)

  • Registered: 2015-02-22
  • Posts: 430

Re: Very different result in builder/MT4 backtest

Irmantas wrote:

Maybe someone downloaded and tested this strategy attached in the first post and tested with your data? Does it show good profits like for me?

Thanks

Just tried your strategy using FSB default data source and here is my result
http://s33.postimg.org/c7mpck62z/Strategy_Screenshot_23052016.jpg

I think the curve represent your MT4 Tester result

  • Registered: 2016-05-16
  • Posts: 190

Re: Very different result in builder/MT4 backtest

Just found how to cut weekend data, it was all the time in the options "Data Sources" smile Probably in mt4 pivot indicator was not looking in weekend data, but in FSB it was looking, besides both used same metaquotes demo data (my guessing here). So it created huge difference in results.

After cutting weekend data, profits are gone in FSB, however I still can't replicate entries/results with mt4(every tick) either... well it seems there is time to leave this strategy alone and to look for new one, and not use Pivots any more. But it still leaves me with big questions about different entries/results between platforms. Sorry for long ramblings. Let's see if I can find 3rd system with unrealistic/bugus results smile

Have a nice day!

  • Registered: 2016-05-16
  • Posts: 190

Re: Very different result in builder/MT4 backtest

Got another different backtesting strategy with bar opening, in mt4 only 250 entries, and in FSB 500 entries... Probably time to import new data. From Dukascopy and see how it performs in both platforms.

  • footon
  • footon
  • ◄≡≡≡►
  • Offline
  • Registered: 2009-08-17
  • Posts: 2,969

Re: Very different result in builder/MT4 backtest

MT doesn't offer tick backtesting. What can I say, in FSB you see and can check everything, MT takes awful lot of time, doesn't let to verify anything, how it works is anyone's guess. That's why I don't trust it.

What I would do - if your strat is on a lower TF and it can trade enough in a reasonable time, I would let it demo trade, then run it through using the same data in FSB and compare demo and backtest results. Only then you can say whether entries match or not. If then there are discrepancies, then it is a serious matter and needs a closer look.

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Re: Very different result in builder/MT4 backtest

Hello Traders,

I want to say that my goal with the export of MT Expert Advisors is to achieve very near results in FSB and in MT.

I tested the strategy attached in the first post and found out that this strategy suffers from the bug reported  by Irmantas here: http://forexsb.com/forum/topic/6034/unr … ime-frame/
   As you know, I tried to post a hot fix to that bug, but the version I released has problems with the digital signature.
  I reviewed the bug again and made some adjustments to the bug fix code. I applied the fixes to the EA code and to the Trader code also.
  Now I assume that the bug is finally fixed. I also fixed (at least hope so) the problem with the digital signature. So, I decided to test the attached strategy with my new code.
  I exported an EA and also run the Trader in order to compare the indicator values. They much completely.

http://s33.postimg.org/ubkrh4lmj/screenshot_1671.jpg

I run the MT backtester in order to compare the deals. And found another bug:

http://s33.postimg.org/ckcehtta3/screenshot_1672.jpg

The position in MT opens very near to the FSB's one (3 ticks slippage), but MT closes it at the end of the same bar.

This strategy uses  Exit Hour for closing. It looks like the Exit Hour is not working correctly. I'll check it and will try to fix it.

...

It is very important to report all issues you find in the program in order to be able to fix them as soon as possible.

...

Footon is completely correct for the crappy MT testing, but I recommend you to run it for a short period of time in order to compare several entries and exits with the FSB Test.
Our experts use the same calculation model as FSB so the results must be near.

Popov's Website

14 Reply by yonkuro (edited by yonkuro 2016-05-23 22:59:45)

  • Registered: 2015-02-22
  • Posts: 430

Re: Very different result in builder/MT4 backtest

Popov wrote:

Now I assume that the bug is finally fixed. I also fixed (at least hope so) the problem with the digital signature. So, I decided to test the attached strategy with my new code.

That's a great news Popov, can't wait to try the official version

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Re: Very different result in builder/MT4 backtest

You can find the new release in the Premium Club.

Please test this strategy in FSB and in MT. It must show almost equal results.

Popov's Website