forex software

Create and Test Forex Strategies

forex software

Skip to forum content

Forex Software

Create and Test Forex Strategies

You are not logged in. Please login or register.


(Page 2 of 2)

Forex Software → Premium Club → Demo vs MT4 vs FSB vs EA Studio

Pages Previous 1 2

You must login or register to post a reply

RSS topic feed

Posts: 26 to 39 of 39

26 (edited by DoCZero 2018-08-19 09:53:37)

Re: Demo vs MT4 vs FSB vs EA Studio

So to come back to the topic of comment about MT4 and FSB not matching. I have done some testing with EA Studio - forcing 4 entry rule and 1 exit rule (which is the most complex setup), I built EA's over different time frames (5M , 30M, 1H, 4H). I am finding the MT4 & EAS backtests match fairly well:

https://preview.ibb.co/eg8yYK/EAS1H.jpg
https://preview.ibb.co/dedyYK/EAS5M.jpg

Another thing to note is the execution time of the EAS mql files is a lot faster on tick data vs the FSB EA files. 

https://image.ibb.co/cutFDK/Time3.jpg


Which the leads me to believe FSB custom indicators are a little bit of a mine field. My guess is the normal indicators will be ok  - and that the custom indicators are having issues. I will focus some tests on FSB custom indicators to see if I can prove it is the case.

But from my thinking at the moment - if you are using EA Studio the execution is very close to MT4 (which means if 2 different platforms are matching, live execution should work also).

Re: Demo vs MT4 vs FSB vs EA Studio

This is excellent news -- thanks, DoCZero.

With regards to execution time of EAS versus FSB mql files -- I haven't carefully tested and compared like you have, but from comparing portfolio EAs created by EA Studio and Portfolio Maker (for FSB-generated strategies) I would certainly agree.  I think it is primarily due to the fact EA Studio can use MT4's built-in indicators.

A question -- you are exclusively using tick data, right?  Since EA Studio strategies always use Bar Opening and Bar Closing then have you considered comparing MT4's  "Every tick" versus "Open prices only" models?  I ask for selfish reasons -- I'll often use "Open prices only", but don't have the patience for "Every tick".

Because of your testing results I have more confidence in back testing -- so, I'm spending less time in Demo testing.  The past couple of weeks I've gone straight from back testing to a Real account -- which saves a huge amount of time.  And, so far, no harm -- I still have my pants on...

Thanks...

Re: Demo vs MT4 vs FSB vs EA Studio

Hi Steve,

I have done some work with Every tick and open prices only - but I haven't done much testing between them at this time. At some stage i'll test the difference between the two.

Currently i'm trying to understand the difference between FSB and MT4 and how to ensure our testing between the two matches.  Basically the idea is - Build in FSB or EAS / Confirm with MT4 / Incubate in demo / Live... along the way you should be able to understand and remove curve fitting , errors etc.

Today i'm just trying to get a consistent mismatch in the two - then I can understand why.

Keep you posted wink

Re: Demo vs MT4 vs FSB vs EA Studio

https://preview.ibb.co/jp5M7e/1.jpg
https://preview.ibb.co/msmTne/2.jpg
https://preview.ibb.co/kNSCDK/3.jpg
https://preview.ibb.co/cYcOLz/4.jpg

Here are four tests i've run over the last four days with custom indicators. I have shown the indicators used - so you can get an idea. Some are similar - but all have areas they dont match performance in one way or another.  I tested changing the slippage and commissions - but found I had it setup fairly correctly.

I am at the stage where I feel the custom indicators are introducing the issues - I guess the only way to tell would be start testing single indicator systems, using the custom indicators.

Popov / Others - do you have any comments , hints which may help us from this point forwards?

30 (edited by sleytus 2018-08-21 02:28:29)

Re: Demo vs MT4 vs FSB vs EA Studio

Interesting...

I have a few comments:

(1) Using FSB there are two versions of every indicator -- one written in C# and one written in MQL.  The one written in C# is used by FSB, and the one written in MQL gets exported for use by MT4.  I have written a couple of indicators but have not published any to the repository because I couldn't get their FSB and MT4 behavior to match -- my point being that, depending on the indicator's algorithm, it may not be so trivial to write two versions in different programming languages and have them behave exactly the same.  Not only is the code different, but one algorithm runs within the FSB executable and the other runs within MT4.

(2) Because of the two versions of every indicator, then I always do a "sanity check" by testing every strategy in MT4 Strategy Tester to confirm its performance roughly matches with the back test statistics computed by FSB.  I don't necessarily compare FSB and MT4 Strategy Tester in great deal -- just only to make sure there aren't any surprises.

(3) Technically it would be relatively easy to test and compare indicators individually in FSB and MT4 -- but it would be very time consuming.  I've always assumed the authors of the indicators performed the necessary testing before publishing them.  If that is not the case and you were to expose big differences then that would "open a big can of worms".  On one hand we should be appreciative of indicator authors for sharing their work, but it could get embarrassing if they didn't do a good job of testing and have wasted people's time and money.

(4) Perhaps Popov could provide a "framework" that would simplify the testing of custom indicators.  I have no idea what form it would take or what it would look like, but if such a "framework" existed then other's could jump in and help test the indicators.  Furthermore, perhaps it should be required that custom indicators pass a test before being published.

Re: Demo vs MT4 vs FSB vs EA Studio

I had such "framework" before. It was very useful, but difficult to be used. I'm working on a way to make unified indicators between EA Studio and MT. There are several ways to be done, and I'm trying to make them work best.

It appeared that the EASL is much slower than the standard JavaScript code (because it is a black box for the V8 and cannot be optimized), so it may not be used for writing indicators directly. There is a possibility to make a code converter: JS -> MQL or MQL -> JS. or to have something in the middle. I'm not sure now how it will work, but definitely the software need a way to be customized better. You see how many future requests we have. If we make the apps more modular it will be great for every one. when we find the best way to make indicators, we will design a corresponding testing platform.

About the current custom indicators, some of them are real crap. I don't understand the users that load all indicators at once. Probably they think the more indicators the greater profit !? 

Recently professional traders requested to make modifications of EA Studio indicators to work like the Strategy Quant 4. They sent me some exported code. I agreed to check them, bu sorry for the colleagues,  it was real bullshit. I want to say that you don't have to believe to third party tools, code ... blindly.

..

Thank you for the hard work for testing the performance and indicators. If you notice a particular bug in the builtin indicators and experts, please report. I'm recently fully occupied with EASL, but will finish the current stage within two weeks. Than I'll return back to the regular EA Studio improvement and maintaining FSB Pro. 

I'm planning the development for 5 years ahead in order to make the tools even better.

32 (edited by DoCZero 2018-08-22 07:42:14)

Re: Demo vs MT4 vs FSB vs EA Studio

To get credibility with the tool - we have to calibrate then validate correctly. Any tool is only to aid the skill of the craftsman that wields it - however if the tool is flawed it can ruin the craftsman's work.

I think if I am to continue with FSB - we will have to validate one by one which indicators are giving similar results between the two platforms.  The issue you then have is there indicators have many settings (Simple / Smoothed / Exponential... etc) and Triggers (Crosses above / higher than... etc). Validating each is going to be a mammoth job.

I have been running single indicator tests for the last few days (it is slow work as MT4 execution on tick is quite slow.. I think I have to cross check with control points / only prices to see if I can speed up the work with still a high level of credibility for the process). All the indicators below are custom add ins:


https://preview.ibb.co/hbWCYK/single1.jpg
https://preview.ibb.co/iG3RtK/single2.jpg
https://preview.ibb.co/hGsKDK/single3.jpg
https://preview.ibb.co/j5f6tK/single4.jpg
https://preview.ibb.co/eTfjfz/single5.jpg
https://preview.ibb.co/j74c0z/single6.jpg


Some are similar (look at the last two) - but if you look at the structure of the peaks / valleys you will find all are different.. (even in the last two) The one that didn't trade was very surprising. I still think FSB is incredibly accurate in the way it executes / backtests and handles the code - but I think the difference between MT4 code and C++ for the indicators are the issue. I would like a third tool to cross validate (again we dont know if any issues are introduced with the MT4 backtester - but I feel I have semi validated it with the demo vs mt4 backtests).

Steve - I remember doing the tests with OnTick vs OnTimer - I wonder now if the issue we were seeing is exactly what we are exploring here.

I'm going to test the different speeds of MT4 backtest next and see if the results match.  If I can speed up this process, i'll begin to test the built-in indicators, and start by validating them.

Re: Demo vs MT4 vs FSB vs EA Studio

Ok so here is the 3 types of MT4 options vs FSB

https://preview.ibb.co/iPEGAz/MT4_options.gif

I'll have to do one more test with a very standard setup (probably a simple moving average cross) - report back shortly.

Re: Demo vs MT4 vs FSB vs EA Studio

Ok so here is the test on the simple system:

https://preview.ibb.co/j0BiOK/Envelope_System.gif

Still showing a big difference between the platforms.

Steve - You can confirm that the equity curves match - but please take note of the difference in values from Tick / Control / Open prices.

Re: Demo vs MT4 vs FSB vs EA Studio

PT -- This is very cool.  Again, thank you.

You brought up OnTick vs OnTimer -- I think that is isolated to MT4 and unrelated to your tests.

With regards to testing all custom indicators -- there are too many combinations of settings to make this feasible -- as you say, a mammoth undertaking.  I don't mean to belittle other authors, but perhaps the task could be simplified if it was assumed the indicators authored by Popov and footon were okay.

I like your most recent post comparing MT4's Tick / Control / Open modes -- that's a nice result. Yes -- the values differ, but couldn't this be due to the fact the source of the tick data is different than your broker's data (used by Control and Open modes)?  I'm more interested in Win Ratio and the overall appearance of the equity curves, and I'd say they match pretty darn close.

How would you summarize the results so far?
It seems that things match, except when they don't...

Re: Demo vs MT4 vs FSB vs EA Studio

Just a quick comment. My Aim here isn't to discredit any developer or the FSB software. The issues could be on my side (as everyone's environment is different) but I really don't know.  I think the FSB software is amazing (better than any other tool I have used before) additionally I think the developers who shared Indicators are very generous - personally I love having access to the many different indicators.

Checking the backtest between 2 platforms doesn't mean either platform is right. MT4 is executing these EA totally different to how FSB does. MT4 could be completely inaccurate.

The issue I have - if both back tests match, there is a level of trust.  But when they don't - I don't know what to trust.

I will keep going on this topic until I understand the root causes (if I can haha) - and hopefully understand what I need to do to have the trust i'm looking for.

PT

Re: Demo vs MT4 vs FSB vs EA Studio

Hi All,

I think i'm coming to the root of the problem - its not the indicators at all. It is to do with - Swap rates / slippage / commissions - and matching these settings between the 2 platforms. This really becomes obvious when you increase the time & number of trades.  I have been using 10+ years of data , and I did a test with Steve where he ran 2 years of data on an indicator (where my MT4 and FSB didnt match). I then swapped the time frames to match his time frame and both equity curves matched his.

https://preview.ibb.co/jZQVA9/matching_1.jpg

To also check at longer timeframe - here is the same test with 8 years of data:

https://preview.ibb.co/jbQacp/matching_2_8_year.jpg



First of all - sorry if I worried any developer.  I still think it is highly important to make sure everything is working correctly. I will spend some time doing checks on indicators over time (but my guess is this wont be an issue).

This really highlights the importance of having all the subtle settings of both platforms matching. To be honest it isn't as simple as matching your brokerage commissions also. My broker charges me 5.5 usd per 1 lot traded. If I input this - I don't get the same results with MT4 backtest and FSB. The other thing is the spread - slippage and swap rates. These also don't match exactly what my broker is giving.  Very strange.  Here is a look at the setup between the 2 platforms (which generated the above equity curves).

https://preview.ibb.co/jA4OV9/settings.jpg

So we can learn from this - I think a standard process should be:

1) Check MT4 Demo swap rates (Symbol properties in the backtester)
2) Adjust FSB Swap rates
3) Input your broker comission rates
4) put a minimal slippage
5) check the average spread on your broker (https://www.myfxbook.com/forex-broker-spreads) input that into FSB.
6) run a simple EA backtest with 5+ years of data (FSB & MT4)
7) check equity curve results between platforms.
8) Make adjustments in FSB to match MT4  (FSB is so much faster and you can tweak the settings very quickly).

One note on MT4 backtesting:  Be VERY careful with the spread setting - MT4 has a setting called "Current" and if you are testing over the weekend - the spreads are often widened greatly. This will destroy your results.

I think it would be wise to run your standard setup process (with a simple EA) almost every time you backtest - 

SO ..... to give credibility:  FSB & EAS & MT4  all execute fairly closely. Setup is critical (and not as simple as getting your brokerage info and plugging it in).

Re: Demo vs MT4 vs FSB vs EA Studio

The following picture is a multi indicator / multi timeframe complex system. This shows that the setup on FSB and MT4 is critical - and once again shows that FSB can match the demo test results.

https://preview.ibb.co/kKr1YU/indi_multi_TF.jpg

Re: Demo vs MT4 vs FSB vs EA Studio

DoCZero,

Thanks for all your work on this.

Posts: 26 to 39 of 39

Pages Previous 1 2

You must login or register to post a reply

Forex Software → Premium Club → Demo vs MT4 vs FSB vs EA Studio

Similar topics in this forum