1 (edited by DoCZero 2018-07-28 13:34:53)

Topic: Demo vs MT4 vs FSB vs EA Studio

Hi Guys,

After reading the endless posts where people post "FSB doesn't match live" I thought I would put together a test to really see if this was the case.  In essence - a backtester compares the strategy against historical data - it's mathematical therefore as accurate as the code , data and setup parameters being run. The way the ticks are replayed (real tick vs hypothetical replay) , how orders are handled (slippage, commissions and spread) and the accuracy of the strategy execution (Ea code against historical data) can all be areas which impact the accuracy of a backtest.

I thought to validate or invalidate the backtesters - you need to start with a very simple mathematical strategy. Complex strategies with many variables will add another area which could cause a difference between live and backtest (especially with the risk of coding errors in the indicators between the historical backtest and the live EA you run).

The System setup for this test:

10 / 20 MA crossover system   
Buy when 10 MA crosses above 20 MA
Close and reverse when 10 MA crosses below 20 MA
No stop or takerprofit.

(please note - EA Studio doesn't have close and reverse, opposite rules were used so execution may be different).


The other variable is the computer power (for execution) and distance from broker (slippage issue). The backtesters give you a hypothetical fill (which may or may not be true) - these need to be setup with an average slippiage and comissions to get closer to the real results from a live account. Demo doesnt have the same issues with slippage (as Demo fills are instantanious - although forex fills are also very instant, due to liquidity - try this on some stocks and it may not be the same results). Pleae note I dont run VPS's for trading with MT4- In my experiences, VPS's often slow down and dont execute - I prefer to pay solid money for a physical PC or server (not a shared machine) based right near the exchange / broker.

Hardware:

DEMO  (Dedicated Server - based in NY)
i7 4890k
32gb Ram
SSD 128 GB
1.2ms Ping to my broker (GLOBAL PRIME).


MT4 / FSB / EA Studio
i7 7700k
32gb Ram
3TB SSD

DATA:

On MT4 I used Tick Data Suite to download the latested dukascopy tick data. This data was converted to both FSB and EA Studio.  Please note EA Studio - I had some issues with the 1 min data (bug report sent) im not sure why , but ended up using the preloaded default data. It was surprisingly close to the tick data and demo data.

To also understand the difference in execution / backtest I have run the test on many pairs:  USDCHF / GBPUSD / EURUSD / USDJPY / USDCAD / AUDUSD / SGDJPY / NZDUSD

And multiple time frames:  1m  / 5m / 15m / 1H

The period of time this was run was from the 25th of June until the 27th of July.

Below I will post some pictures from the test - If you would like a copy of: The EA , The demo data (either Excel or MT4 report), other pair data please PM me. Also I will post some of my observations and understandings after I post a block of pictures.

Stay tuned smile

PT

Re: Demo vs MT4 vs FSB vs EA Studio

https://preview.ibb.co/mRZ7zT/USDCHF1M.jpg
https://preview.ibb.co/b1opDo/USDCHF1H.jpg
https://preview.ibb.co/fBZ7zT/USDCHF5M.jpg
https://preview.ibb.co/c79J68/USDCHF15_M.jpg

This is the first batch from USD CHF.  Some of my observations are as follows:

- All equity curves per time frame between the Demo vs backtest software are similar. 
- All back testers seem pessimistic in their results (i.e show a worse scenario than actual).
- FSB Pro and MT4 are more pessimistic than EA Studio.
- No timeframes had any backtester matching the amount of trades of the Demo.
- The variation between timeframes (with M1 having the biggest variance, and 1H having the least) can't be considered so deeply as the sample size at the higher time frames is too small for credibility.

Re: Demo vs MT4 vs FSB vs EA Studio

https://preview.ibb.co/iyvBto/GBPUSD1H.jpg
https://preview.ibb.co/bGvpeT/GBPUSD1M.jpg
https://preview.ibb.co/gMhrto/GBPUSD5M.jpg
https://preview.ibb.co/kTOxYo/GBPUSD15_M.jpg

-I find it interesting that the MT4 Backtester and FSB execute more trades than demo - additionally EA Studio executes less.  Still the equity curves are very similar.
- In the 1 Hour charts , FSB and MT4 execute 20%+ more trades - where as EA Studio is fairly spot on.
- In the 5min charts - FSB and MT4 are clearly pessimistic, where both stayed underwater, DEMO + Eastudio got into positive territory at similar times.

Re: Demo vs MT4 vs FSB vs EA Studio

https://preview.ibb.co/dM1qm8/EURUSD1H.jpg
https://preview.ibb.co/nr3gto/EURUSD1M.jpg
https://preview.ibb.co/dOLi68/EURUSD5M.jpg
https://preview.ibb.co/d9RO68/EURUSD15_M.jpg

Now for the big boy - EUR USD

- EA Studio on the 1 hour matches the DEMO accurately - FSB and MT4 are again pessimistic.
- On 1 MIN data - Demo / FSB and MT4 are fairly close in number of trades executed - EA Studio seems way off (i assume thats the difference in exit). Please note, the Equity curves are still similar.
- On M5 none of the backtesters match the DEMO exactly. The equity curve style is still similar.
- on M15 the trade number is very close across all - but the equity curves are very different. This points to difference in execution (possibly slippage, commissions or spread at play).

Re: Demo vs MT4 vs FSB vs EA Studio

I wont post the rest unless someone really wants me to - but at this stage I can draw the following conclusions:

- In my opinion, the differences in execution between the demo and 3 backtesters does not invalidate the backtesters. I would be confident in saying that a system shown in the backtester then compared with Demo - the demo always seems to perform better (therefore the risk in using the backtester is mitigated).

- The setup of the backtester is critical - if you dont get the right spread / slippage or comissions - your performance wont be close to the DEMO.

- EA Studio - in my opinion - seems to be the closest match to DEMO.

- The 1 hour time frame was easier to be profitable (with a simple system) - but please note , the sample size is very small (vs the 1min) so this comment is only an observation for this test.

- I was suprised there wasnt much difference between the pairs (i was expecting to see bigger variance on more exotic pairs , like SGDJPY for example).

- The data source you use is critical - it is probably good practice to run a test on demo vs backtester to validate your data source.

- Paying for tick data (tick data suite), in my opinion, didnt yield such a great difference vs the free data (already in EA Studio).

- The speed execution from slowest to fastest:   DEMO < MT4 < FSB < EASTUDIO.

Final point - is to share that this exercise has given me confidence in the tools I am using.  This doesnt give any validity into making profitable systems moving forwards - but checking historical execution of demo vs the backtest yields very similar results.

I'd love to hear your thoughts smile

PT

Re: Demo vs MT4 vs FSB vs EA Studio

Wow -- DoCZero, this is really excellent.  Thank you for taking the time to perform the tests, preparing the results and sharing them.  Now we have something definitive we can refer to.

I want to look through your results again more closely, but I did have a few initial comments:

1. I like that you included the MT4 back tester.  People always trash it, but I do rely on it as a "sanity" check before deploying a portfolio.

2. I have created many strategies that give excellent back test results both in FSB and MT4 Strategy Tester -- yet when deployed to a Demo or Real account totally bomb.  So -- I'm pretty sure there is something else going on. And I suspect that is caused by *overly* curve-fitted strategies -- but there is no rule I am aware of to spot these ahead of time.

3. How well does a Demo account match with a Real account?  For me this is "when the rubber hits the road", as Blaiserboy would say.  What your testing has confirmed is that FSB and EA Studio are good at creating strategies that perform consistently across back testers and Demo accounts.  However, I would still claim there is a "disconnect" between Demo and Real accounts -- though I haven't done testing as thoroughly as you have.

7 (edited by DoCZero 2018-07-28 18:29:15)

Re: Demo vs MT4 vs FSB vs EA Studio

Hey Steve,

First of all - Thanks for http://mt4tracker.com/ smile Without it , I couldnt have easily compared the Demo data with the other systems! Awesome bit of work mate!

For point 1 - please note i'm using the Tick Data Suite (also known as birt's tick data) - which increases the backtest reliability in MT4. I have some additional code I add to FSB or EA Studio's EA's which manages the position sizing for me. Unfortunately I cant have this code added to FSB or EA Studio as it is quite complex. But testing MT4 is important to see it is still viable (if not the code I add could invalidate the system). It also shows MT4 could be another step in validating your system (think different data set).

for point 2 - yes curve fitting is very easy to do with a data mining style system generator. It is possible to use the software without curve fitting but it is very easy to curve fit. The rule to spot this ahead of time is actually incubation (which is talked about in this forum) - running an EA on Demo for 6+ months before allocating real money to it! ... unfortunately - i'm also very skilled at building systems which perform as you say hehe

Now point 3 I haven't looked at yet.  I am currently running around 60 systems with real money - but none is what I would consider simple (unfortunately). 'Im tempted to run the MA test on a live account with the smallest position size possible - but Im sure i'd be burning money ... heheh what to do  smile  - i'm sure this is the next step I will be taking.

I guess the key point here is that the backtesters (FSB / EAS or MT4) arent bad - they arent 100% accurate - and I think its more a situation of selecting the correct variables (Slippage, spread and commissions) vs them not working. This is a nice thing to know as it really gives some confidence in the work being done.  Curve fitting is a totally different topic (as I demo tested then validated with backtester. Most of the time we are building with backtester then hoping demo/live works.. Curve fitting is the challenge I am sure).

PT

8 (edited by hannahis 2018-07-29 15:04:44)

Re: Demo vs MT4 vs FSB vs EA Studio

Thanks PT for your extensive testing....give me some time to digest your findings and to understand the implications behind your findings.  I'll have questions that I'll like to ask you directly for clearer understanding cos it's hard to discuss something like this over the text.  I'll What's app you.

9 (edited by hannahis 2018-07-29 15:18:38)

Re: Demo vs MT4 vs FSB vs EA Studio

DoCZero wrote:

Now for the big boy - EUR USD

- EA Studio on the 1 hour matches the DEMO accurately - FSB and MT4 are again pessimistic.

I'm surprise that FSB and MT4 are "pessimistic" (you mean the trading results are worst off?).  I though most of the users have good FSB backtesting results and when they test it out in demo/live, their strategies in demo results are more "pessimistic" than the FSB's backtesting results.  Are did I misunderstand your statement? (or maybe, as you mentioned, it's the input of the spread, commission, slippage issue that cause the "disconnect"?

This make me wonder, maybe I should set my spread and commission a bit higher than "usual" to improve my trading results (to compensate for the variable spread issues, while FSB Pro uses fixed spread)?

Re: Demo vs MT4 vs FSB vs EA Studio

Hi Hannah,

To explain my comments - Pessimistic meaning worse of (probably in terms of trade execution).  I use the word pessimistic because Its as though all execution , slippage or spread was purposefully worse off than the DEMO. I actually see this as a positive thing because you want the backtester to be executing worst than reality (as in reality there are many potential issues - if the backtester executed better , you would get an unrealistic result).

But the interesting points I saw were as follows:

1) On the 1 hour equity curve of EURUSD , trade 25 of DEMO really returns a lot more than on the other platforms. I havent analysed the time and event data, but I imagine this is due to execution in a faster moving market (Still not sure).

2) Also on the demo 1 Hour here is trade 21>22 it shows one step down, but on the other 3 platforms it shows 2 steps down. I wonder if this is an issue with the demo not executing every signal (I'm not sure the reasoning - but there is a chance for errors in EA execution - so I still feel this result is valid, as that probably would happen in live).

3) On the 1 minute , the number of trades between DEMO / FSB & MT4 are very similar - this is good , but EA Studio is very different (I can only imagine it is due to the lack of "close and reverse" as the exit strategy). Still the bigger take away is the equity curve - EA Stuido's result is still comparitive - showing that the overall performance of the system shines through, even when things dont execute 100%. This gives me more trust in EA Studio.


4) on the 5 minute charts - around trade 320, you can see when the system goes in sync with the market on DEMO we get back into profitability.  But none of the backtesters get close. This is good - because it will keep you away from mediocre strategies. (It also makes me wonder if DEMO is somehow making it easier to make money????)


Regarding setting your spread and comission higher than "usual"  - I would recommend this , I have done the same - as we really want live execution to be better than the backtester. Therefore backtested system which pass onto live should perform better. Please note MT4 has variable spread when using Tick data suite - but I didnt see so much of an impact when comparing vs non-variable spread.

The obvious next step is comparing Demo vs Live. At this stage I dont want to burn any cash with such an exercise.  To keep this exercise valid - you would still need to test 1M / 5M / 15M and 1H  to then compare the performance.  The 1M system would just kill the account too fast.  The reason I use the backtesters is to keep away from this situation.

Anyway Thanks for checking it out. Very interesting topic for me - but the big takeaway for me is that this gives credibility for the backtesters vs Demo.

smile


PT

11 (edited by hannahis 2018-07-30 07:17:23)

Re: Demo vs MT4 vs FSB vs EA Studio

Hi PT,

I think you may have missed out my point.

I understand that the FSB Pro and MT4 results is more pessimistic, i.e. worse off compare with Demo results right?

In another words, your finding suggest that Demo testing show better trade outcome than FSB Pro backtesting results.

But in reality, users experience shows that their demo/live trading results are worst off then FSB Pro backtesting results (which supposed to be worst off, pessimistic).

Lastly, I want to emphasize (as you have mentioned before) the limitations of this testing (as all testing operated under certain assumptions/limitations).  So that if we want to use the implication of this test findings, we need to take note of the following issues.

1. As you have pointed out, this testing is based on using SIMPLE strategies and hence, the same outcome cannot be implied for people who use complex strategies such as multi time frame/Longer Time Frame logic

2.  This testing assumed that one set the spread, commission "correctly" i.e. Broker's variable spread vs software fixed spread, in order to produce a backtesting as close as the demo results?

Thanks once again for taking your time to do such extensive testing and generous to share your findings here

Re: Demo vs MT4 vs FSB vs EA Studio

Ahhhh yes this is a key observation!  We will underline it so it sticks out wink   (and i'm good at missing points hehe).

users experience shows that their demo/live trading results are worst off then FSB Pro backtesting results

This is now a worth while topic!  So what did we do for the above test:

Experiment
1) We designed a system.
2) Demo tested it (to get the results)
3) Validated it with the backtest.

What does the general FSB Pro / Ea studio user do:

Gen User
1) Generate a system.
2) confirm suitability with backtest data.
3) Attempt to live trade  (and the find the future is very different).

So the differences.  With the Experiment we are not trying to look into the future.  The demo test happens before the validation. Therefore the experiment confirms that the Demo and Backtest match in the past.

We don't want FSB or EA Studio to do that!

What we want them to do is to generate a system which works in the past and KEEPS WORKING into the future!!! But the issue we fight against is that no one knows the future (2 months ago - did traders know Facebook would drop 16% on friday?). 

So therefore the systems we build have to be around a theory which could be:

1) Based around a market structure theory (repeatable patterns which are exploitable).
2) Constantly re-adjusted to keep the EA's working as the markets change.
3) Very loosely created (i.e not sensitive to changes with market regime). 

and not be around:

1) Data mined / Generated
2) Highly optimized for previous data.

The issue that people are seeing is that on new data - the idea fails. Curve fitting or the other reason is that they didnt run the system live long enough  (i.e you look at a 5 year backtest , but expect the system to continue to work perfectly in the next 30 days.. the 30 day drawdown could be still in the acceptable range of the system , but you only see a loss over 30 days and kill the system).

My experiment was designed to show that this isn't a fault of the software backtester.  This is more a fault in the way the system has been built. I think EA Studio is going in the right direction with Walk forward testing. Still I am currently convinced that Generated systems (data mined) are more prone to failure in the short term. It comes back to the acceptance criteria on the generated system that will improve the chance of working into the future - and also a validation period via demo testing which will keep you safe.

For your last 2 points:

You have to run this experiment with simple systems first. There is no point gravitating to a complex strategy up front (if you cant prove the simple system works).  On a complex strategy - longer timeframe or multiple time frame wont be the issue. The issue will be introduced coding errors on the indicators therefore different execution between the two.  To explain - FSB has an indicator coded in 1 way , MT4 is in a different language and is effectively a different set of code. The issue is between the two - coding /mathematics errors may come into play. FSB runs the indicator perfectly / MT4 code is slightly different.   It wouldn't be hard to test a complex system - and I am sure the results will be fine , just the variables change. I chose a moving average crossover system as such a simple idea should have a low chance of coding errors.

2) Absolutely.  If you vary the spread, commission or slippage - you will see a similar execution but each profit or loss with be different. Trade times should match.


I'm still working on many ideas at once - overcoming curve fitting is probably the hardest job, but at least I know its not an execution issue with the backtester smile

Re: Demo vs MT4 vs FSB vs EA Studio

Thank you DoCZero for your contribution! Excellent job!! Hannah and Steve - good comments!

I only want to add minor things.

- using exponential or weighted indicators. The traders frequently forget that these indicators give different signals depending on the length of the Data series. It may or may not make the difference on the tests and the trading. EA Studio "doesn't recommend" such indicators by default.

- bugs in the indicators are possible and do my best to rectify them. Please report any observations. By "bugs" I mean not only programming errors but different behavior of the indicators between MT and FSB / EAS.

- the MT backtesting is a necessary step before running a strategy Live (except of a thorough Demo testing before that). However, the MT Tester has some quirks. The EA Studio strategies must be easier for testing.

Re: Demo vs MT4 vs FSB vs EA Studio

So the differences.  With the Experiment we are not trying to look into the future.  The demo test happens before the validation. Therefore the experiment confirms that the Demo and Backtest match in the past.

Good point PT, I missed that out. 

Yes, then your test finding is very reassuring cos it's shows you to what extend you can rely on FSB Pro/MT4/EA Studio backtester and which time frame would yield more "reliable" results such as H1 EA Studio has the closest match.

Can you kindly elaborate more on your findings on 1min time frame cos that's my favour time frame and more findings or comment about the matching issue, variance etc?

Re: Demo vs MT4 vs FSB vs EA Studio

Hi Hannah,

I think there is a few parts to point 1 - Reliable results in terms of accuracy of trade execution - Its hard to say because the sample size changes in between. My observation is I think all executions are similar (in the time frames) so you could trade any time frame. 


The second part to point 1 could be - which time frame would build a better system.  And my honest answer is that I dont know - would have to test effectiveness of building non-curve fit EA's in each time frame.  The Way to analyze would be the number of non-curve fit systems generated in a set criteria over a specific set of time.  This would be a very worth while analysis (also one that would take an even longer time to test and validate the test).


For your second question -  I'll run a test with 1 min, more complex and using multiple time frames.  Come back to you soon wink

PT

16 (edited by sleytus 2018-08-02 08:41:02)

Re: Demo vs MT4 vs FSB vs EA Studio

Hi PT,

I've been thinking about your results some more -- again, thanks very much.  As I consider my own approach to trading and what I plan to do in the future, your test results will play an important role.

There is a related issue I've run into and thought I would pass it along -- but not to make more work for you.

I've been pretty good about using my broker's data when generating and optimizing strategies.  I would go to my Demo account, download the latest *.csv files (for FSB Pro) and add them to FSB-Pro's data folders.  And prior to deploying a portfolio EA I would do a "sanity" check and run MT4 Strategy Tester from my Demo account.  And everything would continue to look good.

And then a couple of months ago when I performed a "sanity" check the results were horrible -- and I couldn't figure out why.  Turns out I ran the "sanity" from my Real account.  When I ran the sanity check from my Demo account then results looked good again.

So, I learned the data that we download from Demo accounts can be different than data downloaded from a Real account.  I tested this on another broker and the difference between Real and Demo data wasn't as marked -- so, this may vary from broker to broker.  Anyways, now I only use data from a Real account when generating and optimizing strategies.  And I wonder if you had used data from your Real account when generating and optimizing strategies if those would perform similarly in your live Demo account -- do you know what I mean?

A quick test would be to take a portfolio EA and back test it using MT4 Strategy Tester within both Demo and Real accounts to see if there is a difference.  Perhaps others could try this as well and let us know.  Maybe it was only this one broker I use -- or maybe this is a general issue.  I don't know.

Re: Demo vs MT4 vs FSB vs EA Studio

Hi Steve,

Thanks for adding to the conversation - I am thinking along the same lines that a demo vs live trade - and live vs FSB / EAS & MT4 is a necessary test.

I'm not using my broker data anymore - i've been running with Dukascopy tick data. My thinking currently is that the system robustness will show when you swap data between brokers. There will be some difference of course (as the market is not a single regulated market but buckets of clearing houses) but see a system robustness when swapping the broker data on the same pair / timeframe.

I will run some tests on these - I have FSB strategies that I have been running live for a while - and it is easy for me to compare FSB / MT4 / Live.   I would have to setup a new test for DEMO Vs Live.  I've got quite a testing backlog (as always) but this is high on the priority list.  Will set something up and keep you posted

Cheers,

PT

Re: Demo vs MT4 vs FSB vs EA Studio

Hi PT,

A couple comments / questions...

1. From your results is it fair to say that OHLC data (e.g. used by FSB and EAS) is good enough?  In other words, since OHLC data sources used by FSB and EAS yield similar results as your Demo account, then does tick data offer any advantages?

2. Suppose you were given a choice between two data sources: (a) tick data that was one year-old (i.e. accurate, but a bit out-dated), or (b) OHLC bar data from the most recent 6 months (i.e. less accurate, but more closely reflects current data patterns), which would you choose?

And since you touched on a related issue -- I can't help bringing this up. We all have different approaches, but from a mile high it seems there are two camps -- those who create strategies *once* and hope they are sufficiently robust to stand the test of time, and those who requirements are less stringent because they are prepared to continually "refresh" their strategies and turn them ON / OFF depending on how they respond to current data patterns and market conditions.  I definitely fall in the latter group.

Regards,
Steve

Re: Demo vs MT4 vs FSB vs EA Studio

DoCZero wrote:

Hi Steve,

Thanks for adding to the conversation - I am thinking along the same lines that a demo vs live trade - and live vs FSB / EAS & MT4 is a necessary test.

I'm not using my broker data anymore - i've been running with Dukascopy tick data. My thinking currently is that the system robustness will show when you swap data between brokers. There will be some difference of course (as the market is not a single regulated market but buckets of clearing houses) but see a system robustness when swapping the broker data on the same pair / timeframe.

I will run some tests on these - I have FSB strategies that I have been running live for a while - and it is easy for me to compare FSB / MT4 / Live.   I would have to setup a new test for DEMO Vs Live.  I've got quite a testing backlog (as always) but this is high on the priority list.  Will set something up and keep you posted

Cheers,

PT

PT,

Thanks for your contributions.

I'd like to confirm what you said. 

To create strategies, you are not using data from your broker but instead data from Dukascopy??  And your EAs are robust enough?

Thanks,

Samuel

Re: Demo vs MT4 vs FSB vs EA Studio

Another point I forgot to make...

A while back, in a different thread, I had questioned why some people advise the importance of using broker data -- I wanted to know why.  And I believe it was Irmantas who mentioned the most compelling reason -- certain indicators use time filters.  If your strategies use one or more of these and if your broker's time zone is different than the time zone of the data source being used for generating, optimizing and back testing, then this becomes a problem.  So -- if you are not going to use your broker's data I think it is important to (a) confirm your broker's time zone is the same as the data source's, or (b) ensure that you avoid using indicators that rely on time filters.

For FSB I have data sources from several different brokers.  When I load a strategy and switch from one broker to another the balance and equity curves do change.  Whether it is enough to significantly alter trading performance, I couldn't say.  However, I do think the time zone issue must be taken seriously.

21 (edited by DoCZero 2018-08-04 09:54:59)

Re: Demo vs MT4 vs FSB vs EA Studio

OK - so a few questions to answer... here we go:

1)  OHLC data vs Tick Data   Depends really on your system - But to give you an idea - The MT4 backtest is run on Tick data.  FSB & EA Studio is OHLC 1min Data - results are very close - therefore my observation would be that there is minimal difference in the 2 for the above strategy (moving average cross with bar closing as the entry point).

2) 1 year tick data vs 6 month OHLC for those 2 choices - i'd chose the OHLC with the up to date data.  Personally I think the rule is the more data the better (I try feed FSB with 10+ years of 1min data, please note I don't test with a 10 year data horizon - I like to work different time brackets and have the last 6-12 months as OOS). But the last few months is the closest to the current market regime (providing it continues).

3)  Steve's comments about 2 camps   I think there is more than 2 camps - I run multiple ideas, one basket systems stops the system before it fails. Another basket I use money management to minimize draw downs  - and they are based on systems that have a strong performance history over a long period of time. The third camp is different - its about using FSB to find bunches of indicators that give the direction of the market for the next N bars (hehe there you go - but believe me , this is very hard to do - i've run trillion's of calculations on this topic and still am not satisfied).

4) Do I use data from my broker  and are my EA's robust enough    This topic of data from the broker is a tricky one.  Ok lets drill into it - Most of us here will have OTC trading accounts (over the counter) - where a broker is matched with a liquidty provider through a true ECN (electronic communications network) exchange such as Hotspot (run by BATS).  A lot of the market advertising says you trade on ECN - but its just marketing.  You will find that most brokers (dukascopy / global prime / tickmill / etc) have a style where they either warehouse the trades (which means they take the other side and hedge - not specifically against your trade, but taking the opposite position of many many traders as a portfolio - using a calculation to minimize their risk) and also take a % directly to their liquidity provider.  Before we start the "I use an ECN discussion"  unless you have a minimum USD100k Account and are doing 100 lots per day - you probably don't have an ECN account, the other way to tell is if you get level 1 / 2 / 3 (market depths) or bank visibility.

Ok - so now we understand there is actually 2 views of the market.  ECN is showing open orders in the market from liquidity providers,  OTC is showing the price and bid ask spread from your broker. 

In the past the brokers were much more aggressive in the tactics of widening spreads, varying from the average price on the ECN and slippage. This has tightened up a lot with the regulations coming in the last 2 years (In Australia its ASIC , USA its FIRA and Germany FFSA) - you will find if your broker is registered in those countries it has to comply with many regulations.  My broker is Australia - they are required to keep records of every trade (and what the price did during that trade)  - if I raise a complaint to the regulatory authority - the broker is required to present that data, and similar data from other traders - to ensure I was treated identical to the other traders.

So the topic of using your broker data was a big topic when you had all kinds of broker infulence going on - these days its minimized. Dukascopy is regulated by SFMSA - therefore their data should be a good representation of the OTC market. I am not concerned about my broker changing prices and widening spreads unethically due to the ASIC regulations - therefore my broker data and Dukascopy should be very similar. 

If you were using an unregulated broker (say one registered in a tax haven) - prehaps you should be more careful.

The last point - Demo = My brokers data,   MT4 = Duksacopy tick    EA Studio + FSB = Dukascopy 1 min... The results are very similar for the system.  If a system is so sensitive that minuscule changes between the brokers stops it from working - I wouldn't want to use it in the markets. 

5) Time settings   Yep - which is why when you setup your FSB , ask your broker what time adjustment to use smile   Personally I don't use time based filters or exits.

Re: Demo vs MT4 vs FSB vs EA Studio

PT -- this is way, way too excellent.  You answered all my questions and brought a smile to my face.  In this one post I've learned years worth of valuable information.  Thank you very much.

Re: Demo vs MT4 vs FSB vs EA Studio

Hey Steve,

smile My pleasure!

I want to clarify one point The MT4 backtest is run on Tick data.   This only happens because I am using Tick Data Suite. Honestly I don't think it is necessary but my process includes - FSB ->  MT4 Backtest -> Demo Test -> live  in that order smile

Re: Demo vs MT4 vs FSB vs EA Studio

To add on the topic of Data - One area you have to be careful is how clean is your data. I was in a skype call with one of the people from this forum - and we were looking through bar by bar of historical data for many years, and it was amazing the amount of gaps and potential data issues.  Using a data that is of quality is a must , but how to understand what is quality and what is not?  Personally I eyeball the charts for gaps, bad ticks and large gaps in dates / time at the lowest time frame. It is time consuming because when you find a gap - you then have to search what happened at that date etc. Hard topic and data it something I'm sure I can improve, this is another reason I like cross checking across multiple platforms- but the best cover is to incubate your systems before running them live. Then you can check does the historical performance and the demo performance seem in a realistic range.

Re: Demo vs MT4 vs FSB vs EA Studio

Hi Hannah & All Interested.

For the last 3-4 weeks i've been running a more complex EA using 5 indicators at different timeframes.  This EA is setup int he following way:

1m Bar Chart
Bar Opening
1 Stoc Indicator H1
1 Stoc Indicator M30
1 Stoc Indicator M15
1 Stoc Indicator M5
1 Stoc Indicator M1
Exit close and reverse.

I ran the test on 8 different currencies.  Please find below the difference between DEMO VS MT4 VS FSB.  Please note I did't post all photos - the outcome is similar

https://preview.ibb.co/jU20TK/EURUSD.jpg
https://preview.ibb.co/dYFn8K/GBPUSD.jpg
https://preview.ibb.co/fFAXaz/USDCHF.jpg

My observations are that the execution is fairly exact. Please note the MT4 backtest is VERY VERY VERY slow. This EA has 6.5k lines of code so I think it is fairly bloated in its execution.. the 4 week test (single backtest - using tick data) took around 30 min!!! Using MT4 to validate your strategies isn't optimal and I'm unsure if it can fit in a process to validate systems.

Please note the Stochastic  indicator is the default one in FSB - This means that the FSB code and the MT4 code will execute identically (due to popov's attention to detail).

I have however been running 10+ plus complex tests and getting totally different results. This is using more indicators (downloaded and non-default) and I think they are having an impact on the accuracy.  Please see below comparison chart:

https://preview.ibb.co/kWXW2e/C1.jpg
https://preview.ibb.co/cO2Lvz/Time2.jpg

I am very worried about this result - as it makes me unsure what to trust. The left hand side is MT4 and right is FSB.  The EQ curves are totally different but it is the same system. Also take a look at the time to test (Yep 20 hours.. UGHHH).  At the moment I think I have more investigation to do. I will also need to create something complex in EAS and to a long time comparison with MT4.. I believe the MT4 backtest will be faster as the code is a lot lighter.