This is the first batch from USD CHF.  Some of my observations are as follows:

- All equity curves per time frame between the Demo vs backtest software are similar. 
- All back testers seem pessimistic in their results (i.e show a worse scenario than actual).
- FSB Pro and MT4 are more pessimistic than EA Studio.
- No timeframes had any backtester matching the amount of trades of the Demo.
- The variation between timeframes (with M1 having the biggest variance, and 1H having the least) can't be considered so deeply as the sample size at the higher time frames is too small for credibility.

Now for the big boy - EUR USD

- EA Studio on the 1 hour matches the DEMO accurately - FSB and MT4 are again pessimistic.
- On 1 MIN data - Demo / FSB and MT4 are fairly close in number of trades executed - EA Studio seems way off (i assume thats the difference in exit). Please note, the Equity curves are still similar.
- On M5 none of the backtesters match the DEMO exactly. The equity curve style is still similar.
- on M15 the trade number is very close across all - but the equity curves are very different. This points to difference in execution (possibly slippage, commissions or spread at play).

Wow -- DoCZero, this is really excellent.  Thank you for taking the time to perform the tests, preparing the results and sharing them.  Now we have something definitive we can refer to.

I want to look through your results again more closely, but I did have a few initial comments:

1. I like that you included the MT4 back tester.  People always trash it, but I do rely on it as a "sanity" check before deploying a portfolio.

2. I have created many strategies that give excellent back test results both in FSB and MT4 Strategy Tester -- yet when deployed to a Demo or Real account totally bomb.  So -- I'm pretty sure there is something else going on. And I suspect that is caused by *overly* curve-fitted strategies -- but there is no rule I am aware of to spot these ahead of time.

3. How well does a Demo account match with a Real account?  For me this is "when the rubber hits the road", as Blaiserboy would say.  What your testing has confirmed is that FSB and EA Studio are good at creating strategies that perform consistently across back testers and Demo accounts.  However, I would still claim there is a "disconnect" between Demo and Real accounts -- though I haven't done testing as thoroughly as you have.

Thanks PT for your extensive testing....give me some time to digest your findings and to understand the implications behind your findings.  I'll have questions that I'll like to ask you directly for clearer understanding cos it's hard to discuss something like this over the text.  I'll What's app you.

Hi Hannah,

To explain my comments - Pessimistic meaning worse of (probably in terms of trade execution).  I use the word pessimistic because Its as though all execution , slippage or spread was purposefully worse off than the DEMO. I actually see this as a positive thing because you want the backtester to be executing worst than reality (as in reality there are many potential issues - if the backtester executed better , you would get an unrealistic result).

But the interesting points I saw were as follows:

1) On the 1 hour equity curve of EURUSD , trade 25 of DEMO really returns a lot more than on the other platforms. I havent analysed the time and event data, but I imagine this is due to execution in a faster moving market (Still not sure).

2) Also on the demo 1 Hour here is trade 21>22 it shows one step down, but on the other 3 platforms it shows 2 steps down. I wonder if this is an issue with the demo not executing every signal (I'm not sure the reasoning - but there is a chance for errors in EA execution - so I still feel this result is valid, as that probably would happen in live).

3) On the 1 minute , the number of trades between DEMO / FSB & MT4 are very similar - this is good , but EA Studio is very different (I can only imagine it is due to the lack of "close and reverse" as the exit strategy). Still the bigger take away is the equity curve - EA Stuido's result is still comparitive - showing that the overall performance of the system shines through, even when things dont execute 100%. This gives me more trust in EA Studio.

4) on the 5 minute charts - around trade 320, you can see when the system goes in sync with the market on DEMO we get back into profitability.  But none of the backtesters get close. This is good - because it will keep you away from mediocre strategies. (It also makes me wonder if DEMO is somehow making it easier to make money????)

Regarding setting your spread and comission higher than "usual"  - I would recommend this , I have done the same - as we really want live execution to be better than the backtester. Therefore backtested system which pass onto live should perform better. Please note MT4 has variable spread when using Tick data suite - but I didnt see so much of an impact when comparing vs non-variable spread.

The obvious next step is comparing Demo vs Live. At this stage I dont want to burn any cash with such an exercise.  To keep this exercise valid - you would still need to test 1M / 5M / 15M and 1H  to then compare the performance.  The 1M system would just kill the account too fast.  The reason I use the backtesters is to keep away from this situation.

Anyway Thanks for checking it out. Very interesting topic for me - but the big takeaway for me is that this gives credibility for the backtesters vs Demo.

PT

Ahhhh yes this is a key observation!  We will underline it so it sticks out    (and i'm good at missing points hehe).

users experience shows that their demo/live trading results are worst off then FSB Pro backtesting results

This is now a worth while topic!  So what did we do for the above test:

Experiment
1) We designed a system.
2) Demo tested it (to get the results)
3) Validated it with the backtest.

What does the general FSB Pro / Ea studio user do:

Gen User
1) Generate a system.
2) confirm suitability with backtest data.
3) Attempt to live trade  (and the find the future is very different).

So the differences.  With the Experiment we are not trying to look into the future.  The demo test happens before the validation. Therefore the experiment confirms that the Demo and Backtest match in the past.

We don't want FSB or EA Studio to do that!

What we want them to do is to generate a system which works in the past and KEEPS WORKING into the future!!! But the issue we fight against is that no one knows the future (2 months ago - did traders know Facebook would drop 16% on friday?). 

So therefore the systems we build have to be around a theory which could be:

1) Based around a market structure theory (repeatable patterns which are exploitable).
2) Constantly re-adjusted to keep the EA's working as the markets change.
3) Very loosely created (i.e not sensitive to changes with market regime). 

and not be around:

1) Data mined / Generated
2) Highly optimized for previous data.

The issue that people are seeing is that on new data - the idea fails. Curve fitting or the other reason is that they didnt run the system live long enough  (i.e you look at a 5 year backtest , but expect the system to continue to work perfectly in the next 30 days.. the 30 day drawdown could be still in the acceptable range of the system , but you only see a loss over 30 days and kill the system).

My experiment was designed to show that this isn't a fault of the software backtester.  This is more a fault in the way the system has been built. I think EA Studio is going in the right direction with Walk forward testing. Still I am currently convinced that Generated systems (data mined) are more prone to failure in the short term. It comes back to the acceptance criteria on the generated system that will improve the chance of working into the future - and also a validation period via demo testing which will keep you safe.

For your last 2 points:

You have to run this experiment with simple systems first. There is no point gravitating to a complex strategy up front (if you cant prove the simple system works).  On a complex strategy - longer timeframe or multiple time frame wont be the issue. The issue will be introduced coding errors on the indicators therefore different execution between the two.  To explain - FSB has an indicator coded in 1 way , MT4 is in a different language and is effectively a different set of code. The issue is between the two - coding /mathematics errors may come into play. FSB runs the indicator perfectly / MT4 code is slightly different.   It wouldn't be hard to test a complex system - and I am sure the results will be fine , just the variables change. I chose a moving average crossover system as such a simple idea should have a low chance of coding errors.

2) Absolutely.  If you vary the spread, commission or slippage - you will see a similar execution but each profit or loss with be different. Trade times should match.

I'm still working on many ideas at once - overcoming curve fitting is probably the hardest job, but at least I know its not an execution issue with the backtester

So the differences.  With the Experiment we are not trying to look into the future.  The demo test happens before the validation. Therefore the experiment confirms that the Demo and Backtest match in the past.

Good point PT, I missed that out. 

Yes, then your test finding is very reassuring cos it's shows you to what extend you can rely on FSB Pro/MT4/EA Studio backtester and which time frame would yield more "reliable" results such as H1 EA Studio has the closest match.

Can you kindly elaborate more on your findings on 1min time frame cos that's my favour time frame and more findings or comment about the matching issue, variance etc?

Hi PT,

I've been thinking about your results some more -- again, thanks very much.  As I consider my own approach to trading and what I plan to do in the future, your test results will play an important role.

There is a related issue I've run into and thought I would pass it along -- but not to make more work for you.

I've been pretty good about using my broker's data when generating and optimizing strategies.  I would go to my Demo account, download the latest *.csv files (for FSB Pro) and add them to FSB-Pro's data folders.  And prior to deploying a portfolio EA I would do a "sanity" check and run MT4 Strategy Tester from my Demo account.  And everything would continue to look good.

And then a couple of months ago when I performed a "sanity" check the results were horrible -- and I couldn't figure out why.  Turns out I ran the "sanity" from my Real account.  When I ran the sanity check from my Demo account then results looked good again.

So, I learned the data that we download from Demo accounts can be different than data downloaded from a Real account.  I tested this on another broker and the difference between Real and Demo data wasn't as marked -- so, this may vary from broker to broker.  Anyways, now I only use data from a Real account when generating and optimizing strategies.  And I wonder if you had used data from your Real account when generating and optimizing strategies if those would perform similarly in your live Demo account -- do you know what I mean?

A quick test would be to take a portfolio EA and back test it using MT4 Strategy Tester within both Demo and Real accounts to see if there is a difference.  Perhaps others could try this as well and let us know.  Maybe it was only this one broker I use -- or maybe this is a general issue.  I don't know.

Hi PT,

A couple comments / questions...

1. From your results is it fair to say that OHLC data (e.g. used by FSB and EAS) is good enough?  In other words, since OHLC data sources used by FSB and EAS yield similar results as your Demo account, then does tick data offer any advantages?

2. Suppose you were given a choice between two data sources: (a) tick data that was one year-old (i.e. accurate, but a bit out-dated), or (b) OHLC bar data from the most recent 6 months (i.e. less accurate, but more closely reflects current data patterns), which would you choose?

And since you touched on a related issue -- I can't help bringing this up. We all have different approaches, but from a mile high it seems there are two camps -- those who create strategies *once* and hope they are sufficiently robust to stand the test of time, and those who requirements are less stringent because they are prepared to continually "refresh" their strategies and turn them ON / OFF depending on how they respond to current data patterns and market conditions.  I definitely fall in the latter group.

Regards,
Steve

Another point I forgot to make...

A while back, in a different thread, I had questioned why some people advise the importance of using broker data -- I wanted to know why.  And I believe it was Irmantas who mentioned the most compelling reason -- certain indicators use time filters.  If your strategies use one or more of these and if your broker's time zone is different than the time zone of the data source being used for generating, optimizing and back testing, then this becomes a problem.  So -- if you are not going to use your broker's data I think it is important to (a) confirm your broker's time zone is the same as the data source's, or (b) ensure that you avoid using indicators that rely on time filters.

For FSB I have data sources from several different brokers.  When I load a strategy and switch from one broker to another the balance and equity curves do change.  Whether it is enough to significantly alter trading performance, I couldn't say.  However, I do think the time zone issue must be taken seriously.

PT -- this is way, way too excellent.  You answered all my questions and brought a smile to my face.  In this one post I've learned years worth of valuable information.  Thank you very much.

To add on the topic of Data - One area you have to be careful is how clean is your data. I was in a skype call with one of the people from this forum - and we were looking through bar by bar of historical data for many years, and it was amazing the amount of gaps and potential data issues.  Using a data that is of quality is a must , but how to understand what is quality and what is not?  Personally I eyeball the charts for gaps, bad ticks and large gaps in dates / time at the lowest time frame. It is time consuming because when you find a gap - you then have to search what happened at that date etc. Hard topic and data it something I'm sure I can improve, this is another reason I like cross checking across multiple platforms- but the best cover is to incubate your systems before running them live. Then you can check does the historical performance and the demo performance seem in a realistic range.