I have not run comparisons, the two are different programs although very similar.

Perhaps you can provide some details of the differences you have discovered..... specifics, pictures, journals, etc so that Popov can take a look at the situation. Also the strategy which is showing the differences.. the actual xml file so that he can load it on his machine for testing.

He will be sure to look into it if you can provide materials for him to work with

Peter, are you having a lot of ambiguous bars in standard FSB?

And I strongly agree with Dave that both FSB's are very similar, I think it was verified at one point that both FSB's had the same backtesting result on the same data, which they definitely should.

Greetings, I am new to the World of FSB, FST, FSB Pro but I really love what you’re trying to do here. I am also part of the asirikuy project which is trying to do something similar but I think FSB is a lot further along. Anyway to my point..
I am having exactly the same issue as PeterChristoph, I have generated a number of strategies with FSB and some of them are showing results which to be frank are not very likely, so to me the whole back testing experience with FSB is now under some doubt.
I have attached a strategy which was generated with FSB on the AUDUSD over a Data Horizon of 2-1-2012 to the 4-2-2014 on an initial balance of $3000 the FSB back tester projected a return of $18147.16 which is pretty unbelievable I would have to say. I have also attached the history data file to match the strategy so you can test it your self.
I open the same strategy in FSB Pro with exactly the same parameters and I end up with a balance of 3657.98 which to be honest seems a lot more likely. 
I am using identical data (using the exact same data directory) so I know it’s not a data issue, I am not using the same indicators code (I am using whatever come in the installer) so that is the only difference other than the application its self. I am more inclined to trust FSB simply because its results seem to me to be more likely, but I will now perfume more testing which will take me some time.
My first test will be to compare the back test results with MT4 using the same strategy with the same data horizon, if this prove conclusive then that will be a great result and hopefully prove definitively which version is correct and which one isn’t.
if I don’t get a clear result then my next step will be to pick a strategy and run 2 demo accounts, one with FSB Pro interfacing to MT4 and the other with the MT4 version of the strategies. I will report back my findings.
Bottom line I think the back tester on FSB in particular is highly suspect. OR there is something wrong with the default Indicators which I am yet to rule out as an issue.

You are saying that if mt backtester and FSB would produce different results, FSB would be the loser?

Why don't you use real tick backtesting in FSB? MT's "every tick" isn't exactly using tick data...

So, the problem is speed? Works fast means doesn't work at all? Scanner loads up slowly, after that it works as it should - fast. Better to point to a specific spot where backtest is not correct, that would be a way forward in every respect.

What do you mean by "very good" strategies? Expectancy to remain profitable or reliable backtest?

What did you exactly test? FST trades compared to FSB's backtest?

do you use different date periods or do you rely on one screen for OOS.

Myself, I prefer to use a Monte Carlo to get an idea as to worst case....  to rely on OOS may not be  good except for an estimate.

Equity curve is what I want to see doing well and I look at several periods to gauge that.

I would think that if you do a backtest in MT4 you will see the quality of your data.

MT doesn't offer not even remotely close to what one would call reliable backtest. Full stop.

FSB is not really non-transparent, mind you, its source code is freely available! You said you can code, I don't see a problem why one couldn't work it out for oneself.

I think you want to know how the generator works, not the optimizer?

You seem to take FSB Pro as a finished product, which it isn't. It's still a beta, it doesn't have all features incorporated, it has bugs to iron out etc.

I think you can have a look at the code on the web and decide where it can be changed..... and then modify your own copy. I do not have the knowledge to affect a change in the code.

I have had good luck using FSB, I feel it to be reliable.

Proviso is.. not too many things to be optimized..... a couple variables only, same as any system.

Robustness achieved by running expert on more than a couple pairs.

I use standard settings only to avoid overfitting. I have only indicators that come with the initial download and quite a few of them are blocked out.

I find that I am doing better with FSB as I can run two or three at a time.

When you mention 'inaccuracy'  do you have any specific metric that would describe  the area of doubt...

What part of the scanning do you feel to be inaccurate, can you elaborate further.

I well know you are not a critic..... I understand that we have to delve into this thing further and that all of us will benefit.

If we do not question. we will never be fully confident.

If you can spell it out exactly what you question, the we can ask Popov to reply, hopefully his answer will set us at ease.

Thanks

I have an impression that some think intrabar scanner should be working all the time, thus reducing FSB's speed, but it is used only when there is a need for it - in the cases of ambiguous bars.

Cooffeesnoob, download the tick data, allow its usage in the scanner and it should be fine. Read more here:
http://forexsb.com/wiki/fsb/manual/jforex_data
http://forexsb.com/wiki/fsb/manual/oanda_data
http://forexsb.com/wiki/fsb/manual/truefx_data

I'm still having a hard time understanding how comparing MT's backtest to FSB's helps to build trust. I understand the trust issues - I have those too from time to time - but I'm solving those with analysing and verifing FSB results and charts.

Footon, thanks very much for you help here. I was expecting something different in regard to tick data, I already know the process you have linked to.  I guess I was more wondering what file name you need to use to make FSB use tick data. This would be settings in the  instrument editor i would expect, for example AUDUSD1 is the aussie USD on 1 min and AUDUSD5 is the aussie USD on 5 min so what would be aussie USD tick?  but thanks again for helping I do appreciate it.

Regarding the interbar scanner, I don't have an issue with how long it takes, and I understand how it works. To be honest this is a brilliant improvement for backtesting, very very well done to the people who drept that idea up.

You may be right. MT4 compared to FSB is probably a waste of time, however in the absence of that perhaps you might know how the backtester in FSB was tested in the first place, if I understand how it was baselined then this maybe all I need.

you see I can't believe that when the backtester was first developed that there was not some kind of method used to determine if it was actually working, if I was developing it I would be using some method to determine that the calculations it is using are written correctly. I would not be so confident as to just expect that something I had created would work without issue, but then again I am not a developer by profession, I am en implementation engineer so I have to fix stuff others have written all the time so that might explain my thinking. 

The very fact that two different versions of the same product (FSB and FSB Pro) give vastly different results using the exact same data and the same strategy (and FSB Pro seems more accurate than FSB which is a surprise because its an Alpha build) seems to indicate that the backtester doesn't actually work at all.

Given that I don't know of any backtester on the planet that actually works properly, then there isn't really anything to baseline against. So it is probably best to go with what is the industry practice that being Mt4 with 99% model quality using a very basic EA (SMA Cross) with identical settings in both MT4 and FSB using the same data.

I would expect that as SMAs can only be calculated one way (as far as I know a simple moving average on the close is still and average over a period, kinda hard to stuff that up) then I would expect the results good or bad to be almost the same. If they are not then this shows at some level that the backtester is inaccurate and then you have to look to testing your strategies an other way.

testing on real data is of cause the best way forward, but honestly that will take a hell of a lot of time if you are trying to validate something the generator built, for example the generator generates a strategy which tested with 18K profit over 3 months, it will take at least 2 months to confirm if the backtest was accurate or not. if you have 10 or so such strategies this is 2 to 3 years. I could have been in the market with something if I was pretty confident the backtest was accurate.

Understand please, I dont expect the backtester to be a perfect predictor of future profit. This is plainly never going to happen (its like trying to predict the next number on a roulette wheel, probability is it will be completely random regardless of how many times the ball has landed on 12) but it needs to be close, maybe 60 - 70% accurate in predicting profits.

at present the FSB backtester might as well be a random number generator because it appears to be about as accurate (this is regardless of how long it takes to produce the results). I hopw you can see my point of view.

I understand your point very well. The questions you are asking are also justified.

About FSB backtest algorithm and its baseline - the go-to reference is obviously the developer Mr.Popov. He devised, planned and executed it. I can only give simplified overview, which is accessible in this forum in one way or another. In over 4 years with FSB, I haven't got to a point where I've needed to delve into the backtest algo, as I got answers to my questions from other sources/methods. Suspicious backtest results, which I've had a few, have always turned out to be bugs on my end, leaving FSB's reputation intact.When the project became open-source, Popov promised to give an overview regarding specifically the backtest algo and answer any questions which may rise. When he comes back from his main job, and you have patience to wait and get to the bottom of this, you can ask him then. He'll be more than happy to do so.

You can use very basic strategy for MT/FSB comparison, but the true virtues of FSB rise with more complex strategies. Do we know how MT interpolates bars? Does MT warn about a possible unreliable backtest? Not really.

I must admit you have caught me with the tick data file names, I haven't felt a need for tick data, I guess I adapted my methods to gain good results without using tick testing! I'll look into it.