Topic: Intrabar Scanning

Hellow
is it possible to start the intrabar scanner to scan the tick data in FSB Pro ?
I have the problem that strategies in FSB work good, but wenn I open the strategy in FSB Pro
it makes very bad results. I use the same CSV files that I have importet from JForex into FSB.
All account and Data Horizon parameters are the same.I have checked everething, but can not find how
i can use the tick data from scanner.
In generally I have notived a diferences between FSB and FSB Pro strategy performance even if I do not use tick data. Can someone help me with some ideas or advices?

thanks

Re: Intrabar Scanning

I have not run comparisons, the two are different programs although very similar.

Perhaps you can provide some details of the differences you have discovered..... specifics, pictures, journals, etc so that Popov can take a look at the situation. Also the strategy which is showing the differences.. the actual xml file so that he can load it on his machine for testing.

He will be sure to look into it if you can provide materials for him to work with

Re: Intrabar Scanning

ok , here ist one strategy , that works with ticks in FSB in 2013 very well, but
in the same data horizon in FSB Pro, gives me complete other results.
i dont know, what program i have to trust smile ?

Post's attachments

Generated_TopBottom.xml 10.76 kb, 14 downloads since 2014-01-06 

You don't have the permssions to download the attachments of this post.

Re: Intrabar Scanning

Peter, are you having a lot of ambiguous bars in standard FSB?

And I strongly agree with Dave that both FSB's are very similar, I think it was verified at one point that both FSB's had the same backtesting result on the same data, which they definitely should.

Re: Intrabar Scanning

Its very strange, i saw now that i have lot of ambiguous bars in  FSB Pro wich gives me the strange results. I have generated the strategy in FSB and reopened it in FSB Pro. Perhaps is there the problem. Aceptance Criteria is also Not fuldilled. How can I solve this strange behaviour ?

Re: Intrabar Scanning

Greetings, I am new to the World of FSB, FST, FSB Pro but I really love what you’re trying to do here. I am also part of the asirikuy project which is trying to do something similar but I think FSB is a lot further along. Anyway to my point..
I am having exactly the same issue as PeterChristoph, I have generated a number of strategies with FSB and some of them are showing results which to be frank are not very likely, so to me the whole back testing experience with FSB is now under some doubt.
I have attached a strategy which was generated with FSB on the AUDUSD over a Data Horizon of 2-1-2012 to the 4-2-2014 on an initial balance of $3000 the FSB back tester projected a return of $18147.16 which is pretty unbelievable I would have to say. I have also attached the history data file to match the strategy so you can test it your self.
I open the same strategy in FSB Pro with exactly the same parameters and I end up with a balance of 3657.98 which to be honest seems a lot more likely. 
I am using identical data (using the exact same data directory) so I know it’s not a data issue, I am not using the same indicators code (I am using whatever come in the installer) so that is the only difference other than the application its self. I am more inclined to trust FSB simply because its results seem to me to be more likely, but I will now perfume more testing which will take me some time.
My first test will be to compare the back test results with MT4 using the same strategy with the same data horizon, if this prove conclusive then that will be a great result and hopefully prove definitively which version is correct and which one isn’t.
if I don’t get a clear result then my next step will be to pick a strategy and run 2 demo accounts, one with FSB Pro interfacing to MT4 and the other with the MT4 version of the strategies. I will report back my findings.
Bottom line I think the back tester on FSB in particular is highly suspect. OR there is something wrong with the default Indicators which I am yet to rule out as an issue.

Re: Intrabar Scanning

Indeed , there are some strange things in FSB project . In short, it generates too fast, too good strategies... lol ... To test how good they really are the strategies generated by the FSB ( in this life ...lol) I proposed completion FSB with a WFA module ( walk forward analyzer ) but the idea has not sparked interest.
Intrabar scanner (FSB ) is particularly suspect. I have made the following experment : I put in a data folder GBPUSD1440 + USDGBP60 for 14 years ( 3662 + 87203 bars ) ONLY , and in another  data folder GBPUSD1440 +  USDGBP1 for  14 years ( 3662 +  4813744  bars) ONLY . [ Well , yes : I ` m using MAXBARS = 5,000,000 ... ] . Then I enabled "generator" . ( In both cases Automatic scan = on ) .
The results were : in the first case were generated ~ 700 strategies / minute. In the latter case was generated ~ 585  strategies / min ( ...lol...) . In both cases " ambiguous bars " = very low ( 0-3) .
To me this means that intrabar scanning does not work. Or maybe it works phenomenally fast. But then I do not understand other things ...[ if I`m wrong please correct me , because I hope that Intrabar scanning works fine and the EAs FSB generated are really very good !]
I tried a comparison between FSB backtest and  MT4 backtest , too ,but the results were not conclusive because it is hard ( for me...) to write exactly the same  EA in both languages  ,  especially because  some indicators of FSB was not found in MQL . But the test remains a very interesting one ! Good Luck!

Re: Intrabar Scanning

B52 wrote:

I tried a comparison between FSB backtest and  MT4 backtest , too ,but the results were not conclusive because it is hard ( for me...) to write exactly the same  EA in both languages  ,  especially because  some indicators of FSB was not found in MQL .

You are saying that if mt backtester and FSB would produce different results, FSB would be the loser?

Re: Intrabar Scanning

footon wrote:
B52 wrote:

I tried a comparison between FSB backtest and  MT4 backtest , too ,but the results were not conclusive because it is hard ( for me...) to write exactly the same  EA in both languages  ,  especially because  some indicators of FSB was not found in MQL .

You are saying that if mt backtester and FSB would produce different results, FSB would be the loser?

Not at all. But  Intrabar scanner  it is one really problem . In MT4 "Every tick" model works (very slow, indeed...). But in FSB "every 1 minutes " model seems not working ...
I tested many (demo&live) , FSB+FST / FSBPro, etc ) FSB generated strategies , but the only ones I trust are those who work with "Bar openig/closing".
I have searched the opinions of traders with more experience in using the FSB on this issue, but to my surprise I found nothing on the forum. In these conditions, your opinion would be very valuable for me.

Re: Intrabar Scanning

Why don't you use real tick backtesting in FSB? MT's "every tick" isn't exactly using tick data...

So, the problem is speed? Works fast means doesn't work at all? Scanner loads up slowly, after that it works as it should - fast. Better to point to a specific spot where backtest is not correct, that would be a way forward in every respect.

What do you mean by "very good" strategies? Expectancy to remain profitable or reliable backtest?

What did you exactly test? FST trades compared to FSB's backtest?

11

Re: Intrabar Scanning

footon wrote:

Why don't you use real tick backtesting in FSB? MT's "every tick" isn't exactly using tick data...?

Generally speaking, I do backtests for at least five years. The large volume of tick data makes backtest  to be very slow. In addition, if I use only "Bars opening / closing" as "Opening / Closing Point of the Position", using tick data is useless.

"So, the problem is speed? Works fast means doesn't work at all? Scanner loads up slowly, after that it works as it should - fast."
Speed is not bad "per se". But a high speed during the process of generating new strategies, makes me to doubt that every time FSB uses all 4,900,000 min1 bars to make intrabar scanning . (see the example in my previous post). It is hard to believe that when FSB considering a strategy  for TF = 1 day,intrabar scan using 87,000 bars  lasts just as long as intrabar scan using 4,800,000 bars ...

"What do you mean by "very good" strategies? Expectancy to remain profitable or reliable backtest? "
Hehe! This question touches a sore point for all traders.
If the FSB, I'm happy with "reliable backtest". To get a strategy "with a good expectancy to remain profitable" is completely different discussion.
I have a vast experience in this area, but unfortunately very little of it is useful for me when I use FSB. All I can say now is that the use of OOS during the generation of new strategies is far, far from enough.

"What did you exactly test? FST trades compared to FSB's backtest?
I have tested a lot of aspects. To compare FSB backtest trades with FST (live forward) is just one of them (and not a quite relevant aspect , because the shortness of human life). I have tested FSB + FST versus FSPro (live for the same strategy), then FSB generator versus FSBPro  generator (here I had trouble because I could not figure out how can I change MAXBARS in FSBPro, although I asked on the forum .. .), how FST and FSBPro working with ECN broker, etc ...
But I spent a long time trying to find out if I can trust "Intabar scanning" or not. Until now I do not have a clear answer to this dilemma.

However, any opinion from some experienced traders are welcome!

Re: Intrabar Scanning

do you use different date periods or do you rely on one screen for OOS.

Myself, I prefer to use a Monte Carlo to get an idea as to worst case....  to rely on OOS may not be  good except for an estimate.

Equity curve is what I want to see doing well and I look at several periods to gauge that.

I would think that if you do a backtest in MT4 you will see the quality of your data.

13

Re: Intrabar Scanning

Blaiserboy wrote:

do you use different date periods or do you rely on one screen for OOS.

Myself, I prefer to use a Monte Carlo to get an idea as to worst case....  to rely on OOS may not be  good except for an estimate.

Equity curve is what I want to see doing well and I look at several periods to gauge that.

I would think that if you do a backtest in MT4 you will see the quality of your data.

To determine the robustness of a strategy ( its ability to make money in the future) is a difficult operation which, due to the large amount of calculations require special software and powerful computers . The WFA which I mentioned is one of the methods. Monte Carlo can be another method . The most advanced in this field are those of TradeStation . There are some timid attempts for MT4 .
But all methods require a reliable backtest . No personal computer ( even if it works with many cores, GPU , etc) can not execute  exhaustive optimizations using tick data . Such that , it requires the use of genetic optimization ,scans intrabar with low TF ( or at xxx  ticks ),interpolations , etc. .
Unfortunately FSB Optimiser works completely non-transparent to the user , so I can not know if a backtest is reliable or not.
In conclusion, I think that before discussing the methods for determining the robustness of strategies  FSB generated , would be more useful to ask Mr. Popov how   FSB  optimizer works (lol)
Or even Intrabar Scanning . ( Or encourage Mr. Coffeesnob  to make comparisons with MT4...)

PS : In accordance with the principles of the Holy Trinity , I make another attempt : Does anyone know how could I change MAX_BARS ( MAX_INTRA_BARS ) for FSB -PRO ?( I emphasize FSBPRO because for  FSB I know how to do ... )

Re: Intrabar Scanning

MT doesn't offer not even remotely close to what one would call reliable backtest. Full stop.

FSB is not really non-transparent, mind you, its source code is freely available! You said you can code, I don't see a problem why one couldn't work it out for oneself.

I think you want to know how the generator works, not the optimizer?

You seem to take FSB Pro as a finished product, which it isn't. It's still a beta, it doesn't have all features incorporated, it has bugs to iron out etc.

15

Re: Intrabar Scanning

@footon :
1. " MT doesn't offer not even remotely close to what one would call reliable backtest. Full stop."
I absolutely agree! For this reason I do not use MT4 backtester than "open prices only" mode. But the real question is whether the FSB backtester is reliable.

2 "FSB is not really non-transparent, mind you, its source code is freely available! You said you can code, I don't see a problem why one couldn't work it out for oneself."
Unfortunately I'm not a coder. So reading the source code exceeds my abilities.

3" I think you want to know how the generator works, not the optimizer?"
My desires are more modest. I would be pleased if I understand how (and if) IntrabarScanning works.

4."You seem to take FSB Pro as a finished product, which it isn't. It's still a beta, it doesn't have all features incorporated, it has bugs to iron out etc."
From your message I should understand that MAX_BARS  can not be changed because FSBPro is in beta?

5.I greatly appreciate the work done by the team that created and maintains the FSB / FST / FSBPro. I appreciate also as experienced traders accept to consume their time to answer the questions asked by beginners.

Re: Intrabar Scanning

I think you can have a look at the code on the web and decide where it can be changed..... and then modify your own copy. I do not have the knowledge to affect a change in the code.

I have had good luck using FSB, I feel it to be reliable.

Proviso is.. not too many things to be optimized..... a couple variables only, same as any system.

Robustness achieved by running expert on more than a couple pairs.

I use standard settings only to avoid overfitting. I have only indicators that come with the initial download and quite a few of them are blocked out.

I find that I am doing better with FSB as I can run two or three at a time.

17

Re: Intrabar Scanning

@ Blaiserboy :
To be clear: I am not a critic of the FSB. On the contrary, I have tested several systems for automatic generation of trading strategies, and can say that the FSB is the best of all.
Right now I use three strategies generated by the FSB on a live account, and another 6 on an demo account
I just posted in a thread called "Intrabar Scanning", some doubts about the accuracy of this process.

Re: Intrabar Scanning

When you mention 'inaccuracy'  do you have any specific metric that would describe  the area of doubt...

What part of the scanning do you feel to be inaccurate, can you elaborate further.

I well know you are not a critic..... I understand that we have to delve into this thing further and that all of us will benefit.

If we do not question. we will never be fully confident.

If you can spell it out exactly what you question, the we can ask Popov to reply, hopefully his answer will set us at ease.

Thanks

Re: Intrabar Scanning

Hi All. very good discussion.

I will definitely do my comparative testing with FSB and MT4 using the same data, one question.. Footoon you mentioned about loading tick data into FSB. how do I do that?

I like B52 am hopeful that the results I have been seeing from the backtester are true because I will do anything to ditch that stupid Mt4 backtester.

for me its about building trust and like B52 I need to understand how the back tester works, I am a coder but to be honest haven't spent a lot of time trawling through the code to understand how it works.

I would have expected that at some point some comparative testing would have been done to determine if the backtester works in FSB. even understanding what method was used for this testing would be helpful.

will update with my results once I get my copy of tick data suite updated to fit the new build of MT4

Re: Intrabar Scanning

I have an impression that some think intrabar scanner should be working all the time, thus reducing FSB's speed, but it is used only when there is a need for it - in the cases of ambiguous bars.

Cooffeesnoob, download the tick data, allow its usage in the scanner and it should be fine. Read more here:
http://forexsb.com/wiki/fsb/manual/jforex_data
http://forexsb.com/wiki/fsb/manual/oanda_data
http://forexsb.com/wiki/fsb/manual/truefx_data

I'm still having a hard time understanding how comparing MT's backtest to FSB's helps to build trust. I understand the trust issues - I have those too from time to time big_smile - but I'm solving those with analysing and verifing FSB results and charts.

Re: Intrabar Scanning

Let me add this that it is best to have data for all timeframes updated, especially for M1 if tick data is also used.

Re: Intrabar Scanning

Footon, thanks very much for you help here. I was expecting something different in regard to tick data, I already know the process you have linked to.  I guess I was more wondering what file name you need to use to make FSB use tick data. This would be settings in the  instrument editor i would expect, for example AUDUSD1 is the aussie USD on 1 min and AUDUSD5 is the aussie USD on 5 min so what would be aussie USD tick?  but thanks again for helping I do appreciate it.

Regarding the interbar scanner, I don't have an issue with how long it takes, and I understand how it works. To be honest this is a brilliant improvement for backtesting, very very well done to the people who drept that idea up.

You may be right. MT4 compared to FSB is probably a waste of time, however in the absence of that perhaps you might know how the backtester in FSB was tested in the first place, if I understand how it was baselined then this maybe all I need.

you see I can't believe that when the backtester was first developed that there was not some kind of method used to determine if it was actually working, if I was developing it I would be using some method to determine that the calculations it is using are written correctly. I would not be so confident as to just expect that something I had created would work without issue, but then again I am not a developer by profession, I am en implementation engineer so I have to fix stuff others have written all the time so that might explain my thinking. 

The very fact that two different versions of the same product (FSB and FSB Pro) give vastly different results using the exact same data and the same strategy (and FSB Pro seems more accurate than FSB which is a surprise because its an Alpha build) seems to indicate that the backtester doesn't actually work at all.

Given that I don't know of any backtester on the planet that actually works properly, then there isn't really anything to baseline against. So it is probably best to go with what is the industry practice that being Mt4 with 99% model quality using a very basic EA (SMA Cross) with identical settings in both MT4 and FSB using the same data.

I would expect that as SMAs can only be calculated one way (as far as I know a simple moving average on the close is still and average over a period, kinda hard to stuff that up) then I would expect the results good or bad to be almost the same. If they are not then this shows at some level that the backtester is inaccurate and then you have to look to testing your strategies an other way.

testing on real data is of cause the best way forward, but honestly that will take a hell of a lot of time if you are trying to validate something the generator built, for example the generator generates a strategy which tested with 18K profit over 3 months, it will take at least 2 months to confirm if the backtest was accurate or not. if you have 10 or so such strategies this is 2 to 3 years. I could have been in the market with something if I was pretty confident the backtest was accurate.

Understand please, I dont expect the backtester to be a perfect predictor of future profit. This is plainly never going to happen (its like trying to predict the next number on a roulette wheel, probability is it will be completely random regardless of how many times the ball has landed on 12) but it needs to be close, maybe 60 - 70% accurate in predicting profits.

at present the FSB backtester might as well be a random number generator because it appears to be about as accurate (this is regardless of how long it takes to produce the results). I hopw you can see my point of view.

Re: Intrabar Scanning

When we are backtesting we are usually generating a system......  the walk forward is what we focus on...

Ensure you have good data and do walk forward carefully....

Sometimes we complex things,

Backtesting is where we sorta look at things, the walk forward is where the proof is.

Re: Intrabar Scanning

coffeesnob wrote:

Footon, thanks very much for you help here. I was expecting something different in regard to tick data, I already know the process you have linked to.  I guess I was more wondering what file name you need to use to make FSB use tick data. This would be settings in the  instrument editor i would expect, for example AUDUSD1 is the aussie USD on 1 min and AUDUSD5 is the aussie USD on 5 min so what would be aussie USD tick?  but thanks again for helping I do appreciate it...

I understand your point very well. The questions you are asking are also justified.

About FSB backtest algorithm and its baseline - the go-to reference is obviously the developer Mr.Popov. He devised, planned and executed it. I can only give simplified overview, which is accessible in this forum in one way or another. In over 4 years with FSB, I haven't got to a point where I've needed to delve into the backtest algo, as I got answers to my questions from other sources/methods. Suspicious backtest results, which I've had a few, have always turned out to be bugs on my end, leaving FSB's reputation intact.When the project became open-source, Popov promised to give an overview regarding specifically the backtest algo and answer any questions which may rise. When he comes back from his main job, and you have patience to wait and get to the bottom of this, you can ask him then. He'll be more than happy to do so.

You can use very basic strategy for MT/FSB comparison, but the true virtues of FSB rise with more complex strategies. Do we know how MT interpolates bars? Does MT warn about a possible unreliable backtest? Not really.

I must admit you have caught me with the tick data file names, I haven't felt a need for tick data, I guess I adapted my methods to gain good results without using tick testing! big_smile I'll look into it.

Re: Intrabar Scanning

thanks Footon..