Create and Test Forex Strategies
You are not logged in. Please login or register.
Forex Software → Premium Club → Out of Sample Testing
What's the recommended way of doing Out of Sample testing with FSB ? The challenge that I'm running into is that I can't seem to filter strategies with the best combined in and out of sample testing. Also when loading the charts, I can't see the out of sample data.
Thanks,
Isie
Read carefully in Wiki.
Everything is there.
GD is right.
Here is a wiki link for Generator and OOS
Hi Popov,
From the attached image you can see that the strategies were generated with a 20% OOS, however, there is no out of sample region in the chart. Also, when loading the strategies from a collection I don't see any OOS region in the chart.
It would be great that if in a future version of FSB you can include a collection filter for best combined In and Out of Sample Performance.
Thanks,
I used your indicators in generator.
I cannot find any problem.
If at the end of generator, you use optimization then the number of bars will increase.
ps/ Miroslav, remeber to fix the vertical scrollbar only in the indicator part of the Chart of indicators. it will be very beautiful and useful to anybody, as he can compare fast and easily with the position of bars and other indicators on the main chart.
When you set OOS (for example 20%) the Generator removes the last 20% of the data and uses the first 80% for testing.
Lets say we have 10 000 bars. When you run the Generator it will cut off the newest 2 000 bars and will only load the first 8 000 bars. During the generation process, the Generator uses all loaded data (8 000 bars). It calculates the strategy and shows the result on the Balance/Equity chart using these 8 000 bars. There is no way the Generator to print the result for the other 2000 bars because it has no access to them. It also cannot send the OOS stats to a collection simply because it is not able to calculate it.
The Generator never sees the green OOS region. When it stops and returns the best strategy, FSB Pro loads the full data set and recalculates the strategy again using all the data (10 000 bars) this time. At that time, the program compares the result from the 8000 bars from the Generator and the result from the full data set and prints the green OOS region. If you press F5 and recalculate the strategy, the OOS are will disappear because the program uses the full data series.
Actually, if you see a zone called OOS with balance in it, it is a lie. IT IS NOT POSSIBLE TO CALCULATE OOS RESULT. If you calculate it, it is not OOS. How you can calculate result on data that doesn't exist? It is not possible. Once the data are available and you calculate a result, it is not OOS anymore. It may be a pseudo or virtual OOS (like in our legacy FSB).
Hi Mr. Popov,
I think, i have a similar problem like imasri.
At the moment, after creating a collection with OOS-feature, i have to look at every single strategy again and sort out manually by acceptance criteria.
In my opinion, the OOS-feature really makes a sense, if the generator only sends strategies to the collection, which the acceptance criteria also meet, when testing (not generating!) with all data.
Is it not possible, always all data to load and use only the OOS-data for generation process? If an appropriate strategy found, it should be tested immediately with the entire data. Does it still matches the acceptance criteria, it would transferred into the collection.
...sorry for my poor english. 
If an appropriate strategy found, it should be tested immediately with the entire data. Does it still matches the acceptance criteria, it would transferred into the collection.
If you switch off the OOS, you'll have exactly that result.
Instead of:
1. Generate and pass the criteria with OOS on.
2. Recalculate and pass the criteria with all data.
3. Collect the strategy.
You can simply:
1. Generate and pass the criteria with all data.
2. Collect the strategy.
This is minimum twice faster than the first approach.
It guarantees that all collected strategies will pass the OOS test as per the first approach.
All strategies in the collections will be calculated and shown with the full data series.
OOS is only a "marketing" trick to make you believing your strategy has a greater chance to make money. I'm not a broker and do not collect your losses, so I may afford telling you the truth.
Backtesting on partial data and later confirming the test on the full data series is a waist of time.
The only thing may working is to test your strategy on the full data series and to confirm the performance on demo trading at your broker.
Your answer here, Mr. Popov, indicates that I have (maybe) misunderstood something basically very important.
1. I will choose to have OOS with value of 25%
2. I will generate a strategy with OOS on (i.e. on 75% of available data set)
3. I will visually check this strategy with OOS of (i.e. on 75% + 25% of the available data set, it is shown in Balance /Equity Chart of Repository Window, Accounts Statistics)
3a. I will accept the strategy if the balance line is similar in both 75% and 25% parts
3b. I will delete the strategy if the balance line is different in 75% and 25% parts
4. I will check the strategy with Multi Markets and Monte Carlo
5. I will check the strategy with some large data set (you could call this step for “EXTERNAL OOS”)
6. I will test the strategy on my demo account.
So, what is wrong with my reasoning? Your forceful denial of usefulness of OOS together with my respect for you knowledge makes me question my basic assumptions.
Forex Software → Premium Club → Out of Sample Testing
Powered by PunBB, supported by Informer Technologies, Inc.