Topic: FSB: estimation period, validation period (OOS), and the future

A great way to test a trading strategy and to realistically compare its forecasting performance is to perform out-of-sample validation, which means to withhold some of the sample market data from the strategy generation process, then use the generated strategy to make predictions for the hold-out data in order to see how accurate they are and to determine whether the statistics data of this strategy are similar to those that the strategy made within the sample of market data that was fitted. In the FSB pro software, you are given the option to specify a percentage of data points to hold out for validation (Out of Sample in the generator panel).The data which are not held out are used to generate strategies which past the acceptance criteria of the control panel. The strategy is then tested on data in the validation period. Unfortunately FSB pro lacks the ability to make forecasts beyond the end of the estimation and validation periods. This ability is known as “forecasts into the future”. But this feature may be possible to be Incorporated in a future release.

    In general, the data in the estimation period are used to help select the best strategy and its parameters. Forecasts made in this period are not completely "credible" because data on both sides of each observation are used to help determine the best strategy. The one-step-ahead forecasts made in this period are called fitted values. (They are said to be "fitted" because the software estimates the strategy so as to "fit" them with less error as possible in a mean-squared-error sense or similar criteria.) The corresponding strategy forecast errors are called residuals. The residual statistics  may underestimate the magnitudes of the errors that will be made when the strategy is used in live, because it is possible that the data have been over-fitted.   Over-fitting is especially likely to occur when either (I) a strategy with a large number of parameters (and/or slots) has been fitted to a small sample of market data and/or (II) the strategy has been selected from a large set of potential strategies precisely by minimizing the mean squared error in the estimation period (e.g.,  all possible subsets regression has been used with a large set of regressors). All of these are aspects well known by experienced traders.

    The data in the validation period are held out during strategy parameter estimation. One-step-ahead forecasts made in this period are what we called back-tests. Ideally, these are true forecasts and their error statistics are representative of errors that will be made in forecasting the future. However, if you test a great number of strategies and choose the strategy which errors are smallest in the validation period, you may end up over-fitting the data within the validation period as well as in the estimation period.
So here are my first suggestion for FSB improvement:

A) The statistics of the strategy in the validation period (in-sample data) should be reported alongside the statistics of the strategy in the estimation period (out-of-sample data), so that we could compare them easily. Furthermore, this statistics should be included as acceptance criteria in the generation of strategies. With this in place we could get really sound strategies.

                                   
                                            Estimation Period    Validation Period
Profit per day,                             170                         145
Sharpe ratio,                               0.27                          0.19
System quality number,           2.18                          1.17
Win/loss ratio…                          0.51                         0.46

        ...

    If the data have not been too much over-fitted, the statistics in the validation period should be similar to those in the estimation period, although they are often at least slightly worst.

    Certainly, FSB can perform this calculation actually but we need to make them separately and obviously it is not possible to use them over the thousands of generated strategies.

B) The linearity of the equity/balance should be an acceptance criteria too. At present time FSB
can include the filter for non-linear balance pattern which is some kind of deviation measure from fictitious lineal behavior (a sort of lineal regression line). This is very useful , because the naked eye can distinguish a strategy that performs equally well during the estimation and validation periods. The problem with the actual feature is that FSB take into account linearity of the strategy into the estimation period but does not performs any test of this linearity after the generation of the strategy in the validation period. Thus we find that after generation of strategies, the vast majority of them do not meet this criterion of linearity and thus must be rejected manually , rather than automatically rejected by the software. Obviously, the generator must not take into account the non linearity of the validation period to perform any parameter calculation. It only should use the non-linearity filter of the validation period at the end before presentation of the strategy to the trader. In other words, FSB should “look at” the final strategy instead of the trader.
Here below is an example of what I mean. These are two strategies generated unattended (great for FSB!). Green color is the validation period (OOS), before green area we have the estimation period.

The first may be good enough an be ready only for further refinements.
http://s19.postimg.org/mnpx1hktf/image.png

The second is simple useless, over-fitted one.
http://s19.postimg.org/4z2rnlyg3/image.png
Obviously this last strategy should be rejected by FSB directly in automatic mode.

Well, these are just some thoughts that maybe be of some assistance in the development of this really good software.

Thank you

Re: FSB: estimation period, validation period (OOS), and the future

So....... a couple questions

What software has these features....?

What says this is something that will work to our benefit?

Is there some examples in real markets that we can have a look at?

Re: FSB: estimation period, validation period (OOS), and the future

What if I make a special tool for the job?

Here is my idea.
- Generate a bunch of strategies with (or without OOS) in a collection (save it)
- Open Multi Calculator tool
- Set Input collection and Output collection
- Let the tool recalculating all strategies from input collection to the output collection with the latest data available.
- Filter and sort the new collection using Acceptance Criteria

In that way you can recalculate and filter all strategies without  compromising the Generators OOS functionality.

Optionally, the tool may allow recalculation of all strategies from a folder.

...

Actually I spent 2 hours on outlining this new tool.

Here is my draft design:

http://s28.postimg.org/dr90gvoc9/screenshot_1257.jpg

If you accept it, I may finish it in two-three weeks.

Edit:
- It may override the original strategies data source,
- It may check if the strategies comply with the Acceptance Criteria after the recalculation and before pushing to the new collection.

Re: FSB: estimation period, validation period (OOS), and the future

Hi Popov, thank you very much for your prompt reply.

Yes, this sounds very interesting suggestion.
With this FSB enhancement , all the strategies that output the Multi Calculator should be fairly consistent.

If you can program it, I think it would be a great improvement.

Great software with great service!

Popov wrote:

What if I make a special tool for the job?

Here is my idea.
- Generate a bunch of strategies with (or without OOS) in a collection (save it)
- Open Multi Calculator tool
- Set Input collection and Output collection
- Let the tool recalculating all strategies from input collection to the output collection with the latest data available.
- Filter and sort the new collection using Acceptance Criteria

In that way you can recalculate and filter all strategies without  compromising the Generators OOS functionality.

Optionally, the tool may allow recalculation of all strategies from a folder.

...

Actually I spent 2 hours on outlining this new tool.

Here is my draft design:

http://s28.postimg.org/dr90gvoc9/screenshot_1257.jpg

If you accept it, I may finish it in two-three weeks.

Edit:
- It may override the original strategies data source,
- It may check if the strategies comply with the Acceptance Criteria after the recalculation and before pushing to the new collection.

Re: FSB: estimation period, validation period (OOS), and the future

Hi Blaiserboy,

There are a few software out there with these features or similar. More of then need some kind of script programming to generate strategies, but they are really goods. The very best are expensive, not for retail traders unless not for mini or micro accounts.

The FSB truly brilliant is its extreme simplicity and speed to generate trategies.

I think the suggestion made by Popov worked to our advantage.

Yes, there are real examples in the market, but I do not consider it appropriate to mention other software in this forum . At least , it is not my intention , sorry.


Blaiserboy wrote:

So....... a couple questions

What software has these features....?

What says this is something that will work to our benefit?

Is there some examples in real markets that we can have a look at?

Re: FSB: estimation period, validation period (OOS), and the future

What name is more suitable:
- Multi Calculator
- Multi Tester
- Mass Calculator
...

I think that Multi Tester is the best. What do you think?

Re: FSB: estimation period, validation period (OOS), and the future

You know Popov,  I'm your secret admirer , oophs it's not a secret anymore.

Your professionalism,  dedication and passion in your work really amazes me!

I take my hat off!  Keep up the good work.

Re: FSB: estimation period, validation period (OOS), and the future

This is the simplified model of FSB as I see it:

1)      Market data and configuration entries
2)      Strategy Selection with in-sample data (generator module)
3)      Deploy strategies (Expert advisor or trader mode)

Now with your new feature you will add the following step:

2bis)Strategy validation with out-of-sample data

So the simplified model now becomes:

1)       Market data and configuration entries
2)       Strategy Selection with in-sample data (generator module)
2-bis)  Strategy validation with out-of-sample data (validator module)
3)       Deploy strategy (Expert advisor or trader mode)

So my suggestion is to name it just VALIDATOR.

I do not see the need to include any adjective in the name of validator , just as none is included in the name of generator ( generator of strategies). Of course it generates multiple strategies and also massively but not name it as multi -generator or mass -generator is simply his nature as it is designed.

But this is nothing more than a simple opinion . Please , for any reason you feel compelled to take it into account. It is your software and your rules.

The important thing about new module is to fulfill its role and filter good and robust strategies.

Popov wrote:

What name is more suitable:
- Multi Calculator
- Multi Tester
- Mass Calculator
...

I think that Multi Tester is the best. What do you think?

Re: FSB: estimation period, validation period (OOS), and the future

The new tool will be useful also with another two cases:
1. Recalculation of all strategies (either from a  collection or from a  folder) with the latest data.
      - In that way the traders can see the updated results no mater if the previous calculations were made with OOS or not.
Let's say if a trader recalculates all his strategies once per month.  It will be "real life OOS".

2. Recalculation with different data sources.
     - If a trader has to move to another broker or to another server with different settings, it will take several clicks to "validate" all the strategies.
    - I see also the following usage case:
              Creating strategies to Collection1 with DataSource1
              Recalculation from Collection1 to Collection2 with DatSource2
              Recalculation from Collection2 to Collection3 with DatSource3
    And finally will receive strategies in Collection3 that fulfills the criteria for all data sources.

Re: FSB: estimation period, validation period (OOS), and the future

Really Great!

Popov wrote:

The new tool will be useful also with another two cases:
1. Recalculation of all strategies (either from a  collection or from a  folder) with the latest data.
      - In that way the traders can see the updated results no mater if the previous calculations were made with OOS or not.
Let's say if a trader recalculates all his strategies once per month.  It will be "real life OOS".

2. Recalculation with different data sources.
     - If a trader has to move to another broker or to another server with different settings, it will take several clicks to "validate" all the strategies.
    - I see also the following usage case:
              Creating strategies to Collection1 with DataSource1
              Recalculation from Collection1 to Collection2 with DatSource2
              Recalculation from Collection2 to Collection3 with DatSource3
    And finally will receive strategies in Collection3 that fulfills the criteria for all data sources.

11

Re: FSB: estimation period, validation period (OOS), and the future

I think Advanced Validator can be a good name

Re: FSB: estimation period, validation period (OOS), and the future

Great idea !
Can be added a tool  correlation  between  strategies?

Re: FSB: estimation period, validation period (OOS), and the future

2. Recalculation with different data sources.

I think this will help to prevent curvefitting.
H1 strategies for example should have the same result on different data sources.

thomas

14

Re: FSB: estimation period, validation period (OOS), and the future

Maybe it is a good idea.

Re: FSB: estimation period, validation period (OOS), and the future

I'll go even further.
I'm afraid of myself.

Re: FSB: estimation period, validation period (OOS), and the future

Popov wrote:

I'll go even further.
I'm afraid of myself.

This is like intelligence explosion -> technological singularity at some point?

Re: FSB: estimation period, validation period (OOS), and the future

It calculates and collects:

http://s13.postimg.org/ypn8ojxtv/screenshot_1262.jpg

The next step is to optimize.

18

Re: FSB: estimation period, validation period (OOS), and the future

beautiful

Re: FSB: estimation period, validation period (OOS), and the future

Now is better.

http://s13.postimg.org/3p6w1h8g3/2015_11_11_10_07_37.jpg

http://s13.postimg.org/xidwa2x37/2015_11_11_10_07_37.jpg

I have to make minor fixes and think to release a demo on this Saturday.

Do you think the tool needs something more? Is the user interface clear enough?

20 (edited by GD 2015-11-12 03:16:38)

Re: FSB: estimation period, validation period (OOS), and the future

Mr. Popov, it looks fine to me. Nice and useful job.

By the way (as I become older and you stay young)

1> I cannot discriminate the color of lines  in Multi Market. Can you make it a bit thicker the thickness of lines next to the names of currencies in the list in the right column in the Multi Market screen or use color names together??? The thickness of lines inside Graph is fine.
2> In SMALL Indicator chart (first screen) the Color of Volumes does not help me to recognize in it, the LONG and SHORT positions. Can you change it to a bit more light color?
3> At which file I can change or add logics in indicators? Is it possible?
4> Can you add a small button for Edit to the indicators labels inside strategy in first screen (even a link i.e. Edit is fine)? I can use notepad++ and CTRL+I for compiling and see changes in graph.

Re: FSB: estimation period, validation period (OOS), and the future

It looks fine to me, too.

I am impressed by your quick response and the quality of your work .

When testing the new demo 'll keep commenting .


Popov wrote:

Now is better.

http://s13.postimg.org/3p6w1h8g3/2015_11_11_10_07_37.jpg

http://s13.postimg.org/xidwa2x37/2015_11_11_10_07_37.jpg

I have to make minor fixes and think to release a demo on this Saturday.

Do you think the tool needs something more? Is the user interface clear enough?

22 (edited by hannahis 2015-11-12 12:22:49)

Re: FSB: estimation period, validation period (OOS), and the future

Impressive Popov,

For the output log, is there a clearer way to present the data in a more visually easier way to glance at the data?  For eg, the Balance, profit/loss or win/lose ratio etc, can these headings be highlighted while the results stay as it is so that it is more visual to capture the attention...these are really just minor issue for presentation, not a big deal but easier for my eyes after staring at numbers for so long.

Re: FSB: estimation period, validation period (OOS), and the future

I'm using a text box for the out data. It allows easier selection and copying of the records. Unfortunately it does not allow any formatting.  However, the main idea is to put the strategies in a collection where a user can sort and filter very easy.

I'm almost ready with the tool. I may publish a demo today.

Re: FSB: estimation period, validation period (OOS), and the future

Parrondo

Hey.......... thanks for the suggestion!

Your idea has really improved the software.

daveM