forex software

Create and Test Forex Strategies

forex software
Forex Software
Products
Expert Advisor Studio Forex Strategy Builder Professional Top 10 Robots App Express Generator CoinRang Trading Courses Historical Forex Data
Purchase Partners Repository Docs & Guides Forum

Skip to forum content

Forex Software

Create and Test Forex Strategies

You are not logged in. Please login or register.

Forex Software → Forex Strategy Builder Professional → Out of Sample using missing in Videos?

Pages 1

You must login or register to post a reply

RSS topic feed

Posts: 7

  • From: Sweden
  • Registered: 2014-05-08
  • Posts: 84

Topic: Out of Sample using missing in Videos?

I have been viewing your videos, learning new tricks and deepening my knowledge of FSB Pro - it is very usefull way to learn FSB!. I am not through all of the videos yet (part 12 of 15). But I found out that you (Mr. Popov) are not mentioning or applying Out of Sample in your  examples (I heard you mention OOS once, but only casually, not in the main line of the story, and without applying OSS to the strategy).

In my opinion, Out Of Sample (I will use 35%) is a very good filter to avoid over-optimization. After generating a strategy I will always use it as the first step to discard bad strategies. Then I will use Comparator, Mulit Markets or Monte Carlo to check quality of the strategy. So why this OSS ommission in your videos?

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 8,014

Re: Out of Sample using missing in Videos?

Thank you for that question. It is a great opportunity to share some thoughts about OOS.

But first, can you explain how are you exactly using OOS?

Popov's Website

3 Reply by LeonHerd (edited by LeonHerd 2015-03-25 09:57:31)

  • From: Sweden
  • Registered: 2014-05-08
  • Posts: 84

Re: Out of Sample using missing in Videos?

What I will usually do, when developing a strategy with FSB, is exactly what you are showing in your videos. But I am always additionally switching on OSS, requesting that the FSB Pro program will start searching for a strategy applying OSS (as I wrote, I will often choose 35% as the parameter for OSS). So all strategies in a Collection are developed on the initial part (65%) of the total amount of the available historical data.

When I am ready and all 100 strategies are collected in in a Collection, I will go through best strategies in the Collection, one by one. I will choose some strategy in the Collection (by one mouse click) and then Account Statistics sheet (column) for this strategy will open on the right side of FSB program window. This column will show the balance and equity drawings for the strategy, but now for the whole period of 100% of historical data.

So now I can compare two drawings, the one in the middle, and the one to the right. The right one has additional period of 35% of historical data added. And here, on the part of drawing now added (35%) I can directly see if the strategy behaves properly, continuing to grow in similar way as in the initial 65% (shown in the left column) or changes behaviour radically, implying in this way that this strategy was over-optimized and should be discarded.

In my opinion this “filter” is very important and should be always applied when developing sound strategies. Over-optimization is potentially very dangerous factor in a strategy and it is not easy to filter out. Other methods for such analysis, such Comparator, Multi Markets or Monte Carlo will add additional level of security against over-optimization, but those methods work in a different way, adding additional strength to our primary (OSS) defences against over-optimization.

  • From: Sweden
  • Registered: 2014-05-08
  • Posts: 84

Re: Out of Sample using missing in Videos?

I will try here to exemplify previous description with two cases; first a bad strategy:

http://s1.postimg.org/tnwn7vu6z/Bad.jpg

The blue marked strategy, visible in the collections column, shows equity line growing in a nice way up to 3084€ and SQN=2,22 Sharp=0,18. This strategy was automatically generated utilising only 65% of available historical data. In right column we can follow the same strategy, but now with the result for whole data period (100%). The first part of the curve in the right column is the same as the blue one, but it is obvious that last part (35%) is not behaving well. You can also observe that both SQN and Sharp went down, SQN=1,42 and Sharp=0,13. Equity is down to 2729. This strategy has to be discarded.

  • From: Sweden
  • Registered: 2014-05-08
  • Posts: 84

Re: Out of Sample using missing in Videos?

And here is an example of a good strategy:

http://s7.postimg.org/giw1hcx07/Good.jpg

Please observe here the last 35% of the equity curve. This strategy is continuing to work in the same way on the last 35% of the data period as it has been working on initial 65%. I will keep this strategy and make some additional tests.

  • Blaiserboy
  • Blaiserboy
  • Junior Part-Time Aspiring Space Cadet
  • Offline
  • From: Toronto
  • Registered: 2010-09-06
  • Posts: 2,849

Re: Out of Sample using missing in Videos?

Some people actually use other features of the program to establish robustness.

I will share a couple ideas......  once you get your strategy looking good...... you can

change the dates in the current time frame.. that is quite similar to OOS

see how it does in other time frames.......

see how it does on other instruments and varied time frames.

One thing that i like to do is to export to MT5, lots of timeframes and instruments.

Easy to do forward testing because the data goes back a long way. (MT5 is not as accurate as FSBPro)

If you are into coding, you can take the EA and make a few modifications so you can optimize in MT5; further, you can build an EA to optimize your EA on multiple instruments and multiple time frames.

FSBPro has many tools and many ways to accomplish what we need. Working with each tool for a while will allow you to have ideas to create your own solutions.

No matter what we do to create our EA, the final test is what happens in a real account, live trading.

Popov has shared many ideas and methods, it is up to us to use those and others and create our own solutions.

I like to do a backtest in MT5 with the indicators brightly coloured and examine every bad entry to understand why it failed and how to compensate for the error. This takes a lot of time because I like to look at many bad entries...to get the cause of the error. You can do this also in FSBPro, I prefer MT5 for my own reasons, mainly because I will perhaps adapt the EA so that I can optimize it in MT5, that is just my personal preference.

In summary...... you can do OOS or you can use different periods and different time frames and different instruments.

I hope that gives some ideas.

Blaiserboy's Website

  • From: Sweden
  • Registered: 2014-05-08
  • Posts: 84

Re: Out of Sample using missing in Videos?

Thank you Blaiserboy for commenting!

The reason I will post a note about some subject is a possibility of achieving a discussion on the forum. Only in discussions with others I can test my ideas and thoughts and also learn something new. So I am not trying to be critical when writing on the forum, rather I am trying here to ask a question: am I right interpreting this in that way?

So I am interpreting your answer as: Yes, you can use OSS in FSB Pro in the described way, but also there are many other methods of testing. Thank you Blaiserboy for sharing those methods with us. After your comments my toolbox is large now.

Posts: 7

Pages 1

You must login or register to post a reply

Forex Software → Forex Strategy Builder Professional → Out of Sample using missing in Videos?

Similar topics in this forum