1 (edited by dedok3000 2015-08-25 18:43:10)

Topic: OOS question

Hey guys!

I set 20% OOS in the generator but it does not draw that green zone on the charts:

http://i.imgur.com/MJIDd9C.png

Also it's not shown on a strategy page.
However if I stop the generator right after start it draws the green zone:

http://i.imgur.com/gbWrFLS.png

Is it normal or a bug? Maybe I'm doing something wrong sad

Re: OOS question

It is a normal behavior. The green zone appears only after the Generator or Optimizer are stopped.

When you set OOS and start Generator or Optimizer, the program loads only data up to the OOS date. For example if you use OOS = 20%, the program will load the first 80%. The Generator and Optimizer use these 80% of the data without having access to the OOS part. That makes the OOS calculations "real". After the calculations are finished, the program loads the full data set and this is the only moment the program can compare the performance from the partial data (first 80%) with the performance with the complete data set.

In reality there is no such thing as OOS. However this topic is difficult for understanding and the popular writings do not help much.

Re: OOS question

Popov wrote:

It is a normal behavior. The green zone appears only after the Generator or Optimizer are stopped.

When you set OOS and start Generator or Optimizer, the program loads only data up to the OOS date. For example if you use OOS = 20%, the program will load the first 80%. The Generator and Optimizer use these 80% of the data without having access to the OOS part. That makes the OOS calculations "real". After the calculations are finished, the program loads the full data set and this is the only moment the program can compare the performance from the partial data (first 80%) with the performance with the complete data set.

In reality there is no such thing as OOS. However this topic is difficult for understanding and the popular writings do not help much.

So... there is no way to see OOS zone on the other tabs like repository or strategy tab?

Re: OOS question

No way. OOS data and stats are out of sample. If you see them, they are not OOS smile

It is simple. There is absolutely no way to calculate stats on OOS (missing / not yet existing) data.
Once the data is available, it is not more OOS. Therefore we always calculate in sample.

Probably you want to calculate stats with less data. Later to calculate stats with more data and the program to mark the zone from where the new data starts.

It is possible only if the program makes two calculations every time when calculates the strategy. Such feature is not available in the current version.

Re: OOS question

Popov wrote:

No way. OOS data and stats are out of sample. If you see them, they are not OOS smile

It is simple. There is absolutely no way to calculate stats on OOS (missing / not yet existing) data.
Once the data is available, it is not more OOS. Therefore we always calculate in sample.

Probably you want to calculate stats with less data. Later to calculate stats with more data and the program to mark the zone from where the new data starts.

It is possible only if the program makes two calculations every time when calculates the strategy. Such feature is not available in the current version.

Well... looks like the only way to do what I want is to generate a strategy using cut data horizon and then recalculate it on extended data horizon..

Re: OOS question

Oh.. actually looks like I got what you mean.. So the generator just uses 80% of the data set in data horizon and then I see the chart with 100% data?

Re: OOS question

dedok3000 wrote:

Oh.. actually looks like I got what you mean.. So the generator just uses 80% of the data set in data horizon and then I see the chart with 100% data?

Yes. It is correct.

It uses the partial data only during the generation. After finishing, it loads all data and calculates the full stats that you see on the Editor tab hence no green zone in the Editor (full data calculation).

During the generation, it sends stats to the Collections repo. The collections has only the partial data (first 80%). This is the reason why the collection doesn't show the green zone.

Re: OOS question

Popov wrote:
dedok3000 wrote:

Oh.. actually looks like I got what you mean.. So the generator just uses 80% of the data set in data horizon and then I see the chart with 100% data?

Yes. It is correct.

It uses the partial data only during the generation. After finishing, it loads all data and calculates the full stats that you see on the Editor tab hence no green zone in the Editor (full data calculation).

During the generation, it sends stats to the Collections repo. The collections has only the partial data (first 80%). This is the reason why the collection doesn't show the green zone.

Oh... but my idea was to show this green zone in the editor because it's useful to check when the OOS period starts on the chart...

9 (edited by dedok3000 2015-08-25 21:29:21)

Re: OOS question

And one more question - what about acceptance criteria? The generator applies it to the 80% part of data only?

Re: OOS question

I mean.. that would be a nice option in generator to generate strategies that fulfill the acceptance criteria INCLUDING the OOS part.

Re: OOS question

generator to generate strategies that fulfill the acceptance criteria INCLUDING the OOS part.

Switch off OOS and you'll get it.

Re: OOS question

Actually, when the generator finishes it shows the stats calculated on the full data set.

13 (edited by dedok3000 2015-08-25 21:55:16)

Re: OOS question

Popov wrote:

generator to generate strategies that fulfill the acceptance criteria INCLUDING the OOS part.

Switch off OOS and you'll get it.

No smile it's different.. I mean it should find a strategy that fulfills the acceptance criteria on 80% of data, then add the missing 20% and check for acceptance criteria AGAIN. And push the strategy to repo only after that.

Re: OOS question

There are very few cases where a strategy that fulfills the criteria on full data set doesn't fulfill the criteria on the partial data. So you can easily safe the time for the first calculation.

Imagine you have criteria for max DD and max consecutive losses.

If you have max DD of $200 for the full data set, it is not possible to have greater DD for the partial data. The same is for the max consecutive losses. If you have 6 consecutive losses for the full set, it is not possible to have more for the smaller set.

However the opposite is fully possible. For example to have 6 consecutive losses for the partial set, but after loading the full data, the losses to become 8.

15 (edited by dedok3000 2015-08-25 22:37:06)

Re: OOS question

Popov wrote:

There are very few cases where a strategy that fulfills the criteria on full data set doesn't fulfill the criteria on the partial data. So you can easily safe the time for the first calculation.

Imagine you have criteria for max DD and max consecutive losses.

If you have max DD of $200 for the full data set, it is not possible to have greater DD for the partial data. The same is for the max consecutive losses. If you have 6 consecutive losses for the full set, it is not possible to have more for the smaller set.

However the opposite is fully possible. For example to have 6 consecutive losses for the partial set, but after loading the full data, the losses to become 8.

I mean that on the 80% of data the strategy may be extremely over-optimized so when you check it on the 100% data the balance chart drops down at those last 20% and this is actually a very often case... until you check "use default indicators values" of course.

So I have to go to the Editor of each strategy and check if the chart drops down at the end (over-optimized) or not (probably good strategy) smile and my idea was to save time by making a new feature like "double acceptance criteria check" option smile Hope you understand what I mean.

Re: OOS question

Yes, I understand. You can achieve that when you do not use OOS. It will work on 99% of the cases.

Summary:

If you want the Acceptance Criteria to be fulfilled for the partial (first 80%) of the data, use OOS.
If you want the Criteria to be fulfilled for the partial data (80%) and full data (100%) with one calculation, do not use OOS.

Re: OOS question

Popov wrote:

Yes, I understand. You can achieve that when you do not use OOS. It will work on 99% of the cases.

Summary:

If you want the Acceptance Criteria to be fulfilled for the partial (first 80%) of the data, use OOS.
If you want the Criteria to be fulfilled for the partial data (80%) and full data (100%) with one calculation, do not use OOS.

With the second case we will more likely get an over-optimized strategy smile anyway.. thanks for replying

Re: OOS question

In a way, everything is over optimized.....  best test is the real market or having the expert perform on other instruments.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....