1 Topic by Popov (edited by Popov 2021-11-05 05:40:30)

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Topic: Sort Collection by profit on unknown data (Work in Progress)

Hello traders,

I think to make it possible to sort the strategies in a Collection by the profit made on a newer data.
When we export a Collection, it stores the time, and the stats on the last backtest in the file. (it is possible to add these records in the Portfolio Expert also)

Then, when we re-import the Collection, it shows the updated area of the chart in green (from the previous test to the moment of the newer test). However, we cannot see the actual performance.

My idea is to calculate the Net Profit made in the newer data period and add it to the sorting criteria.
These stats will be handy for re-validating of older collections.

It is also possible to add an OOS zone in days to serve the same purpose.

I think it will be able to test a prototype this week.
Do you think a proper name of the stat can be "OOS Profit"? It can be also available after generating strategies with OOS.

The OOS days options can be:
- Automatic
- 7 days
- 14 days
- 21 days
- 28 days
- 45 days
- 60 days
- 90 days

Where "Automatic" means from the previous backtest or from the Generator's OOS settings (also Validator and Reactor).

Any thoughts on that?

Popov's Website

2 Reply by geektrader (edited by geektrader 2021-10-25 20:17:36)

  • From: Germany
  • Registered: 2010-04-05
  • Posts: 835

Re: Sort Collection by profit on unknown data (Work in Progress)

Sounds perfect, always was hoping for such a feature. Right now, I do it by recalculating the collection on a limited data horizon to sort it by "OOS". If you could implement it the way you describe it (including the "Automatic" one which uses the OOS amount from the backtest settings), that would be an absolutely great and welcome feature :-)

However, it would be nice to have the same sorting criteria like for the whole backtest. E.g., not just "OOS Profit"; but also "OOS return/drawdown ratio", "OOS r-squared" (that´s what I always use), etc.

The actual best and most organized way would be to keep the sorting menu as it is, and simply add a checkbox below it that says "Apply to OOS". This way the dropdown list keeps its current size and we can easily apply all these criteria to OOS instead of the whole backtest by a simple checkbox.

As for the length for OOS, I would not mess with days, but simply make it "Auto", "10%", "20%", etc. in an extra selectable box under the "Apply to OOS" checkbox.

geektrader's Website

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Re: Sort Collection by profit on unknown data (Work in Progress)

> "Apply to OOS"

A great idea!
It can be in the OOS menu instead of a checkbox (see below)

> I would not mess with days, but simply make it "Auto", "10%", "20%

I want to have days because it will allow us to choose them regardless of the length of the data series for recalculation.
It can be something like

-  Full backtest
-  OOS (Automatic)
-   7 days ( 0.97%)
-  14 days ( 1.94%)
-  21 days ( 2.92%)
-  28 days ( 3.89%)
-  45 days ( 6.25%)
-  60 days ( 8.33%)
-  72 days (10.00%)
- 144 days (20.00%)
- 216 days (30.00%)
- 288 days (40.00%)
- 360 days (50.00%)

Additionally, I'll make the sorting stats always visible in the 5-lines backtests stats table. If it is not among the chosen params, it will replace the 5th line of the table.

I think to add an up or down arrow to show the sorting stat. Upp-arrow when higher is better. Down-arrow when lower is better (Drowndown, Stagnation...)

Popov's Website

  • From: Germany
  • Registered: 2010-04-05
  • Posts: 835

Re: Sort Collection by profit on unknown data (Work in Progress)

That sounds very good!

As long as we can possibly have both, % and days (maybe even simply let us enter how many days or % instead of dropdown entries), that should be a new great feature :-) Eager to see what you come up with... The arrows are a great idea too.

geektrader's Website

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,988

Re: Sort Collection by profit on unknown data (Work in Progress)

If we decide to use OOS sorting, EA Studio will need to recalculate the collection to have the necessary stats.
Fortunately, the Collection calculation is more performant now.

It looks like we can make not only OOS sorting but OSS filtering as well. It will be handy when the  "Use performance filters." is on.

I was thinking about how to make it faster.
One possible solution is to calculate the necessary OOS stats when the Generator pushes a new strategy to the Collection. Then to store that data in the Collections records.

There are two benefits with that:
- It will be fats because we will calculate the OOS stats only once. The generator will without changes and will calculate the OOS info only if we have the OOS sorting enabled.
- The OOS info will be available when we reload EA Studio or when we import a Collection file.


https://image-holder.forexsb.com/store/sorting-metrics-scope-thumb.png


...

Actually, it may be possible to calculate the OOS stats without re-running a complete backtest. I'll investigate the possibilities.

Popov's Website

  • Registered: 2015-02-22
  • Posts: 430

Re: Sort Collection by profit on unknown data (Work in Progress)

OOS sorting is a really great and convenient feature, I dont need to cut data horizon again, thank very much for this update