Forex Software — Methods for Evaluation Strategy Performance

Re: Methods for Evaluation Strategy Performance

2012-03-24T13:12:31Z

I think that we are looking at 3 different topics in this thread now:
1. Requested metrics and stats for strategy performance;
2. The reasons why there really is no such thing as OOS; and
3. Some form of way to product a highy detailed strategy performance report.

I will comment on #1 and #3, and stay out of the other discussion.

Metrics and Stats

I ignore almost all the math and statistical values that are currently in FSB, and look for the least 'bumpy' equity line. I'm not paticularly interested in the various ratios available, as net profit doesn't really mean anything in FX as we can use higher leverage and/or more a different lot size to ramp up the profitability of a strategy that has a good equity line. There are some methods (http://en.wikipedia.org/wiki/Correlation_and_dependence) to achieve this. Saying this, I am always interested in the minimum number of trades, and also the % accuracy. Having a small number of trades, or a very high % accuracy will generally result in correlation values that are meaningless so I really just use these as filtering mechanisms.

As far as making changes to the generator process and modifying the top 10 output goes, this is a lot more difficult than it looks (I made some attempts to do exactly this), although I believe that there is a solution for users who wish to do their own post-generator analysis. As there are only a small number of users (judging by the interest in the thread) interested in this feature, it may not be worthwhile adding it to the core application, but incorporating it in another way.

I have been experimenting with the development of a 'plugin' interface for FSB that would allow that application to work with external compiled DLL's as extensions in a similar way to how the application can currently work with custom indicators. The plugin interface would allow for smaller development efforts to be maintained without the need to incorporate additional complex functionality into the core application. Problems with these external plugin modules could be isolated by disabling the 'Enable Plugins' option in FSB (that would need to be added), allowing the main codebase to the maintained without adding (or maintaining) logic that is only used by a small number of people.

What may be possible is to develop a plugin that would be called after each strategy is developed by the generator - what occurs in the DLL plugin would be independent of FSB, and any sort of analysis and filtering could be performed by the plugin (such as calculating the stats, and recording the strategies based on out custom criteria).

This would require considerable work, but once the plugin interface code in incorporated into FSB, there would be very little maintenance overhead involved.

Strategy Performance Report

In the same way, a custom strategy performance report could be developed using a plugin module that doesn't involve a larger number of FSB changes for a small number of users.

I have a lot of ideas on this topic and the comments above probably just scratch the surface ... I'll join the Skype session tomorrow and we can discuss this more if there is come interest.

Re: Methods for Evaluation Strategy Performance

2012-03-24T10:37:59Z

I thought the Out of Sample testing would help with generating good strategies but I find that what it does is to bring the OOS data into the in sample data because it is still rejecting strategies based on bad OOS performance.

This is a very important conclusion and it's completely true. There is no such thing as OOS.

That's the reason we want to develop an algorithm for evaluating strategies. I think on this matter for last 10 years and I have some ideas, but not a practical application for now. We can discuss this on our Sunday meeting tomorrow.

Re: Methods for Evaluation Strategy Performance

2012-03-24T08:23:54Z

For sometime now I have been concerned about how strategies are generated and how they often end up curve fitted to the extent that the strategies fail to work on walk forward data. FSB has developed some marvelous strategies for me that produce spectacular equity curves, that is until they are applied to Out Of Sample data. By Out of Sample Data I mean data not part of the test set (including the OOS bars).

I thought the Out of Sample testing would help with generating good strategies but I find that what it does is to bring the OOS data into the in sample data because it is still rejecting strategies based on bad OOS performance. I think what we need to do is run the the strategy generator to develop a good strategy and only then should it be tested against OOS data. If it fails then all the parameters should be reset and it should start again.

By doing it the way it now generates I feel the generator effectively 'contaminates' the in sample data with OOS bars.

Re: Methods for Evaluation Strategy Performance

2012-03-24T05:03:42Z

I'd like it if we could select our own stats to sort by in the top 10.

This will be very useful, I'll make some kind of tool that will allow selection and arrangement of stats parameters for the Account Statistics info window.

Another idea is that perhaps these stats could be more modular, like custom indicators, such that one could create his own module and just drop in a folder to be able to add that stat on the master list of available choices.

This is also another useful approach. Such modules can have access to all backtester statistics and to return a Parameter Name and a Parameter Value. At the beginning we can experiment with showing such custom stats params in the statistic screens and later we can add them in Generator / Optimizer limitations.

Re: Methods for Evaluation Strategy Performance

2012-03-24T01:15:13Z

I'd like it if we could select our own stats to sort by in the top 10. For example, what if we had a collection of various stats to choose from, and then we could multi-level-sort them in order of importance. Limit the user-selectable amount to 5 perhaps.

So for example, I might tell it to sort by:
1) highest efficiency stat (profit per day / max DD)
2) lowest max equity % drawdown
3) highest equity
4) highest reward ratio
5) lowest % time in market

Another idea is that perhaps these stats could be more modular, like custom indicators, such that one could create his own module and just drop in a folder to be able to add that stat on the master list of available choices.

Re: Methods for Evaluation Strategy Performance

2012-03-23T18:40:38Z

Popov wrote:

Nice link.
We'll discuss these parameters on our Sunday Skype chat.
We have to decide what parameters to show on Account stats on the main screen and what to leave in the Overview.

Would it be easy enough to make it user-selectable what stat is shown and which isn't? I know Krog modified the Account stats that it would show straightaway what he needed to see without scrolling the stat window.

Re: Methods for Evaluation Strategy Performance

2012-03-23T18:34:56Z

Blaiserboy wrote:

We want to add some meaningful formulas to limitations and perhaps overview... just to make analysis more thorough.
There may be a way to change limitation screen so as to make the results much better, ie. use Sharpe Ratio as once criteria.

I was thinking that it would reduce "the thorough analysis" part of strategy development - better to have a pool of strats with desired evaluative stats in the first place, rather than going through a pool of strats to find which one actually qualifies. Certainly there is a downside to this approach, but does it outweigh the upside?

Re: Methods for Evaluation Strategy Performance

2012-03-23T18:12:35Z

We want to add some meaningful formulas to limitations and perhaps overview... just to make analysis more thorough.

There may be a way to change limitation screen so as to make the results much better, ie. use Sharpe Ratio as once criteria.

Re: Methods for Evaluation Strategy Performance

2012-03-23T17:59:48Z

Popov wrote:

Nice link.
We'll discuss these parameters on our Sunday Skype chat.
We have to decide what parameters to show on Account stats on the main screen and what to leave in the Overview.

Does this mean those parameters will not be included in generator/optimizer limitations?

Re: Methods for Evaluation Strategy Performance

2012-03-23T17:35:54Z

So far, there is not a lot of interest, I am reluctant to ask that you change anything until we see more people comment as to what would be desirable.

Re: Methods for Evaluation Strategy Performance

2012-03-23T17:05:31Z

Nice link.
We'll discuss these parameters on our Sunday Skype chat.
We have to decide what parameters to show on Account stats on the main screen and what to leave in the Overview.

Re: Methods for Evaluation Strategy Performance

2012-03-23T04:31:16Z

here is a list of trading system metrics

http://www.earnforex.com/report-analysis/

Re: Methods for Evaluation Strategy Performance

2012-03-22T23:12:52Z

Does anyone have anything to add to this thread.... any useful formulas that can be added.

At some time the developer will want to have a look at this thread with a view to making changes to FST, hopefully whatever we want to have included will be listed for him to decide on.

Re: Methods for Evaluation Strategy Performance

2012-03-17T05:28:33Z

Blaiserboy is right
Once we have a complete formula or algorithm, example and description, we can add the parameter to the program's account statistics or Generator limitations. For time consuming algorithms we can use the Analyzer tool.

Re: Methods for Evaluation Strategy Performance

2012-03-16T16:59:31Z

I think that if we can define these carefully and completely that they can be incorporated into the limitations screen.

We will have to outline a formula for each and then we can ask developer to add them