Topic: Methods for Evaluation Strategy Performance

This thread will be specifically for developing methods and sequences for evaluating strategies.

It is hoped that we can come up with several techniques which will enhance our trading, shloud we be able to provide good information and methods, the developer of FSB may be able to incorporate some of them into the program.

Feel free to contribute any and all ideas and some background and formulas if you have those available.

People with an investment background use certain measures and ratios, Quant type people use different ones.

Perhaps we will be able to gather many ideas that can be enhanced and that will be useful to the whole community.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

I'd like something about the losing streaks, because when the strategy goes live and it loses 5 times in a row, it is a big question if that is within expected results of the backtest, or if it indicates the strategy no longer works.

Re: Methods for Evaluation Strategy Performance

Good point....... maybe program can be changed to reflect number of consecutive losses and number of consecutive wins.

It is pretty discouraging to have an expert ready to go and it gets clobbered as soon as it starts live trading....

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

Performance Summary Example

part 1

Post's attachments

daveM - SNAG-00146.png
daveM - SNAG-00146.png 88.99 kb, 1 downloads since 2012-03-14 

You don't have the permssions to download the attachments of this post.
My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

Performance Summary example

part 2

Post's attachments

daveM - SNAG-00147.png
daveM - SNAG-00147.png 27.97 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.
My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

How to know when your strategy is 'busted'

The first thing to assess is whether the strategy is catching the moves for which it
was designed. If your strategy ever misses a move of the type for which you
designed it, the strategy has busted.
A trend-following strategy is designed to lose money in sideways markets make it
all back and more in the trend. If your trend-following strategy misses the big
move, it clearly has busted. A number of losing trades in a row does not mean the
strategy is not working. Missing the big move does.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

Sharpe Ratio  Already included in FSB Strategy Overview (Bottom Line)

A ratio developed by Nobel laureate William F. Sharpe to measure risk-adjusted performance. The Sharpe ratio is calculated by subtracting the risk-free rate - such as that of the 10-year U.S. Treasury bond - from the rate of return for a portfolio and dividing the result by the standard deviation of the portfolio returns. The Sharpe ratio formula is attached

Sharpe Ratio

The Sharpe ratio tells us whether a portfolio's returns are due to smart investment decisions or a result of excess risk. This measurement is very useful because although one portfolio or fund can reap higher returns than its peers, it is only a good investment if those higher returns do not come with too much additional risk. The greater a portfolio's Sharpe ratio, the better its risk-adjusted performance has been. A negative Sharpe ratio indicates that a risk-less asset would perform better than the security being analyzed.

A variation of the Sharpe ratio is the Sortino ratio, which removes the effects of upward price movements on standard deviation to measure only return against downward price volatility.

Post's attachments

daveM - SNAG-00148.png
daveM - SNAG-00148.png 12.33 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.
My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

Sortino Ratio

A ratio developed by Frank A. Sortino to differentiate between good and bad volatility in the Sharpe ratio. This differentiation of upwards and downwards volatility allows the calculation to provide a risk-adjusted measure of a security or fund's performance without penalizing it for upward price changes. It it is calculated as follows:

Post's attachments

daveM - SNAG-00149.png
daveM - SNAG-00149.png 19.47 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.
My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

It may be that the ratios etc could be exported to a preformatted spreadsheet as FSB already has export capability, that may be simpler than revising the program.

I have no experience with spreadsheets........ woud it be possible to export a spreadsheet that is formatted with the various formulas and ratios......?

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

10 (edited by dusktrader 2012-03-15 16:18:58)

Re: Methods for Evaluation Strategy Performance

krog wrote:

I'd like something about the losing streaks, because when the strategy goes live and it loses 5 times in a row, it is a big question if that is within expected results of the backtest, or if it indicates the strategy no longer works.

Moreso than losing streaks, I think it is important to identify a non-subjective way to know when a live-trading bot has become "out of spec" per the FSB testing.

According to some other reading I have done (actually per your suggestion elsewhere) -- I now consider strategies that "flat line" for a period of OOS to be acceptable whereas ones that decline too much during an OOS test are not-acceptable to me.

I am currently tracking new-found bots using 50,000 bars of M1 data with 30% OOS enabled, but rolled back 2 weeks into the past via Data Horizon. This allows me to evaluate strategies from a 2-weeks-ago point of view.  I use the 30% OOS as a pre-screen only. If the strategy looks GREAT, including the 30% OOS portion, I then move to the next phase.

In this next phase, I am still working with 2-weeks-old data. I run Optimizer for the indicators and also find the optimal Permanent SL/TP/BE.

Lastly, I take this developed strategy and use Data Horizon to see how it would handle OOS data in the next week and also next-next week. These two tests are the most important factors in my decision of whether or not to go live. During this phase I record all stats and also screenshot the equity curve. In the stats, I'm looking for these specific and non-subjective things:

  • max balance DD -- want to see this the same

  • max equity DD -- want to see this the same

  • max equity DD% -- want to see this the same

  • reward ratio -- bonus if it has improved

  • efficiency -- bonus if it has improved

After the non-subjective checks, if all still looks good, I evaluate (subjectively) the change in the equity curve plot. I'm looking for a flat or preferably rising equity line. Declining line is ok ONLY if its decline was muted such that it did not increase those hotspots listed from the stats.

This method gives me about 3.5 weeks of OOS analysis. I'm not sure if it's "enough" for trading M1 or not.

Anyway... back to the original concept here --- what would be great is if FST could automatically trigger an alert when certain metrics fall out of spec. These could be self-programmed for example. In some ways, I already do this by enabling the minimum account balance feature of the expert. I key the bots to trade at a level where the tested-max-equity-DD% equals my comfort level (say 10%), using the formula I've posted elsewhere. Then, I set the expert to stop trading if it exceeds this level (say 12% for cushion). In that way, I know if a bot stops trading then it has exceeded a critical level (max equity DD%) and has fallen out-of-spec.

I also analyze every live-trading bot on weekends to determine whether or not they are still in-spec or have fallen out-of-spec. I rotate the live-traders weekly.

Re: Methods for Evaluation Strategy Performance

Blaiserboy wrote:

It may be that the ratios etc could be exported to a preformatted spreadsheet as FSB already has export capability, that may be simpler than revising the program.

I have no experience with spreadsheets........ woud it be possible to export a spreadsheet that is formatted with the various formulas and ratios......?

Yes, this can be done, exporting spreadsheets is not too difficult, the FSB class for exporting is Common\Exporter.

Re: Methods for Evaluation Strategy Performance

dusktrader wrote:

...
Moreso than losing streaks, I think it is important to identify a non-subjective way to know when a live-trading bot has become "out of spec" per the FSB testing.
...

That makes sense, if the critical losing streak is 5 and it performs as 4 L - 1 W - 4 L - 1 W ... the critical streak number doesn't help very much. It might be detectable if it draws down the win/loss ratio too much.
I like Blaiserboy's common-sense approach of verifying it catches the moves it was designed to catch. I don't know how that could be automated though, sounds like it would have to be manually reviewed.

Re: Methods for Evaluation Strategy Performance

I think that if we can define these carefully and completely that they can be incorporated into the limitations screen.

We will have to outline a formula for each and then we can ask developer to add them

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

Blaiserboy is right
Once we have a complete formula or algorithm, example and description, we can add the parameter to the program's account statistics or Generator limitations. For time consuming algorithms we can use the Analyzer tool.

Re: Methods for Evaluation Strategy Performance

Does anyone have anything to add to this thread.... any useful formulas that can be added.

At some time the developer will want to have a look at this thread with a view to making changes to FST, hopefully whatever we want to have included will be listed for him to decide on.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

here is a list of trading system metrics

http://www.earnforex.com/report-analysis/

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

Nice link.
We'll discuss these parameters on our Sunday Skype chat.
We have to decide what parameters to show on Account stats on the main screen and what to leave in the Overview.

Re: Methods for Evaluation Strategy Performance

So far, there is not a lot of interest, I am reluctant to ask that you change anything until we see more people comment as to what would be desirable.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

Popov wrote:

Nice link.
We'll discuss these parameters on our Sunday Skype chat.
We have to decide what parameters to show on Account stats on the main screen and what to leave in the Overview.

Does this mean those parameters will not be included in generator/optimizer limitations?

Re: Methods for Evaluation Strategy Performance

We want to add some meaningful formulas to limitations and perhaps overview... just to make analysis more thorough.

There may be a way to change limitation screen so as to make the results much better, ie. use Sharpe Ratio as once criteria.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Methods for Evaluation Strategy Performance

Blaiserboy wrote:

We want to add some meaningful formulas to limitations and perhaps overview... just to make analysis more thorough.

There may be a way to change limitation screen so as to make the results much better, ie. use Sharpe Ratio as once criteria.

I was thinking that it would reduce "the thorough analysis" part of strategy development - better to have a pool of strats with desired evaluative stats in the first place, rather than going through a pool of strats to find which one actually qualifies. Certainly there is a downside to this approach, but does it outweigh the upside?

Re: Methods for Evaluation Strategy Performance

Popov wrote:

Nice link.
We'll discuss these parameters on our Sunday Skype chat.
We have to decide what parameters to show on Account stats on the main screen and what to leave in the Overview.

Would it be easy enough to make it user-selectable what stat is shown and which isn't? I know Krog modified the Account stats that it would show straightaway what he needed to see without scrolling the stat window.

Re: Methods for Evaluation Strategy Performance

I'd like it if we could select our own stats to sort by in the top 10. For example, what if we had a collection of various stats to choose from, and then we could multi-level-sort them in order of importance. Limit the user-selectable amount to 5 perhaps.

So for example, I might tell it to sort by:
1) highest efficiency stat (profit per day / max DD)
2) lowest max equity % drawdown
3) highest equity
4) highest reward ratio
5) lowest % time in market

Another idea is that perhaps these stats could be more modular, like custom indicators, such that one could create his own module and just drop in a folder to be able to add that stat on the master list of available choices.

Re: Methods for Evaluation Strategy Performance

I'd like it if we could select our own stats to sort by in the top 10.

This will be very useful, I'll make some kind of tool that will allow selection and arrangement of stats parameters for the Account Statistics info window.

Another idea is that perhaps these stats could be more modular, like custom indicators, such that one could create his own module and just drop in a folder to be able to add that stat on the master list of available choices.

This is also another useful approach. Such modules can have access to all backtester statistics and to return a Parameter Name and a Parameter Value. At the beginning we can experiment with showing such custom stats params in the statistic screens and later we can add them in Generator / Optimizer limitations.

Re: Methods for Evaluation Strategy Performance

For sometime now I have been concerned about how strategies are generated and how they often end up curve fitted to the extent that the strategies fail to work on walk forward data. FSB has developed some marvelous strategies for me that produce spectacular equity curves, that is until they are applied to Out Of Sample data. By Out of Sample Data I mean data not part of the test set (including the OOS bars).

I thought the Out of Sample testing would help with generating good strategies but I find that what it does is to bring the OOS data into the in sample data because it is still rejecting strategies based on bad OOS performance. I think what we need to do is run the the strategy generator to develop a good strategy and only then should it be tested against OOS data. If it fails then all the parameters should be reset and it should start again.

By doing it the way it now generates I feel the generator effectively 'contaminates' the in sample data with OOS bars.