Topic: System Quality Number

Regarding the sort in FSB, here is a guide to the values in System Quality Number (Van Tharp)

The SQN number can be interpreted as your overall trading 'grade'. This data should not be deemed reliable until (> 30 trades). Stop prices must be entered for each trade to be accurate.

score
1.6 - 1.9  Below average but trade-able
2.0 - 2.4  Average
2.5 - 2.9  Good
3.0 - 5.0  Excellent
5.1 - 6.9  Superb
7.0 and above..... wow Maybe you have the holy grail...!!

Refer to the work of Van Tharp for further.

Hopefully this helps as we develop our systems

Re: System Quality Number

Blaiserboy wrote:

Regarding the sort in FSB, here is a guide to the values in System Quality Number (Van Tharp)

The SQN number can be interpreted as your overall trading 'grade'. This data should not be deemed reliable until (> 30 trades). Stop prices must be entered for each trade to be accurate.

score
1.6 - 1.9  Below average but trade-able
2.0 - 2.4  Average
2.5 - 2.9  Good
3.0 - 5.0  Excellent
5.1 - 6.9  Superb
7.0 and above..... wow Maybe you have the holy grail...!!

Refer to the work of Van Tharp for further.

Hopefully this helps as we develop our systems

Hi,

overall it works well but sometimes it put on the top the strategy with very few trades and very littel profit and mark the quality number as infinit.

Re: System Quality Number

I read in a couple forums that there has to be a minimum of 30 trades for it to be valid......

I have seen the infinity also..... and discarded the result.

I appreciate having this metric, I tossed my old robots and started anew when this became available.

Re: System Quality Number

Sorting in FSB Pro will be much better.
I think to make three criteria where the first will be Net profit and other two will be chosen by the user.
The program will keep all strategies that passed the Acceptance Criteria and they will be sorted according to the three criteria.
Each strategy will have a rating showing the per cent value of the sorting parameter. The top strategy will have 100% and each strategy with lower param will have a proportional rating.

Finally, there will be a forth column that will sort the strategies according to the total rating as a sum of all three ratings.

I have not decided yet how to implement this sorting. It can be in Generator page or in a separate Tab, which can compare all generated strategies (from different Generators).

Re: System Quality Number

I wonder if some of these results could be exported.. say the top 3 or 5 results, so that we can examine the components of each strategy.

System Quality Number is most valuable, I really appreciate its inclusion in the program.

Re: System Quality Number

I wonder if some of these results could be exported.

I may add a Save As button on strategies records.

All these options will be available in the futures Alpha and Beta versions so we can correct them.

Other variant I'm considering is that the strategies can be shared on cloud storage. After that a user may sort thousands generated strategies.

Re: System Quality Number

Strategies are based on data and as  data and market conditions shift a strategy becomes inoperable. I am not so sure a strategy downloaded would be of a lot of use for much more than a few weeks.

Building strategies, developing techniques of using the software may be more valuable than storing strategies and would be more economical, that could be done in this forum.

Some people should comment on this as this could be an added expense if strategies are stored online.

The basis of my thinking is.... 'Would it be better to show people how to use the software than to give them results which they will not understand'

Just my opinion.

Re: System Quality Number

I added SQN in standard stats in FSB Pro.
SQN will be added also to Acceptance Criteria for Generator and Optimizer.

I made some modifications to the FSB Pro  formula comparing with FSB.

New code is:

private void CalculateSystemQualityNumber(IBackTester backtester)
{
    // Build the Trades List Curve
    var trades = new List<double>();
    for (int pos = 0; pos < backtester.PositionsTotal; pos++)
    {
        Position position = backtester.PosFromNumb(pos);

        if (position.Transaction != Transaction.Close &&
            position.Transaction != Transaction.Reduce &&
            position.Transaction != Transaction.Reverse)
            continue; // There is no profit/loss taken.

        trades.Add(position.ProfitLoss);
    }

    if (trades.Count < 30)
        SystemQualityNumber = 0;
    else    
        SystemQualityNumber = Math.Sqrt(trades.Count)*trades.Average()/StdDev(trades);
}

Edit:
Custom Account Stats file is updated with SQN. You can try it  with current FSB Pro:
Edit: SQN added in Acceptance Criteria.

Re: System Quality Number

These make FSB Pro a most powerful tool for a developer.....!

Thanks.

Re: System Quality Number

The generator, when sorting on System Quality Number, will end up with a result that does not have a trailing stoploss or hard stop as it grinds through the numbers.

In order to ensure that there is a stop or stop loss in every result, I unchecked everything in 'Closing Point of the Position' that did not have a  trailing stop or stoploss in it...

And that resulted in some nice numbers for both System Quality Number and Sharpe Ratio.

The bad news is that I do not like the ratio of take profit to stop or trailing stop.

The good news is that I have a starting point to develop a good ratio.


Maybe that will help someone who is having difficulty controlling System Quality Number in the generator.

If someone else has an idea as to how to use this, please add your suggestion.

11 (edited by mel8331 2014-05-26 14:57:46)

Re: System Quality Number

http://blog.bigmiketrading.com/2009/07/van-tharps-max-expectancy-and-system.html
============================================
Quoting caprica, here is a quick definition of Max Expectancy:

So what is expectancy?

Expectancy is your profit percentage per win multiplied by your win rate minus your loss percentage per loss multiplied by your loss rate.

Expectancy tells you what you can expect to make (win or lose) for every dollar risked. Casinos make money because the expectancy of every one of their games is in their favor. Play long enough and you are expected to lose and they are expected to win because the “odds” are in their favor.   

Example, you could have 99 losing trades, each costing you a dollar. Thus, you would be down $99. However, if you had one winning trade of $500, then you would have a net payoff of $401 ($500 less $99)—despite the fact that only one of your trades was a winner and 99% of your trades were losers.

And again quoting caprica, here is a quote on System Quality Number:

The only way to seriously qualify and optimize any system is through its System Quality Number (SQN). I advice you to refer to Van Tharp for reference on the subject.

Assuming a set of N trades (N>30 for being statistically significant), SQN is defined as follow:

SQN= Squareroot(N) * Average (of the N Profit&Loss) / Std dev (of the N Profit&Loss).

The large the N, the more trading opportunities you have.
The large the average P&L, the better you are obviously.
The smaller the Std dev (P&L), the more regular are your results and the smaller are the drawdowns.

Note here that if you optimize for the largest SQN, you maximize in fact the product N*average P&L and you minimize the Std dev (P&L) and the drawdowns at the same time.

This is exactly what all good traders should be looking for their system.
============================================

Can we have the  Std dev (P&L) displayed below the SQN in the STB Pro for better info since it's already calculated in arriving at the SQN ?

Re: System Quality Number

Can we have the  Std dev (P&L) displayed below the SQN in the STB Pro for better info since it's already calculated in arriving at the SQN ?

It can be added very easy because FSB Pro supports a custom Account Statistics code.


http://s7.postimg.org/okkbhb62f/screenshot_380.jpg

I set the stdev of P&L to a global variable and added a new record for the info panel:

records.Add(new InfoRecord
{
    Name = "Std dev profit loss",
    Value = stdProfitLoss.ToString("F2"),
});

The code is available in the Repository: http://forexsb.com/repository/repositor … cs-default

Re: System Quality Number

Thank you, with this information one can chose the better strategy for use in trading.

Re: System Quality Number

More explanation on the SQN.

http://www.blueowlpress.com/WordPress/articles/measuring-system-quality/

http://www.blueowlpress.com/WordPress/articles/measuring-system-quality/measuring-system-quality-page-2/

In the second part of the article, it has a chart for Maximum Drawdown as percentage of Maximum Balance and another chart for Terminal Wealth after 500 trades.

Can FSB Pro generate the two charts shown in the article, I believe it will provide use more information on how much to fund the system to be able to cater for the expected drawdown and if it is worthwhile to trade the system on a long term basis?

Re: System Quality Number

Found an article " Profitable ETF Trading Strategies: reflections on System Quality Number" @ http://kansasreflections.wordpress.com/2009/05/18/profitable-etf-trading-strategies-reflections-on-system-quality-number/
===========================================================================

here is my take on the IITM System Quality NUmber idea (SQN): and fat right tails. it’s what I said to chuck whitman on this topic

on my -1R loss exits: this is the result of a single trade decision cycle on that trade that is very effective

now, on the possibility of a 10R win that “skews” the histogram of results and increases the variability of the data set and therefore lowers SQN and therefore lowers recommended risk

the essential question is this: was the 20R a consequence of the act of entry, of the fact of entry, independent of any trader decisionmaking along the way?

if you have no influence over the achievement of 20R in the trade, then the argument that the increased variability suggested by the 10R return should reduce the SQN is warranted

because upside variability that is a result of the market’s “decision” ought to imply the potential of the same kind of downside variability suprising you

that is the essence of the marble game, that once you decide to play that the result is pure random function generator

BUT: what if you had played the washout pattern in SPY on 10 March and followed every rule and were able to manage the trade thru 6 iterations of successive Washout Cointinuation patterns and as a result of 7 cycles of trade decisions, were able to bring in 10R, yet never risked more than 1R on the downside

everything is now a function of how you classify that trade batch

if you say: that is one episode, and the trade is 10R, and the 10R implies i could have taken a 10R loss which is beyond my management ability, then i will say i disagree

if we have traded a WO pattern system 200 times and my 100 losses show an avg loss of less than 1R, AND my worst loss was 1.4R, and my trading mgt skills enabled me to manage risk properly all the way up the ladder, then simply treating the WOCs as 6 separate trades on avg of 1.6R (6x 1.6R = 10R) gives a true picture of the quality of the system

the Sortino ratio which examines the stats of just the losses with StDev is the right way to understand the losses and since computation is free you SHOULD do both the standard SQN AND a Sortino to better understand your loss pattern to make your decision on how much risk to take on

now, if i can get a 10R thru intraday trade mgt by getting a carefully engineered, risk controlled, very manageable morning hook which gives me a boost and my trade is a single continuous episode but Inever was in a position to experience a 10R loss, then you would be nuts to penalize that system

you MUST really know your edge, and where the 10R comes from and decide if it truly represents the possibility of expereincing a 10R loss

you MUST know your system thoroughly in order to create meaning, to fully understand its risks (as much as it can be understood)

portfolio heat rules and other heuristics must be in place to protect us against -10Rs beyond our control such as power outages, discontinuity in the mkts, so that we never commit the hubris error of LTCM

we are no where near that in our application

here is where SQN is VERY VALUABLE: when i have 5 sets of mechanical rules that i am testing, iindependent of trader discretion, as a check on the robustness of the mechanical framework           

SQN can give me the basis for deciding which to pursue, to revise and extend

what your studies are revealing is that you are understanding how the SQN math works

this is a good thing

example: this morning in the seminar we looked at a Triple Screen on GOOG which if executed mechanically would have given a .8R win going in to the close, but which had “open risk” ie trader initial cpaital at risk AND was in the red for 5 hours until near the close

by applying trader Quality to it, we engineered the entry timing in order to get a 2R iStop, and scratched the 1st trade, and then earned 6R on the identical setup on the next leg up.

the open risk was 15 minutes, when we moved to no lose, and then we spent 4 hours in the green until deciding to cash a 5R before the close

i guarantee that the 5R win should not be interpretted as implying increasing downside volatility and thus lowering SQN

coming to that conclusion would demonstrate IN MY OPINION, an inferior understanding of interpreting SQN

let me suggest what true Quality measure could/should include: amount of open risk x # of minutes

1st GOOG example: 15 min red all of which was <1R and then all green rest of the day up to a 5R win

correction that was the 2d GOOG example: the 1st GOOG exmaple eas mostkly red all day = lots of pain, low quality; compare the “time area in the red” to the “time area in the green” to really understand the “quality of each system.

charts to follow

bottom line: you better know where your outsize R comes from

you must be ruthlessly focusing on your R losses to ensure you are calibrated in identifying mkt risk to your idea

you must be open to the potentials of achieving risk managed high R wins, ie fat right tails

the end

=========================================================================

The statistic is beyond me, but I felt that the FSB Pro should advice us on the risk based on the criteria above of the systems found so we reduces the risk and understand more of it.

Re: System Quality Number

Blaiserboy wrote:

Regarding the sort in FSB, here is a guide to the values in System Quality Number (Van Tharp)

The SQN number can be interpreted as your overall trading 'grade'. This data should not be deemed reliable until (> 30 trades). Stop prices must be entered for each trade to be accurate.

score
1.6 - 1.9  Below average but trade-able
2.0 - 2.4  Average
2.5 - 2.9  Good
3.0 - 5.0  Excellent
5.1 - 6.9  Superb
7.0 and above..... wow Maybe you have the holy grail...!!

Refer to the work of Van Tharp for further.

Hopefully this helps as we develop our systems

The interpretation of the score is only good if the number of trade is equal or higher than 100 ( http://tradingdiary.wikispaces.com/System+Quality+Number+(SQN) ).

17 (edited by mel8331 2014-05-30 00:25:30)

Re: System Quality Number

PunBB bbcode test This I believe is for a SQN=3 and at the 64th trade the result will have a big variation in result .

I suspect that strategy found by FSB that has low System Quality Number cannot be expected to use for trading too long into the future as the expected result can be much different from that given by FSB.

It also mean that you will have to use the latest data if you wanted the result to have some resemblance to that indicated by FSB.

Re: System Quality Number

OMG.. is this system quality number reliable?? Coz I have 3 strategy with SQN >8, 2 strategy with SQN > 6

I trade only DAILY data, all my strategy produce 200+ trade in 13 years daily data..

My market instrument on 5 major PAIR specification already PESSIMISTIC scenario (Spread, n swap larger than actual)..

This is my equity curve..

https://fbcdn-sphotos-g-a.akamaihd.net/hphotos-ak-xaf1/v/t1.0-9/s720x720/10556521_4571839389571_731074474455655538_n.jpg?oh=3fd1d9bb3ecc2fc6c59047d655b53f57&amp;oe=547D210F&amp;__gda__=1417527325_16bab19549770320b2e51ee82971eb24

https://fbcdn-sphotos-f-a.akamaihd.net/hphotos-ak-xpf1/v/t1.0-9/q82/s720x720/10615637_4571839549575_8644377634147801185_n.jpg?oh=67e396d1b93c41e4cb0f4a2bb2234046&amp;oe=5463C6A3&amp;__gda__=1416589237_56045cb3daf89fcdb11f6cafc8893c63

is my strategy too good to be true?? I alr check the robustness by moving paramater and TP and it's seems that the performance not quite sensitive when I change the input parameter..

Any Suggestion where my mistake is??

Re: System Quality Number

The only way you will know if your strategy is reliable is with extensive forward testing and walk forward.

The numbers may look great but the system can still fail.

For example, I have a scalping system that may endure close to 90 small losses before it gains..... the numbers are atrocious, but the profits are lovely.

I tend to look at the equity curve as the main determinant.

All of the ratios can be perfect and when the market throws a new face........ there can be a disaster, there fore, extensive walk forward and live forward testing is essential.

I suggest that you use the numbers as, perhaps, a benchmark only, and devise a thorough method of walk forward analysis using OOS in order to analyze your system.

Just an opinion.

here is a system with some terrible looking numbers, except that the equity curve is lovely as are the profits

Post's attachments

DaveM-0027.png
DaveM-0027.png 45.24 kb, 1 downloads since 2014-08-31 

You don't have the permssions to download the attachments of this post.

Re: System Quality Number

Blaiserboy wrote:

...when the market throws a new face........

Determine when and how that happens, and everything will be immensely easier.

Re: System Quality Number

Will it be included also to free version?

Re: System Quality Number

kazik1616 wrote:

Will it be included also to free version?

System Quality number is already in the free version, it has been there for many, many months.

You will see it in the 'Generator', under 'Sorting'.

You will also see it in the strategy overview.

Re: System Quality Number

Blaiserboy wrote:

For example, I have a scalping system that may endure close to 90 small losses before it gains..... the numbers are atrocious, but the profits are lovely.

I tend to look at the equity curve as the main determinant.

here is a system with some terrible looking numbers, except that the equity curve is lovely as are the profits

I know this post is quite old but those numbers don't really look that bad when you take into account the equity drawdown, clearly your stop loss is very tight and if the trade doesn't fire off like a rocket immediately the trade is nullified and stopped out.

Anyways that's not really what my main question is on this whole thread. I generally try to optimize a strategy which I think will work well with wide parameters using SQN as the "search best" then when it finishes I generally use something like +-10 with net balance to further optimize it in hopes it'll find more trades at the expense of W/L ratio, possibly drawdown and in the end be more profitable over time. The thing is how do I know if my strategy is over-optimized or curve fitted? I know I could use the %OOS to see how those inputs work and obviously forward testing in real time. But is there a way to help avoid over optimizing strategies?