Topic: New statistics: P-value

Hello, I would like to propose the insertion of a new statistic: the P-value.
It helps to understand the significance of a result within a hypothesis system.
In excel starting from "number of operations", "mean" and "standard deviation" you can reach the T-value and then the P-value.
I think it's a very useful tool.
Thank you, I am looking opinions :-)

Re: New statistics: P-value

I would add that a p-value assists in rejecting a null hypothesis but at the same time not offering probability on its truthiness. What would be your null hypothesis?

As far as I understand, results must follow certain distribution (usually normal distribution) for p-value calculation, so that poses a problem for trade data. There are a few methods which are more suited to evaluate trading strategies and I'm quite interested in them. The main issue or question for me personally is its usefulness as it would be better to know what can be realistically expected from developing such a feature before devoting a lot of time and effort into it. I see you are already convinced in its usefulness, would you be so kind and share your results and experience? smile

Re: New statistics: P-value

Hi Footon, thanks for your reply! :-)
Thanks to Grove Under, Jon Joh and Michael Harris.

How do you know if your trading system has an edge or whether it's just random luck? A lower p value is better; the lower the p value simply means that based on the series of trades analyzed, the results are less likely due to random chance or luck, in general usage, a p value of < 0.05 is often considered to be statistically significant (Fisher). Adam Grimes discusses that "standardizing for risk removes the position sizing effect" so that it could reveal that a system or trader via the p value could be "trading with a clear statistical edge, even though it was completely obscured by his position sizing decisions".
Now the current means of calculating statistical significance is by using first the t-statistic (Sharpe ratio (for the year) * sqrt(N years)) => if you have a few years with a high sharpe, then that's great; however some strategies that have a decent sharpe ratio but plateau or have negative performance for a year and don't rebound that much or rebound for a little but don't exhibit the same return distribution.
The S&P 500 example: the t-statistic for testing the null hypothesis that the average return, excluding 2017, is drawn from a population of zero mean, is given by:
t-statistic = (mean return/standard deviation) × sqrt(number of years)
= (8.722/16.735) × sqrt(66) = 4.2342
For a one-tailed hypothesis, the corresponding p-value is 3.3E-05. Therefore, the probability that the mean return of the sample is drawn from a population with zero mean is 0.0033% and the null hypothesis is rejected.

Talking about it is a good thing :-)

Re: New statistics: P-value

Popov is quite amenable to adding things to the program, especially if you can justify the addition.

I am sure he will consider your idea as he works on FSBPro, please keep in mind that he has a huge list of priorities so the addition may take a while to see.

He is reluctant to add things that slow the processing substantially as he wants many users to be able to generate strategies, not only the people with the high powered machines.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: New statistics: P-value

Hi Blaiserboy, I agree with you! I had the pleasure to note the quality of work done by Popov :-) With this post I wanted to send a new idea that can make more profitability at all. FSB is very good, there will be time to improve it

Re: New statistics: P-value

D5QM54S7 wrote:

...if you have a few years with a high sharpe, then that's great...

Hi -- I'm not a statistician, so I can't add anything about p-values.  However, something you mentioned did raise a red flag for me.  You refer to years of sharpe data as a good thing.  Forex data is different than solving other types of statistical problems where the available outcomes are finite and have a certain probability.  Since "optimizing" a strategy is analogous to tuning a mathematical formula to fit some input data, and since future data has infinite possible outcomes and we have no clue about the next bar, then it seems that using large chunks of historical data to draw insights would not be so valuable -- even, counter-productive.

In other words, statistical metrics are most useful when creating strategies and finding the best parameters to fit the data horizon you've chosen -- that is, solving y = mx + b to find the best 'm' and 'b'.  But once you've done that then there is nothing more to do.  If the 'm' and 'b' continue to work well for the next incoming data (i.e. the data continues its current pattern) then you win -- otherwise you lose.   I don't understand how analyzing results after the fact to determine whether trades happened due to good design or chance would provide an edge since I can easily explain good results by saying my strategy was well-trained and bad results by saying the data's pattern changed.

Re: New statistics: P-value

Working with t-stat is a bit more complex, the following article is covering all points - https://medium.com/@mikeharrisNY/the-s- … 2eb559abed

Distribution is most relevant, some kind of sampling is needed, bootstrapping or monte carlo are most common. Otherwise it will be with no real value, considering statistical point of view.

I have similar work in progress and for starters I think I can make p-value available in accountstats, so stay tuned smile

8 (edited by D5QM54S7 2017-10-10 16:02:06)

Re: New statistics: P-value

Hello Footon! That's right, working with the p-value is more complex. I study economics and encounter often statistical material, in addition I read many texts to evade from university reality. I gained little with manual trading on the daily market timeframe, so I bought this software after a few years (the university leaves me a little time). I love reading the texts that reject hypotheses (the power of mathematical thinking / freakonomics etc), rather than texts that teach something and I noticed that this kind of value is present everywhere, something will mean. No instrument can guarantee us profits, not even quantitative analysis. but here we try to have a simple statistical advantage and I think p-value can help in this
I like to have attracted the attention of more competent people than me :-)

Re: New statistics: P-value

Yes, p-value is widely used in almost all spheres of science, especially in statistical research and inference. To carry this to trading, it needs to be modified and this work has already been done more than a decade ago (most well-known is White's reality check) and there are couple of improvements developed over the past 5 years or so. The main problem is still there - statistically significant strategy is far from being a guarantee of future profitability. Second point is the bigger the dataset the easier to produce statistically significant results. Third point is the possibility to evaluate strategies one by one or as a pool of strategies, whose results contribute collectively in establishing statistical significance. Having minor experience with the former, I tend to believe that the latter is the better option, because individual evaluation produced p-values in the region of 0-0.0001 too often making me doubt in this approach. Unfortunately only accountstat file can be edited and this means individual evaluation, it is impossible for me to automate pool evaluation.

Lastly, about competence. Having a degree in humanitarian sciences makes work involving math harder than it should normally be smile I know I'll give my best but it remains to be seen if it'll be enough.

10 (edited by hannahis 2017-10-10 17:29:01)

Re: New statistics: P-value

I'm not a statistician (though I took 1 course in my uni and also needed it for my Master research thesis...but still it's not my cup of tea.  However, I did economics and I love it...so here are just some of my thoughts

1. I do believe in the midst of all those "random" price fluctuation, there are some "repetitive" patterns that doesn't arise due to chances.  I see them all the time in my charts, whenever these "conditions" happened, there are usually a significant trend.  The problem or challenge is how to exit timely as these trends are sometimes weaker than another.  Hence, I do believe we can statistically find these repetitive conditions that are not due to random chances.  Just like Google analytics, it tracks buyers profiles and buying patterns and by understanding their repetitive search or purchase patterns to predict what kind of advertisement would "suits" these users.  Though most retail purchases may seem to be an issue of impulses, there is some "predictive" behaviours behind the sum of all these random behaviours.

2. As much as we often think that forex is so "unpredictable", it is still often governed by basic economic factors or market forces such as Demand and supply.  It's the psychological part of speculation that fuel the volatility.

3. There are big players who dominate/dictate forex prices/trends and I believe these people's decisions are more often ruled by economic factors rather then whimsical and psychological (which most retail traders tend to react).  I don't think they wake up one day and decided to sell because they "feel" like it.  Since their decisions are more likely governed by forces of demand and supply (or fundamental issues), therefore it is possible to "see" a pattern behind every breakout or big movement made by these big players.  Hence, RSI is a good indicator to use to lookout for these "pressure" points.  Imagine if you happened to use the same "indicators" or whatever tools they use for their decision making, it's like "insider" trader whereby you have a close estimate of when/timing (not in terms of day and time but via indicators), whereby these big players are going to enter/exit the market. 

Hence, in a nutshell, I do believe there is predictive value (if there isn't, then we are all gamblers here) in forex and usually the shorter the time interval, the "stronger" the predictive value.  For example, which is easier to predict? To predict whether EURUSD prices will go up or down 1 year from now or 1 day from now?  Of cos, it's much "easier" to predict the prices for the next minute than for the next hour.  Hence, I believe if we focus on how much we can scalp out of the market at shorter intervals, we have a better chance of increasing our EA predictability because the longer we hold, the less certain we are about when the trend will start to change.  Thus, most of my EA trade very frequently at shorter intervals. 

However, from a position trader's point of view, there is also predictive value if you look at the market on a "bigger" picture.  And use D1 and higher chart as reference point.  Such trading method isn't my cup of tea because one need to have really large SL to stomach all those high and lows.  And the larger the SL, the smaller the lot size, relative to your capital, you are able to trade "safely".  Eg if your SL is 3000 pips and you only want to risk 10% of your capital, then you can only enter 1 lot size per $30k capital.  I personally believe such method isn't "economical" use of capital for optimum return and if you divide the returns over the long holding time needed, it may not be really that attractive investment considering the high element of risk vs return.

Re: New statistics: P-value

I don't mean to distract the discussion -- I read the article footon linked to below.

I have a fundamental question about applying statistics such as this to Forex data -- because I think Forex is a different beast.  When considering the S&P 500 the actual data is tied to companies whose only goal in life is to increase their value.  So, there is a built-in bias for the numbers to increase.  But in Forex there is no built-in bias.  Doesn't the nature of the data have some influence over which statistics are valuable and what insight they provide?

Now -- back to walking my dog...

Re: New statistics: P-value

It's not about evaluating market data but returns.

One thing about next bar - I disagree about having no clue. I know that tomorrow's daily bar can be one of three: up, down or doji bar wink

13 (edited by sleytus 2017-10-10 20:20:06)

Re: New statistics: P-value

Thanks, footon.  I always appreciate your insight.

Regarding tomorrow's bar -- yes, 3 possibilities.  But what about their probabilities?  When tossing a coin I know the next toss has two possible outcomes -- so, in that respect I can sort of predict the future.  Furthermore, I know if I toss a coin enough times the outcome will tend to 50 / 50.   But that is where I begin to lose the connection with applying statistics to Forex data or returns.  Even though you know there are 3 possible outcomes -- up, down, doji -- you have no clue about their probabilities.  All you can do is hope the pattern continues.  Come to think of it -- perhaps there is a statistic that can provide a clue as to the probability of a pattern continuing.  Is that where the p-value would come in?

14 (edited by D5QM54S7 2017-10-10 22:53:53)

Re: New statistics: P-value

If we stop before you can observe the T-test => T-test is a simple statistical test that tells you how likely your trading results are to have happened by chance alone. You do not want random results due to chance. The t-test is less than 1.6 favors chance (not so good). The t-test above 1.6 and you are more likely to have found a strategy that is useful and reliable as a trading tool. You need to calculate using roughly 30+ samples and see if you have a tradeable approach. Jeff Swanson reports the formula as T-stat = Sharpe x (square root of trades). Notice how: (t-test) 1.60 to 2.0 = average system; 2.01 to 3.0 = good system; 3.01 to 5.0 = excellent system. The T-test has two problems: 1) missed discoveries are high and 2) fall when the market conditions change (so a periodic check seems necessary). However its use in trading system validation seems to grow.

Re: New statistics: P-value

Thanks for the simplified explanation...

So -- I have an idea.  Suppose I have a strategy and 30 arbitrary data sets.  I then test my strategy against the 30 data sets.  And let's say my overall Sharpe value is an average (or median) from the 30 data sets.  Then,

T-stat = Sharpe x (square root of trades),  where Sharpe is the average from the 30 data sets and 'square root of trades' is the sqrt of all the trades from the 30 data sets.

Would this allow me to calculate a T-stat (or something similar) for this particular strategy?

16 (edited by D5QM54S7 2017-10-11 18:09:12)

Re: New statistics: P-value

"Trading systems and methods" (Perry J. Kaufman)
When fewer prices or trades are used in a distribution, we can expect the shape of the curve to be more variable. For example, it may be spread out so that the peak of the distribution will be lower and the tails will be higher.
How do you know if your trading system has an edge or whether it's just random luck?
A way of measuring how close the sample distribution of a smaller set is to the normal distribution is to use the t-statistic.
When testing a trading system, degrees of freedom can be the number of trades produced by the strategy. When you have few trades, the results may not represent what you will get using a longer trading history. When testing a strategy, you will find a similar relationship between the number of trades and the number of parameters, or variables, used in the strategy: more variables used, the more trades are needed to create expectations with an acceptable confi dence level.
When using the t-test to find the consistency of profi ts and losses generated by a trading system, replace the data items by the net returns of each trade, the number of data items by the number of trades.
"The Art and Science of Technical Analysis" (Adam Grimes) modified by Grove Under
The second step is to get the p-value, why? Once the p-value is obtained, it is simply a matter of deciding which threshold qualifies for statistical significance. Scientists usually determine the statistical significance threshold at 0.05 (ie, the null hypothesis would be rejected for any p-value less than or equal to 0.05). How do you calculate?
For example, if your p value is 0.01, that means based on the data set analyzed, there's a 1% chance of seeing the analyzed results due to random chance or luck.  If your p value is 0.50, then there's a 50% chance your results are based on luck.
These purely statistical tools do not guarantee profits, but can be used to exclude overfitting strategies or provide greater selection.
I've seen that I'm already available online tools to quickly calculate p-value by having t and n available.
I've already tried to test some strategy but I'm just beginning.

Post's attachments

Area1.png 23.35 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

Re: New statistics: P-value

comment.
I really appreciate this discussion.

thank you

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: New statistics: P-value

First step is done - t-ratio.

File goes into Code folder, custom code reload has to be enabled in control panel.

Post's attachments

AccountStatistics.cs 23.18 kb, 5 downloads since 2017-10-12 

You don't have the permssions to download the attachments of this post.

Re: New statistics: P-value

I based the calculation on info found here - http://jukebox.esc13.net/untdeveloper/R … le_49.html

My theoretical thinking leads me to an opinion that the mean value should be compared to a benchmark, I'm talking about this formula:
http://jukebox.esc13.net/untdeveloper/RM/Stats_Module_4/lessonimages/equation_image41.gif

If now it is "mean - 0", I think it should be "mean - benchmark". Would like to hear what other statisticians are thinking smile

Re: New statistics: P-value

Hi, Footon, you've been too fast! :-) I read your formula but the benchmark does not get me back. I found a document that could answer our questions: we find the T test, the P value, and the adjustment for the portfolio. I've already tested the values that go back to chapter "AN EXAMPLE WITH STANDARD
AND I LOVE CAPITAL IQ "and I've solved it without problems. I'm waiting for your reading to discuss. I appreciate this document as the authors refer to some quant, I think it's a good starting point.

Post's attachments

CAMPBELL R. HARVEY AND YAN LIU.compressed.pdf 523.92 kb, 3 downloads since 2017-10-12 

You don't have the permssions to download the attachments of this post.

Re: New statistics: P-value

I have the p-value done.

I read through quite a number of statistical articles and to some questions I found answers, but only to realize that those answers raise even more questions. Anyhow, after number of tests I settled with the current p-value calculation. I didn't use t-ratio for the calc as it uses a table of corresponding p-values. Instead it resamples the returns using bootstrapping method. Some papers, which actually lack scientific validity, use returns as they are, but for the sake of more reliable results I zero-centered the returns.

Post's attachments

AccountStatistics.cs 24.63 kb, 7 downloads since 2017-11-04 

You don't have the permssions to download the attachments of this post.

Re: New statistics: P-value

Thanks, footon.

I've updated my AccountStatistics.cs and started poking around.  And I have a several questions:

1. The T-Ratio that I'm getting is *exactly* the same as the SQN.  Do you get that also?

2. My P-values are always 0.00.

3. I have a pretty good feeling for how to use and compare SQN and sharpe values -- though I couldn't come up with the formulas on the top of my head.  From previous posts, it sounds like the "simple-man's" description for a p-value is the probability a result is strictly due to chance -- i.e. they should range between [0,100].  Is that correct?

4. And if it is a probability, then which metric or metrics does it refer to?  Does it refer to the overall result?

5. In your hands, what kind of P-values are you seeing.  I have no reference.  I have no idea what it means to have a good or bad P-value or how to use the value in making a decision about a strategy.

6. Suppose you had a winning strategy -- i.e. it made money.  But the P-value was high, indicating it was likely due to luck.  Would you stop trading that strategy?  And, if not, then I'm curious how one would use this metric to make decisions. 

7. The P-value seems related to SQN -- how would you compare them?

Sorry for all the questions -- feel free to tell me to do my own Googling and I'd understand.  I guess the most important questions for me at the moment are (a) why all the P-values are 0.00, and (b) what kind of range of P-values are you seeing?

Thanks

Re: New statistics: P-value

1. Good catch, sleytus! They really are the same if there's a profitable strat and there are more than 30 trades. Calculation on "the outside" look different but same variables are used. I can't now really give a mathematical reason for this... But it looks to me that it's better to drop the T-ratio as it has no value.

2. You mean like for strategies? Good strategies from an equity curve point of view?

3. Almost. P-value shows statistical significance with a degree of probability. The threshold is actually more or less arbitrary (sorry statisticians for loose understanding and use of words). It depends on the field of study and the test procedure and so on, but usually the statistical significance starts from 0.05 or 5%, the lower it is, the better. In essence, yes, the idea is to reject strats, which luck into winning and/or are heavily biased.

4. There's no metric in this sense, it is strat's mean in relation to bootstrapped distribution. If the mean location is on the edge of the distribution curve and the area, which follows it, is less than 5% of the remaining area, one can conclude the results are statistically significant, in other words it cannot be due to luck or chance to have such outstanding result. That's the idea.

5. It is fresh out of the oven, I haven't had time to put it through its paces. I know it works, it produces numbers I deem ok. But I haven't made proper evaluation on selected collections and further analysis. Firstly, 0.05 mark is the starting point, everything above it should be rubbish, so that will be an interesting test to see and find out.

6. Last answer applies to this as well.

7. Currently, only relation is the underlying data that is equity curve. Certainly, a statist. significant start will produce a high sqn also, so there is this correlation. Let me put it this way - it remains to be seen if the concept of statistical significance is of value to algorithmic strategy mining smile

I have some coding tasks to finish first and then I'm delving deeper in testing and then we can compare notes, hopefully.