Forex Software — machine learning and curvefitting

Re: machine learning and curvefitting

2016-02-28T15:40:17Z

Trying to solve the problems, while researching I found other people who are doing similar stuff, an example how good FSB is: one guy is running the bootstrap test for 5x less loops than I am and he says it takes 3 minutes. In FSB I make 5x more calculations in less than 3 seconds!

Re: machine learning and curvefitting

2016-02-25T21:38:43Z

Look how the equity line hugs the underlying price curve, still position bias? In this case if we would trade on beyond MC, it should be profitable again, lets use some of that FSB magic...

No, it's just ****. Hmm, is Aronson's theory also ****?

Re: machine learning and curvefitting

2016-02-25T21:31:49Z

Now I have some meaningful results. It's a generated strat on m5 EU, its p-value is 0.0014! Highly significant in other words, it should retain its profitability in the following OOS test, but the cold hard truth is - total failure! Equity charts below...

Re: machine learning and curvefitting

2016-02-18T17:18:37Z

i do not know much about ea development but perhaps seeing the performance of the strategy in oos would make it clear if any curve fitting took place.

if the performance in oos is not good the strategy is not tradeable:this has been asserted by Popov.

i discard any strategy which does not do well in oos.....but i have an experience of two days in ea development!!!!!!!!!!!!!!!!!!!!!!!!!!

Re: machine learning and curvefitting

2016-02-18T14:56:40Z

I tried H's 03 strat. I got only 100k data, which is only a third of what I deem minimum. As it really trades seldom I think maybe 600k might show something meaningful, we can't forget the law of large numbers after all. I searched for export function in pro for quite some time, after I found the button it turned out csv is not supported, so I turned to old guns and my stress level lowered significantly. Another issue was that backtest result was similar but not exactly the same, fewer trades and some discrepancies in stats. My conclusion is that due to these factors my results are inconclusive.

Returning to the theoretical side of things, again the bootstrapped distribution is not zero-centered, which means that something is not right. I'm thinking it is an indication of too few original samples (trades). This makes me think how many samples is enough? Definitely more than 1000... Another thing I noted - higher TF results look more "normal", maybe I'll ditch M5 and work with higher TFs after all.

Re: machine learning and curvefitting

2016-02-17T16:47:18Z

Thomas is using some software that generates a ton of strategies and has several filters..... I think it will be great if you can refine a couple approaches for FSBPro.

We can certainly use this type of tool.

In my own stuff I am seeing that perhaps one strategy in 2000 makes the grade..... and that is after a lot of playing around. What I get though is something that can trade more than one time frame and a few currency pairs.. so the time investment is worth it.

As your procedure gets incorporated into FSBPro, will be interesting to see what metrics in Control Panel will give a satisfactory result in your project.

Thanks huge for your efforts!

daveM

Re: machine learning and curvefitting

2016-02-17T16:26:02Z

Dave, I'm experiencing a great statistical method to assess trading strats, I'm not banging my head against the wall or something. What does Thomas ask in his first post? He asks for (automated) methods to aid the process of picking strats, which have actual ability to operate profitably. We can call it anti curve-fit test or robustness test or whatever, it doesn't change the objective.

How this might come into play? Think of a man, a trader, who wants to trade but lacks resources. For instance, he lives up North, every spare moment he spends to make firewood to be able to live through the cold winter (he uses a handsaw and an axe). Therefore, he just doesn't have that capacity to develop strats manually. What can he do? He can use machines to do the work for him.

Like Thomas, he uses FSB to generate strats. At one point he has 15000 strats. Now he wants to build a portfolio of 10 strats. He knows that data-mining has its pitfalls, you can have great returns on a backtest, but these won't translate into profits on a live account with a flick of a switch. What next? Demo trade them all? Take a guess? No, he determines the statistical significance of the backtested strats and from that point he'll build his portfolio.

I hope you understand the project now.
-----------------------------------------

I increased the bootstrap cycle from 5000 to 15000. The p-value for the last strat is the same, 0.54! Or like Ahmed says - 0.54!!!

Now the most difficult part - I need to have a strat with a sufficiently low p-value, which I then put on unseen data. Given the theory of the concept, a strat with a sufficiently low p-value has to maintain its profitability. If I succeed in this, I have proof this works and implementation to FSB can begin!

Re: machine learning and curvefitting

2016-02-16T22:18:08Z

footon

I think you are experiencing the reason that I gave up on statistical methods.

I studied many things, worked with MatLab.. and my brain was fried worse than before...

I even enlisted my son's help.........I just could not get results, I guess I should have done my math homework in grade 8 instead of chasing the school cheerleaders...

Re: machine learning and curvefitting

2016-02-16T21:27:26Z

Corrected p-value is 0.54, pretty awful, just like the out of sample proves.

Re: machine learning and curvefitting

2016-02-16T20:53:02Z

In addition, I would love to test out your EAs but my M1 is too sketchy and lacks in numbers.

Re: machine learning and curvefitting

2016-02-16T16:53:46Z

Frustration? Why? I'm getting the results that this statistical significance test works! It has merit. Certainly there is still a lot of work, but this approach seems to offer remarkable ability in assessing a trading strategy. And it proves how big pitfalls there are in datamining.

I see now that there might be a slight error in p-value, I'll check the numbers later.

Re: machine learning and curvefitting

2016-02-16T16:17:24Z

Footon,

I sense your frustration. I think nobody can comment whether the strategy is workable because nobody knows.

I tried to put the strategy in an EA n run it, nothing happens. I looked at the parameters, doesn't looked good to me, but dare not make presumptuous comments till I run the EA.

If you need a profitable EA to test out your statistics, you may consider using EA 33703 free download section. This EA, though doesn't trade frequently but does have rather profitable results.

Rest if you must but don't quit. That's what I tell myself whenever I bang my head against a wall in my EA development

Keep up the good work.

Re: machine learning and curvefitting

2016-02-16T15:07:54Z

Bloody hell, some technicalities turned out to be rather fundamental, but the numbers arrived finally. Unfortunately no one offered opinion whether the strat presented above should be regarded as potentially good or a total curvefit.

First discovery: bootstrapping results are not zero centered, I wonder why.

P-value is over 3, so this suggests the strat is a waste-bin material. But we have the opportunity to test whether this inference holds true or not, equity curve for the remaining 20k bars below.

The statistical significance test on this strat told us that it is not a good strategy, and it really isn't!

Re: machine learning and curvefitting

2016-02-09T22:59:03Z

I found good data on my MT, 65k bars of M5, so I adjusted the generation period to 45k bars, the last 20k's will be used for confirmation that would prove a statistically significant strat is really a tradeable strat, not the result of a backtest gone lucky or a curve-fitted misfit.

I got my first strat spat out by FSB, and this is a little teaser - what do you think, will it make the test? Balance curve is OKish, but look at those parameters, that can't work, can it?

I've got a few technical problems to solve before the numbers pop out, but the truth will come out sooner than later.

P.S. Thomas, if you mind my experimentation being in your thread, let me know and I'll move myself politely out to a new thread.

Re: machine learning and curvefitting

2016-02-09T15:22:49Z

Keep it up Footon!

Thanks for everything