Re: machine learning and curvefitting
Footon, this is very interesting.
If you have success, we can make some or all of the operations automatic in FSB.
Create and Test Forex Strategies
You are not logged in. Please login or register.
Forex Software → Forex Strategy Builder Professional → machine learning and curvefitting
Footon, this is very interesting.
If you have success, we can make some or all of the operations automatic in FSB.
Can I ask which software you are working with for this project?
thanks
FSB and openoffice!
Well.. you are doing an amazing job..
Looks like Popov will want you to document your research a bit and he will try to incorporate into FSB.
Footon, this is very interesting.
If you have success, we can make some or all of the operations automatic in FSB.
I'll do my best! It's actually quite simple and not much resourceful. But the main point for now is to have results which have real value. Aronson in his book tested like 7000 rules (strategies) and he didn't find none, which were statistically significant. I'm not getting my hopes up, that's for sure. But I'll keep reporting of my findings.
Well.. you are doing an amazing job..
Looks like Popov will want you to document your research a bit and he will try to incorporate into FSB.
Thanks, but at points I feel as dumb like a bag of nails when theory overwhelms me, but I'll soldier on.
Keep it up Footon!!!
So proud of you and it's such an encouraging news...seeing some bit of the fruit of your labour. Wishing the very best that you will have a clarity of mind to sieved through all the information to find the essence of what you need.
Hang on there
Next small step done. A (profitable) strat with only 159 trades assessed, p-value 0.09 is too much and the strat should be rejected as unfit. A small illustration of balance line and distribution below.
I'm pretty convinced my framework holds, I'll now automate the process a bit more so that I get a p-value right away. The next step is to generate sufficient number of profitable strats, which bear significance in this sense that they produce sufficient number of trades on larger dataset. Wish me luck!
Keep it up Footon!
Thanks for everything
I found good data on my MT, 65k bars of M5, so I adjusted the generation period to 45k bars, the last 20k's will be used for confirmation that would prove a statistically significant strat is really a tradeable strat, not the result of a backtest gone lucky or a curve-fitted misfit.
I got my first strat spat out by FSB, and this is a little teaser - what do you think, will it make the test? Balance curve is OKish, but look at those parameters, that can't work, can it?
I've got a few technical problems to solve before the numbers pop out, but the truth will come out sooner than later.
P.S. Thomas, if you mind my experimentation being in your thread, let me know and I'll move myself politely out to a new thread.
Bloody hell, some technicalities turned out to be rather fundamental, but the numbers arrived finally. Unfortunately no one offered opinion whether the strat presented above should be regarded as potentially good or a total curvefit.
First discovery: bootstrapping results are not zero centered, I wonder why.
P-value is over 3, so this suggests the strat is a waste-bin material. But we have the opportunity to test whether this inference holds true or not, equity curve for the remaining 20k bars below.
The statistical significance test on this strat told us that it is not a good strategy, and it really isn't!
Footon,
I sense your frustration. I think nobody can comment whether the strategy is workable because nobody knows.
I tried to put the strategy in an EA n run it, nothing happens. I looked at the parameters, doesn't looked good to me, but dare not make presumptuous comments till I run the EA.
If you need a profitable EA to test out your statistics, you may consider using EA 33703 free download section. This EA, though doesn't trade frequently but does have rather profitable results.
Rest if you must but don't quit. That's what I tell myself whenever I bang my head against a wall in my EA development
Keep up the good work.
Frustration? Why? I'm getting the results that this statistical significance test works! It has merit. Certainly there is still a lot of work, but this approach seems to offer remarkable ability in assessing a trading strategy. And it proves how big pitfalls there are in datamining.
I see now that there might be a slight error in p-value, I'll check the numbers later.
In addition, I would love to test out your EAs but my M1 is too sketchy and lacks in numbers.
Corrected p-value is 0.54, pretty awful, just like the out of sample proves.
footon
I think you are experiencing the reason that I gave up on statistical methods.
I studied many things, worked with MatLab.. and my brain was fried worse than before...
I even enlisted my son's help.........I just could not get results, I guess I should have done my math homework in grade 8 instead of chasing the school cheerleaders...
Dave, I'm experiencing a great statistical method to assess trading strats, I'm not banging my head against the wall or something. What does Thomas ask in his first post? He asks for (automated) methods to aid the process of picking strats, which have actual ability to operate profitably. We can call it anti curve-fit test or robustness test or whatever, it doesn't change the objective.
How this might come into play? Think of a man, a trader, who wants to trade but lacks resources. For instance, he lives up North, every spare moment he spends to make firewood to be able to live through the cold winter (he uses a handsaw and an axe). Therefore, he just doesn't have that capacity to develop strats manually. What can he do? He can use machines to do the work for him.
Like Thomas, he uses FSB to generate strats. At one point he has 15000 strats. Now he wants to build a portfolio of 10 strats. He knows that data-mining has its pitfalls, you can have great returns on a backtest, but these won't translate into profits on a live account with a flick of a switch. What next? Demo trade them all? Take a guess? No, he determines the statistical significance of the backtested strats and from that point he'll build his portfolio.
I hope you understand the project now.
-----------------------------------------
I increased the bootstrap cycle from 5000 to 15000. The p-value for the last strat is the same, 0.54! Or like Ahmed says - 0.54!!!
Now the most difficult part - I need to have a strat with a sufficiently low p-value, which I then put on unseen data. Given the theory of the concept, a strat with a sufficiently low p-value has to maintain its profitability. If I succeed in this, I have proof this works and implementation to FSB can begin!
Thomas is using some software that generates a ton of strategies and has several filters..... I think it will be great if you can refine a couple approaches for FSBPro.
We can certainly use this type of tool.
In my own stuff I am seeing that perhaps one strategy in 2000 makes the grade..... and that is after a lot of playing around. What I get though is something that can trade more than one time frame and a few currency pairs.. so the time investment is worth it.
As your procedure gets incorporated into FSBPro, will be interesting to see what metrics in Control Panel will give a satisfactory result in your project.
Thanks huge for your efforts!
daveM
I tried H's 03 strat. I got only 100k data, which is only a third of what I deem minimum. As it really trades seldom I think maybe 600k might show something meaningful, we can't forget the law of large numbers after all. I searched for export function in pro for quite some time, after I found the button it turned out csv is not supported, so I turned to old guns and my stress level lowered significantly. Another issue was that backtest result was similar but not exactly the same, fewer trades and some discrepancies in stats. My conclusion is that due to these factors my results are inconclusive.
Returning to the theoretical side of things, again the bootstrapped distribution is not zero-centered, which means that something is not right. I'm thinking it is an indication of too few original samples (trades). This makes me think how many samples is enough? Definitely more than 1000... Another thing I noted - higher TF results look more "normal", maybe I'll ditch M5 and work with higher TFs after all.
i do not know much about ea development but perhaps seeing the performance of the strategy in oos would make it clear if any curve fitting took place.
if the performance in oos is not good the strategy is not tradeable:this has been asserted by Popov.
i discard any strategy which does not do well in oos.....but i have an experience of two days in ea development!!!!!!!!!!!!!!!!!!!!!!!!!!
Trying to solve the problems, while researching I found other people who are doing similar stuff, an example how good FSB is: one guy is running the bootstrap test for 5x less loops than I am and he says it takes 3 minutes. In FSB I make 5x more calculations in less than 3 seconds!
Forex Software → Forex Strategy Builder Professional → machine learning and curvefitting
Powered by PunBB, supported by Informer Technologies, Inc.