Re: machine learning and curvefitting

I think you just have to learn to modify the indicators so that you have a group of indicators that are custom designed for your purpose.

The indicators are easy to modify......... but there is some sort of naming convention that you have to know and understand and then there is some way to prepare the new indie for mt4 and mt5, I  am unfamiliar with those. however, there is a thread for that purpose.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

27 (edited by hannahis 2016-01-26 05:54:17)

Re: machine learning and curvefitting

But that's what FSB is suppose to do...for traders who are clueless about programming...to "tell" FSB what we want, and FSB will do the "programming part" and code the instruction for us.  Furthermore, the combination are not hard and fast rules to apply per se.  When one input interacts with another input, the combination changes all the time (eg, you don't use the same combination of smoothing and base price for "rising" as compare with "cross upward", or "higher than", especially when you use different LTF too.  Each time you add in additional conditions, the interaction change.

Re: machine learning and curvefitting

I think a trader has to also do some coding.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: machine learning and curvefitting

Many of the indicators that we use have been designed by  Footon and Ahmed

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

30 (edited by hannahis 2016-01-26 06:02:44)

Re: machine learning and curvefitting

Blaiserboy wrote:

I think a trader has to also do some coding.

Sorry Dave, it's beyond me...I'm not so computer savvy nor very mathematical skillful, though I wished.  Trading is not my training, it's just my hobby...I'm a psychologist in training and I happened to be curious to apply my analytical skill in trading.  Maybe it's a blessing in disguise...I see through the "smoke" and see better without too many trading jargon smile

Re: machine learning and curvefitting

Footon will see this in a couple hours and comment, I am sure.

I am not so sure that the program can be easily altered to fit your needs...... but indicators can be changed  if you just want to delete parts of them....

You can name them as u wish........ and have a selection of them custom designed for your purpose.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

32 (edited by hannahis 2016-01-26 07:02:03)

Re: machine learning and curvefitting

Thanks, that's very kind of you. 

However, I'm hoping FSB can incorporate such mini optimizer for the benefits of all user.  If this is incorporated, many of us will be surprised that a number of our input is actually rather profitable to begin with...and after some adjustment, refine it better.

Ps: I forgot to mention signal line too in addition to smoothing and base price for optimization.

Re: machine learning and curvefitting

what do you mean by FSB "non-fit strategies", do you mean not genuine predictive ability?

Hannah,

Yep I meant those strats without genuine profitability. You know, these kinds of strats which like fall from the cliff if put on OOS data or live trading. Basically the gems are rare and hidden among strats which have lucked into great equity curve.

About the list parameter optimizer - cannot be done by indi modification.

I agree that optimizing is more a curve-fitting activity than generating, hence isn't the list parameter optimizer just another brick in the curve-fitting wall?

I'm so anxious to test out these statistical methods I've been researching, but the little math things haven't clicked in yet, I hope I can make some progress tonight.

34 (edited by hannahis 2016-01-27 05:49:00)

Re: machine learning and curvefitting

Footon,

I think there is a grey line between fine tuning an EA and curve fitting.  Both in action seem very much the same.  But the intention is different although both using an "instrument" to derive an outcome.  Fine tuning start with a fix concept and but needed some "instrument" to align it with the price movement.  Whereas, curve fitting has no concept to begin but will randomly change whatever it is to fit into the price movement and from there derive it's concept or formula...deduction method.

It's like a principled man who cope in this world and circumstances with discernment in order to achieve "happiness" without compromising his principles.  Whereas, an unprincipled man will do anything and do all it takes to achieve his "happiness' in whatever manner or "short cuts" to attain it.  Hence, you will find a principled man's behaviours consistent, reliable and predictable, whereas an unprincipled man's behaviours the opposite.  Similarly, curve fitting EA has the characteristics of an unprincipled man because he lacks the "trading concepts or rules" to guide his behaviour.  His aim is money making without principle, thus unreliable.

I do hope you somehow with certain enlightenment found the right "click" in your mathematical/research and finally would get to reap the fruit of your labour.

I personally don't think there is A formula to be sieved out of the data mining.  Pricing is liken to human behaviour, there is a pattern but not a formula per se.  Hence, if we think there is a Fix formula in the pricing, we might be aiming for a futile search.

What we need to look for is not A formula but the Rules of Interaction, this is the so call formula we are searching.  Pricing is like human being, not determined by a set of formulated rules...for example for every ten steps you take, you will to turn left and then next 20 steps you going to turn right, etc we are not subjected to fix rules.  However, as much as human behaviour seems to be random, we don't defer too much from our Rules of Interaction or Engagement. Hence, in counselling, I listened to people's speech and look out for their Rules of Interaction and I would know the person better and hence, "predict" what are the likely problems/issues this person would encounter in his life or interaction with people.


That's the same with my Trading Theoretical Framework, I discovered the Rules of Interaction.  Each individual curve in the MT4 chart will following it's own characteristic of behaviour and dictated by the price movement, hence very fluid.  However, whenever two or more curves come together, they seldom or never failed to "produced" a kind of interaction...in my case a Breakout, my challenge is now to look for their rules of interaction when they (the curves" signal that they have ended their interaction and parted ways and till they meet again, another interaction or breakout resulted.

Thus each of my opening conditions is to "choose" who are the main "characters/actors" that caused such interaction that will result to breakout and who are the supporting "characters" that would help increase the likelihood of such a situation.  Hence, layer by layer, precepts by precepts (with each added opening conditions), I placed my "characters" into the picture and when all these characters come together, there is almost always a "fight" or breakout, smile

The challenge is to find the main characters and don't forget the supporting ones because they enhance entry/exit accuracy.  That's why LTF is very important, many just look for the main character in higher time frame but neglect the supporting characters in lower time frame.  When I say I'm perfecting my EA, some may think I'm optimizing my EA, but actually, I'm just adding more and more characters into my opening conditions to fine tune the interaction so that I can enter as close as possible during the "fight", you don't want to go to a venue where the "show" already long started (thus 1 min time chart is extremely important if you want to arrived timely), you try to go slightly before the "show" or slightly later or be on time, that's my fine tuning process.  If your mode of transportation is so infrequent, eg H4, H1 etc, you can never arrived at your destination on time, you are restricted by the hourly train schedules.  But if you mode of transportation allows you to alight anywhere, anytime, isn't it better?  That's why we favour Taxis more than train or buses, we can get to our show on time.

Sorry, I know my illustration is rather unorthodox in terms of speaking of trading jargon.  I see trading from my psychologist training perspective, not a trader's perspective.

Re: machine learning and curvefitting

I will add this to the discussion.

Data mining has been able to find patterns or tendencies in price bar data, and been able to make relatively accurate predictions.

When I worked with a couple of the programs, I was able to make predictions from the data of the S&P 500 and those had quite a bit of accuracy.

The problem I experienced was that I did not have the computer for the job and I was too cheap to spend the money for the software.

Patterns can be located with the right software, that is what the high frequency traders are using every day.

FSBPro offers us the same opportunity through the use of the generator and rigorous testing of results.

Many patterns have predictable results....., our task is to discover those... whether we set up potentials in advance and then test them or whether we run the generator until we locate what works.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

36 (edited by hannahis 2016-01-27 09:45:53)

Re: machine learning and curvefitting

Thanks Dave for enlightening my mind...may I know which software is it so that I can test my EA on those software to fine tune it?

i know you aren't here to promote other software, but we do share resources and knowledge to enhance our trading even if it is not related to FSB, just like I discuss FSB in other forum too.  Do let me know which you would recommend and it's strength etc, if you are not comfortable to share here openly, can PM me.  Thanks

Re: machine learning and curvefitting

You can have a look at Matlab or Rapid Miner, there are a few of them, Wolfram is another.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: machine learning and curvefitting

Thanks

Re: machine learning and curvefitting

Curve Fitting and optimization are great subjects for investing time and study.

FSBPro has the capacity to research and develop price action patterns that will work repeatedly.

Price action patterns can work on all time frames and in conjunction with each other.

The problem with price action patterns is that they have to be an exact fit....... and the tendency to use oscillators appears. and then comes trouble.

Omit the oscillators and curve fitting is dramatically reduced.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: machine learning and curvefitting

Years ago I bought a software called FCPro..... a real clunker. written in VB6. The thing came equipped with a few indicators and a facility for the user to make his own indicators.

I think I can make 100 indicators and use them in formulas.

These indicators can be used to structure bars and bar patterns.

I maintain daily the stocks information for NYSE Nasdaq ETF's Indexes and about 35 Forex pairs.

And what I use this thing for is to test my creations against these databases...

I dream up whatever and run the thing against say 100 stocks or 1000 to see what happens.....

And that is why I have deducted that price bar patterns are far more effective than oscillators.. and need far less maintenance.

Today a person can use Worden except the cost is close to $100 per month and Forex feed is extra.

In any event, curve fitting can be complex when using any oscillators and I recommend that people focus on price bar patterns.

FSBPro allows us to create complimentary patterns on multiple time frames for both entries and exits.

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

41 (edited by hannahis 2016-01-28 05:46:03)

Re: machine learning and curvefitting

When I looked at Matlab and RapidMiner, I'm overwhelmed, my brain is not wired for this direction, I'm a social art grad, or maybe I'm too tired now to learn new things, all the manual "optimizing" thousands of EA with the smoothing, base and signal line has taken a toil on me mentally.

I appreciate FSB Pro simplicity and user friendly platform...idiot proof smile

Now I wish one of my children will be interested in this direction as their career path to learn data mining and analysis so that he/she can use my trading framework and work on it to improvised further...that's a number of years of waiting for them to grow up, though my girl is already interested to develop her own EA, but she is still very young.

Thanks for your tips Dave.

42 (edited by hannahis 2016-01-29 10:02:17)

Re: machine learning and curvefitting

Blaiserboy wrote:

And that is why I have deducted that price bar patterns are far more effective than oscillators..

In any event, curve fitting can be complex when using any oscillators and I recommend that people focus on price bar patterns..


Hi Dave,

When it comes to using oscillators in FSB, there is some confusion...

Let use MA 14 and MA 21 as an eg.

In opening condition, if MA 14 is above MA 21, oscillator rising is when the two curve spread out further from each other, right?

And what if in an opening condition,  if MA 14 is below MA 21, and the two curve spread out further from each other, too, do we call that rising or falling?  Can't be failing because the oscillator between these 2 curves are going further apart, but if I put it as rising, then how do FSB differential between MA 14 above MA 21 rising from MA 14 below MA 21 rising? Likewise, I wonder how it works for closing condition.

Because of this possible confusion, whenever I use oscillator, I input the direction of the crossing first before input the rising condition.  In other words, I use 2 opening conditions.  1st opening condition is to determine MA 14 is above MA 21 and then 2nd opening condition is to use rising.  Please enlighten me if I got this whole concept wrong.

If there is no such differentiation, then when we use generator to search, whether FSB can differentiate which is a "better rising to use for opening and closing conditions?

Another possible reason why Oscillator may not be suitable to use (by itself) in searching for predictive EA is that prices do continue to increase despite even in conditions when oscillation falls or remain flat, hence, oscillator rising by itself isn't very predictable.  Popov, came up with a better alternative, that's the use of convergence and divergence, it's very powerful and predictive tools that is part and parcel of my EA success.  In the past, I relied much on Oscillator but now Convergence and Divergence is by far my most important tool and oscillator is supporting tools.  Convergence ensure both the curves are in the same direction despite the oscillation not rising.  Hence, when used together with oscillator, it would enhance the results and predictability of the EA.  That's my observation and personal experience.

As for Price Bar action, I personally find it too "complicated" to use and in my own Limited opinion, lacks predictability because market condition is so volatile that such patterns always repeat by itself in both direction and hence, I find it uncertain to predict the trend and when market moves so fast, I don't have time to examine each bar and then make my decision, too long and too late (that's why I didn't venture into this direction and hence, didn't develop your level of expertise to use it properly).  I use indicators on my MT4 chart in such a way that at a glance without too much effort or calculation, my chart is able to clearly show me direction of the trade, no need any drawing of lines, finding the resistance, support or look for higher close, lower highs etc.  At a glance, the trend is clear but now my EA is far better and faster than me (of cos) in predicting the direction because it can scan all the time frames at split of seconds.

As for Worden, yeah my dream is one day convert my theoretical framework into (formula) signals/software so that it can scan all stocks, indices etc and select those that are about to enter into breakout.  With my chart, I can see it coming (or ending)...in whatever indices, forex, stock.  Just that I don't have the mathematical expertise of Popov to put these into formula and calculation.  Maybe, I'll have to leave it to my children to bring this version further.  Family secret, hence can't trust it into others who will take off with the ideas and leave me dry, sad

Re: machine learning and curvefitting

Maybe it is better to get to the bottom of what oscillator means, guesswork on this part seems such a waste of everything, really.

Example of 2 MAs can be best shown with Awesome Oscillator and the calculation is a s follows:

adAO[iBar] = adMAFast[iBar] - adMASlow[iBar];

That is how oscillator value is derived and what is used in signal calculation.

44 (edited by hannahis 2016-01-29 15:56:39)

Re: machine learning and curvefitting

Its one thing to know the formula, but we also need to know in FSB contex, how it classified or label the, opening conditions for oscillations.  if MA 14 is lower than MA 21 and the curves are growing further apart, what is that classified as? Rising or falling? So that I can choose the correct option for my opening conditions.

This clarification is useful when we want to set up buying conditions in the situation of a divergence  For eg when MA 14 is below MA 50 and the gap between them are closing (divergence), its signaling a possible buying condition. Because by the time there is a cross over, it's usually too late, thus I want to set up divergence condition using oscillator, so when I input the fast period as 14 and slow period as 50, what do I choose? Rising or falling  as my option?  And if I choose falling, how does FSB differentiate in terms of classifying them between MA 14 higher than MA 50 falling from MA 14 below MA 50 falling? Both has opposite  implications when it comes to buying and selling.  How to set up this kind of opening divergence condition using oscillaton in FSB?

Or should I just reverse the input eg fast input 50 and slow as 14 and use falling as the option?

Please bear with me,  I'm a language student, not a math student  I understand things by concepts rather than numbers  so even if you show me the formula, I still need someone to explain the formula in words to understand its function.  My brain not wire to see patterns in numbers but pattern in human behaviours and visual presentation.  So in this forum that is filled with so many mathematical inclined people, I know my questions may sound so stupid to the obvious.  But ask me in my field of expertise,  I'm very sure I can pick up things you can't see or perceived.  That's why there's a saying  "we connect through similarities and grow through differences"

That's why when I was searching for my theoretical trading framework,  I skip all those complicated calculation and zoom in for visual aids to present my trading plan at a glance, so that any person, with or without trading background can see and understand my theory clearly without too much explanation, a picture is worth a thousand words.  That's where my visual perception skills come in handy.

Re: machine learning and curvefitting

Do I get this right.....

You want to measure the momentum of the MA's, ie becoming closer or further..

Did u have a look at the gator oscillator, would that serve your purpose...?

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

46 (edited by hannahis 2016-02-02 08:17:15)

Re: machine learning and curvefitting

Dave,

I want to highlight whether FSB distinguish the difference between 1) MA14 above MA21 rising from 2) MA14 Below MA21 rising in the opening conditions and vise versa, 1) MA14 above MA21 falling vs 2) MA14 below MA21 falling.

And if FSB does make the difference, how to input it as, eg Fast MA 14 and Slow MA 21 and choose rising for MA14 above MA21 and then how about MA14 below MA21 rising, do I put Fast MA21 and Slow MA 14 and choose rising?

And If FSB doesn't distinguish the difference, would it affect the statistic calculation of the results? (sorry, I'm not familiar with formula calculation, just want to highlight whether this is an area of blind spot).  If this issue is taken care of, I'm fine with it.  Then I just want to know what's the correct input to differentiate these two types opening conditions.

Likewise for the issue of convergence, it means two curves are going at the same direction, but does FSB differentiate between of eg RSI MA convergence for 14 above 21 from 14 below 21 convergence in the opening conditon?  So for such cases, do we reverse out input in the opening conditions eg Fast 14 and Slow 21 for 14 above 21 convergence, but Fast 21 and Slow 14 for 14 below 21 convergence?

Re: machine learning and curvefitting

There is no difference between Fast MA and Slow MA. The names are nothing more than words shown for the user. The FSB bakctester calculates the MAs according to the provided parameters. If you want, think for the MAs as First MA and Second MA. The program allows you to set any parameters in the provided limits. It is perfectly possible to set the period of the Fast MA = 14 and the period of the Slow MA = 9.

48 (edited by GD 2016-02-03 03:02:10)

Re: machine learning and curvefitting

We have to see and the subjects

1. digital filters and
2. no lag indicators

for the lower TFs

Re: machine learning and curvefitting

I think I've ironed out most problems and hiccups concerning the determination of statistical significance of a trading strategy (pick a strat with genuine predictive ability instead of a strat which has lucked into great back-test, but in OOS or live testing fails miserably). Basically I've been trying to emulate ideas presented in Aronson's Evidence-based TA, I've done my best but some mistakes might be present as I'm a novice in statistics, I hope it's not the case and they're not fundamental.

What I did (this is the first try, like a proof of concept):
1. I developed a "perfect" strat, its balance curve is the lower one on the pic, it's diagonal meaning you can't do much better than that. My idea is that this strat should prove to be statistically significant no matter what.

2. Then I adjusted the results by detrending the data, this is removing trend from data and therefore position bias if it's present. The M15's 20k bars were good enough, so no significant impact on trading results.

3. After that a zero-centering procedure in order to proceed with bootstrap test.

4. Then I conducted the bootstrap test, I kept the sequence at 5000, meaning I have 5000 bootstrapped samples, each sample has the same size as the original sample calculated on detrended data and is deduced to one measure - its mean. So I have 5000 mean returns from 5000 bootstrap samples.

5. Then I made a simple frequency distribution or more of a probability density function (the first pic) of the 5000 bootstrapped samples, from that it is possible to calculate and visualize the p-value.

Results: The p-value is zero. A low p-value is desirable, but having it a zero can cast a shadow on the test validity. On the other hand I was waiting for this kind of result for this type of "perfect" strat.

My next steps: I will do the same with a real strat and see the results. Then I have to see if increasing the bootstrapping cycle from 5000 to say 10000 makes any kind of difference. If the results are encouraging, I'll try to establish whether a statistically significant strategy is for real! The plan is to take at least 50k bars, 30k bars will be used for strat development, then will come the significance tests, and if I'm able to have at least a few significant strats I can test out their performance on the remaining 20k bars.

Post's attachments

bal.JPG 77.15 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

Re: machine learning and curvefitting

Can I ask which software you are working with for this project?

thanks

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....