forex software

Create and Test Forex Strategies

forex software

Skip to forum content

Forex Forum

Forex Software, Forex Strategies, Expert Advisors Generator

You are not logged in. Please login or register.


Forex Forum → Expert Advisor Studio → Multi market validation

Pages 1

You must login or register to post a reply

RSS topic feed

Posts: 18

Topic: Multi market validation

I wanted to share other peoples Experian es with using multi market validations.

I currently am generating strategies that have less than 10% drawdown (sometimes as low as 2%) with 20% OOS data over 16 years M30 (190xxx) bars with 7 squared over 80. With no SL or TP

These are straight out of the generator with no optimisation what so ever and pass 4 multi markets from a possible 24.

The issue I’m having is I cannot get any strategies generated that have more than 4 markets with an r squared of over 75. The markets are all forex markets and all built on majors and related over the big 6-7 currencies.

I’m just wondering other peoples experiences?

2 (edited by GD 2019-03-26 02:59:11)

Re: Multi market validation

I do not use multimarkets as I cannot use different brokers with EA studio, as I do with FSB pro. FSB pro does not use R squared. For me Multimarket I cannot use.

3 (edited by ats118765 2019-03-26 13:09:20)

Re: Multi market validation

Michael1 wrote:

I wanted to share other peoples Experian es with using multi market validations.

I currently am generating strategies that have less than 10% drawdown (sometimes as low as 2%) with 20% OOS data over 16 years M30 (190xxx) bars with 7 squared over 80. With no SL or TP

These are straight out of the generator with no optimisation what so ever and pass 4 multi markets from a possible 24.

The issue I’m having is I cannot get any strategies generated that have more than 4 markets with an r squared of over 75. The markets are all forex markets and all built on majors and related over the big 6-7 currencies.

I’m just wondering other peoples experiences?

I hear you Michael. I love EA Studio but this is one of the limitations that really holds me back.

At the moment EA studio only offers fixed size position sizing and stops and profit targets defined in pips in the reactor and generator 'strategy properties setting'. What this means is that if you want to test the strategy across multi-markets, you can therefore only test the efficacy of the strategy across those markets that possess very similar volatility profiles.

For example, it is illogical to test EURUSD against any JPY pair or against any metal as the relative pips in terms of standard volatility are wildly different. You therefore need to select the nearest equivalent market/s in terms of possessing a similar ATR expressed in pips assuming the same fixed position sizing such as GBPUSD for example,

If we had different position size options such as % of trade equity or fixed $ risk allocation using ATR per trade for position sizing calculation,  stop/profit target placement  then we would have a standardised method that would apply across the entire range of multi-markets on offer.

Having this feature then truly would give EA studio a leg up as a multimarket strategy data-mining solution from which you could create very robust portfolios. Until this day comes, we are stuck with the current limitations so you need to do a bit of homework in assessing what instruments have a similar ATR to select for multi-market purposes.

I do have a fairly unique workflow process however that is suited to datamining 'divergent' strategies as opposed to 'convergent' techniques that require a different process with more emphasis based on strategy design than in statistical treatment and optimisation. As a result I place more emphasis on diversification than Popov and the majority of users on this forum and hence are looking for common strategies that work across a very large data set (or multimarkets). I can therefore accept that this 'wish list item' is perhaps not so much of a priority for the masses.

Cheers

Rich B

Diversification and risk-weighted returns is what this game is about

Re: Multi market validation

Rich - wouldn't a strategy be far more robust if it works on all types of volatility markets - rather than cherry picking markets with similar volatility profiles? Thanks

5 (edited by ats118765 2020-06-04 13:16:24)

Re: Multi market validation

zikka1 wrote:

Rich - wouldn't a strategy be far more robust if it works on all types of volatility markets - rather than cherry picking markets with similar volatility profiles? Thanks

Hi Z

I agree with you mate....and that is what I was suggesting :-)

What you could do is use Average True Range as a basis for stops, move to breakeven, trailing stops, take profit targets etc. At the moment EA studio only uses pips.

So say we use an ATR of a 14 bar lookback and set our stops on any liquid market to be 2 x ATR. The ATR represents the 'actual volatility' of the market and is variable. A pip based method is 'static' and independent of volatility.

Lets say we apply a fixed $ position sizing method where we risk $100 for our trade by having a stop placed 2 x ATR away.  The current method uses standard lots and pips (eg.we define a stop as being 100 pips away where we apply a fixed position size of 0.01 lots.)

When markets get highly volatile, an ATR of 2 x (the 14 day lookback) stretches and the distance to stop is greater giving more room to breathe while still risking $100. When markets have low volatility the ATR contracts so the distance to stop of $100 loss is less.

Now the lot size will change accordingly with this different volatility adjusted method and with small account balances you will run into minimum lot thresholds of your Broker.....however....having this option would be great for those that want to use it.

This approach normalises volatility across multi-markets and allows you to have a common strategy applied to any liquid market such as all Forex, Cryptos, CFD's etc. It is pip 'independent' and based on ATR only.

The process is the opposite of cherry picking and allows you to test across a large multi-market universe.

For example the multi-market test that is applied currently may only be appropriate for 4 markets that have similar pip based stops, trails etc. That is why in the original post Michael1 found that only 4 markets would validate even though you may have had multi-market validation on for say 26-30 different markets. With the ATR based method, it would be applicable to all liquid markets and you will find that say 10 out of 28 are now valid for the particular strategy.

At the moment EA Studio focuses on convergent workflow processes...so multi-market is not that useful anyway....but it does become essential for trend following methods that want to achieve diversification benefits from cross asset exposure.

I hope this makes sense mate.

Cheers

Rich

Diversification and risk-weighted returns is what this game is about

Re: Multi market validation

Thanks Rich - very clear.

If I understand what you're saying, the strategy might generally work across 10-20 markets, if only the stop had been "normalized" to reflect the volatility dynamics of that market, but EA advisor discards it without trying the ATR stop adjusted to the particular market.

I've been following your posts with great interest on this forum (and also Copernicus on another forum - I think that's you also?) and really enjoy the discussion.

7 (edited by ats118765 2020-06-13 04:15:48)

Re: Multi market validation

zikka1 wrote:

Thanks Rich - very clear.

If I understand what you're saying, the strategy might generally work across 10-20 markets, if only the stop had been "normalized" to reflect the volatility dynamics of that market, but EA advisor discards it without trying the ATR stop adjusted to the particular market.

I've been following your posts with great interest on this forum (and also Copernicus on another forum - I think that's you also?) and really enjoy the discussion.

That's it Z :-)

I like to use ATR in my workflow method to 'normalise' a parameter set across markets which allows me to achieve very large trade sample sizes to see if an underlying edge (bias) is present in the technique. Each market therefore becomes just another possible 'backtest' data set.

Also my initial tests exclude the impact of trading costs and simply work off the Gross Profit return stream so that I can get 'apples for apples' in my testing. If they demonstrate an edge, then I will back in the trading costs (SWAP, Commissions, Slippage etc.) as a later part of the workflow process to determine how trading costs 'bias' the overall technique.

In doing this I can get between 25 to 27 valid markets in a possible 31 market universe that all display positive expectancy (on a Gross Profit basis). With say 20-35 years of years of data across say 25 markets, this really helps to lift the market data sample for you to get a large trade sample result.

Given the difficulty in seeing an 'edge' with a small sample size, it really is helpful if you can 'boost' the sample size using this technique.

I pinched the following explanatory method from a great series on Darwinex that Blaiser Boy pointed out to me....

Consider the following series of charts which highlight 30 random return streams and a single return stream (in red) that actually has a slight edge. All these return streams also have cost drag built in to them. If you look at the different charts using different sample sizes you can see how difficult it really is to select an 'actual' return stream with an edge...unless you have a seriously large data sample. That is why multi-market is so important and why ATR really helps the cause in the market data normalisation function.

Example 1: Trade Sample Size 500 per Return Stream - 30 Random Return streams (with cost drag) that demonstrate how many equity curves appear to have a 'fictitious edge'.

https://atstradingsolutions.com/wp-content/uploads/2020/06/Random-Sample-500.png

Example 2: Trade Sample Size 500 per Return Stream - The same 30 Random return Streams with an additional system included (in red) that actually has been configured with a modest edge. Note how in that particular array of 31 return streams that you would have no hope of selecting the return stream with the 'actual edge'. A sample size of 500 does not tell the 'actual story'.

https://atstradingsolutions.com/wp-content/uploads/2020/06/Random-Sample-with-Edge-500.png


Example 3: Trade Sample Size 2000 per Return Stream - The same as Example 2 but this time using 2000 trades per return stream. You still would be hard pressed to select the return stream with an edge. You might also notice a slight deterioration in the average performance of the random return streams. This is the cost drag kicking in to bias the overall series.

https://atstradingsolutions.com/wp-content/uploads/2020/06/Random-Sample-with-Edge-2000.png

Example 4: Trade Sample Size 5000 per Return Stream - The same as Example 3 but this time using 5000 trades per return stream. Thanks to compounding it is pretty clear now which return stream has an edge.....but look at the trade sample size you needed to actually distinguish this? Also look at the natural volatility that results from compounding. This is created through the impact of 'noise' on the underlying edge and cannot be eliminated. You won't see this on return streams with a short lifespan...but you do see this on return streams when compounding starts to take a dominant role in wealth building.

Note: The following is really important to understand. The slight edge is prevalent across the entire data series (I know it is because I programmed it that way)....but the drawdowns and volatile curve is simply from the impact of randomness (noise) on that permanent edge.  It is a very normal symptom arising from complex markets. You cannot avoid drawdowns. Techniques that do are simply curve fitting the effect away. A drawdown must be expected over a large sample size. . Smooth equity curves are a fiction over a large trade sample size and are simply a symptom of a short data sample.

Many would think during the drawdown periods that the edge was lost....but it hasn't. It is still there. The drawdown was simply a result of the noise that was present in that market and it's volatile influence on that permanent edge.

https://atstradingsolutions.com/wp-content/uploads/2020/06/Random-Sample-with-Edge-5000.png

Example 5: Trade Sample Size 10000 per Return Stream - The same as Example 4. Now it is very clear to all what is going on.

https://atstradingsolutions.com/wp-content/uploads/2020/06/Random-Sample-with-Edge-10000.png


These examples are simply used to demonstrate how hard it is to extract an edge from this market using data mining and how important a large trade sample size is in your data mining efforts. It really is an eye opener.

Cheers mate

Rich

Diversification and risk-weighted returns is what this game is about

8 (edited by zikka1 2020-06-13 13:44:47)

Re: Multi market validation

Thanks Rich - very helpful.

"You cannot avoid drawdowns. Techniques that do are simply curve fitting the effect away. A drawdown must be expected over a large sample size. . Smooth equity curves are a fiction over a large trade sample size and are simply a symptom of a short data sample.

Many would think during the drawdown periods that the edge was lost....but it hasn't. It is still there. The drawdown was simply a result of the noise that was present in that market and it's volatile influence on that permanent edge."

***Fully agree - if there were edges that have no risk (whether volatility/drawdown/years of underperformance/inability to exit/enter without moving the market), they will be arbitraged away.

Based on your posts, I absolutely want my workflow to be multi-market based - goal is to generate strategies that have longevity without tinkering (walk forward, tweaking every x months/years etc.) the strategies, albeit the position sizing/portfolio management/generation of new strategies could be an ongoing process:

However, I have some questions:

    Data mining for trading strategies: The presumption is guilty until proven innocent. I get that and if you're trying to curve-fit/weed out all the drawdowns/risk, even worse.What is their fundamental role? Is it to speed up backtesting (and save time finding edges) and avoid coding? My sense is that most people use these softwares, check the performance over 10 years of data (maybe as few as 20 trades), see beautiful curves with beautiful monte-carlos on single symbols and think generating strategies is super easy.

  Logic: What if the software generated a strategy that made no intutitive sense but works across, in your example, 20-25 markets and is robust otherwise: large sample of trades, 2 parameters only etc.; should such a strategy be discarded? I see that even QIM say that their collection of strategies explains certain behaviours, which sounds like they are looking for logic also; and Rentech also at least tries to find logic. The answer may be quite simple though i.e. " strategies that generalize well ACROSS markets and have only 2 parameters/are simple are very easy to understand anyway so the question is moot"  I know a couple of CTAS with a couple of decades of experience each and they wouldn't generally trade a straegy they don't understand - something like High[2] > [Low[5] and Low[3] > Low[8] would make them very skeptical.

Single market "edge": what if the strategy only works on 1 market or only across a series of CORELATED markets e.g. an overnight premium strategy only working on index futures in the U.S. - that would certainly not be multi-market. Couldn’t there be other effects that are specific to a particular market? Won’t an insistence on a strategy generalizing across markets lead to many missed discoveries? Perhaps, the answer is that the missed discoveries are a price worth paying as it will generally also weed out of a of false discoveries, which are more dangerous. Alternatively, one could say that if it doesn’t generalize well, one takes higher "overfitting risks"  with such a strategy and if he is cognisant of that, should perhaps only allocate a much smaller % of capital to strategies that do not generalize well.

No need for rigour - focus on survival: Generate 100's/1000's of strategies through data-ming quickly, no need to worry about logic or multi-market validation (only slippage/costs i.e. feasibility) and then run them on "demo" over 6 months (or 3 years years if you need more proof!) and select the ones that survived - lots of hours saved and lots of "false negatives" avoided. Some people swear by this approach vs other "rigorous" approach; their view is if it works (survival of the fittest) in live market, it is proof enough and the ultimate test Any thoughts?

Testing across markets: If one is testing across markets to validate a strategy, would it be better to a) test on one market (e.g. corn) and then validate on the other ten futures markets (which would then be the out of sample then) or b) test on all (all would be in sample but considering each market is so different, if it works on hundreds of years and trades of combined data even if in IS, does one need an out of sample) or c) test on all the markets in sample but split the data for each market such that each market has a portion of data, say 30pc, reserved as out of sample at the end of the data period (I.e. 30 pc of each symbols ending data would be oos), such that the in sample would be 70% of the data across the 10 symbols and out of sample would be the 30% of data across the 10 symbols?

Realistic returns: 20pc CAGR over a sustained period of time is considered world class for a hedge fund. The CTA performance graph you often post shows the average is closer to 8%. However, I often hear trades say "yes, that's because they manage billions of dollars, you can do much better as a retail trader". Is there any basis to this statement or is this another one of those myths with no evidence - Sure, I'd be super happy generating a 100% each year on my $100k and reach my million super fast (not greedy for the hundreds of millions smile ), if this were true!

FX: Professional manager friends have advised me to stay away from spot FX as a retail trader and focus on futures. Any thoughts on this?

Many thanks

Re: Multi market validation

nice zikka,

you throw in many food for thoughts

10 (edited by zikka1 2020-06-13 13:33:18)

Re: Multi market validation

Thanks hannahis

I think clarity of direction and philosophy (before approaching the problem) can be quite useful in confidence building/mitigating psychological weaknesses (and actually getting somewhere, of ocurse!)

I also learnt a lot from your exchange with geektrader (who I see is another veteran of the market and found success after almost a decade, though he also says that it probably wasn't worth the effort!)

11 (edited by ats118765 2020-06-14 05:27:41)

Re: Multi market validation

Hi Zikka1. Mate a lot in that post of yours. Awesome :-)

zikka1 wrote:

    Data mining for trading strategies: The presumption is guilty until proven innocent. I get that and if you're trying to curve-fit/weed out all the drawdowns/risk, even worse.What is their fundamental role? Is it to speed up backtesting (and save time finding edges) and avoid coding? My sense is that most people use these softwares, check the performance over 10 years of data (maybe as few as 20 trades), see beautiful curves with beautiful monte-carlos on single symbols and think generating strategies is super easy.

I think convenience has got a lot to do with the popularity of the method for many and particularly the ability to generate trade-able solutions which do not require coding......but there is a very useful element of the process that tends to be overlooked by many. It provides a much faster basis for hypothesis testing assumptions you make about the behaviour of a market.

No matter if we use a data mining method or simply our own discretionary or other systematic method for developing strategies, the results are always curve fit from the get go. Curve fitting is a necessity for any successful strategy but the key factor that determines whether an edge is present is associated with whether that curve fit result is associated with a causal or non-causal relationship with the market data. We need our strategies to be curve fit to a particular emergent market condition that is enduring into the future rather than simply being curve fit to past market data that has no causative relationship and future longevity.

The latter non-causal relationship is what you have got to watch out for and data mining without a bit of preliminary guidance actually exponentially increases that possibility of simply generating a solution that is linked to 'noise' as opposed to an underlying causal pattern that has future endurance if you simply let data mining do it's own thing.

There are many more possible 'emergent patterns' arising from the impacts of noise than the emergent patterns that arise from a causative 'push factor'. Left to it's own devices, data mining will simply multiply your problems in seeking a system with an edge. The universe of possibilities generated by data mining is huge given the speed and scope of it's capabilities with current technology....but the vast majority of the outcomes generated are unlikely to lead to an enduring edge but they all will be deftly 'fitted' to historic market data.

But provided you restrict it's capabilities within a predefined logic that you can clearly see will result in a desired outcome if the market exhibits a particular behaviour, then you can significantly narrow down the scope of outputs that fit within the realm of your hypothesis. This is when data mining then becomes an invaluable tool to test that logic and validate whether the majority of solutions generated have a slight positive expectancy over that market condition that can test that hypothesis within high confidence levels.


zikka1 wrote:

  Logic: What if the software generated a strategy that made no intutitive sense but works across, in your example, 20-25 markets and is robust otherwise: large sample of trades, 2 parameters only etc.; should such a strategy be discarded? I see that even QIM say that their collection of strategies explains certain behaviours, which sounds like they are looking for logic also; and Rentech also at least tries to find logic. The answer may be quite simple though i.e. " strategies that generalize well ACROSS markets and have only 2 parameters/are simple are very easy to understand anyway so the question is moot"  I know a couple of CTAS with a couple of decades of experience each and they wouldn't generally trade a straegy they don't understand - something like High[2] > [Low[5] and Low[3] > Low[8] would make them very skeptical.

Given the complexity of markets IMO it is possible that there are solutions of no intuitive sense that actually have an edge.... but there is just no way that I can confidently test that this is the likely case. I think the reason the professionals also avoid these solutions is that they like to tie their conclusions to an underlying hypothesis of how they feel markets actually operate. This gives them confidence to stick to their rules when times get tough and also allows them to apply this principle in different contexts. Once you have an understanding of the 'causative' factors that give rise to the result, then you tend to trust in those rules and stick with them and can also apply them as general market principles.

For example if I assume that trends are a phenomenon associated with an 'enduring' participant behaviour of fear and greed, and design a model that simply follows price with a trailing stop and open ended profit condition  (around which I loosely data mine to provide design variations) ..then I can assume that in an uncertain future, that the 'causative' behaviour will remain in place. This gives me confidence to hold onto these strategies even during tough times. It also gives me confidence that this 'causative factor' can be tradeable in other contexts (eg. is a behaviour common to all liquid markets).

It is a bit like learning a new language. If like a robot you simply utter a phrase that has no connection to the underlying logical meaning then it cannot be applied to different contexts , even if by luck the uttered phrase actually makes sense in that context. If however you understand the 'causal roots' of that language and how to then apply that language to different contexts then meaning can be applied. Simply restating that phrase in a different context may lose the causal connection of the intent. So in the data mining context....a solution (a phrase) needs to be connected to a causal factor to be able to be confidently applied in an uncertain future.

I personally feel that understanding the 'causal' factors to that solution is imperative as it then gives a basis to understanding the why's that result from performance. Why did it fail or why did it succeed?  It gives you a context for your experiment and a firmer grip on the inherent strengths and weaknesses of your system before you then you step into an unknown future with risk capital at stake.

zikka1 wrote:

Single market "edge": what if the strategy only works on 1 market or only across a series of CORELATED markets e.g. an overnight premium strategy only working on index futures in the U.S. - that would certainly not be multi-market. Couldn’t there be other effects that are specific to a particular market? Won’t an insistence on a strategy generalizing across markets lead to many missed discoveries? Perhaps, the answer is that the missed discoveries are a price worth paying as it will generally also weed out of a of false discoveries, which are more dangerous. Alternatively, one could say that if it doesn’t generalize well, one takes higher "overfitting risks"  with such a strategy and if he is cognisant of that, should perhaps only allocate a much smaller % of capital to strategies that do not generalize well.

Totally agree mate. There are certain strategies that respond to a particular market condition that occur from time to time and can be attributed to a type of resonance that is market specific. These tend to be 'convergent' in nature. For example when markets are directionless and can exhibit a rhythmic oscillation around a fairly stationary equilibrium point. These oscillations are generally unique to a particular market and associated with a particular participant behaviour mix but are unlikely to be successfully applied across multi-markets...... unless of course that there is some very strong causal link between other markets to this phenomenon. With central bank intervention across asset classes (buying the dips and selling the tips), we have seen this feature spread across asset classes...but it is unlikely to be an enduring feature that can be relied on.

I personally prefer hunting for more permanent market features that are market ambivalent as opposed to less enduring market patterns that tend to be market specific. IMO there is more wealth gains to be had in mining for divergence than convergence. It is a far less competitive space as you really need patience to play it and confidence in your system....but that is why I feel that the edge is so enduring with divergent techniques and why they still haven't been arbitraged away.

That is just my preference though mate....and should not dissuade people from data mining for either divergent or convergent solutions. Both are valid IMO...however they are at either ends of the spectrum and as a result require different workflow processes to mine for them. It is not a one size fits all process. The process you use needs to tie to the logic of what you are trying to achieve.

zikka1 wrote:

No need for rigour - focus on survival: Generate 100's/1000's of strategies through data-ming quickly, no need to worry about logic or multi-market validation (only slippage/costs i.e. feasibility) and then run them on "demo" over 6 months (or 3 years years if you need more proof!) and select the ones that survived - lots of hours saved and lots of "false negatives" avoided. Some people swear by this approach vs other "rigorous" approach; their view is if it works (survival of the fittest) in live market, it is proof enough and the ultimate test Any thoughts?

Not sure mate. When in doubt I always refer to the long term track record of the best FM's in the world. This particular approach is yet to feature in that track record. It might still be a long time coming...or never at all.

While I like the idea of 'Survival of the fittest' it isn't a process I would deem worthy as a rigorous empirical method. There is almost an Artificial Intelligence (AI) flavour to the rationale where you pass the heavy lifting over to the machines but I have yet to see significant success in this arena. One of the big reasons it doesn't fly with me is that to actually demonstrate an edge....you need a really big trade sample size. A 3-5 year track record could just have easily been a simple 'lucky result'.

zikka1 wrote:

Testing across markets: If one is testing across markets to validate a strategy, would it be better to a) test on one market (e.g. corn) and then validate on the other ten futures markets (which would then be the out of sample then) or b) test on all (all would be in sample but considering each market is so different, if it works on hundreds of years and trades of combined data even if in IS, does one need an out of sample) or c) test on all the markets in sample but split the data for each market such that each market has a portion of data, say 30pc, reserved as out of sample at the end of the data period (I.e. 30 pc of each symbols ending data would be oos), such that the in sample would be 70% of the data across the 10 symbols and out of sample would be the 30% of data across the 10 symbols?

Yes mate. You have nailed it IMO. I actually separate my universe into 3 groups. Commodities, Equity Indices and Forex. I then undertake multi-market tests within each of these categories. These broad groupings are constructed differently and exhibit different global behaviours. I therefore would not recommend a one size fits all here. Equity Indices in their construction have a long bias and need to be treated differently in the data mining process. There is more commonality between Forex and Commodities, but I still prefer to split them up to play it safe.

Regarding OOS. This IMO depends on the confidence you have on the method. When I started the trend following workflow process without multi-market I would reserve about 50% for OOS. Over time and with the introduction of multi-market I became more and more confident that the 'core design' was actually preventing unfavourable curve fitting results...so I then tested the results using all In Sample with no OOS but with Multimarket. I then compared and contrasted the impact using Map to Market and found that the greater available market conditions used to build the strategies (In Sample) achieved the outcomes I was looking for. It became more important to me to have a larger In Sample than to reserve an amount of market data for OOS to test the validity of the strategy on unseen data.

When you get to a trade sample of say 10,000 trades using multimarket then it becomes clear if you have an enduring edge or not....Provided your trade sample size is sufficient enough then the need for OOS to test the validity of the strategy on unseen data loses its significance.

So in a nutshell I have more confidence in an edge that can be demonstrated by a 10,000 simulated trade backtest than an edge we might presume exists in a 200 live or demo trade out of sample phase. That OOS test suffers from the same lack of sample size that creates the problem in the first place. The emphasis therefore needs to be placed on ensuring that backtests correctly simulates the live trade environment and you can factor in conservative cost assumptions such as Slippage, varying SWAP and spread etc. to validate that simulation and ensure it is conservative in nature.

I don't place much validity in a method that rotates say monthly in and out of EAs that succeed or fail in that very short time interval based on short term data mining methods....but I would place more emphasis on a method that rotates between algorithms to 'sharpen the edge' regularly provided those same EAs can also demonstrate a very long term track record (say 20 to 30 years) under simulation. I admit that markets do adapt so you need some method to keep that edge sharp over time....even with trend following.

The parameters of the strategy need to stay but the values of those parameters need to be dynamic over time to reflect adaptive market conditions.

zikka1 wrote:

Realistic returns: 20pc CAGR over a sustained period of time is considered world class for a hedge fund. The CTA performance graph you often post shows the average is closer to 8%. However, I often hear trades say "yes, that's because they manage billions of dollars, you can do much better as a retail trader". Is there any basis to this statement or is this another one of those myths with no evidence - Sure, I'd be super happy generating a 100% each year on my $100k and reach my million super fast (not greedy for the hundreds of millions smile ), if this were true!

I hear you mate. When I hear this bold rhetoric I tend to smile and quickly dispatch the discussion into the 'graveyard of trading myths'. It is not that people are necessarily 'lying' but rather that they simply do not know how to distinguish between random luck and an edge.  I think an appreciation of the randomness that exists within complex system only really comes from a pretty deep understanding of how complex systems work.

Also I think people might confuse annual returns with CAGR. An 8% CAGR over a 20 year plus track record allows for some great years (60% plus) but it needs to be put in the context of those bad years or average years where you may get a 4% loss or a 2% gain any particular year. Once again it is a symptom of volatile equity curves and particularly attributed to the impact of compounding (leverage) on that return stream over the long term. Most traders never experience the windfall of compounding as they are not alive long enough to experience it.

All we need to validate this myth is proof of the pudding over say a 20 year trading history or say 10,000 live trades....then I would be prepared to at least consider what is being said.....the ultimate arbiter of truth is the validated track record which shows a very large trade sample size.

zikka1 wrote:

FX: Professional manager friends have advised me to stay away from spot FX as a retail trader and focus on futures. Any thoughts on this?

They are right to a degree but forget that the finite capital available to us retail traders is a really big obstacle to achieve sufficient diversification. With a small capital base you simply cannot obtain the required degree of diversification in Futures that you can with Forex and CFD's through the retail broker.

The microlot offering in Forex and CFDs is a great thing that you don't have in Futures that allows you to heavily diversify....but there is a greater trading cost to this convenience which does put some extra challenges on the retail trader.

I personally prefer the Forex and CFD space to play this game.

Cheers Z. Thanks for the discussion. :-)

Diversification and risk-weighted returns is what this game is about

Re: Multi market validation

A workaround - what about normalizing data that all pairs can be used effectively in the multi-market? Then it would be pip-for-pip, comparing apples to apples. It can be easily done for Pro, a bit of a hassle for Studio, though.

Re: Multi market validation

Thanks Rich

Thanks; all very well noted.

Logic and multi-market: I think we are "convering" (no pun intended) that multi-market testing acros a variety of markets solves some of the questions in the post e.g. statistical significance, random noise and logic. With regards to logic, in particular, a) my intutition is that it won't be a problem as it is likely that it will be apparent what the logic of a stratetegy is if it works on 10-20 markets as it will be quite simple/intuitive; however, even if it is not apparent, *if it is working on other markets* (especially uncorelated ones), I would be OK to accept that I don't understand it, although, repeating myself, I do think it is unlikely that one will run into a problem. Another point is of course that if something is not easily understandable but works across 15 (ideally uncorelated) markets, it will also be looked over by others (and hence that is why it perhaps persist). From the Zuckerbook it appears that they would trade something they don't understand but then again, they will likely be subjecting the "illogical strategy" to so many other cross-validations (perhaps including multi-market) that it is probably OK.
However, if there is something that doesn't "make sense" but only works on 1-2 markets, fo course the risk of a "non-causal" relationship and curve-fitting is higher; again, showing the usefulness of multi-market testing.

The book "Finding Alphas" says one of the characteristics of a good alpha is "its profits hits a recent new high" - not sure I fully agree with that. It also days "the longer the period one cosndiers, the more likely that the alpha will exhibit signs of decay" - any thoughts about that as I would have thought "long lasting", simple alphas of the kinds I understand you are looking for endure?

Survival of the fittest: On the point about the "let some strategies run" and see which one's survive, I agree that it is unlikely to be empirically rigorous.

Data split: Also agree that is hard to see the point of splitting the data between IS/OS if you're testing across so many markets; if a strategy "works" across multiple market regimes and time periods, then what purpose would OS serve? Might as well run it in "incubation" going forward from today for a period of 3/6 months for a OS.

One more question that arises from the above is the issue of statistical significance/t-stat; so the more combinations you try the less significant the "authenticty" of the results become is what I understand. However, that is the whole premise of the data-mining programs. Or again, perhaps you think whilst this is true in theory, a multi-market test also mitigates this risk significantly?

Small trader: On the "small traders can make 20pc a month", my feel is that it is possible for SOME discretionary day traders who trade a few thousand dollars, getting in and out of micro-cap/iliquid biotechs etc. but the success rate in that is probably tiny and a wholly different ball game to professional/systematic trading.

Forex/CFD - so I'm guessing that you can find "honest" broker in the space? I may have to choose this option over futures due to small account size (alongside micro contracts). I have a IG account. Is CFD data reliable/rigorous enough since the period would be quite short I imagine vs FX/futures?

Re: Multi market validation

footon wrote:

A workaround - what about normalizing data that all pairs can be used effectively in the multi-market? Then it would be pip-for-pip, comparing apples to apples. It can be easily done for Pro, a bit of a hassle for Studio, though.

Good thinking footon. That might help...but I am thinking that it may only be applicable to the backtest context. Not really sure how to apply your thoughts to the live trading context. :-)

Diversification and risk-weighted returns is what this game is about

15 (edited by ats118765 2020-06-15 05:47:33)

Re: Multi market validation

zikka1 wrote:

With regards to logic, in particular, a) my intutition is that it won't be a problem as it is likely that it will be apparent what the logic of a stratetegy is if it works on 10-20 markets as it will be quite simple/intuitive; however, even if it is not apparent, *if it is working on other markets* (especially uncorelated ones), I would be OK to accept that I don't understand it, although, repeating myself, I do think it is unlikely that one will run into a problem. Another point is of course that if something is not easily understandable but works across 15 (ideally uncorelated) markets, it will also be looked over by others (and hence that is why it perhaps persist). From the Zuckerbook it appears that they would trade something they don't understand but then again, they will likely be subjecting the "illogical strategy" to so many other cross-validations (perhaps including multi-market) that it is probably OK.
However, if there is something that doesn't "make sense" but only works on 1-2 markets, fo course the risk of a "non-causal" relationship and curve-fitting is higher; again, showing the usefulness of multi-market testing.

Good points Z :-)

PS. It is interesting that Renaissance has posted a 20% loss for 2020 so far. I wonder what the Medallion fund has produced? I wonder whether the Renaissance clients are actually the product used by the Medallion fund....*sniggers* It all sounds very fishy to me? I remember Long Term Capital Management too clearly :-)

zikka1 wrote:

The book "Finding Alphas" says one of the characteristics of a good alpha is "its profits hits a recent new high" - not sure I fully agree with that. It also days "the longer the period one cosndiers, the more likely that the alpha will exhibit signs of decay" - any thoughts about that as I would have thought "long lasting", simple alphas of the kinds I understand you are looking for endure?

I seem to have found this symptom as well Z so I do agree with them. There appears to be a degree of adaption to the markets which influences even stable models like trend following. This is probably attributed to the varying participant mix and behaviours over time. There is more noise in the market today than there was in say the late 1990s to 2000's. This seems to be altering the nature of general principles such as the trending condition and making them 'less easy' to catch.

The volatility that generally accompanies trending conditions appears to be much greater than it was, and as a result does influence the values you use for the common parameter sets. The CTAs express how in the previous century a short term looback and a tighter trail was sufficient to catch the trends (eg. 50 day lookback)....but most have recognised in today's market that you need longer term look-backs and more breathing room for trends. Nearly all the longer term juggernauts have progressively extended their lookbacks to the medium to long term ternding environment. The core strategy hasn't generally changed but the values used for the parameters have been adjusted.

For example the Turtles strategy still works today...but not when you apply the exact same value sets for each of the parameters of that strategy. This appears to be symptomatic of an adapting market. I can also visually see the effect when I look at monthly charts over a long data series. The trends used to be far more defined and less volatile than the trends of today. You really need to step outside the noise to capture them today which demands longer lookbacks in general and wider trailing conditions.

The way I deal with this adaption is by:
1. Using a process that first establishes whether the return stream is robust over a very extended data set.......and if it is....then it goes into a pool of robust return streams. If not it gets excluded.
2. I then take this entire pool which is re-generated on an annual basis and use a shorter term method (say past five years) and select those return streams that have floated to the top of the list in terms of their short term performance metrics.

For example let's say that my robust pool of multimarket candidates is 500 or so return streams. I can't trade all of them on a single MT4 installation with my Broker so I need to make a 'selection' decision to reduce this pool to say the top 100 non-correlated return streams. Rather than applying a discretionary method for selection, I prefer to use a systematic rules based approach.

So the first step is to apply the recency test to reduce this initial pool to say the top 200....and then from within this filtered pool I can then iterate to determine the best 100 of these return streams that offer the best bang for buck as a consolidated portfolio.

I would fairly regularly redo the recency test so that I rotate across the possible solutions...which adds a bit of extra diversification into the process.

To confirm this opinion I find the following. Let's say you develop a trend following strategy based on a 20 year market sample. Then let's say you have a further 20 years on either side of this In Sample data sample. What you find is that there is a progressive deterioration in performance the longer you look back and the further you look forward. This deterioration appears to result from the fact that there is definitely an element of curve fitting going on in the 20 year in sample period and that markets adapt over time. In backtest land....your worst drawdown is always behind you....and in live trading land your worst drawdown is always ahead of you.

Markets never repeat but they do rhyme. The past data sample only contains a fraction of the future possible paths. The uncertainty of the future introduces new market conditions that are never exactly a repeat of the past.


zikka1 wrote:

One more question that arises from the above is the issue of statistical significance/t-stat; so the more combinations you try the less significant the "authenticty" of the results become is what I understand. However, that is the whole premise of the data-mining programs. Or again, perhaps you think whilst this is true in theory, a multi-market test also mitigates this risk significantly?

I really have issues with applying traditional statistical tests to determine the validity of the sample. Given that I predate on 'fat tailed' conditions where non-Gaussian rules dominate, the validity of statistical techniques that apply Gaussian assumptions may just be outright wrong!!!! I far prefer using logic and visual methods as opposed to statistical techniques in my decision making.....however I am a big fan of using MAR as a performance method.

For example I am actually looking to catch outlier events which traditional methods suggest you exclude due to their dominant impact they produce on your overall trading result. I actually need to test that my strategies are designed to avoid the 'common' trend of possible no enduring substance and only focus on attacking those significant trends that possibly turn into an outlier.

Contrary to popular opinion I therefore need to reduce my sample size of trading events rather than increasing them. The way I compensate for this is by increasing methods of diversification to lift overall trade frequency. This conundrum therefore forces me to use other methods as opposed to traditional statistical methods as a means to address these unpredictable and infrequent 'outlier' events.

I have real issues with Sharpe and Sortino and also methods that determine what a suitable sample size is to reach a statistically valid conclusion. They all tend to be based around Gaussian distribution premises.

zikka1 wrote:

Small trader: On the "small traders can make 20pc a month", my feel is that it is possible for SOME discretionary day traders who trade a few thousand dollars, getting in and out of micro-cap/iliquid biotechs etc. but the success rate in that is probably tiny and a wholly different ball game to professional/systematic trading.

Yep agreed. Small accounts might have higher overall CAGR but they are also accompanied by higher volatility and bigger drawdowns. Risking a small amount makes you less risk averse so can support higher leverage than what might be recommended on large accounts where capital preservation is a priority.

We must also remember that Professional traders report on a Net basis (NAV) after deducting management fees and performance fees and also excluding fund operating costs such as audit fees etc........so in comparing apples for apples between the Funds and the Retail trader you can add a few percent to the CAGR and reduce the Max Draw a few percent in comparing directly.

zikka1 wrote:

Forex/CFD - so I'm guessing that you can find "honest" broker in the space? I may have to choose this option over futures due to small account size (alongside micro contracts). I have a IG account. Is CFD data reliable/rigorous enough since the period would be quite short I imagine vs FX/futures?

IG are good and have a great diverse offering that really helps diversification objectives...but micro-lot offering is absent. You need a large capital base to heavily diversify. I have had a long history with them...but the product may have changed in the last few years. The only problem I found was the limited data history that is generally available from the feed.

I now use Pepperstone (costs are razor sharp and rebates are good but product diversification is more limited, Data history and quality is good).....but Avatrade is Ok if you are after broad diversification (but costs are generally higher. Data quality can be dodgy).

Diversification and risk-weighted returns is what this game is about

16 (edited by zikka1 2020-06-15 14:42:54)

Re: Multi market validation

Thanks Rich,

The point about volatility regime change and therefore systems needing to adapt (e.g. changing the parameters in the moving average) is noted and if you are running a collection of uncorrelated strategies or at least with low corelations and diversified across currencies/markets/factors,short/long/timeframes/parameters (let’s say there’s 5 MACD strategies spread out across 50 instruments), should this really be a problem?

And of course, you could do the above AND rebalance the systems as I think you’re suggesting. In the vocabulary I better understand (as my live experience to date is with systematic investment in stocks), there are studies showing that the best predictor of which factor will outperform over the next period is “what factor is outperforming at the moment”. Of course, this assumes serial auto-corelation of sorts in factors; if that changes (and anything could happen ofcourse) then the factor momentum theory won’t hold.

Which brings us back to the point about running cross-over systems with multiple parameters and model diversification. As an example, are you familiar with the “dual momentum” model; if so, I think Gary uses only a 12 month lookback (i.e. go long particular ETFs when 12 month lookback > 0) and he says it has worked best over the longest stretch of data and is simple and has lower chance of curve-fitting. Someone else has come up with a model that uses his approach but averages 3/6/9/12 months, so goes long only if the average is > 0. Alternatively, you could run 4 models (cost allowing) – one would work on 3months >0, another 6 months > 0 and so on. Although Gary doesn’t like these “modifications”, they intuitively appeal to me as they reduce model specification risk and specifically the problem of market behaviour changing and I would sleep much easier!
To make it more sophisticated, ideally you’d have a collection of systems, some of which do well in multiple combinations of sideways, trending + low vol/high vol + low autocorrelation/high autocorrelation etc. etc.

So drawing an analogy with my stock modelling (which by far I think is easier as there is a defined space where institutions find it difficult to enter, no leverage, LOTS of studies on what factors work/don’t work and most importantly a LONG bias). Sometimes I think why put so much more effort into developing trading (vs investing) systems – just continue to develop stock “investment” models and diversify across tactical allocation/investment strategies which at least in my opinion are easier and will probably achieve the 10-15% CAGR. Further, I don’t need my equity curve to be smooth, although managing money in the future (where in that requirement of smoothing comes in) is appealing. I don’t know – I haven’t found a convincing answer yet!

On the point about “statistical tests”, I don’t quite understand them that well and I do think something could show a high “t-stat” and still be completely bogus, as you allude. The danger with these kinds of tests borrowed from the “real sciences” is that they can mislead you into thinking what you have is robust and “scientific” (I may be wrong, just a thought) by providing all kinds of complex “affirmations”. For example, you may have a rule that has a t-stat of 5 what does that mean if there are a few trades and it is market specific and it has 5 parameters.

As you say, a few simple metrics and an eye ball of an equity curve and distribution (and perhaps most important of all, hundreds of years of multi-market data and robustness across markets) is probably the best affirmation…..?
My capital base for trading for now is probably about 25-30k USD – maybe I should consider Pepperstone then? Not looking at “intra-day” at all for now – happy to hold for longer periods and just want to minimize my headache through the “non-model-errors” that can happen e.g. data errors, slippage, results not being replicating.

Thanks,

17 (edited by ats118765 2020-06-15 15:55:29)

Re: Multi market validation

zikka1 wrote:

The point about volatility regime change and therefore systems needing to adapt (e.g. changing the parameters in the moving average) is noted and if you are running a collection of uncorrelated strategies or at least with low corelations and diversified across currencies/markets/factors,short/long/timeframes/parameters (let’s say there’s 5 MACD strategies spread out across 50 instruments), should this really be a problem?

The problem with correlations is that they are a moving feast over the time series. This in part can be attributed to adaptive market behaviour. If you use a single correlation statistic to determine the price relationship between two return streams it really tells you nothing other than the degree of correlation over the entire time series.

For example below is a correlation matrix between 10 separate sub portfolios (at the market level).

https://atstradingsolutions.com/wp-content/uploads/2020/06/Correlation-between-10-systems.png

This makes the relationship appear static which is misleading and why I always prefer to visually map the relationship between return streams over a time series.

https://atstradingsolutions.com/wp-content/uploads/2020/06/Visual-Mapping-of-return-streams.png

When you visually compare you will find there are particular regimes where return streams become highly correlated and other regimes where they become uncorrelated. By using a visual mapping process, you can see how the nature of the return streams adapts to different market regimes.

In my trend following world I actually like high positive correlation when it is beneficial for my systems...such as during times of market crisis where I ride multiple return streams to wealth heaven.....but I do concern myself with adverse correlation when it is unfavourable to my portfolio. During adverse market regimes I like low correlation between return streams.
You need to identify those regimes where things turn ugly as this reveals the weakness in your portfolio solution. A single correlation statistic simply won't do this.

You will be familiar with this phenomenon in equities markets. During bull markets, equities tend to be far less correlated than during bear markets where nearly all equities turn south at the same time.

So having a portfolio that adapts offers a way around this dilemma. Simple broad diversification might not necessarily address this 'adaptive' issue.


zikka1 wrote:

And of course, you could do the above AND rebalance the systems as I think you’re suggesting. In the vocabulary I better understand (as my live experience to date is with systematic investment in stocks), there are studies showing that the best predictor of which factor will outperform over the next period is “what factor is outperforming at the moment”. Of course, this assumes serial auto-corelation of sorts in factors; if that changes (and anything could happen ofcourse) then the factor momentum theory won’t hold.

Yep mate. What I tend to find in 'reality' is that serial correlation tends to be clustered at particular zones of a time series and separated by large tracts of noise. For example, recurrent favourable news events might be one of the causative agents of this repeated bias to the time series. When I refer to a persistent edge..what I really mean is a repetitive auto-correlated signal.

I actually think (but cannot prove) that momentum signals are probably less prone to change but it is actually the noise that actually is the dominant contributor to adaptive markets. IMO it suppresses or amplifies the signals creating variation over time. (I have spent too much time in the laboratory wave tanks splashing around and looking at the effect of disturbances on repetitive wave patterns...LOL)


zikka1 wrote:

Which brings us back to the point about running cross-over systems with multiple parameters and model diversification. As an example, are you familiar with the “dual momentum” model; if so, I think Gary uses only a 12 month lookback (i.e. go long particular ETFs when 12 month lookback > 0) and he says it has worked best over the longest stretch of data and is simple and has lower chance of curve-fitting. Someone else has come up with a model that uses his approach but averages 3/6/9/12 months, so goes long only if the average is > 0. Alternatively, you could run 4 models (cost allowing) – one would work on 3months >0, another 6 months > 0 and so on. Although Gary doesn’t like these “modifications”, they intuitively appeal to me as they reduce model specification risk and specifically the problem of market behaviour changing and I would sleep much easier!
To make it more sophisticated, ideally you’d have a collection of systems, some of which do well in multiple combinations of sideways, trending + low vol/high vol + low autocorrelation/high autocorrelation etc. etc.

Yep mate. I have pinched lots of ideas from Gary Antonnaci. Rob Carver and Nick Radge have also had their heads deep in this rotational principal. The idea relates to using 'signal strength' as a basis to select your particular markets to trade from a broader available universe. It certainly makes sense to me. Signals tend to ebb and flow (impact of destructive/constructive interference from noise) and are never stationary so an assumption that the stronger signal has more durability into the future is probably a wise move....though as you say....it can all change on a pin.

zikka1 wrote:

So drawing an analogy with my stock modelling (which by far I think is easier as there is a defined space where institutions find it difficult to enter, no leverage, LOTS of studies on what factors work/don’t work and most importantly a LONG bias). Sometimes I think why put so much more effort into developing trading (vs investing) systems – just continue to develop stock “investment” models and diversify across tactical allocation/investment strategies which at least in my opinion are easier and will probably achieve the 10-15% CAGR. Further, I don’t need my equity curve to be smooth, although managing money in the future (where in that requirement of smoothing comes in) is appealing. I don’t know – I haven’t found a convincing answer yet!

I hear you Z :-)....but I think that a long or short and heavily diversified method across asset classes, geographies, systems etc. are probably more robust considering what history has delivered over the past 500 or so years. I see how the Nikkei has performed since the late 80's or the FTSE....and it makes me shudder. It is not a problem for those that committed to the US Equities....but even the greatest empires have a finite duration. Why not trade both the ups and possible downs.....everywhere you can?

zikka1 wrote:

On the point about “statistical tests”, I don’t quite understand them that well and I do think something could show a high “t-stat” and still be completely bogus, as you allude. The danger with these kinds of tests borrowed from the “real sciences” is that they can mislead you into thinking what you have is robust and “scientific” (I may be wrong, just a thought) by providing all kinds of complex “affirmations”. For example, you may have a rule that has a t-stat of 5 what does that mean if there are a few trades and it is market specific and it has 5 parameters.

As you say, a few simple metrics and an eye ball of an equity curve and distribution (and perhaps most important of all, hundreds of years of multi-market data and robustness across markets) is probably the best affirmation…..?
My capital base for trading for now is probably about 25-30k USD – maybe I should consider Pepperstone then? Not looking at “intra-day” at all for now – happy to hold for longer periods and just want to minimize my headache through the “non-model-errors” that can happen e.g. data errors, slippage, results not being replicating.

Yep mate. I agree. There are damned lies and statistics. Pepperstone certainly has been a good choice for me....but is a bit limited in product range. Still beggars can't be choosers. 

Nice chatting Z  :-)

Diversification and risk-weighted returns is what this game is about

Re: Multi market validation

Ah yes, my view was naive on corelation....all understood.

"I hear you Z :-)....but I think that a long or short and heavily diversified method across asset classes, geographies, systems etc. are probably more robust considering what history has developed over the past 500 years. I see how the Nikkei has performed since the late 80's or the FTSE....and it makes me shudder. It is not a problem for those that committed to the US Equities....but even the greatest empires have a finite duration. Why not trade both the ups and possible downs.....everywhere you can?"

Absolutely, not disagreeing - if equities go sideways for 15 years or down, I'll be in trouble, so diversify across approaches (trading, investing) strategies, markets, assets, geographies, timeframes etc etc. is key (heck, even buy and hold is a strategy and I have exposure to that through my pensions - everything has a place in life!)

Posts: 18

Pages 1

You must login or register to post a reply

Forex Forum → Expert Advisor Studio → Multi market validation

Similar topics in this forum