1 Topic by hannahis (edited by hannahis 2017-08-24 15:03:04)

  • Registered: 2010-11-18
  • Posts: 1,179

Topic: Analytic Trade Management (ATM)

To discuss the advantages of combining trade analytic into automated trade management

Below is a software I envision that would help algo traders like us.

My wish is to have a program that can analyse my strategies performance and then use certain criteria to turn the strategies on and off. I call it Analytic Trade Management. Combining the function of Trade analytics into our trade management.

1. Run hundreds of our EA on a demo/nano account to collect trade performance statistics/metric. (eg using Fxblue to collect our EA trading performance)

2. Based on these trading performance, use certain criteria or metric such as net profit, profit factor, R-expectancy etc to select which EA qualify for live trading portfolio.

3. This software will constantly monitor the trade performance of all our testing EA in demo/nano accounts and when EA meet the criteria such as min Pf = 1.5, it will automatically "turn on" copy trading function from demo/nano to live account for all EA that meet this criteria.

Advantages

1. By constantly running these EA on a demo/nano account, we can start to accumulate trading statistics and monitor their trading performance over time.  Such additional and detailed information and help us understand under what kind of situations, our EA did well or badly.

2. I can do weekly comparison and understand why my EA did badly on certain weeks and go to the corresponding MT4 time period/date and examine the MT4 charts to see what additional trading conditions I can add in or modify to improve my EA further.

3. By doing weekly comparison, I can spot "un-corelated" strateiges, i.e. I can identify ranging EA and Breakout EA and thus intermix my portfolio with EA from these different categories.  Some weeks, a group of EA will do better than another (eg. Ranging EA will do well in ranging week while Breakout EA may suffer some loses etc).

4. Having the trading Statistics, I can therefore choose which EA to be added into my portfolio and produce better overall results.

5. If the software can automatically "turn on and off" which EA to copy trade from demo/nano to live accounts, it will spare us the trouble of having to keep monitoring and keep changing and setting up our MT4 charts to create our portfolio every now and then.

For Eg, If the software can copy trade my top 100 EA (collect the trade statistics from fxblue/excel file) that has minimum pf value of 1.5 and average trade win of certain amount, then if will keep those good EA constantly running on live account and if any EA started to deteriorate or decline in it's performance and "fall out" of the top 100 list, the software will turn off that particular EA but it will still keep running in the demo/nano account to continue to collect trading statistics so that when the EA starts to improve in it's performance (because it's formula fits the current market situation).


I'm suggesting 3 methods for portfolio selection criteria

1. Base on preset criteria (Net Profit, PF, MAE/MFE, Win loss, etc), i.e turn off strategies when PF fall below 1.5

2. Base on 2 criteria, select Top 30 (can be pre define by user) Strategies with PF equal and above 1.5 This method select strategies that are robust and versatile to fit into various market trends (based on monthly or weekly results). If strategy's results fall below top 30 position, turn strategy off. Using this method, the portfolio is constantly having the "best" fitting strategies for the current market trend. (edited)

3. Base on percentage growth. Select strategies that has the largest percentage growth (weekly or monthly comparison, again can be preset by users). This method is to turn on and off strategies that doesn't fit into current market trend. Often when we switch off strategies, we wonder when to switch it back again.

With the strategies constantly running all the time (switch on or off for live but still running in demo) and with the software doing ongoing performance analytics, no more guessing game as to when to turn back on, our decision will be made based on statistical results.



A dear friend of mine had created a java programme and it can accomplished what I have mentioned above, except it can't automatically collect the trade statistics and automatically "communicate" with Fxblue copier to turn strategies "on/off".

If anyone has the expertise and willing to further improve this project, it would be great.

So here is a Semi-auto version.

Go to this you tube watch the demo https://www.youtube.com/watch?v=52x40re … e=youtu.be

Step 1. Download the manage EA zip file.

Step 2. Attached the "Magic Number Fetcher" EA to the experts folder of your MT4 (nano/demo/live account whereby you are using to collect your trade analytic data). Then install this EA onto a mt4 time chart, like the way u install any EA.

Step 3. Copy the Link to Java runtime and go to website to download the window off line file

Step 4. Open the "Executable Jar File" inside the Manage EA folder.

Step 5. Go to your MT4 MQL and under the "files" folder, you will see a "Dump History file". The file contain all the magic numbers that are installed in that particular MT4 terminal. Open the file (via excel, note pad, word pad etc) and copy the list of magic numbers (ctrl c).

Step 6. Go to the Java programme and paste (ctrl v) this list of magic number under the csv MT4 section (the right hand column). NOTE: the mouse right click function is not working properly here. So to paste, press "ctrl v"

Step 7. Go to your trade analytic software or website, such as Fxblue, Myfxbook, which ever programme you use to collect your trade analytic data. Select your EA/strategies via whatever metric you prefer. Once you have done, highlight the list of EA's magic number, copy and paste this list unto the section in the Java programme, under the section "Fxblue".

Step 8. Now press "Get Magic" number and you will see a string of magic numbers. Each row of magic numbers is for each Fxblue Copier (available free for download). Copy and paste each string of magic number onto the Fxblue SENDER copier's "include magic number" section.

If you have more EA/strategies, you will have more rows of magic number and you need more Sender Copier per MT4. However, if you only have less than 25 or so EA/strateiges, you most likely only need 1 Sender Copier and 1 row of string of magic number to copy.

Please note that the mouse right click function isn't working properly, so you need to press ctrl c for copy and ctrl v for paste on your keyboard.



For setting up of the Fxblue Sender and Copier, kindly refer to Fxblue manual to read and understand it's function. Basically, each copier need to have it's own "channel" to differentiate one copier from another, even if you are going to send it to the same receiver account.

So now you can semi-automate your copy trade your strategies based on your trade analytic criteria for your selection of strategies and you don't have to manually set up these strategies, one by one onto your MT4 terminal.

You can update this top selection list of EA/strategies/magic number yourself according to your personal preference. Such as update your top strategies selection once a week or once a month. Based on Net profit, Pf or your favourite R/DD or whatever you deem fit.


NOTE:

The above java programme is useful only to those who have hundreds of EA running in several MT4 accounts. 

This programme was meant to solve the problem I faced because I have many  MT4 terminals running with a couple of hundreds of EA.  And I have several portfolio and thus, it is very trouble when I have to keep pruning each portfolio every now and then.

So if you have less than 50 EA, then this method/programme is rather redundant for you.

Post's attachments

manage Eas (2).zip 8.68 kb, 9 downloads since 2017-08-24 

You don't have the permssions to download the attachments of this post.
  • Registered: 2010-11-18
  • Posts: 1,179

Re: Analytic Trade Management (ATM)

Hi Sleytus (Steve),

I understand you have created a MT4 Tracker, is there any way you can integrate this MT4 Tracker with trade statistics for turning on/off auto trading in MT4?

Fxblue Copier has the "select Magic number" section whereby we instruct the copier what EA/magic number to copy over. 

This Fxblue Copier has take "text" information whereby if you know some programming, you can make it retrieve a file such as excel file or whatever to automatically retrieve those magic number from the text file.  So all we need to do is to keep update the list of magic number in the file and fxblue copier will automatically try to retrieve it.  I hope I got this understanding correctly from my friend.  However he was too busy to do this second part, so I'm thinking whether you would be interested to improve this project further or do you have any suggestions or idea to add, given your expertise (no obligation).

  • Registered: 2017-03-14
  • Posts: 709

Re: Analytic Trade Management (ATM)

Hello Hannah,

You're stealing my thunder... (smile)

I've been working on a new indicator that comes very close to what you describe.  It's called "Adaptive Pruning".  I've mentioned it in passing in a couple of posts.

As an indicator it can continuously monitor transactions, and then based on a policy -- for now, something simple like Win Percent -- it turns Magic Numbers off.   Based on the policy, the indicator creates a file that contains a list of pruned (or excluded) Magic Numbers.  A few lines of code need to be added to the EA's *.mql to check this file prior to opening a position.  If the current Magic Number is a member of that file, then the request to open a position is skipped.  In time I will improve the indicator to support more sophisticated policies.

I'm currently using the indicator in my own trading and will update MT4 Tracker so it to can also read and modify the list of excluded Magic Numbers.

With regards to FxBlue -- I've used it in the past and wasn't so impressed.  Also, I don't want to go the route of remote monitoring -- that is, I prefer that MT4 Tracker runs on the same machine running your instances of Metatrader terminal.

I'll have more to share in a month or so -- I'm still testing.  Thanks to Popov and his software it opens up many new doors -- not only for allowing an inexperienced trader like myself to be successful trading forex, but also developing new tools to help manage this new trading paradigm.

4 Reply by hannahis (edited by hannahis 2017-08-24 19:18:22)

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Analytic Trade Management (ATM)

Wow Steve (great minds think alike but you are lightning years ahead of me smile )

I have yet to try your MT4 Tracker and I've the impression it is for EA Studio (correct me if I'm wrong).

I prefer your idea, having the "pruning device, ie. Code" attached to the EA itself, that a far better solution (that's the difference between a layman and a programmer.  A programmer always has a better solution because he knows what's available in his tools box).

Can this "code" be attached to a FSB Pro EA?

Can you kindly create a file whereby I can add the list of EA allowed (inclusion, instead of exclusion) for Live Trading as an alternative (for FSB Pro EA)?

In another words, when I use FxBlue statistics and decided to shortlisted my EA manually, I want to add my EA's magic number to the list and this "code" will check what are the list of EA allowed to be turned on.  Is this possible?

Well, I may steal your thunder only but for a while cos the final encore will go to you and I'll be the one standing and clapping and cheering you on smile


NOTE:

Popov, is there such a way whereby you can add a feature in FSB Pro so that when we export our EA, it will automatically attached an "external/additional" code such as Steve's code into our EA so that we don't have to do it one by one?

or

Can you kindly incorporate this "automatic turn on and off" trade management function as one of FSB Pro's unique additional feature, the FSB software will be one of it's kind to offer such complete automated trading from developing to trading, A to Z.

That would be every algo traders dream come true.  Whereby not only is our trading automated but our portfolio selection method automated as well.  Then we can go for our long vacation and truly hands off and let our EA and this analytic trade management feature do they jobs hand in hand smile

5 Reply by hannahis (edited by hannahis 2017-08-24 20:31:18)

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Analytic Trade Management (ATM)

Hi Steve

There are some Key differences between your concept and mine but they both has the same goal, using trade analytics to select our portfolio.  Neither one is better than another, it only depends on our priorities and trading preferences.

1. I want to constantly run all my testing EA on demo/nano account so that I can continually collect trade statistics to track the performance of my EA over a period of time, eg for 1 year.  And only good performing EA are added to the list for these EA to be traded in another account, Live account. 

Hence there is always two set of MT4 terminal running.  One for collection of trade statistic and another for Live trading.  The advantages of having a MT4 constantly running and collecting trade statistics irregardless whether EA is performing well or not, is that not all EA that decline in it's performance need to be discarded/stopped. 

Some EA declined in performance due to "misfit" of market conditions and it will pick up it's performance when market conditions are favourable (to it's trading rules/market, such as ranging EA vs Breakout EA).  And when they performance begin to pick up, I can put these EA back (recycling).  By discarding them prematurely, we ended up this constant and tiresome cycle of keep creating and testing new EA endlessly.



2. You are running your EA already on a live account and you are using your "code/device" to Exclude those poor performing EA (which means these EA may have "drained" your account to certain extend and then it's performance dropped and thus you remove them.  Hence you are using an "Exclusion" list, in another words, it's reactive measure.


3. Whereas my portfolio selection are based on past good performance or certain other criteria and then this EA are added into the list, hence I'm using an "Inclusion" list, in another words, it's a pro active measure.  And since these EA are being "tested" in demo/nano account first, those poor performing EA didn't have the "chance" to drain my live account before I pull them out.

4. Thus by turning your EA off, you also stopped collecting their trade performance and thus you will never know whether this EA will ever perform better again.  So what are you going to do with it, discard it?


NOTE:

There are 2 Schools of thoughts when it comes to portfolio selection.

1. Choose the "LOUSY" EA.

Select those poor performance EA because these EA has reached it's max draw down (it's trough) and hence, it's going to peak soon.  Hence, putting these poor performing EA in Live account is pitching it ready to ride the peak. 

However, the challenge is to identify/track each EA's max dd (or max consecutive losses) so as to pin point when it has reached it's potential lowest point and then put these EA to live trading.

If you belong to this group of traders, you will therefore not select the past best performing EA because these EA has reached it's peak and if you put them into Live trading, they are all going down hill because after it's peak it is time for it to hit it's trough.  Thus, by putting past good performance EA into your Live account, it will "blow" your account.


2. Choose the BEST

The 2nd School of thoughts is the most conventional perception.  Choose the best EA for your portfolio.  Assuming that the reason why these EA are the best is because they have whether through the numerous onslaught market conditions and thus these EA are the survivor of the fittest and thus they are selected on the basis of it's robustness and long durability/stability


In conclusion, whether you belong to the 1st or 2nd school of traders, it's a matter of perspective and I personally think it's very much depend on your personal inclination of how you develop your EA.  A statistical approach person would then to look at the statistics and see a pattern in the peak and trough performance and thus most likely pick the 1st school of thoughts.

Whereas for me, I develop EA from a manual trader's perspective (looking at market conditions rather than statistics to develop EA, ie. Rule based design) and hence I believe a robust EA is one that has strong and strong trading rules that can withstand volatility and thus become the survivor of the fittest.

Secondly I have hundreds of EA, I find it extremely tedious to know each EA's max dd or max consecutive losses to keep track which has reach it's peak and it's trough, I find such tracking method (and due to my inexperience) too much for me to handle.  And my strength is in designing good EA, I know how to improve my EA and hence, it's much easier for me to tackle the issue at the EA's development stage.

Lastly, I track my EA's weekly performance and this help me understand which EA are good for ranging and which are good for breakout and which are all weather EA.  Thus such trade analytics/statistics help me "balance" my portfolio with a good intermix of EA so that when 1 group is down, there is another group of EA to push up the performance.  But don't mix it too well till your portfolio is always at break even smile 


Understand your EA's strength and the domineering market conditions and thus select your EA based on these factors.

1. Know how much your EA earns, Ave Trade Win$ and it's potential win in each "right market" conditions

2. Understand the period your EA will start profitable, Eg is it a ranging EA or a Breakout EA

3. Know how much is your EA's Ave Trade Loss$ and max loses during "off market" conditions.

If it's a ranging EA, how much can it earn in between a breakout.  And for a Breakout EA, how much it will lose during the ranging.

Once you know an estimate figure, you can then add in the number and types of EA for your portfolio. 

For Example, if my Breakout EA earns $3000 per breakout (due to Winner options) but lose $1000 during ranging.  Then for every Breakout EA, I add a number of Ranging EA that will cover up or exceed my breakout loses during ranging.


Another simple method...look at ave weekly gain/loss

By looking at the weekly performance, I can identify which is the ranging weeks and thus see which EA did well/badly.  It makes my portfolio selection so much easier and with the excel file, it is much easier to visualize and select. 

As I continually improve my EA, I begin to notice there are more and more EA that begin to perform well in every consecutive weeks irregardless ranging or breakout.  This is my ultimate aim and my ideal EA.

  • Registered: 2017-03-14
  • Posts: 709

Re: Analytic Trade Management (ATM)

Hello Hannah,

It's a long post and it will take me awhile to digest it all.

I don't have any objections or disagreement with what you suggest.  It is all do-able.  The only thing I would add is that when developing a product it first needs to walk before it can run.  It is something that continuously evolves.  Since Forex is rather empirical in nature and since we are doing something new,  then there is bound to be many starts and stops along the way.  The important thing is to keep things simple at first so it becomes possible to implement.  If it is overly complex or engineered then it just won't happen.  Also, if it is overly engineered and along the way we learn something new, then you have to remove features.

So -- yes, I'm game.  But slowly, slowly we'll build it up...

  • Registered: 2017-03-14
  • Posts: 709

Re: Analytic Trade Management (ATM)

hannahis wrote:

I have yet to try your MT4 Tracker and I've the impression it is for EA Studio (correct me if I'm wrong).

MT4 Tracker is not just for EA Studio.  Since it uses the *.htm file created by Account History then it monitors all strategies (Magic Numbers) in an account.  However, it does have an important feature that is tailored for EA Studio -- and this is physically pruning an EA Studio-generated Portfolio Expert *.mq4 file.  It does this by parsing the original *.mq4 file and substituting a "dummy" strategy that will never trade for those Magic Numbers that have been tagged for exclusion.

One advantage to this approach is everything is self-contained -- that is, after compiling the resulting *.ex4 file can be moved around and the excluded Magic Numbers remain excluded.  However, one disadvantage is that if a pruned Magic Number has any pending transactions then they will never be closed -- that is, they will become zombies.

An alternative is to instead create an external file that includes a list of Magic Numbers to be excluded.  This is how the Adaptive Pruning works.  Using this approach, excluded Magic Numbers will not open a new position, but if there were any pending transactions then they will close gracefully -- as if the strategy was still active.

8 Reply by hannahis (edited by hannahis 2017-08-25 07:56:03)

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Analytic Trade Management (ATM)

Hi Steve,

Yes, I totally agree with you....all great things begin with little steps.

I look forward for your completed or semi-completed version of your "Adaptive" programme (hopefully it is applicable to all types of MT4, FSB Pro and EA Studio)

As for the MT4 Tracker, I agree the most important feature is the pruning part.  And now I remembered why I didn't try it out, cos I have my excel file as an alternative for portfolio selection process and if I can't make use of the pruning feature (meant only for EA studio), then it's rather limited.

Nevertheless, it's an excellent piece of work and your contribution to FSB's community is awesome.  Wish to have more people like you smile


NOTE:

With regard to your "zombies" position, that are "orphan" trades because it is being replaced by "dummy ea".

Can you "code" your "dummy EA/replacement EA" with some basic features such as SL, TP and trailing?

So that those "orphan" open trades have SL as a ultimate SL feature and if it is in profit, it has the Trailing or TP so that either direction, you already have a preset features to protect these open positions.

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,982

Re: Analytic Trade Management (ATM)

Can you "code" your "dummy EA/replacement EA" with some basic features such as SL, TP and trailing?

Hannah beat me this time smile because I was too lazy to answer on a tablet at 4:00.

It is easy to make strategy replacements that:
- close a position, if any open with that magic number, at the next bar open
- sets "no entry" rules, but closes the position at the current strategy closing rules
- sets SL and/or TP or BE (if at profit).

There are many variants.

Steve, please contact me if you like to implement some of those and have questions.

Popov's Website

  • Registered: 2017-03-14
  • Posts: 709

Re: Analytic Trade Management (ATM)

hannahis wrote:

Can you "code" your "dummy EA/replacement EA" with some basic features such as SL, TP and trailing?

Popov wrote:

There are many variants.

Two ways to "prune":
1. Modify *.mq4 and replace original strategy with a "dummy", non-trading strategy.
    -This is what MT4 Tracker currently can do.
2. Create an external file that contains a list of Magic Numbers to exclude.
    -This is what the new "Adaptive Pruning" indicator does.

As I mentioned in the earlier post, (1) has the disadvantage that "zombies" may remain.

There is a 3rd approach:
3. Modify *.mq4 so it includes a list of Magic Numbers to exclude.  And then, within the OpenPosition() function, check whether the current Magic Number is a member of that list.  This method will NOT result in "zombies" -- that is, since the pruned strategy is still sensitive to close signals, then it will die gracefully using whatever rules belong to the original strategy.

Popov -- thanks for your offer.  I'm still experimenting and testing.  As things become more firm then I will check with you again for some input and advice. 

Adding multiple features to support different ways of killing-off a strategy has minimal benefit.  The most common scenario is this -- a strategy has been running for awhile and you've now determined it is no longer of value, so you want to prune it.  As to when you prune it is totally arbitrary -- today, last week, next week, two months from now, who knows?  So, once you decide you no longer want to use it then it seems perfectly fine to allow the last instance of it to simply die gracefully, using its original rules.  It's only one last trade -- right?   In the end, I will probably modify MT4 Tracker to use method (3).

11 Reply by hannahis (edited by hannahis 2017-08-25 12:14:40)

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Analytic Trade Management (ATM)

Hi Steve,

If you are using Winner option, then the "Orphan/zombie" open trades may not be just 1 last trade (depending how many addition you use for your Winner option).  Furthermore, if this "last" trade is not handle properly, it can also blow your account or suffer unnecessary lose.  Don't ever take a open position trade likely.  Imagine for each EA, there is 1 last trade, the combination can result to many last trade if I am using 100 EA.

Secondly, if I have hundreds of open positions, finding and monitoring those "orphan" trades may not be that easy.

Thirdly, adding a trailing as an additional exit plan has proven to be a very good option for me.

Thanks for taking your time to experiment and no hurry.  Let us know if you need us to bounce off some ideas with you.

Keep up the good work.

  • Registered: 2017-03-14
  • Posts: 709

Re: Analytic Trade Management (ATM)

I understand that last trade may actually include more than one position.

The point I was trying to make was that the point at which one prunes a strategy is very arbitrary.  All we know is there is a strategy that has been running for awhile -- days, weeks, months, whatever -- and that it has completed N number of trades.  And then one day, based on some arbitrary criteria, we decide we don't want to use it anymore.  That is, there isn't some "sweet spot" that represents the perfect time to remove it.   The fact that strategy is in your portfolio in the first place means at some point you had some confidence in it.  Those last remaining positions may also close in your favor -- right?   So, if the strategy was good enough to add to a portfolio in the first place, then letting the last instance of it die according to the rules you chose for it really isn't so bad.  Yes -- one could add a bunch more logic to perhaps make the death a little less painful, but the goal is to prune and look forward to the future, not to take a simple idea and make it more complex than it needs to be.

Obviously I'm coming from a developer's point of view.  When developing software one is constantly making design decisions and weighing the time and effort involved versus the payoff.  I want to create a tool that adapts to a portfolio's performance and automatically excludes strategies that no longer perform according to some policy specified by the user.  At this point I'm not even considering the finer points of how a strategy can be excluded.  As long as the portfolio doesn't open a new trade using that strategy, then that is good enough.  Not opening a new position is my definition of pruning.

  • Registered: 2014-04-01
  • Posts: 25

Re: Analytic Trade Management (ATM)

I agree, simple is the way to go to close orphans.  Based on old criteria is ideal but even a forced close at the next bar.  As you say, the trade could well be in profit at close (although less likely).