Re: MT4 Charts Do Not Trade Equally

PT -- this is excellent!  I recall also seeing a lot of 'Context Busy' errors...

Could you try another test to see if OnTimer fixes the problem on your VPS.  It requires 3 minor changes:

1. In the OnInit() function, near the bottom add this line:

    EventSetTimer(10);

2. Rename OnTick() to OnTimer().
    NOTE: It is important to actually rename OnTick() so that the 'OnTick' symbol no longer exists in the *.ex4 and MT4 knows not to keep calling it.

3. Near the bottom of the OnDeinit() function add this line:

    EventKillTimer();

Of course you'll need to reclone your EA so that all 32 are using OnTimer (and OnTick is no longer present).

I'm curious to see if all the 'ContextBusy' errors go away.  Substituting OnTick with OnTimer that fires every 10 seconds still gives strategies lots of opportunities to execute, but MT4 is no longer under the burden to keep calling OnTick.  Instead, each EA keeps calling its own OnTimer every 10 seconds.  Also, from TaskManager, I think you'll see overall CPU usage go way down for MetaTrader.

Re: MT4 Charts Do Not Trade Equally

Steve - thanks for the instructions. I've run the test , but see the results as worse.  What I think is happening is the ontimer function is missing the trigger event (in this case - on bar open)  , so some systems don't seem to send the order. I have more missed trades than errors.   Logs below


https://s9.postimg.org/xjd8ejra3/tradelog4.jpg

and the trade log:

https://s9.postimg.org/inep7231n/All_Mixed_currency_32_test_2.jpg

Steve - i'll email you the modified EA (I cant seem to upload on here) so you can take a look. I'll also change the timer to 1 second as maybe that eliminates the missed trade issue.  Will report back shortly

PT

Re: MT4 Charts Do Not Trade Equally

Hmmm...  That's interesting.  In my hands it seemed to improve things.

The OnTick handler is called by MT4 when it thinks there is new data the EA might be interested in.  The OnTimer handler is called from within the EA (I believe) because it has the capability to know when 10 seconds have elapsed and doesn't need MT4 to tell it to wake up.

Ticks can come at any time -- e.g. they might even come at 10 second intervals.  That's why I'm sort of surprised things are worse.

With regards to missing a bar open -- I'm still sort of fuzzy on this.  I thought a bar open occurs with the first tick when Time[0] is larger than the previous Time[0].  So, even if OnTimer is called at 10 second intervals I would think it could still figure out whether there is a new bar. 

If you could send me your EA then I will also test it to see if I get similar results.

Re: MT4 Charts Do Not Trade Equally

I have just run the tests with 1 second interval  - same issue.


https://s9.postimg.org/os67k52sb/All_Mixed_currency_32_test_3.jpg

https://s9.postimg.org/ut3wh7x4b/tradelog5.jpg

It seems a lot better - please note magicnumber 306 didnt fire as I had the chart set to 1 hour by accident.

Steve - do you think you could help add the code regarding -  Tradeallowed()  (info: https://www.mql5.com/en/articles/1412)  into the 1 min EA I just sent to you?   It may be a way to get this fixed?

If you read half way down the post - the author has packaged the code as a function, it may be easy to cut and paste into the EA. The code looks to check if its able to trade, if not it waits until free then sends the order.

If you dont get a chance to look at it - i'll try and knock it together tonight (in around 12 hours) and test

55 (edited by sleytus 2018-01-10 02:22:15)

Re: MT4 Charts Do Not Trade Equally

PT -- I received your attachment -- thanks.  I will test your EA and let it run overnight.

With regards to checking if trades are allowed -- Popov's code already includes that in both EA Studio- and FSB-generated EAs.  And I think Popov's implementation is more robust.

Also, I don't think testing whether trades are allowed will solve the problem -- here's why.  Each chart runs in its own thread.  Furthermore, each time MT4 calls OnTick it launches a new, asynchronous thread (I believe).  As a result, if I have 40 EURUSD strategies attached to charts then their OnTick handlers are called at the same time and they all execute their code in unison -- as a result this creates a "bottle-neck" jam.  It's the equivalent of dropping-off 40 hungry teenagers at McDonalds at the same time.  The broker can handle only one order at a time and the code in all 40 EAs may become blocked.  Or something along those lines.

OnTimer seemed to work for me because I think it breaks the bottleneck -- the 40 EAs are no longer running in unison.

Another solution would be to create a separate EA whose only job in life is to process orders.  The way this would work is whenever an EA wants to send an order it doesn't actually send it but, rather, adds it to a queue.  The separate EA checks the queue and processes orders one at a time.  FxBlue has a nice library called QuickChannel (https://www.fxblue.com/appstore/9/quickchannel) that makes it relatively easy to create EAs, indicators and applications that perform inter-process communication using named pipes.  I've used it before.  Though more complicated than OnTimer, it might offer another path in the event OnTimer doesn't work out.

Re: MT4 Charts Do Not Trade Equally

Hello Sleytus,

You are right about Popov's code (funny what you find what you actually look lol)!  I can also see the retry set to 4 and wait time 100 (which I believe is 0.1 sec)  -  I wonder if the answer lies in the 0.1 sec retry,  this means it shoots all retries in 0.4 sec.

Increasing this maybe a way to overcome the bottleneck.. to put it in your context.

- 40 teenagers get dropped to McDonalds - 10 get served, 30 get rejected.
- 30 remaining teenagers come back 0.1 sec later - another 5 get served - 25 get rejected.
- 25 remaining teenagers come back 0.1 sec later ....... and so on smile

i'll try playing with a few of these options tonight and report back smile

Re: MT4 Charts Do Not Trade Equally

So a bit of an update it - i've changed the time between retries and number of retries is an order fails - here:


https://s9.postimg.org/8hkccs6cb/retry.jpg

and hopefully done it in a fashion that should be fool proof. But still find that the systems are missing a fair number of trades - as per this:


https://s9.postimg.org/uuwocy6u3/slow_retry.jpg

I was thinking that it could be a slippage issue - i.e the price moves from when the order is sent (like a limit order) - but when I check the order we are using market orders (so that shouldn't be the issue).

I'm out of ideas at the moment - but we could travel back to this issue being linked to the number of EA's operating at once.  I'll run a test tomorrow only using 10 EA's and see what happens.

If anyone has any other ideas to check, please post them here.

Re: MT4 Charts Do Not Trade Equally

You can write a specific test to detect the correct work of the OnTick event. In order to do it, make a dummy expert that only logs the OnTick event and does not make any calculations or order executions. I'm pretty sure you will find that all OnTicks are executed correctly. I think so because I imagine how I would program MT. I imagine that all OnTicks subscribe to a single event invocator and MT doesn't have load balancer. It means that all OnTicks must be executed straight one after the other. I also would add a cancellation in case a new tick arrives during the execution of the previous one. It means that if the total time of execution of all OnTick event handlers is lower than the time between the tick, all mus be executed. On dynamic markets and frequent ticks, MT will execute OnTicks until a new tick terminate the job. It means that the cancellation will have a greater effect on the last event handlers. I can bet that this is the case.

Trade Safe!

Re: MT4 Charts Do Not Trade Equally

Thanks Popov,

it makes sense to log the events to see - at this stage I haven't done any work in MQL language - but I see it wouldn't be hard what you describe.

That would validate the on-timer principal - you just need to make sure the time set is adequate.

Steve is running a longer on-timer test at the moment, and i'll be working on another one early next week (i'm a little tired at the moment - and the market closes over the weekend).

I wonder if someone from the community would like to help out with coding the above idea so we could check? (if not no big issues smile )

PT

60 (edited by sleytus 2018-01-12 13:25:22)

Re: MT4 Charts Do Not Trade Equally

Popov -- I would like to add to your explanation...

The OnTick event handler is called from MT4 -- so, yes, MT4 may choose to cancel it if a new tick arrives -- and this can be a major problem.  On the other hand, OnTimer is called from within the EA since it is capable of determining elapsed time on its own -- that is, OnTimer does not need MT4 to wake-up the EA to begin executing code.  As a result, OnTimer is not subject to being cancelled and should run until it has completed all of its calculations and orders.  This is why I think OnTimer holds some promise to solving the problem.  Not only does it avoid a logjam (since charts are no longer executing their code in unison), but it also makes EAs immune from MT4 cancelling the OnTick thread before all strategies have had a chance to run.

Re: MT4 Charts Do Not Trade Equally

dear Sleytus,
your last update version of "Portfolio Maker" "1.0.9973.0", does it include the "onTimer" issue?
any new updates about the trial of the "Ontimer"?
regards

Re: MT4 Charts Do Not Trade Equally

Hello alamehmazen123,

I don't think the last update of Portfolio Maker includes the use of OnTimer.  I will send you a private build that does include it.  Please give me a day or so and I will send it to you.  This would be for testing purposes only.

I continue to test the use of OnTimer.  In my hands and under my specific scenarios, it seems to always work better than OnTick (for reasons I've written in previous posts).  This applies to tests using MT4's Strategy Tester and Demo Accounts.  I have not yet applied it to a Live Account.

DoCZero seemed to be getting different results -- so, I'm curious to learn more about his testing and whether he has anything more to add.

Regards...

Re: MT4 Charts Do Not Trade Equally

Hi Steve,

Im currently travelling in China and wont resume testing for 1 week. If its ok , share with me the private build and i'll use that for my next batch of testing (that way its semi controlled - and my own coding with ontimer wont get in the way wink  )

PT

Re: MT4 Charts Do Not Trade Equally

For those who are interested, there is an updated version of Portfolio Maker.  This new version *does* use OnTimer in place of OnTick.  Here is a link:
http://nutsaboutnets-downloads.s3.amazonaws.com/PortfolioMaker_Setup.exe

For a brief explanation why I now prefer OnTimer, please refer to the FAQ "OnTick versus OnTimer Events -- which works better for portfolio EAs?" on the following link:
http://mt4tracker.com/faq/

Re: MT4 Charts Do Not Trade Equally

A portfolio EA does use a single time frame.  Also, one of the more unique features of FSBPro and EA Studio is their ability to generate collections of strategies that are then converted into a single, portfolio EA *.ex4.

Others prefer to create a single, perfect strategy and stick with it.  However, creating a single perfect strategy is easier said than done.  Using a portfolio EA that contains many strategies relieves you of having to create a single, perfect strategy.

Re: MT4 Charts Do Not Trade Equally

Ok All - coming back to this topic. I have built a very simple system to test - Logic here:
https://s7.postimg.cc/n13ck30tz/Simpletest.jpg

And I have run this on live demo test (Global prime is my broker - I'm running the EA on a i7 4980 - 32gb Dedicated server based in NY - 1.5ms ping to my broker) for the last few months. I have just pulled up the data / reloaded new data into FSB and done a comparison... This is my findings:

https://s7.postimg.cc/8jaqphr3b/MT4_VSFSB.jpg


As you can see - most of the time it executes correctly - minus some slippage and time setting differences (??).  Where I get a bit confused is where I have marked exit issues. I'm not really sure what is going on here.  When i pull up the chart in FSB:



https://s7.postimg.cc/qbwaxgvhz/1604_FSB.jpg

And then compare with MT4:

https://s7.postimg.cc/aqezdmm5j/1604_MT4.jpg

To my eyes it looks similar.  So at this stage - I'm not really sure why there is a different in execution.  Either MT4 is missing trades (could be) or FSB is somehow taking trades when it shouldnt (unlikely).  I'm not at a definitive root cause for different at this stage (probably sleep on it - and maybe think of a better test to try uncover the difference)

if you want the EA file / FSB file or raw data to look at - please let me know (PM).. Happy to share.

67 (edited by GD 2018-05-08 17:36:47)

Re: MT4 Charts Do Not Trade Equally

Sleytus

why we cannot secure if some trades are not completed in first tick to be completed in the next one?

IsTradeContextBusy can be used somehow?

I think this is what Popov means.

Re: MT4 Charts Do Not Trade Equally

DoCZero wrote:

I have built a very simple system to test

I'm not clear about your test system.  Are you testing a single strategy or a portfolio EA created by Portfolio Maker.

In the comparison -- I understand the left side is from the Demo Account and the right side from FSB.  Are you comparing a single strategy in the Demo and FSB or a portfolio?

To me it looks like a single strategy -- in which case the results show that FSB backtesting matches closely with trading in a Demo account, which is good to know.  But I'm pretty sure Popov has already verified that.

This issue has to do with testing a portfolio EA of 40 strategies attached to a single chart, versus the same 40 strategies attached to their own, separate charts.

Re: MT4 Charts Do Not Trade Equally

GD wrote:

why we cannot secure if some trades are not completed in first tick to be completed in the next one?

GD -- I think I understand your question, but I'm not sure.

I believe that MT4 has a threading / performance problem.  With each tick an EA is given a certain amount of time to execute, and if it exceeds that time or if the next tick arrives before it has completed executing, then that first thread is killed. 

When the next tick arrives the EA starts at the beginning again -- it does NOT begin where it left-off.  As a result, the code at the end of the EA never gets executed.  This probably explains the observation that an FSB-generated portfolio EA seems to miss trades.  Using OnTimer (instead of OnTick) appears to solve the problem.

70 (edited by GD 2018-05-08 17:50:19)

Re: MT4 Charts Do Not Trade Equally

Sleytus

I attach this from MQL4 Refernce manual

https://docs.mql4.com/runtime/event_fire#newtick

NewTick

The NewTick event is generated if there are new quotes, it is processed by OnTick() of Expert Advisors attached. In case when OnTick function for the previous quote is being processed when a new quote is received, the new quote will be ignored by an Expert Advisor, because the corresponding event will not enqueued.

All new quotes that are received while the program is running are ignored until the OnTick() is completed. After that the function will run only after a new quote is received. The NewTick event is generated irrespective of whether automated trade is allowed or not ("Allow/prohibit Auto trading" button). The prohibition of automated trading denotes only that sending of trade requests from an Expert Advisor is not allowed, while the Expert Advisor keeps working.

The prohibition of automated trading by pressing the appropriate button will not stop the current execution of the OnTick()
function.

My English is not GOOD.... I am confused now!!!!

Re: MT4 Charts Do Not Trade Equally

Hello GD,

Yes -- I've seen that.  And your English is fine...

I've seen other posts on mql5.com where others have seen a similar problem.   Also, that statement may be true as long as the EA doesn't take up too much time executing all of its code.  Once an EA or Indicator have exceeded a time limit then MT4 kills the thread that was executing.

The arrival of new ticks is non-deterministic and they can arrive very quickly, one after the other.  If MT4 killed a thread when each new tick arrived then if you were trading EURUSD during a busy period your EA would never complete (since ticks would arrive many times per second).  So, I think that statement is true as long as the EA completes within a fixed period of time.  However, once that time has been exceeded then MT4 will kill the thead.   I've seen this happen with indicators.  The strategies created by FSBPro are very HEAVY -- i.e. complex and take some time to execute.  When you place 50 FSBPro-generated strategies in a single, portfolio EA then it takes a long time to execute all 50.  From my testing, the first 7 or 8 strategies will always get executed, but then it drops off very dramatically from there.  So, strategies 20-50 very rarely get executed.

I don't have access to MT4 code, so I can't prove this -- this is simply based on observations and testing.

72 (edited by GD 2018-05-08 18:29:54)

Re: MT4 Charts Do Not Trade Equally

Hi Sleytus

case 1

In this case even onTimer solution cannot possibly full execute all the portfolio EAs as threading can stop. Am I right?
The situation should be the same.

case 2

With onTimer, I think the problem is when many orders asked to be done with portfolio  cannot be completed cause of bottleneck
Maybe if you can add some delay between  executions of orders. I am not sure.

Sidekick can be used even with one by one. But it is very hard to do it... for a big number of EAs.

Re: MT4 Charts Do Not Trade Equally

Hello GD,

OnTimer can fully execute.  This is because it is called by Windows and is not under MT4's thread control.

OnTimer does not run into the same problems as OnTick.  I believe the reason is because OnTick forces threads to be synchronized and they execute code at the same time.  When using OnTimer, the charts are not necessarily executing their code at the same time -- they are a few seconds apart.  It depends on how frequently OnTimer is called.  In Portfolio Maker the default is once every 10 seconds.

Again -- this is based on mine and other's observations.  It's fine to question it.  Portfolio Maker has been coded to use OnTimer because it misses too many trades when using OnTick.  If you prefer to use OnTick, that is fine.  I was just sharing my test results and observations.  It's easy enough to do your own testing to verify whether or not OnTick can handle a portfolio EA with many strategies.

74 (edited by GD 2018-05-08 18:49:56)

Re: MT4 Charts Do Not Trade Equally

Sleytus

I ask the questions to make everything clear.
I use onTimer.
Normally all orders should execute. But why few of them not as you also mentioned?
I try to find reasons

Suppose that some specific orders corresponded to specific EAs in portfolio cannot be executed cause the moment of execution the broker's server is busy, what will happen?

Can we increase thread time in windows to be dead?

Re: MT4 Charts Do Not Trade Equally

Hello GD,

I understand.

Trades will occur if the code that opens new positions gets to execute.  If the code that opens new positions doesn't have the opportunity to execute, then there will be less trading.

A program is just like a recipe.  For example, suppose the recipe to make a cake has 20 steps.  But each time when you get to step #7 you are interrupted.  And when you return to the kitchen you have to start over again at step #1.  In the end, the cake will never be baked.