forex software

Create and Test Forex Strategies

forex software

Skip to forum content

Forex Software

Create and Test Forex Strategies

You are not logged in. Please login or register.


Forex Software → Expert Advisor Studio → Supercharging the Data Export Script: From 5 Minutes to 7 Seconds!

Pages 1

You must login or register to post a reply

RSS topic feed

Posts: 5

Topic: Supercharging the Data Export Script: From 5 Minutes to 7 Seconds!

Hey guys,

As many of you know I am working with some pretty hefty historical datasets. When using Popov's excellent MT4 Data Export script to export data to Expert Advisor Studio, it was always taking very long. I'm talking about a good 5 minutes just to get the M15 (900000 bars), M30 (450000 bars), and H1 (225000 bars) timeframes exported on my 37 years of historical data.

Now, I don't know about you, but I'm all about efficiency, so I decided to take a look under the hood and see if I could give the script a bit of a boost.

The main bottleneck was the way the script was building the JSON strings using the StringAdd function. Every time StringAdd is called, MT4 needs to resize the string's memory, which takes a lot of time, especially when dealing with massive amounts of data like in my case.

So, what did I do? I implemented a much faster approach: pre-allocating memory for the strings using uchar arrays and copying the characters one by one. This way, we avoid all those unnecessary memory reallocations, and the script can just focus on getting the job done quickly.

I also added some error handling to make the script even more robust and user-friendly. Now it checks if the data is retrieved correctly, and if the files are opened and written properly.

The bottom line: the modified script takes just 7 seconds to export the same dataset that took 5 minutes before. That's a whopping 40x speed improvement! I'm making this modification freely available so that everyone can benefit from this turbocharged version and so that Mr. Popov can possibly implement it to future EA Studio releases directly.

I hope this optimization helps everyone streamline their data export workflow.

Cheers,

Lorenz

Post's attachments

Data Export (performance mod by geektrader).mq4 15.43 kb, 7 downloads since 2024-09-02 

You don't have the permssions to download the attachments of this post.

Re: Supercharging the Data Export Script: From 5 Minutes to 7 Seconds!

This is nice! In coding I find it most satisfying making something perform quicker, so much quicker that you could almost physically feel it, this goes into that category smile I making it a sticky topic for the time being.

Re: Supercharging the Data Export Script: From 5 Minutes to 7 Seconds!

I do something different but in MT5 I leave only the pairs I want in the market watch and it exports all automatically as I work with the spreed of 30 I leave in the watch of mr always the spreed of 30

Re: Supercharging the Data Export Script: From 5 Minutes to 7 Seconds!

aqui esta o arquivo usado

Re: Supercharging the Data Export Script: From 5 Minutes to 7 Seconds!

footon wrote:

This is nice! In coding I find it most satisfying making something perform quicker, so much quicker that you could almost physically feel it, this goes into that category smile I making it a sticky topic for the time being.

Thanks for the sticky, footon! I totally agree, the feeling of making something this much faster is incredibly satisfying time and time again. Especially when you're working with datasets as large as mine – 28 pairs over 37 years. What used to take me over 2 hours each week, now finishes in 4 minutes! A real game-changer for me in my workflow. Hope it helps others too.

Posts: 5

Pages 1

You must login or register to post a reply

Forex Software → Expert Advisor Studio → Supercharging the Data Export Script: From 5 Minutes to 7 Seconds!

Similar topics in this forum