Topic: Supercharging the Data Export Script: From 5 Minutes to 7 Seconds!
Hey guys,
As many of you know I am working with some pretty hefty historical datasets. When using Popov's excellent MT4 Data Export script to export data to Expert Advisor Studio, it was always taking very long. I'm talking about a good 5 minutes just to get the M15 (900000 bars), M30 (450000 bars), and H1 (225000 bars) timeframes exported on my 37 years of historical data.
Now, I don't know about you, but I'm all about efficiency, so I decided to take a look under the hood and see if I could give the script a bit of a boost.
The main bottleneck was the way the script was building the JSON strings using the StringAdd function. Every time StringAdd is called, MT4 needs to resize the string's memory, which takes a lot of time, especially when dealing with massive amounts of data like in my case.
So, what did I do? I implemented a much faster approach: pre-allocating memory for the strings using uchar arrays and copying the characters one by one. This way, we avoid all those unnecessary memory reallocations, and the script can just focus on getting the job done quickly.
I also added some error handling to make the script even more robust and user-friendly. Now it checks if the data is retrieved correctly, and if the files are opened and written properly.
The bottom line: the modified script takes just 7 seconds to export the same dataset that took 5 minutes before. That's a whopping 40x speed improvement! I'm making this modification freely available so that everyone can benefit from this turbocharged version and so that Mr. Popov can possibly implement it to future EA Studio releases directly.
I hope this optimization helps everyone streamline their data export workflow.
Cheers,
Lorenz