Topic: Broker Data Collection

I would imagine that the best data you can use is that directly from your broker.

Unfortunately, as far as I have seen, the brokers are in no hurry to dispense his info, and the furthest back that seems to be obtainable is about ten days for the M1 chart, twenty days for the M5 chart, forty days for the M15 chart, and so on.
Not very useful.

Does anyone here collect the broker data, say once a week for instance, and condense it into the files read by FSB?

Or is there a better, faster way to get hold of decent quantities of data from your broker?

Re: Broker Data Collection

Certainly it depends on the broker. But yeah, the best method so far is to make sure one keeps MT working for at least once per week to have price data up to date.

Re: Broker Data Collection

So a good way to do this would be to CTRL S the new data, save file as EURUSD1a, go to FSB data folder, open old EURUSD1 and EURUSD1a files in notepad, chop out duplicate data in EURUSD1a, copy what is left into EURUSD1, delete EURUSD1a?

Re: Broker Data Collection

No, just save the file from MT to FSB data folder. Max bar setting and number of displayed bars have to be set up correctly in MT, then it's only a matter of periodically updating the data and saving it.

Re: Broker Data Collection

Ok. Is it the case that the data runs from the date MT is installed on your machine, and stays there forever? Or does it vanish as time goes on? For instance when you shut MT down and relaunch?

If it does stay forever, then it's a very simple process to just overwrite existing files in FSB, if I understand correctly?

Thanks for your help on this.

Re: Broker Data Collection

Also, my machine currently saves data using Adobe Acrobat, whereas I see in the tutorials that notepad++ is being used, does this matter?

Re: Broker Data Collection

You can't never have full confidence that the data stays there forever, many things can happen.

But usually just overwriting the data is good enough, if all bars are displayed on chart.

Re: Broker Data Collection

Simon wrote:

Also, my machine currently saves data using Adobe Acrobat, whereas I see in the tutorials that notepad++ is being used, does this matter?

It saves in a defined format, I think, that's what's important.

Re: Broker Data Collection

Thanks Footon.

10 (edited by Simon 2016-10-30 14:11:03)

Re: Broker Data Collection

Still thinking about this..

I've looked at some data that I saved from MT to FSB at the beginning of the month, and it goes back further than what I am able to load into the MT chart today, so the data definitely isn't around for long.

It looks like I would need to regularly manually download data on the dot to avoid gaps.This presents a perilous problem,  seeing as what we really want is seamless data, so that our backtesting is as accurate as possible, but if I were unable to attend to my regular downloading session, for whatever reason, then we have flawed data, for good.

I've been digging around on google for something to automate this process, (a) to remove some of the burden of performing the task of downloading the data for each timeframe for each pair, every week, open it into notepad++ to find the end of the last import, the beginning of the new one, cut out the duplications, and stitch them together, and (b) to hopefully give the data more accuracy by removing the human error element...

Hopefully it's ok to post a link to a third party site, it is free I think, but I just wondered if the script on here: https://www.currencysecrets.com/blog/20 … ownloader/
had any merit at all, and if some of the more experienced FSB users had any thoughts on it?

Re: Broker Data Collection

I do not think you should have a problem with gaps..... if you are updating weekly..

I have my charts set to 65,000 bars. which is more than enough for one week.....

And if you miss a week, you can increase the number of bars in your charts to accommodate the 'missing' data.

You can make the charts hold a ton of data, I see no reason for you have to use notepad.

I think that Popov has written a script for downloading from MT4 and I think you can set that to 100,000 bars... That should look after any situation.

Have you tried the script?

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Broker Data Collection

Not tried the script, is it the 'MT4 HST Import' tab in the History Centre? Anything to make it more failsafe and less labour intensive!

What worries me about simply saving the data from the chart on a weekly basis is that by doing so, I will overwrite and erase the last batch I saved, so I only ever have a tiny snippet of data..

My broker typically only offers:
26,344 bars for M1
256 bars for M5
2048 for M15 and M30
2685 for H1
8542 for H4
2084 for D1
1024 for W1
256 for MN

If I'm going to get the minimum of 20,000 bars for backtesting a timeframe, the only way I can see to get reliable gapless data is, on a weekly basis, saving it to a temporary file, finding the last line of the last batch, deleting it and everything before it, and copying what's left into the symbol/timeframe file in the relevant data directory...

I'm not shy about doing the work, I'll do what needs to be done, but I'm also very busy running a business!

I searched the web until my eyes went bloodshot yesterday for a method to do this, and along the way discovered what a common problem this is, and how many forum posts there are about it, so I apologise for dragging you all through it again.

Seems like it was fine until around build 600 of MT, the most recent posts of people happily automatically exporting data seem to be about four years old now!

Re: Broker Data Collection

check this out for download script
http://forexsb.com/wiki/bot_guide/data-import?s[]=script

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Broker Data Collection

This looks good, I can't find the script though, its says there are two on the Data Import page, is that the Data Import page in the FSB History Center? I can't see any scripts there?

Re: Broker Data Collection

Hello Simon, you have to use the scripts attached in the topics in the Premium Club.
There are two scripts: One for exporting Data Source settings and one for exporting data files.

See the sticky topics in the Premium club.

Re: Broker Data Collection

Fantastic, Thank you Popov.

As you have probably read above, my broker only allows me to export tiny quantities, so I must slowly gather it, adding to it each week, until I have enough to backtest with.

Perhaps a good way to ensure that my data is seamless, and minimising the task of gathering it, is to follow the steps outlined here: https://developers.google.com/apps-scri … duplicates

So, using Google Sheets (or spreadsheet of your choice), having a separate sheet for each timeframe for each pair:

-save the new MT chart data to a MT file (Files>Open Data Folder > MQL4>Files)
-copy new data into the correct google sheet for symbol/timeframe (all timeframe folders can be selected and opened in MT 'Files' with notepad++ at once, making it quicker to copy the .csv files to google sheets)
-run 'removeDuplicate' script (must be created first)
-copy the remaining data (which should be ALL data since collecting began) to the correct Data Source for profile in FSB

Is this adequate, or is there a better way?

Thanks,
Simon

Re: Broker Data Collection

Could FSB Pro read the Data Directory if it was Excel spreadsheets?

18 (edited by Simon 2016-11-02 22:21:50)

Re: Broker Data Collection

Ok, I've figured out how to set Notepad++ up to be a more efficient method..

I tried it a few days ago, but Notepad++ wouldn't show up in the Settings of my machine as an option to be made into the default editor for .csv files, but the info/tutorials in the following links has solved it, I think;

How to set it up as default editor for .csv files:
https://www.raymond.cc/blog/replace-mic … n-windows/

then (in Windows 10), Go to:
Settings > System > Default Apps > Choose default apps by file type > scroll down to .csv > select Notepad++

Now symbol/TF files in FSB data directories are Notepad++ files.

How to remove duplicate lines in notepad++:
http://stackoverflow.com/questions/3958 … n-notepad

Now I can export data from MT to MT "Files", and copy it straight from there to FSB data directory, and remove duplicates quite easily with a few clicks.

Hopefully this will be of use to others with brokers who are tight with the data! (or me in the future if I forget)