Topic: Anyone has 1min and/or 5min historical data to share?

Does anyone has accumulated 1min and/or historical MT4 data to share with the community here for the benefit of those using 1min and 5min time chart...that would spare us the time to collect and enable the rest of us to test out our 1min/5min strategies with a longer historical data for better EA consistency.

Thanks in advance for the kind soul out there smile

You can either post it here to share for the greater good of the community or PM me...thanks

Hannah

Re: Anyone has 1min and/or 5min historical data to share?

As an alternative you can download the data from Dukas

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

3 (edited by vidagig_13 2015-07-17 15:17:57)

Re: Anyone has 1min and/or 5min historical data to share?

Hi Hannahis

From my limited experience and understanding (for what it's worth) - and with apologies for the length of this rambling post.
Someone else may have something useful to add to this (or to correct).

Firstly - especially small timeframe Excel files e.g. 1 minute over multiple years - are generally too big to post here.
(e.g. 10 years 1 min = 300mb; 5 min = 60 mb)

I've spent some time trying to learn about this question of historical data.
E.g. The question of 'tick' data is unresolved for me - e.g. is it (measurably) better, how to get it and import it, etc.
From what I understand, the FSB and e.g. MT4 backtesters 'interpolate intrabar data' - FSB apparently does it better than MT4.

First, I've noticed that strategies give wildly different results depending on the historical data source, which concerns and confuses me; and remains an unresolved question for me, at this stage - in terms of confidence in the robustness of a strategy vs. it being (over-)optimized for a particular historical data source (and the credibility of the data source becomes questionable too).
So, I'm placing the most weight on the strategy results using my own broker's data - even though it's not very long - because that's where I'm running my live strategy tests.
- I notice on my broker - there's much more historical data available on a demo account than on a real account (and they appear to be identical - I checked) so I use both.
So, I have like 6 x data sources on FSB

Data sources:
1. Alpari: install an MT4 from Alpari - it has loads of historical data also on small timeframes - like 10 years worth.
2. GCITrading.com MT4 install also has a fair bit of data (and a variety of instruments other than FX
(E.g. Indices, commodities, bonds, individual shares, etc)
2. Your own broker - download as much as u can and start accumulating it in MT4 then update FSB data periodically
- try a demo and real account and see if there's more / different data on demo vs. real
3. I've tried a)Tickstory Lite and b)StrategyQuant Tick Data Downloader (free) with limited success - one or both gets Dukascopy data only (can't remember which)
4. The FSB default data source is also Dukascopy I think.
Observation: FSB default data source and Alpari data appear to be 'kinder' to a strategy than e.g. my own broker's data.

My notes below:

1. My most efficient Method to get historical data from MT4:
NB In MT4 History center (shortcut f2) - double (multiple clicks!) click on a symbol (on all timeframes)
eg EURUSD5 = to update it’s historical data for each timeframe (from your broker).
-  "Export" (each timeframe for each Currency you want) from MT4 History center may be easier than using FSB MT4 HST Import Tool. It's a repetitive process but you can do it pretty quickly e.g. using keyboard shortcuts.
NB: Do not use MT4 History Centre 'Download' button as this downloads historical data from Metaquotes (and not your broker) - people seem to question the quality of this data.

NB: May need to remove [.] from filenames when exporting to FSB Data Folder for Each Data Source
(sometimes MT4 exports filename with a dot)

NB: Alpari = big data set / file size e.g. 1 minute data: may be better to use FSB MT4 Import for these - because it seems to shorten the files - seems to prevent FSB crashing when working with big e.g Alpari 1 min data sets on a computer with little RAM.

2. How to increase amount of historical data in FSB: (manually over-ride the default 50 000 bars)
(as described elsewhere on this forum)
NB: The bigger the historical data set the more strain on your computer when using FSB - can cause instability.
FSB needs lots of Ram generally I'm learning the hard way to reduce my demands on FSB and also to pre-emptively restart when I see things getting sluggish and before a computer crash. Note to self - Buy more RAM.
How to:
Quit FSB first - then - In FSB Application Folder open in a basic text editor e.g. Notepad -  User Files / System / "Datasource" file for (each) Data Source and set all Max Bars at the bottom to e.g. 1 000 000
- then Restart FSB

3. More notes and FSB video tutorial:
NB: Note FSB Setting: Control Panel / General / for each profile -
- Check box “use all available shorter data periods” = improved accuracy(?)
- Now you can see intrabar stats in History Centre; and green bar below equity chart in an FSB Strategy

NB: FSB Tutorial: Describes saving data from MT4 charts: & setting chart max bars in FSB; and scroll ‘home’ to load all data on chart first in MT4.
Part 6 - Importing Historical Data From MetaTrader | Create & Trade Expert Advisors
https://www.youtube.com/watch?annotation_id=annotation_2376520001&feature=iv&index=6&list=PLWSqhQc0bfFAqy6h5qPWldbCHs10pT3qP&src_vid=aHETvA2RcaU&v=BH-NNE9UO38

Re: Anyone has 1min and/or 5min historical data to share?

Perhaps set up several history sources......... for different periods.....  that might cut down the need for huge amounts of data being processed at the same time

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

5 (edited by hannahis 2015-07-17 18:48:35)

Re: Anyone has 1min and/or 5min historical data to share?

Hi Vidagig

Thanks for your input. 

I tried to download from my broker and for 1 min data the oldest was till May 2015...so very limited and I thought if I'm going to wait for 1 year from now to accumulate these historical data, I would have to wait for a year later to fine tune my EA, boy that's a long wait.  I'm glad to know some broker's MT4 has 10 years.  Thanks for your info.

As for posting the historical data here, I thought we could zip the file...

Hannah

Re: Anyone has 1min and/or 5min historical data to share?

Could post the file on Google Drive or Drop Box for download

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Anyone has 1min and/or 5min historical data to share?

Blaiserboy, www.wetransfer.com lets you upload a file up to 2gb w/o registration, and share it over email or with a link. They keep the files only 1 week before they delete them but that's enough for most things. Actually I consider the 1-week-to-deletion rule the best part. If I upload to another service like Google Drive I have to keep in mind to remove the files to free up space but not here.. smile

8 (edited by vidagig_13 2015-07-18 15:38:14)

Re: Anyone has 1min and/or 5min historical data to share?

Various EURUSD historical price data to 7 July 2015 zip file (166mb) including Alpari 1 Minute from January 1999.
Also Alpari, GCI & PFD Metatrader 4 & 5 installers that allow multiple installs to locations you specify. (19mb)

http://www.sendspace.com/filegroup/HYHZ … P4EHKGK6PQ

"We do not delete active files. We will host the file for an unlimited time, until it becomes inactive, in the case of sendspace Lite uploaders, unless they have purchased file bandwidth for that file. A file becomes inactive if it has not been downloaded at least once during a 30 day period."

How to get or update Historical price data from MT4 History Center (f2) to FSB Data Folders is in my previous post.

For EURUSD
Alpari Demo MT4 currently shows 1 Minute chart from January 1999.
GCI Demo MT4 currently shows 1 Minute chart from March 2015.
PFD Demo MT4 currently shows 1 Minute chart from January 2010. (Demo account expires after 15 Days)
Alpari MT5 currently shows 1 Minute chart from 23 May 2014. Not included in the .zip file above.

MT5 doesn't have a History Center that I can find or figure out, so in MT5 scroll to the beginning of e.g. a 1 Minute Chart using the 'Home' button - to download all available data - and use File / Save (Alt F; Ctrl S). Works the same on an MT4 chart too, or use MT4 History Center method.

Metatrader 5 'help' on Price data
https://www.mql5.com/en/docs/series/timeseries_access

9 (edited by hannahis 2015-07-18 18:10:54)

Re: Anyone has 1min and/or 5min historical data to share?

Hi Vidagig,

Thanks a zillion for your kind soul...saved me and many others lots of time.


Now I'll be very excited to re-run my existing EA on the longer historical data to see it's consistency and profitability.  Finally one of my wishlist come true.  Thanks once again for your willingness to share smile

Hi Popov,

Do you think this can "extra" service can be a permanent features to FSB Pro whereby FSB Pro users have a link to download past historical data (that are constantly updated monthly, quarterly etc)?

This would definitely benefit new comers who join and didn't have the advantage of knowledge and time to accumulate the data but yet with this added "extra" feature, they can instantly download and maximize the potential use of the 1 min time chart to create EA with higher consistency due to longer historical data to generate their strategies. 

FSB Pro is liken to a powerful sports car (especially the 1 min time chart) and the 1 min historical data is liken to the fuel.  And right now, the FSB Pro 1min time chart only has about 1.5 month of historical data to work on.  It's like having such a powerful sport car but with limited fuel to run on.  And if we have this added feature of having a permanent link to download historical data for 1 min chart, it is like injecting unlimited fuel for people to work on such a powerful 1 min time chart EA creator.  A powerful car with sufficient fuel to run a good race, it goes hand in hand (especially, in my opinion, with the 1 min time chart, one can create very responsive and efficient EA out from this FSB tool).  Hence, a powerful 1 min time chart EA creator (FSB Pro) needs also sufficient "fuel" i.e. historical data to run on.

This is just a wishlist and suggestion (on top of your many other wishlist)...not a demand.  Already you have quite a handful things in hand to improve the FSB.  It's your decision, I'll respect whatever your choice.

Meanwhile, I'm grateful for people like Vidagig who took the effort to set up this link to share with the rest of the community here.  This is such a great community with so many people (such as Dave and Footon, who regularly) willing to chip in to help one another smile

Hannah

10 (edited by hannahis 2015-07-18 19:50:41)

Re: Anyone has 1min and/or 5min historical data to share?

vidagig_13 wrote:

First, I've noticed that strategies give wildly different results depending on the historical data source, which concerns and confuses me; and remains an unresolved question for me, at this stage - in terms of confidence in the robustness of a strategy vs. it being (over-)optimized for a particular historical data source (and the credibility of the data source becomes questionable too).
So, I'm placing the most weight on the strategy results using my own broker's data - even though it's not very long - because that's where I'm running my live strategy tests.
- I notice on my broker - there's much more historical data available on a demo account than on a real account (and they appear to be identical - I checked) so I use both.


Hi Vidagig,

Thanks for your detail explanation and also pointing out the need to change the default maximum data bar, otherwise FSB pro will not load more historical data without the increase.  Now I realised just by increasing the maximum data bar, the FSB demo data also increase the period of data available (about 3months with 1,000,000 maximum data bar).

Vidagig, you mentioned change the maximum data bar to 1,000,000.  How much more can I increase?  Is there a recommended maximum etc.

After I loaded the historical data and compare over various data sources you provided...I understood what you mean in your quote above....yes, I'm confused too...not knowing which data source to rely on (for a start, I'll use your Alpari  1 which has the longest period for me to utilise, about 3 years with 1,000,000 maximum bar)

Guess at the end of the day, I have to rely on my own trading theory (whom I've observed for many years how this trading theory of mine "behave").  I guess that explains why, sometimes, despite the FSB pro showing negative results from my own "trading theory" conditions, I still went ahead to export my EA and true enough, my EA performs well.  Don't ask me why...this is something I'm still figuring out (I was wondering is it because of limited 1.5mth historical data etc).  So I intend to run my EA over a longer period of time to see how well it can withstand market volatility....till I run my EA long enough to go through various market cycle, then I can truly be very confident that I've successfully translate my theory into my EA.  However, without FSB Pro, this wouldn't be a possibility cause it's hard to be a trader and a programmer at the same time, in addition to my other professional job that I'm doing.

Hannah

Re: Anyone has 1min and/or 5min historical data to share?

Popov created something for MT5 data, it is on this forum somewhere.

Also there are scripts available for MT5

Look in this thread


http://forexsb.com/forum/topic/5379/met … ta-export/

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....

Re: Anyone has 1min and/or 5min historical data to share?

Just to add in case its helpful

Perhaps try install one of the MT4's I uploaded (or a few) and try getting and updating data from the history center as described. Or the script that's now available for MT5; but there may be more data on Alpari MT4 than their MT5.

In FSB:

Control Panel / General / Check "Use all available shorter periods..." - may improve test accuracy.

Control Panel / Data Horizon: For each data source you can set the start and end date 'segment' you want to use.
Helpful for controlling the amount of data used as you may hit the limits of your computer.
With 1 000 000 I see Alpari 1 minute going back to Sept 2012 - so I guess if you increase this you will see the 1 minute back to 2010.

History Center / Intrabar Stats - shows data stats and missing data - you may see warnings about this in the 'Output Log' at the bottom when you load e.g. Alpari 1 minute on a Strategy. Not sure how serious this is - guess it depends what and how much is missing.

There's different ideas on how much data you need; whether you should try make your strategy work on other currencies to make it more robust - using Multi Markets; Monte Carlo Analysis; etc etc posted here.

Re: Anyone has 1min and/or 5min historical data to share?

Keeping it simple.... some of us have purchased the data......... that saves a lot of time

My 'secret' goal is to push EA Studio until I can net 3000 pips per day....