1 Topic by hannahis (edited by hannahis 2017-08-15 06:34:59)

  • Registered: 2010-11-18
  • Posts: 1,179

Topic: Free Historical Data for ALL Time Frames

Dear All,

A number of us may be looking for longer historical data to generate robust EA.

I personally use only 1min Time Chart with LTF features to develop my EA.

The FSB default 1 min data has about 3mth of historical data which is insufficient for you to develop EA that can withstand the test of time.

Hence, it is necessary to search for longer historical data without the need to purchase them.

Here is the link whereby you can get a free tool to download your historical data in any time frame you prefer.

https://strategyquant.com/tickdownloader/

What is Tick Downloader?

Tick Downloader is a freeware tool that allows to download quickly tick data from Dukascopy, which has a good quality and is reliable for backtest of your strategies.

It allows to download data very quickly
All data from Dukascopy is available: forex, comodities and indeces
Tick data allows backtesting with 99-per-cent accuracy
It is compatible with Birt‘s TD Suite,
It calculates automatically higher timeframes as well: M1, M5, M15, M30, H1, H4, D1
It is possible to change the timezone of the data and if necessary, then remove weekend data
The downloaded tick data can be used for Birt‘s TD Suite, MetaTrader 4, or in FSB to perform a backtest with 99-per-cent accuracy.

  • From: Germany
  • Registered: 2012-02-17
  • Posts: 122

Re: Free Historical Data for ALL Time Frames

Hi,
yes I know this.
This is a usefull tool.

But I don´t need Tickdata for searching strategies.
I use most time M1 Data precision. It is faster to find strategies.
thomas

tnickel's Website

  • Registered: 2017-07-24
  • Posts: 99

Re: Free Historical Data for ALL Time Frames

This tool downloads data for use on all TFs. It can export M1-D1.

  • Registered: 2014-04-01
  • Posts: 25

Re: Free Historical Data for ALL Time Frames

Thank you Hannah! V useful!

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Free Historical Data for ALL Time Frames

Welcome, I wished I had this at the very beginning of my EA development journey, that's why I put this up here cos I believe there are many who aren't aware of this free and useful tools.

  • araza
  • Licensed Member
  • Offline
  • Registered: 2017-08-29
  • Posts: 145

Re: Free Historical Data for ALL Time Frames

Thanks for the link.
I keep on hearing that back testing should be done on data from my broker. Since I am new so my account does not have much data to give me. maximum so far is 66,000 bars.
I am assuming that this data will work irrespective.

hannahis
I like your posts as they are very detailed and well thought out. Learning a lot from them.
One question though...you use the abbreviation LTF. Can you please explain that.
Thanks

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Free Historical Data for ALL Time Frames

Hi Araza

Thanks for your words of encouragement.  I usually don't like general comments because it is as good as nothing.  Hence I try to be as specific as I can with workable steps that really can be of any help.  However, I have to strike a balance between helping others shorten their learning curves and yet at the same time avoiding spoon feeding those who never intended to try but just want hand outs.  I don't like to reward laziness. 

Imagine just walk up to Coke company and say "Hey I really love your drink, can you please tell me what's in your secret recipe?"  Although Coke company may conduct tours to curious customers/tourists to visit their factory to see the workflow.  They aren't going to tell you their secret ingredients.  Likewise, I'm fine sharing with people my workable workflow...just hope they don't ask me anything about my indicators/parameters smile  (I will pretend to ignore these questions).


Yes ideally if you have your broker's data to work on that will be great.

However, I don't think you have the time nor will you wait till you have gathered sufficient data from your broker (such as 1 yr, 5yrs etc) for you to start wanting to develop your EA.

As for the use of LTF (sorry, I'll make my efforts to explain the abrev. in futures).  LTF refers to Longer Time Frames.

When you use FSB Pro to input your opening/closing conditions you have the choice of using the Default (time frame as according to the time chart you use such as 1min, H4, D1 etc) or you have a choice to incorporate other Longer/higher time frames indicators too (remember it has to be higher, if you use 1min time chart, you can choose anything above or more than 1min but if you choose D1 as your time chart, you can't choose anything lower than W1 which is higher).

  • araza
  • Licensed Member
  • Offline
  • Registered: 2017-08-29
  • Posts: 145

Re: Free Historical Data for ALL Time Frames

Thank you for the explanation. Cleared a lot of what I have read in your other posts. Totally agree with your philosophy on hand outs. We will get along fine. Looking forward to learning a lot from you.
Cheers

  • araza
  • Licensed Member
  • Offline
  • Registered: 2017-08-29
  • Posts: 145

Re: Free Historical Data for ALL Time Frames

I read somewhere that the data imported has to be adjusted for time zones or has to match the time of my broker. Can someone please elaborate on that.
Thanks

  • Registered: 2017-08-01
  • Posts: 20

Re: Free Historical Data for ALL Time Frames

Hi Hanna
How do you import the data file from tickdata to FSB?
Thanks


hannahis wrote:

Hi Araza

Thanks for your words of encouragement.  I usually don't like general comments because it is as good as nothing.  Hence I try to be as specific as I can with workable steps that really can be of any help.  However, I have to strike a balance between helping others shorten their learning curves and yet at the same time avoiding spoon feeding those who never intended to try but just want hand outs.  I don't like to reward laziness. 

Imagine just walk up to Coke company and say "Hey I really love your drink, can you please tell me what's in your secret recipe?"  Although Coke company may conduct tours to curious customers/tourists to visit their factory to see the workflow.  They aren't going to tell you their secret ingredients.  Likewise, I'm fine sharing with people my workable workflow...just hope they don't ask me anything about my indicators/parameters smile  (I will pretend to ignore these questions).


Yes ideally if you have your broker's data to work on that will be great.

However, I don't think you have the time nor will you wait till you have gathered sufficient data from your broker (such as 1 yr, 5yrs etc) for you to start wanting to develop your EA.

As for the use of LTF (sorry, I'll make my efforts to explain the abrev. in futures).  LTF refers to Longer Time Frames.

When you use FSB Pro to input your opening/closing conditions you have the choice of using the Default (time frame as according to the time chart you use such as 1min, H4, D1 etc) or you have a choice to incorporate other Longer/higher time frames indicators too (remember it has to be higher, if you use 1min time chart, you can choose anything above or more than 1min but if you choose D1 as your time chart, you can't choose anything lower than W1 which is higher).

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Free Historical Data for ALL Time Frames

1. Hi once you have downloaded the data, you need to tick which time frames you want these data to be converted to.

2. Save these files in the same format as FSB Demo Data (go to FSB, click "User's File Folder and go to "Data", "FSB Demo Data" to view how these files are names.

Eg. EURUSD1 = D1, EURUSD5 = M5 etc

3. Create another new folder name (eg TickDown 3yrs) under "Data", same folder where you can locate "FSB Demo Data"

4. Transfer all those files you have downloaded with the correct name format into this new folder (always use the FSB Demo Data as your sample or reference to see whether you did it the same method).

5. Go to FSB software and under the tab "Market" look for "Data Sources"

6. Create a new data sources (name whatever you like) and go to browse to locate where you save this folder in (i.e. Data folder, under the new name you created such as "TickDown 3yrs).

7. Close the software and when you re open it, go to "FSB demo Data" tab (which is next to the "Default profile" tab), click the scroll down menu and if you have set up your data sources correctly, you will see your new folder's name there.

Here is the instruction on how to set up Data sources, if my explanation isn't clear enough.
https://forexsb.com/wiki/fsbpro_guide/data_sources

  • Registered: 2017-08-01
  • Posts: 20

Re: Free Historical Data for ALL Time Frames

Thanks a lot Hannah.
one more question. The limit of hystorical data are 50.000 bars, so for M15 the initial date will be different that for M30, for example. Is it correct?

hannahis wrote:

1. Hi once you have downloaded the data, you need to tick which time frames you want these data to be converted to.

2. Save these files in the same format as FSB Demo Data (go to FSB, click "User's File Folder and go to "Data", "FSB Demo Data" to view how these files are names.

Eg. EURUSD1 = D1, EURUSD5 = M5 etc

3. Create another new folder name (eg TickDown 3yrs) under "Data", same folder where you can locate "FSB Demo Data"

4. Transfer all those files you have downloaded with the correct name format into this new folder (always use the FSB Demo Data as your sample or reference to see whether you did it the same method).

5. Go to FSB software and under the tab "Market" look for "Data Sources"

6. Create a new data sources (name whatever you like) and go to browse to locate where you save this folder in (i.e. Data folder, under the new name you created such as "TickDown 3yrs).

7. Close the software and when you re open it, go to "FSB demo Data" tab (which is next to the "Default profile" tab), click the scroll down menu and if you have set up your data sources correctly, you will see your new folder's name there.

Here is the instruction on how to set up Data sources, if my explanation isn't clear enough.
https://forexsb.com/wiki/fsbpro_guide/data_sources

13 Reply by hannahis (edited by hannahis 2017-09-21 10:39:03)

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Free Historical Data for ALL Time Frames

Hi dperales

Yes, under different time chart, the initial starting date will be different.

If you want to adjust to the same, I think you can use the function under "Data Horizon" to set the start and end period

Eg. Use start date limit and Use end date limit.  I'm not that sure about this cos I only use 1 min time chart (with LTF features) for all my EA and I don't use any other time chart, so this isn't an issue for me. 

Your FSB historical bar limit can be changed under the "Account setting", "Data Horizon"

Another method is go to"file" and under the scroll down menu click "user's file folder" and under "system" look for your data folder you saved under then scroll all the way down and you can see

Here is just a sample


  },
  "StartDate": "2000-01-01T00:00:00",
  "EndDate": "2020-12-31T00:00:00",
  "IsUseStartDate": false,
  "IsUseEndDate": false,
  "MaximumBars": 50000000,
  "MinimumBars": 300,
  "MaximumBarsLimit": 5000000,
  "MaxIntrabarBars": 10000000,
  "MaxIntrabarBarsLimit": 100000000,
  "IsCheckDataAtLoad": true,
  "IsCutOffBadData": false,
  "IsCutOffSatSunData": false,
  "IsFillInDataGaps": false,
  "IsCacheDataFiles": true
}

To read up more about setting the bar limit click the link below

[url]https://forexsb.com/wiki/fsb/manual/data_horizon?s[]=historical&s[]=bar&s[]=limit[/url]

  • Registered: 2017-08-01
  • Posts: 20

Re: Free Historical Data for ALL Time Frames

I have change the start date, but nothing happens in the Strategy periods....


hannahis wrote:

Hi dperales

Yes, under different time chart, the initial starting date will be different.

If you want to adjust to the same, I think you can use the function under "Data Horizon" to set the start and end period

Eg. Use start date limit and Use end date limit.  I'm not that sure about this cos I only use 1 min time chart (with LTF features) for all my EA and I don't use any other time chart, so this isn't an issue for me. 

Your FSB historical bar limit can be changed under the "Account setting", "Data Horizon"

Another method is go to"file" and under the scroll down menu click "user's file folder" and under "system" look for your data folder you saved under then scroll all the way down and you can see

Here is just a sample


  },
  "StartDate": "2000-01-01T00:00:00",
  "EndDate": "2020-12-31T00:00:00",
  "IsUseStartDate": false,
  "IsUseEndDate": false,
  "MaximumBars": 50000000,
  "MinimumBars": 300,
  "MaximumBarsLimit": 5000000,
  "MaxIntrabarBars": 10000000,
  "MaxIntrabarBarsLimit": 100000000,
  "IsCheckDataAtLoad": true,
  "IsCutOffBadData": false,
  "IsCutOffSatSunData": false,
  "IsFillInDataGaps": false,
  "IsCacheDataFiles": true
}

To read up more about setting the bar limit click the link below

[url]https://forexsb.com/wiki/fsb/manual/data_horizon?s[]=historical&s[]=bar&s[]=limit[/url]

  • Registered: 2017-08-01
  • Posts: 20

Re: Free Historical Data for ALL Time Frames

Hello Hannah
What do you mean when you said that you use M1 with LTF. How do you do so?

Thanks


dperales wrote:

I have change the start date, but nothing happens in the Strategy periods....


hannahis wrote:

Hi dperales

Yes, under different time chart, the initial starting date will be different.

If you want to adjust to the same, I think you can use the function under "Data Horizon" to set the start and end period

Eg. Use start date limit and Use end date limit.  I'm not that sure about this cos I only use 1 min time chart (with LTF features) for all my EA and I don't use any other time chart, so this isn't an issue for me. 

Your FSB historical bar limit can be changed under the "Account setting", "Data Horizon"

Another method is go to"file" and under the scroll down menu click "user's file folder" and under "system" look for your data folder you saved under then scroll all the way down and you can see

Here is just a sample


  },
  "StartDate": "2000-01-01T00:00:00",
  "EndDate": "2020-12-31T00:00:00",
  "IsUseStartDate": false,
  "IsUseEndDate": false,
  "MaximumBars": 50000000,
  "MinimumBars": 300,
  "MaximumBarsLimit": 5000000,
  "MaxIntrabarBars": 10000000,
  "MaxIntrabarBarsLimit": 100000000,
  "IsCheckDataAtLoad": true,
  "IsCutOffBadData": false,
  "IsCutOffSatSunData": false,
  "IsFillInDataGaps": false,
  "IsCacheDataFiles": true
}

To read up more about setting the bar limit click the link below

[url]https://forexsb.com/wiki/fsb/manual/data_horizon?s[]=historical&s[]=bar&s[]=limit[/url]

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Free Historical Data for ALL Time Frames

Hi dperales,

I've answered your question in this post so that I don't have to repeat myself all over the places.

I hope this help.

https://forexsb.com/forum/post/46563/#p46563

  • Registered: 2017-08-01
  • Posts: 20

Re: Free Historical Data for ALL Time Frames

Really useful.
So, as I understand, you allways select M1, and never use the random generator?
You build your own EA's, is it OK?

Thanks

hannahis wrote:

Hi dperales,

I've answered your question in this post so that I don't have to repeat myself all over the places.

I hope this help.

https://forexsb.com/forum/post/46563/#p46563

18 Reply by hannahis (edited by hannahis 2017-09-22 11:04:01)

  • Registered: 2010-11-18
  • Posts: 1,179

Re: Free Historical Data for ALL Time Frames

Yes, I build my own EA based on my own Trading rules.

I don't use the Generator or the Optimization functions to build my EA (only now and then to test out how "good" it can improve my EA).  And also I already have my own workflow and don't have the spare time to experiment with it.  I believe these functions have improved only time though.  May one day put more effort to try but not now cos I'm very busy with my current workflow and don't want to derail myself to work on a new method.

Is it OK?

What do you mean? Yes of cos it is ok to build your own EA

Yes, my EA is doing well, so yes it is Ok

Yes, it's Ok to give it a try

NOTE: Using M1 Time Chart doesn't mean you are going to have fast EA, you can still build very slow EA in M1 time chart using Longer Time Frames indicators such as D1 or larger parameter settings.

Honestly I can't imagine why would anyone want to check the market are longer interval than M1.  It's like switch ON your EA  only once a day if you use D1 time chart.  Such method only works for positional traders.