....

The current stitching works by gluing an entire dataset to its next dataset. Another way that would be interesting to test would work like this:

bar1: EURUSD
bar2: USDCHF
bar3: GBPUSD
bar4: USDJPY
bar5: EURUSD ...

...

How about blocks of bars at a time? The analogy I'm thinking of is when DJ's mix songs, they mix sets of notes from each song together, not individual notes, they preserve some pattern characteristic of the source song. I wonder if a single-bar mixing would remove the anomalies and non-random patterns from naturally occurring datasets? Interesting to research.

In the example I posted, I noticed they are in this order: EURUSD, GBPUSD, USDCHF, USDJPY

perhaps I should have alternated them to be: EURUSD, USDCHF, GBPUSD, USDJPY so that the dollar-strength would be more equally distributed.

The current stitching works by gluing an entire dataset to its next dataset. Another way that would be interesting to test would work like this:

bar1: EURUSD
bar2: USDCHF
bar3: GBPUSD
bar4: USDJPY
bar5: EURUSD ...

Between each bar would be the normalized actual movement in pips. The theory being that the strategy would have to be robust enough to deal with any type of movement, at any given time.

We should also test synthetic data formats on a specific group of non-correlated pairs.

I'm attaching the stitched (synthetic) data that contains the following:

• exactly 7 weeks of EURUSD m1 beginning Sunday 1/8/2012 @ 23:00 GMT

• exactly 7 weeks of GBPUSD m1 data

• exactly 7 weeks of USDCHF m1 data

• exactly 7 weeks of USDJPY m1 data ending Friday 7/20/2012 @ 21:59 GMT

There are 197691 bars in this synthetic dataset. Pricing has been normalized so that between seams the data MOVEMENT integrity remains (price is shifted accordingly). Yen pairs are divided by 100 before normalizing. I worked very hard on this normalizing portion to ensure that it was "correct" and takes into account ONLY the pip movement of the pairs, which is the most important aspect I think.

I'm attaching a screenshot here as well showing the Generator chugging. This strategy is not "good enough" yet, but you can see the price curve in background (light grey line) looks normal.

Next steps for me:

• Find a good strategy with Generator

• Test strategy on non-synthetic (ie real life) data for these pairs, compare stats

I believe this will be a good test to prove or disprove the theory that a strategy generated with synthetic-stitched data is useful.

Shortly thereafter, I plan to tweak the code a little and do that other variation I mentioned above. In that scenario, I will merge the data across all datasets per minute. I think I will probably average the data for each bar. Resultant synthetic bars should be about 50,000. There will be some issues to deal with, such as missing bars in one of the datasets.

<?php
//stitch together data sequentially
//FSB probably doesn't care about weekends or leapyear, so don't worry about this for now
//prerequisite: ensure all datasets begin/end at exactly start/end of week

//login to database
$linkid = @mysql_connect("localhost", "root", "") or trigger_error(mysql_error(), E_USER_ERROR);
mysql_select_db("dusktrader", $linkid);
$stitched = array(); //holds all stitched, time-shifted bar data
$i=1; //array index, keeps track of stitched bars

//datasets to stitch together, each set prepared with exactly 7 weeks of data
processdata('eurusd_m1', 0); //EURUSD not shifted
processdata('gbpusd_m1', 7);
processdata('usdchf_m1', 14);
processdata('usdjpy_m1', 21);

//insert stitched data to MySQL (easy to export from MySQL in FSB-ready format)
print "$i bars processed<br>";
$j=1;
while ($j < $i)
   {
   $sql = "INSERT INTO stitched_m1 " .
      "(feeddate, feedtime, open, high, low, close, vol) " .
      "VALUES (" .
      "'".$stitched[$j]['feeddate']."'," .
      "'".$stitched[$j]['feedtime']."'," .
      $stitched[$j]['open']."," .
      $stitched[$j]['high']."," .
      $stitched[$j]['low']."," .
      $stitched[$j]['close']."," .
      $stitched[$j]['vol'] .
      ");";
   $resultid = mysql_query($sql, $linkid);
   $j++;
   }

exit;


//read and process bar data from an existing table
//shiftweeks specifies the amount of calendar weeks data will be shifted
//from the original feeddate/feedtime
function processdata($table, $shiftweeks)
   {
   global $linkid, $stitched, $i, $prevclose;   
   
   //get initial opening value so we can calculate the difference
   $sql = "SELECT open FROM $table WHERE id=1"; //first record
   $resultid = mysql_query($sql, $linkid);
   $resulttext = mysql_fetch_object($resultid);
   if (floatval($resulttext->open) > 10) $divisor = 100; else $divisor = 1; //divide Yen pair values by 100
   $open = floatval($resulttext->open) / $divisor;
   if (!$prevclose) $prevclose = $open; //synthetic data start point
   if ($open > $prevclose) $difference = ($open - $prevclose) * -1;
      else $difference = $prevclose - $open;
   
   //loop through all data with price normalizing and time shifting
   $sql = "SELECT * FROM $table"; // ******** APPLY LIMIT HERE for testing
   $resultid = mysql_query($sql, $linkid);
   while($resulttext = mysql_fetch_object($resultid))
      {
      $open = floatval($resulttext->open) / $divisor;
      $high = floatval($resulttext->high) / $divisor;
      $low = floatval($resulttext->low) / $divisor;
      $close = floatval($resulttext->close) / $divisor;
      $timeshifted = timeshift($resulttext->feeddate, $resulttext->feedtime, $shiftweeks);
      $stitched[$i]['table'] = $table;
      $stitched[$i]['feeddate'] = $timeshifted['feeddate'];
      $stitched[$i]['feedtime'] = $timeshifted['feedtime'];
      $stitched[$i]['open'] = ($open + $difference);
      $stitched[$i]['high'] = ($high + $difference);
      $stitched[$i]['low'] = ($low + $difference);
      $stitched[$i]['close'] = ($close + $difference);
      $stitched[$i++]['vol'] = $resulttext->vol;
      }
   $prevclose = $close + $difference; //to be used on next dataset
   }

//parse and shift time by specified weeks into future
function timeshift($feeddate, $feedtime, $shiftweeks)
   {
   $unixtime = strtotime(str_replace(".","-",$feeddate)." ".$feedtime);
   $unixtime += (604800 * $shiftweeks); //# weeks into future

   $timeshifted['feeddate'] = date('Y.m.d', $unixtime);
   $timeshifted['feedtime'] = date('H:i', $unixtime);  
   return $timeshifted;
   }
?>

I don't know, this may work out or not, this just gives us something to work with to see what actually happens. I suppose one thing to look for is whether the Generator gives you a strategy with fairly linear equity curve, meaning consistent across pairs, or a curve that looks split in quarters, meaning the pairs are different? I don't know, this is interesting to research , and try to interpret.

Here is a screenshot with my synthetic 5-min data:

For example (correct me if I'm wrong) I believe that at any given place within the synthetic dataset, you would still find the same degree of movement in currency. Even the chart should look identical if you were to plot it. The only change that should take place when stitching the datasets would be adjusting the price level to match (I think) at the splice points.

So the theory is that Genetator would be looking at price action only. The next step would be to take that strategy that works well over the synthetic data and run it with actual data from one of the pairs. If this theory works, it should perform well.

EDIT: after thinking some more on this, I think all we are doing is shifting the next dataset to start at the same price level that the last one ended at. So if dataset1 ends at 1.2350 and dataset2 starts at 1.3468, then I believe the correct way to shift dataset2's data would be to simply subtract .1118 from each value. You'd then do the same shift at each new splicepoint.

First Data Set, 50,000 lines, this references every value as against 1:
1) in I1, enter 1
2) in I2, enter    =(B2-$B$1) + $I$1
3) in J1, enter    =(C1-$B$1)+$I$1
4) copy J1 to K1 and L1
5) for Volume column, in M1, enter   =F1
6) copy J1 to M1 down to next line
7) copy I2 to M2 down to 50000

Second Data Set, lines 500001 to 100000:
1) I50001, enter L50000   // start this set from the last close of the previous data set
2) in I50002, enter    =(B50002-$B$50001) + $I$50001
3) in J50001, enter    =(C50001-$B$50001)+$I$50001
4) copy J50001 to K50001 and L50001
5) for Volume column, in M50001, enter   =F50001
6) copy J50001 to M50001 down to next line
7) copy I50002 to M50002 down to 100000

Third Data and Fourth Data Set:
do the same steps above, but change formulas to divide by 100:
= ((C100001-$B$100001)/100)+$I$100001
this will make it suitable for 2 or 3 digits instruments, like EURJPY and GBPJPY

Yeah, I should probably mention that these are just my subjective observations, that are not backed with any measures, so if anyone is interested in serious work, you might want to check it more thoroughly

I meant, this feature is a good idea because if a strategy works across more instruments it is probably more likely to have found something good. I was just thinking for coding the feature, add a couple checks that use less code to improve the performance of the Generator -- which does 10's to 100's of thousands of combos.

I made a spread sheet to splice the data together, but can't post it here because the forum says it's too big. I think you can download from:
https://github.com/krogKrog/Forex-Strat … s?raw=true
and a sample file:
https://github.com/krogKrog/Forex-Strat … p?raw=true

Unfortunately is an OpenOffice ods file, OpenOffice says it was too big to save in Excel xls format.
How to use:
1) In A1, paste in 50,000 lines of data from a _USD data file
2) In A50001, paste in 50,000 lines of data from a _USD data file
3) In A100001, paste in 50,000 lines of data from a _JPY data file
4) In A150001, paste in 50,000 lines of data from a _JPY data file
5) Open new spreadsheet
6) from splicer, select H1 to M200000, copy
7) in new sheet, right click, Paste Special
8) deselect formulas, select numbers, paste
9) save as EURUSD5.csv  (or can change EURUSD to any instrument you use for FSB

It's pretty rough, so might take a few tries. Mine crashes every 5 minutes. But with that csv file, I was able to run the Generator on it.

Whereas the original idea was to stitch-together a synthetic dataset, this idea would be to create a synthetic dataset by merge.

If you loaded all the minute bars for each desired pair into a separate table, you could then recurse through each bar one step at a time and average them together to get the synthetic bar data. The resultant dataset would be still 50,000 bars, so it would be more efficient from the Generator standpoint.

The theory in this is that a robust strategy that could perform well on the blended synthetic dataset would therefore also perform well on the individual pair (same objective as the stitched dataset).

If that worked well, it would be easy enough to create similar synthetic data from tickdata as well, except that you'd probably want to weight the ticks in case one pair was more active than another (equal weightings for the movement of each pair).

One issue I can think of would be the Yen pairs that only have 3 digits after the decimal instead of 5 for non-Yen pairs. An easy way to fix this (I think) would be to pad them with zeroes just after the decimal (ie, one point of movement remains the same). The actual value of the pair is irrelevant I think, as we would only be concerned about the movement.

For example:
80.936 --> 80.00936

So that means all you'd have to do is ensure that the synthetic start date was aligned correctly with the actual dataset's day-of-week.

Btw, what does FSB do (if anything) when it comes across a gap in data?  What if a broker feed skips an entire day?  Does it do anything at all?

http://www.fxmri.com