• Registered: 2025-03-19
  • Posts: 1

Topic: Where does the Express Generator start for each run?

Hello,

If I have, let's say, 10 years worth of history data for a symbol and I run the express generator. 

But for whatever reason I need to stop it before it gets through all 10 years.

And I run it again with the exact same parameters and same history data, will the Express Generator pick up where it left off or start from the beginning?

Does the EG start from the beginning of the data set every run, randomize where it starts or pick up from where it left off?

If it starts from the beginning each time, is there a way to tell where it stopped so that I can put in that date and start from there?

Thank you

  • Registered: 2023-11-26
  • Posts: 118

Re: Where does the Express Generator start for each run?

Ele comeca sempre do início a única hora que ele randomiza e quando monte Carlo é ligado

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 8,071

Re: Where does the Express Generator start for each run?

Express Generator calculates the backtest on all of the provided data by default.

We have several means to control the data set.

1) Limit the count of the data bars:

We can do it when fetching the data or when running the Generator.

> node .\bin\fetch.js --server Eightcap --symbol EURUSD --period M30 --max-data-bars 100000

         ..:: Express Generator Fetch v2.65 ::..

Fetched: Eightcap EURUSD M30, From: 2017-03-08 00:00, To: 2025-03-21 22:30, Bars: 100000


> node .\bin\gen.js --server Eightcap --symbol EURUSD --period M30

We see that the Generator uses the complete data set.

https://image-holder.forexsb.com/store/ex-gen-run-eightcap-100000-bars.png


We obtained the same result by downloading the complete data set and limiting the data bars when running the generator.


> node .\bin\fetch.js --server Eightcap --symbol EURUSD --period M30

         ..:: Express Generator Fetch v2.65 ::..

Fetched: Eightcap EURUSD M30, From: 2009-02-04 17:00, To: 2025-03-21 22:30, Bars: 200000

Data fetch ready!

> node .\bin\gen.js --server Eightcap --symbol EURUSD --period M30 --max-data-bars 100000

2) Limit the data's start and end dates.

We have four parameters to control the start and end dates of the data set. The 

use_data_start  = false
data_start      = "01 Jan 2020 00:00 UTC"

use_data_end    = false
data_end        = "30 Jun 2023 00:00 UTC"

We enable the corresponding limitation and set the desired date and time.

Usually, we limit the "data end" and use the count of bars limitation.

https://image-holder.forexsb.com/store/ex-gen-data-end-max-bars-back-tick.png

( I'm using "backtick `"  and shift+enter to make a new line)

We see that the Generator loads 100,000 bars ending at 2024-12-31 23:00.


We can also make it by setting the count of days for the data end. This is useful because we can have it in a settings file, which will work with the time.

PS C:\express-generator> node .\bin\gen.js --server Eightcap --symbol EURUSD --period M30 `
>> --use-data-end true --data-end -90 --max-data-bars 100000

--use-data-end true --data-end -90 - it sets the Generator to load data up to 90 days back from the current time.


3) Out of sample

We have a convenient way to cut part of the data.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Out of sample            ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; Use these parameters to set the data range.
; The program cuts data after applying Data Horizon.

data_start_percent = 0
data_end_percent   = 100

The out-of-sample parameters are applied after the data-sart, data-end, and max-bars limitations.

Here is how to have the oldest 80% of the data

PS C:\express-generator> node .\bin\gen.js --server Eightcap --symbol EURUSD --period M30 `
>> --data-start-percent 0 --data-end-percent 80

                        ..:: Express Generator v2.65 ::..


    Market : Eightcap EURUSD M30
    From   : 2017-03-08 00:00, To: 2023-08-11 06:30, Bars: 80000
    Spread : 20, Swap long: -7.44, Swap short: 2.9, Commission: 6 USD
    Account: 10000 USD, Leverage: 100, Entry: 0.01 lots

Popov's Website

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 8,071

Re: Where does the Express Generator start for each run?

> If it starts from the beginning each time, is there a way to tell where it stopped so that I can put in that date and start from there?

Express Generator generates new collections every time you run it by default.

We can also use it to validate previously generated collections. We do it by setting the --input parameter.

 node .\bin\gen.js --server Eightcap --symbol EURUSD --period M30 --input my-generated-collection.json

(Provided that we have a collection named "my-generated-collection.json" )

The principle is the same — Express Generator will use the complete data set for the calculation, with the limitation we provide, as I mentioned in my previous post.

Forward Testing

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Forward testing          ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; Validate strategies on new unseen data.

use_forward_testing = false
preload_data_bars   = 0

We can set the Generator to calculate Backtest statistics only on the new data with these parameters.
It will work only if the loaded collection was tested with an Out-of-sample or with data limited by other means.

Let's have a collection generated on data ending on 31st December 2024. We can set the backtest calculating only on the new data (since January 1st 2025).

PS C:\express-generator> node .\bin\gen.js --server Eightcap --symbol EURUSD --period M30 `
>> --input my-generated-collection.json --use-forward-testing true --preload-data-bars 1000

The --preload-data-bars 1000 sets the Generator to load 1000 bars more. The trades will still start from the time the calculation was calculated.

Let's recap:

The Generator will use the complete data set when you validate an already generated collection.

You can set it to calculate on the new (unseen or Out-of-sample) data in two ways:

1) Limit the data start:

PS C:\express-generator> node .\bin\gen.js --server Eightcap --symbol EURUSD --period M30 `
>> --input my-generated-collection.json `
>> --use-data-start true --data-start "2025-01-01 00:00 UTC"

https://image-holder.forexsb.com/store/ex-gen-data-start-back-tick-1.png

2) Forward testing 

PS C:\express-generator> node .\bin\gen.js --server Eightcap --symbol EURUSD --period M30 `
>> --input my-generated-collection.json --use-forward-testing true --preload-data-bars 1000

The Generator will load data 1000 bars earlier than the time the original strategy was generated and will perform trading only on thenew data.

Popov's Website