C:\express-generator>node  ./bin/fetch.js  --fx-rates true  --symbols ADAUSD BNBUSD BTCUSD DOGEUSD ETHUSD SQLUSD XLMUSD XRPUSD   --periods M1 M5 M15 M30 H1 H4 D1  --max-bars 200000  --time-zone 0

         ..:: Express Generator Fetch v2.66 ::..

RangeError: Invalid array length
    at parseData (C:\express-generator\bin\lib\data-manager.js:156:42)
    at C:\express-generator\bin\fetch.js:224:28
    at IncomingMessage.<anonymous> (C:\express-generator\bin\lib\request.js:94:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

C:\express-generator>pause
Press any key to continue . . .

Are you able to add the Maximum average position length to express gen please. I like to use this and it would be ace if it was avaliable in express gen.

Cheers,

Sam

What is "optimize_protection" setting for in Express Generator?

When I run ExpressGenerator Combinestrategies(see code below image) with same 7916 collected strategies only 5812 strategies are ascended (see gen settings file attached).

Here is the Combinestrategies script I use:

node ./bin/gen.js --settings --server Oanda --symbol EURUSD --period M15 --collection_capacity 300 --input .\collections\ --output collections/combinedcollection.json

I have also attached the modified gen settings ini file I used below.

What additional changes do I need to make that will allow all 7916 strategies to be ascended?

Alan-

I'm getting it on a computer that I used before with no issues.

The error message is "getaddrinfo EAI_AGAIN data.forexsb.com"

I can use the gen functions but not the fetch. 

Not sure how to fix this.

I have uploaded my EA studio settings as the attachment, and below is the command for Express Generator

node bin/gen.js --settings bin/gen.settings.ini ^
--server Eightcap ^
--symbol AUDJPY ^
--period M1 ^
--use_data_start true ^
--data_start "20 Sep 2024 00:00 UTC" ^
--use_data_end true ^
--data_end "29 Jan 2025 00:00 UTC" ^
--output test.json ^
--min_count_of_trades 10 ^
--min_profit_factor 1.01 ^
--session_open 12:30 ^
--session_close 21:00 ^
--close_at_session_close true ^
--close_at_friday_close true ^
--trade_direction_mode LongAndShort ^
--leverage 200 ^
--entry_lots 0.1 ^
--opposite_entry_signal Ignore ^
--trade_on_sunday false ^
--spread 12 ^
--commission 0

Because in the above command, the min_profit_factor is 1.01 so Express Generator should not generate any losing strategies, however, when validating the strategies generated by Express Generator in EA studio, a lot of them results in losses.

To reproduce the problem, simply run the above command for a minute to generate around 10 strategies and then validate them with EA studio.

If I have, let's say, 10 years worth of history data for a symbol and I run the express generator. 

But for whatever reason I need to stop it before it gets through all 10 years.

And I run it again with the exact same parameters and same history data, will the Express Generator pick up where it left off or start from the beginning?

Does the EG start from the beginning of the data set every run, randomize where it starts or pick up from where it left off?

If it starts from the beginning each time, is there a way to tell where it stopped so that I can put in that date and start from there?

Thank you

i have build a small tool where you can create your settings files by click.

You can Download INI file or Upload Existed Files and make changes.

Maybe @Popov can put it online on website or whatever . but it works local.

Its an html file but i dont know how i can upload it here. so i change name to .json

You have to change .json to .html

As the subject line says, I have accumulated some random questions as follows

1. What are these particular metrics? I can't seem to find a definition
min_m1_count_of_trades = 0
min_m1_profit          = 0
min_m1_profit_factor   = 0
min_y1_count_of_trades = 0
min_y1_profit          = 0
min_y1_profit_factor   = 0

2. Why are some entry and exit indicators set to false in the default settings? Don't we just want to test them all?

Thanks in advance

I used to have my-setting file customized to my needs, and I use my broker's data to generate strategies and there is a link inside the script to point to that folder, but after the latest update, when I run the workflow, the engine is reading from the original setting file and not my setting file.

It was working before normally.

REM Generate Strategies using Exness data files

node .\bin\gen.js  ^
 --settings mysettings-insample.ini         ^
 --server C:\Users\stali\AppData\Roaming\MetaQuotes\Terminal\6C3C6A11D1C3791DD4DBF45421BF8028\MQL5\Files\exness\
 --symbol %symbol%                         ^
 --period %period%                         ^
 --max-data-bars 50000                     ^
 --data-start-percent             0                ^
 --data-end-percent                   25                 ^
 --max-working-minutes           60                  ^
 --collection-capacity            100000        ^
 --correlation_threshold           0.98          ^
 --resolve_correlation             true                  ^
 --resolve_similar_rules           true                  ^
 --max-ascended-strategies       100000        ^
 --max-calculated-strategies     0             ^
 --output-replace                 true           ^
 --show-top                1            ^
 --output          ./collections/Premium-in-Sample-%symbol%-%period% 

TIMEOUT /T 5

I'm getting the above error when creating large collections (250k+ strategies) by joining JSON files togther and trying to reload them for validation.

From a bit Of googling this relates to some node.js limit of 512Mb. This corresponds to the file sizes I am working with <500Mb works, anything above that generates the error.

I've taken to zero-ing out many of the fields (balance, equity and any backtestsstats) to reduce the file size, as I figure these fields would be updated during the validation anyway.

Is there a better way to deal with this? Is zeroing out the fields actually a bad idea from a strategy validation point of view?

Thanks.

I've recently got into EG and I'm trying to automate as much as possible. I started with DOS cmd, but I am now moving all my programs onto Powershell, so I can then move to Linux, so I can then get a Cloud VPS to run things 24/7. The Powershell on Linux part is to avoid expensive Windows cloud licences, but still remain compatible with my local machine.

Attached is a validate and generate powershell script. It runs, but it can't find any existing collection to load - and I know they exist. It ouputs the collection at the end of the run perfectly well. It can't then read it back in.

Any ideas? Maybe I've gone bug-blind and can't see something obvious in my code :-/

Many thanks.

[I can't seem to attach the ps1 file, but hopefully the code is readable / will run if someone were to put it in a ps1 file]

# Define the parameters for each run
$runs = @(
    @{ Symbol = "AUDJPY"; Period = "M15"; DataStart = "2014-01-01"; DataEnd = "2023-04-01"; RunTimeMinutes = "1" },
    @{ Symbol = "AUDJPY"; Period = "M15"; DataStart = "2014-10-01"; DataEnd = "2023-10-01"; RunTimeMinutes = "1" }
)

# Run each process in parallel
$runs | ForEach-Object -Parallel {
    # Define the function for Node.js process execution
    function Run-NodeProcess {
        param (
            [string]$Symbol,
            [string]$Period,
            [string]$DataStart,
            [string]$DataEnd,
            [string]$RunTimeMinutes,
            [string]$BaseDirectory = "D:/EG",
            [string]$SettingsFilePath = "$BaseDirectory/genmaster.ini",
            [string]$ScriptFilePath = "$BaseDirectory/bin/gen.js"
        )

        # Generate input and output filenames using string interpolation
        $inputFileName = "$BaseDirectory/collections/$Period/Coll-IS-$Symbol-$Period-$DataStart"
        $outputFileName = "$BaseDirectory/collections/$Period/Coll-IS-$Symbol-$Period-$DataStart"

        # Ensure the output directory exists
        $outputDir = Split-Path -Path $outputFileName -Parent
        if (-not (Test-Path -Path $outputDir)) {
            New-Item -ItemType Directory -Path $outputDir -Force | Out-Null
        }

        # Debugging: Log paths
        Write-Output "Input File:  $inputFileName"
        Write-Output "Output File: $outputFileName"

        # Start the Node.js process with the provided variables
        Start-Process -FilePath "node" -ArgumentList @(
            "`"$ScriptFilePath`"",
            "--settings", "`"$SettingsFilePath`"",
            "--symbol", $Symbol,
            "--period", $Period,
            "--use_data_start", "true",
            "--data_start", $DataStart,
            "--use_data_end", "true",
            "--data_end", $DataEnd,
            "--max_working_minutes", $RunTimeMinutes,
            "--validate_then_generate", "true",
            "--opposite_entry_signal", "IgnoreOrReverse",
            "--stop_loss_usage", "AlwaysUse",
            "--stop_loss_type", "Fixed",
            "--stop_loss_range_min", "20",
            "--stop_loss_range_max", "200",
            "--take_profit_usage", "AlwaysUse",
            "--take_profit_range_min", "20",
            "--take_profit_range_max", "200",
            "--take_profit_gte_stop_loss", "true",
            "--min_count_of_trades", "100",
            "--min_profit", "10",
            "--min_profit_factor", "1.2",
            "--enable_monte_carlo", "false",
            "--input", "`"$inputFileName`"",
            "--output", "`"$outputFileName`""
        ) -WindowStyle Normal -Wait
    }

    # Call the function
    Run-NodeProcess -Symbol $_.Symbol -Period $_.Period -DataStart $_.DataStart -DataEnd $_.DataEnd -RunTimeMinutes $_.RunTimeMinutes
} -ThrottleLimit 4

Script:

set symbol=%1
set period=%2

IF "%symbol%" == "" EXIT
IF "%period%" == "" EXIT

TITLE OOS Workflow %symbol% %period%

REM Fetch forex data

node .\bin\fetch.js                         ^
--fx-rates   true                             ^
--symbol  %symbol%                         ^
--period  %period%                         ^
--max-data-bars 200000                     ^
--time-zone  0
...

Error:

""""Expected property name or '}' in JSON at position 1 (line 1 column 2)"""""

another quick suggestion for the next EG release regarding collection file handling.

As I´ve mentioned a while ago, I've been patching collection.js to allow loading collections from any /collections subdirectory relative to the working directory. It's super helpful for organizing strategies across different projects. Here are the exact changes needed:

1. Add 'resolve' to the path imports:

const {join, parse, isAbsolute, resolve} = require("path");

2. Modify the loadCollection function to handle paths relative to the current working directory:

function loadCollection(inputPath, inputKeys, dataId, dataEndTime, suppressInputError) {
    if (typeof inputPath === "string" && inputPath.length > 0) {
        try {
            let collectionPath = replacePlaceholders(inputPath, dataId, dataEndTime);

            // If path is not absolute, prepend the collections directory
            if (!isAbsolute(collectionPath) && 
                (parse(collectionPath).dir === "" || !collectionPath.includes("collections"))) {
                collectionPath = join(".", "collections", collectionPath);
            }

            // Resolve the path relative to the current working directory
            collectionPath = resolve(process.cwd(), collectionPath);

            // Check if the path exists and is a directory or file
            if (existsSync(collectionPath)) {
                const stat = statSync(collectionPath);
                if (stat.isDirectory()) {
                    return readCollectionDir(collectionPath, inputKeys);
                } else if (stat.isFile()) {
                    return readCollectionFile(collectionPath, inputKeys);
                }
            }

            // If we reach here, try adding .json extension if missing
            if (!collectionPath.toLowerCase().endsWith(".json")) {
                collectionPath += ".json";
                if (existsSync(collectionPath)) {
                    return readCollectionFile(collectionPath, inputKeys);
                }
            }

            if (!suppressInputError) {
                console.error(`${color.Red}Cannot open input file: ${color.Bright}${collectionPath}${color.Reset}`);
            }
            return [];
        } catch (e) {
            console.error(`${color.Red}${color.Bright}${e.message}${color.Reset}`);
            return [];
        }
    } else if (Array.isArray(inputPath) && inputPath.length > 0) {
        return inputPath.flatMap(file => 
            loadCollection(file, inputKeys, dataId, dataEndTime, suppressInputError)
        );
    }

    return [];
}

This replaces the current path resolution logic in loadCollection and makes it much more flexible. The changes maintain backward compatibility while allowing users to organize their collections in subdirectories.

Would you consider adding this to the next release? Would save me from patching after each update :-)

Thanks as always!