1 Topic by geektrader (edited by geektrader 2023-06-15 07:05:53)

  • From: Germany
  • Registered: 2010-04-05
  • Posts: 789

Topic: Bug(?): "RangeError: Maximum call stack size exceeded"

Hi Mr. Popov,

with EG 2.34 I am regularly getting this error:

                        ..:: Express Generator v2.34 ::..

    Market : Unknown GBPJPY H1
    From   : 1986-12-19 14:00, To: 2017-03-21 00:00, Bars: 189249
    Spread : 12, Swap long: 15, Swap short: -17.2, Commission: 11 USD
    Account: 100000 USD, Leverage: 500, Entry: 1 lots

Generator started...

  - Ascended: 5, Calculated: 15279, Opt: 5, Time: 18:00, Coll: 5
Collection exported: ./collections/Unknown_GBPJPY_H1_2017-03-21_str_5_(5172).json

  - Ascended: 10, Calculated: 30019, Opt: 10, Time: 30:54, Coll: 5
Collection exported: ./collections/Unknown_GBPJPY_H1_2017-03-21_str_5_(5195).json

RangeError: Maximum call stack size exceeded, Time: 51:45, Coll: 5
    at existsSync (node:fs:300:1)
    at addSuffixToJsonFile (C:\PortablePrograms\ExpressGenerator\bin\lib\collection.js:321:10)
    at addSuffixToJsonFile (C:\PortablePrograms\ExpressGenerator\bin\lib\collection.js:329:16)
    at addSuffixToJsonFile (C:\PortablePrograms\ExpressGenerator\bin\lib\collection.js:329:16)
    at addSuffixToJsonFile (C:\PortablePrograms\ExpressGenerator\bin\lib\collection.js:329:16)
    at addSuffixToJsonFile (C:\PortablePrograms\ExpressGenerator\bin\lib\collection.js:329:16)
    at addSuffixToJsonFile (C:\PortablePrograms\ExpressGenerator\bin\lib\collection.js:329:16)
    at addSuffixToJsonFile (C:\PortablePrograms\ExpressGenerator\bin\lib\collection.js:329:16)
    at addSuffixToJsonFile (C:\PortablePrograms\ExpressGenerator\bin\lib\collection.js:329:16)
    at addSuffixToJsonFile (C:\PortablePrograms\ExpressGenerator\bin\lib\collection.js:329:16)

I think it happens if there are a lot of collection files in the destination directory already (5000+) and it has to cycle through a loop to find the next free number. This can happen if EG runs for a long time and I hope you can fix it to still be able to handle that amount of files (and much more if its running unattended for weeks).

Please also see https://stackoverflow.com/questions/209 … e-exceeded (in case it helps anything).

Thanks for the fix in advance.

Unlock Your Edge in Automated Forex Strategy Development
Historical Forex Data Starting From 1987, 28 Pairs, M1, 99% Error-Free, Lifetime Free Updates

geektrader's Website

2 Reply by geektrader (edited by geektrader 2023-06-15 08:08:13)

  • From: Germany
  • Registered: 2010-04-05
  • Posts: 789

Re: Bug(?): "RangeError: Maximum call stack size exceeded"

After re-coding the "addSuffixToJsonFile" function that causes the stack overflow from:

function addSuffixToJsonFile(colPath) {
    if (!existsSync(colPath)) {
        return colPath;
    }

    const match = colPath.match(/\((\d+)\).json$/);
    if (match) {
        const index   = Number(match[1]);
        const newPath = colPath.replace(`(${index})`, `(${index + 1})`);
        return addSuffixToJsonFile(newPath);
    }

    const newPath = colPath.replace(".json", "_(1).json");
    return addSuffixToJsonFile(newPath);
}

to

function addSuffixToJsonFile(colPath) {
    // RegExp that matches the last number before the ".json" extension, with and without the index in parentheses
    var regex = /(?:_(\d+)(?:_\((\d+)\))?)?\.json$/;
    var newPath = colPath;

    // Check if the initial file exists
    if (!existsSync(newPath)) {
        return newPath;
    }

    // Loop until a non-existent file path is found
    while (existsSync(newPath)) {
        // Use the replace function with a callback to generate the new file path
        newPath = newPath.replace(regex, function (_, baseIndex, suffixIndex) {
            // If a suffix index is found, increment it by 1, otherwise set it to 1
            var newIndex = suffixIndex ? parseInt(suffixIndex, 10) + 1 : 1;
            // Return a new file path with the updated index in parentheses and the base index unchanged
            return '_' + baseIndex + '_(' + newIndex + ').json';
        });
    }

    // Return the non-existent file path
    return newPath;
}

The error has gone away. Reason: The error occurred because the function addSuffixToJsonFile used recursion to find a suitable file name, which caused a stack overflow for large numbers like in my case with 5000+ files in the directory. To fix this issue, I have converted the recursion to a loop, which will avoid the stack overflow issue.

Unlock Your Edge in Automated Forex Strategy Development
Historical Forex Data Starting From 1987, 28 Pairs, M1, 99% Error-Free, Lifetime Free Updates

geektrader's Website

  • Popov
  • Popov
  • Lead Developer
  • Offline
  • From: Bulgaria
  • Registered: 2006-11-30
  • Posts: 7,895

Re: Bug(?): "RangeError: Maximum call stack size exceeded"

You are right.

Node.js executes only a limited count of recursions sad.

I'll upload an update tomorrow.

Popov's Website