forex software

geektrader · 2022-10-19 03:03:59

geektrader
Licensed Member
Offline

From: Germany
Registered: 2010-04-05
Posts: 835

Topic: 100% Speed Boost for Express Generator

Hi,

so I spent the evening testing Express Generator on different JavaScript Runtimes. The default one that Popov uses is NodeJS from https://nodejs.org/en/download/. However, there are several more runtimes like Deno, Bun, and GraalVM Enterprise JS. Unfortunately, I couldn´t get Express Generator to run on either Deno or Bun, because they are not fully compatible with the node.js modules and NPM that Mr. Popov uses.

However, GraalVM Enterprise JS is praised to be 100% compatible with Node JS and since I´ve been using the Java runtime part of it with all the advanced compiler optimizations for many years already - always gaining about 15% in each and every Java application, I thought it´s time to also look into their Javascript Node JS compatible engine, which also uses all the optimizations, and see what it does for Express Generator.

The result is an almost 100% speed boost for Express Generator if running the exact same settings side-by-side between Node JS and the GraalVM Node JS implementation. Have a look at this video:

Here is how you can get this too:

1) Go to https://www.graalvm.org/downloads/

2) Grab the latest GraalVM Enterprise Edition (faster than the community edition, so make sure you use this one) and make sure to select the Java 17 version (their Java 19 is bugged and has an issue at the moment).
Note: in order to download the Enterprise Edition, you need to create a *free* account with Oracle. It will prompt you for your login credentials when you are trying to download, so just create a new account there, then try the download again.

3) Uncompress the ZIP file to a directory of your choice, go to the /bin subfolder, open a command prompt and run

gu install nodejs

4) Afterward you will have a new node.cmd in the /bin directory, which you can use to run Express Generator the usual way. Just make sure that your original Node JS runtime is not on your system-wide %PATH% variable because otherwise, Windows will always first execute the node.exe and ignore the node.cmd. It´s best if you simply specify the full path to the node.cmd. So, for example:

C:\PortablePrograms\Java\bin\node.cmd .\bin\gen.js --settings .\my-settings.ini

5) Enjoy the 100% speed boost :-)

geektrader · 2022-10-19 15:20:51

geektrader
Licensed Member
Offline

From: Germany
Registered: 2010-04-05
Posts: 835

Re: 100% Speed Boost for Express Generator

A little update: I´ve been running the above setup for the last 8 hours on 31 years of GBPJPY H1 data (189000 bars) on a Ryzen 5950x (PBO activated and slightly overclocked). Running 32 instances to max out the CPU fully, each instance has generated 700000 strategies, which equals 22.4 million strategies in just 8 hours, which will result in 67 million strategies per day and is more than double than running Express Generator on Node JS, as the GraalVM JS engine keeps on optimizing the code during runtime, which speeds it up further and further the longer it runs.

Accepted strategies per instance are currently 10, with a min R-Squared of 96(!), a min return/drawdown of 10, and a max stagnation of 900 days. This means 320 potential high-end strategies in just 8 hours. This is just insanity if compared to just running it on Node JS, not to talk about other platforms where just a few thousand strategies per hour are normal if maxing out all my cores. A paradise for the quant has become reality...

footon · 2022-10-20 10:14:13

footon
◄≡≡≡►
Offline

Registered: 2009-08-17
Posts: 2,969

Re: 100% Speed Boost for Express Generator

Do you think a door for brute-force approach is open now? Given the speed and the number of strats generated, it seems that random generation is wasting resources, would brute-force be quicker to the results?

Roughey · 2022-10-20 16:24:07

Roughey
Licensed Member
Offline

Registered: 2018-12-16
Posts: 346

Re: 100% Speed Boost for Express Generator

hmm i dont know why but with graal vm its is slower for me..node is 4x faster as graal

geektrader · 2022-10-20 17:05:32

geektrader
Licensed Member
Offline

From: Germany
Registered: 2010-04-05
Posts: 835

Re: 100% Speed Boost for Express Generator

Have you downloaded the Enterprise Edition of GraalVM? The Community Edition is much slower.

Possibly an older CPU with no AVX2 support? GraalVM makes heavy use of AVX2 compared to Node JS, if your CPU does not support that instruction set, it could be a lot slower indeed.

Also, make sure to really launch the GraalVM Javascript engine, as Node JS being in your path might prevent it. It´s best to use the full path to the node.exe of GraalVM, it resides in <GraalVM Root>\languages\nodejs\bin\node.exe.

Additionally, try to launch it with these parameters, which gave me another little speed boost as well (adjust your path to node.exe and your settings.ini from EG):

C:\PortablePrograms\Java\languages\nodejs\bin\node.exe ^
--native ^
--vm.XX:+DisableExplicitGC ^
--vm.Dgraal.Vectorization=true ^
--vm.Dgraal.ConsiderVectorizableLoops=true ^
--engine.InliningRecursionDepth=10000 ^
--engine.Mode=throughput ^
--experimental-options ^
.\bin\gen.js --settings .\lori-settings.ini

Also, make sure to let it run about 30 minutes, GraalVM is slower than Node JS in the first few minutes, but then catches up and surpasses Node JS, because the JIT compiler starts profiling, compiling, inlining code to binary code, which Node JS will never do.

If it still is slower, try to record a video when you compare both and upload it, then we can try to find out what the reason might be.

Roughey · 2022-10-20 17:09:34

Roughey
Licensed Member
Offline

Registered: 2018-12-16
Posts: 346

Re: 100% Speed Boost for Express Generator

geektrader wrote:

Have you downloaded the Enterprise Edition of GraalVM? The Community Edition is much slower.
Possibly an older CPU with no AVX2 support? GraalVM makes heavy use of AVX2 compared to Node JS, if your CPU does not support that instruction set, it could be a lot slower indeed.
Also, make sure to really launch the GraalVM Javascript engine, as Node JS being in your path might prevent it. It´s best to use the full path to the node.exe of GraalVM, it resides in <GraalVM Root>\languages\nodejs\bin\node.exe.
Additionally, try to launch it with these parameters, which gave me another little speed boost as well (adjust your path to node.exe and your settings.ini from EG):
C:\PortablePrograms\Java\languages\nodejs\bin\node.exe ^
--native ^
--vm.XX:+DisableExplicitGC ^
--vm.Dgraal.Vectorization=true ^
--vm.Dgraal.ConsiderVectorizableLoops=true ^
--engine.InliningRecursionDepth=10000 ^
--engine.Mode=throughput ^
--experimental-options ^
.\bin\gen.js --settings .\lori-settings.ini
Also, make sure to let it run about 30 minutes, GraalVM is slower than Node JS in the first few minutes, but then catches up and surpasses Node JS, because the JIT compiler starts profiling, compiling, inlining code to binary code, which Node JS will never do.
If it still is slower, try to record a video when you compare both and upload it, then we can try to find out what the reason might be.

ahh looks like i have the community edition...how can i deinstall it now?

geektrader · 2022-10-20 17:13:41

geektrader
Licensed Member
Offline

From: Germany
Registered: 2010-04-05
Posts: 835

Re: 100% Speed Boost for Express Generator

Just delete it´s folder or just install the EE in another folder. They can exist in parallel. Just make sure to call the right node.exe (the one of the EE, once you´ve installed the Javascript Engine via "gu install nodejs" as per my first post).

geektrader · 2022-10-20 17:15:16

geektrader
Licensed Member
Offline

From: Germany
Registered: 2010-04-05
Posts: 835

Re: 100% Speed Boost for Express Generator

Roughey wrote:

hmm i dont know why but with graal vm its is slower for me..node is 4x faster as graal

That would be nice, wouldn´t it? But the possible combinations with all indicators and all SL / TP values is a number with almost endless 0´s still, so that´s impossible even with 72 million strategies per day.

Roughey · 2022-10-20 18:45:20

Roughey
Licensed Member
Offline

Registered: 2018-12-16
Posts: 346

Re: 100% Speed Boost for Express Generator

which of the enterprise edition did i have to download..the first in the list? This Oracle GraalVM Enterprise Edition JDK or Oracle GraalVM Enterprise Edition JavaScript Runtime Plugin or Oracle GraalVM Enterprise Edition Node.js Runtime Plugin

timelleston · 2022-10-21 05:13:57

timelleston
Licensed Member
Offline

From: Australia
Registered: 2022-07-22
Posts: 127

Re: 100% Speed Boost for Express Generator

I used the first one in the list and ran the install on both Mac and (virtual) PC.

I ran a side-by-side test of node and graalvm-node but I got different results...the enterprise graalvm was a lot slower for me (on my mac). And it was terrible in Parallels (not surprised by that though)

Roughey wrote:

which of the enterprise edition did i have to download..the first in the list? This Oracle GraalVM Enterprise Edition JDK or Oracle GraalVM Enterprise Edition JavaScript Runtime Plugin or Oracle GraalVM Enterprise Edition Node.js Runtime Plugin

geektrader · 2022-10-21 06:16:54

geektrader
Licensed Member
Offline

From: Germany
Registered: 2010-04-05
Posts: 835

Re: 100% Speed Boost for Express Generator

The exact download link that I am using is https://download.oracle.com/otn/utiliti … 22.3.0.zip You´ll still need a free Oracle account to get it. Afterward, you need to run the command mentioned in post #1 to install the Javascript Engine within it.

I can´t speak for Mac or any other platform, I am on Windows Server 2022 only, AMD Ryzen 5950X CPU, 128GB DDR4-3800 RAM. These are my results, just retested and even with different NodeJS versions the GraalVM Javascript Engine smokes them all, even just 6 minutes in it's ahead almost 50% already, which extends to 100%+ the longer they run aside, at least on my config.

geektrader · 2022-10-21 06:24:19

geektrader
Licensed Member
Offline

From: Germany
Registered: 2010-04-05
Posts: 835

Re: 100% Speed Boost for Express Generator

20 minutes in, GraalVM is 55% ahead now already.

And on top of that, GraalVM uses half the RAM that both NodeJS instances use, as well as only 60% of the CPU (3% vs 5%) that NodeJS uses, which is crazy given that it is already 55% ahead of the both NodeJS instances too.

geektrader · 2022-10-24 17:07:48

geektrader
Licensed Member
Offline

From: Germany
Registered: 2010-04-05
Posts: 835

Re: 100% Speed Boost for Express Generator

Pure speed insanity! Mr. Popov´s excellent work, combined with GraalVM´s Enterprise Javascript engine is a quant´s heaven.

forex software

100% Speed Boost for Express Generator

Posts: 13

1 Topic by geektrader 2022-10-19 03:03:59 (edited by geektrader 2022-10-19 04:43:02)

Topic: 100% Speed Boost for Express Generator

2 Reply by geektrader 2022-10-19 15:20:51 (edited by geektrader 2022-10-19 16:40:26)

Re: 100% Speed Boost for Express Generator

3 Reply by footon 2022-10-20 10:14:13

Re: 100% Speed Boost for Express Generator

4 Reply by Roughey 2022-10-20 16:24:07

Re: 100% Speed Boost for Express Generator

5 Reply by geektrader 2022-10-20 17:05:32

Re: 100% Speed Boost for Express Generator

6 Reply by Roughey 2022-10-20 17:09:34

Re: 100% Speed Boost for Express Generator

7 Reply by geektrader 2022-10-20 17:13:41 (edited by geektrader 2022-10-20 17:14:27)

Re: 100% Speed Boost for Express Generator

8 Reply by geektrader 2022-10-20 17:15:16

Re: 100% Speed Boost for Express Generator

9 Reply by Roughey 2022-10-20 18:45:20

Re: 100% Speed Boost for Express Generator

10 Reply by timelleston 2022-10-21 05:13:57

Re: 100% Speed Boost for Express Generator

11 Reply by geektrader 2022-10-21 06:16:54 (edited by geektrader 2022-10-21 06:20:08)

Re: 100% Speed Boost for Express Generator

12 Reply by geektrader 2022-10-21 06:24:19 (edited by geektrader 2022-10-21 06:28:02)

Re: 100% Speed Boost for Express Generator

13 Reply by geektrader 2022-10-24 17:07:48

Re: 100% Speed Boost for Express Generator

Posts: 13

Similar topics in this forum