forex software

Create and Test Forex Strategies

forex software

Skip to forum content

Forex Software

Create and Test Forex Strategies

You are not logged in. Please login or register.


(Page 1 of 2)

Forex Software → Express Generator → Express Generator - Design

Pages 1 2 Next

You must login or register to post a reply

RSS topic feed

Posts: 1 to 25 of 40

Topic: Express Generator - Design

Hello Traders,

I've been working on a new backtesting engine and a strategy generator for the last six months.
Happily, I achieved some success and already have a working prototype.

Goals:
- work offline on Windows, Linux, and Mac
- allow automation via simple options, settings, or a script file
- accept data as FSB Pro and EA Studio
- export strategy collections to FSB Pro and EA Studio
- fast and secure


I'll present the Express Generator prototype online in a Zoom meeting on Friday, Aug 26, 2022 at 10:00 UTC

Meeting link: removed



Express Generator will be commercial software designed to work as an addition to FSB Pro and EA Studio.
Of course, all Ultimate Pack users will receive it as a free upgrade.


You can find here some design decisions: I'm Yet to Build the Best Backtester and Generator


Have fun and Trade Safe!

2 (edited by geektrader 2022-08-22 12:18:41)

Re: Express Generator - Design

That sounds like a dream coming true! Automatable workflow by scripts, offline, possibly even usage of multi-CPU-cores without having to launch several instances to max out modern CPUs? Eager to see the presentation on Friday :-)

Re: Express Generator - Design

geektrader wrote:

That sounds like a dream coming true! Automatable workflow by scripts, offline, possibly even usage of multi-CPU-cores without having to launch several instances to max out modern CPUs? Eager to see the presentation on Friday :-)


"Sounds like a dream"

I agree -- I see great potential.  I've kind of adapted a workflow recipe that I repeat each weekend in preparation for the next week.  It's not much work and I enjoy using the software -- it satisfies my urge to remain "engaged" and I no longer spend time fiddling with individual strategies.  If Express Generator allows us to automate customizable workflows that can be launched by script at any time this would be yuuuge. 

As an aside -- I can't see new users taking advantage of Express Generator right away because one would first need to be familiar with FSB Pro and / or EA Studio and come up with a desired workflow before even thinking about automating a script.

Re: Express Generator - Design

> As an aside -- I can't see new users taking advantage of Express Generator right away

Yes, you are right.

The Express Generator is for people that are not afraid to write commands in a black rectangle or to change numbers in text files with an odd extension.

However, if a geek wants to run 8 Generators on EURUSD at 01:30 in the morning. Then validate all of the exported collections on GBPUSD and merge them into another collection. And to repeat this on 10 different currency pairs only by settings a Cron job or a Task schedule... Just to see what their performance will be in a month... Big fun is coming.

Some very early examples:

Generating a new collection

https://image-holder.forexsb.com/store/express-generator-generating-thumb.png


Recalculated 4 Collections

https://image-holder.forexsb.com/store/express-generator-recalculate-thumb.png


Multiple scripts run with a click. Note: network = 0, disk = 0 smile

https://image-holder.forexsb.com/store/express-generator-multiple-scripts-thumb.png


...

Please join our meeting on Friday. Some surprises are always possible.

5 (edited by sleytus 2022-08-24 02:26:34)

Re: Express Generator - Design

Popov wrote:

>.

However, if a geek wants to run 8 Generators on EURUSD at 01:30 in the morning. Then validate all of the exported collections on GBPUSD and merge them into another collection. And to repeat this on 10 different currency pairs only by settings a Cron job or a Task schedule... Just to see what their performance will be in a month... Big fun is coming.

Please join our meeting on Friday. Some surprises are always possible.


Oooooh, aaaaah -- I like it!!   I'm now addicted to EA Studio, but I've run into the problem (as you pointed out in another thread) that the page has to be active (and visible) otherwise the browser puts it to sleep.  To get around this problem I have resorted to "baby-sitting" multiple instances of the browser each running EA Studio.  However, now with Express Generator, once one has settled on a desired workflow then being able to launch a script from the command line is a "game changer".

I will try to join the meeting on Friday -- Popov, it would be nice to chat with you again.  However, 10:00 UTC is 3:00a in my PST timezone.  Not really a problem since sleep is over-rated...  smile

Re: Express Generator - Design

Nice stuff coming, eager to test it! What I am really interested in is the speed. Right now, EA Studios beats it all if being run on Thorium AVX2 builds from here: https://github.com/Alex313031/Thorium-AVX2/releases I wonder if Express Generator will be faster than that...

Re: Express Generator - Design

We will see very soon.

I assume Express Generator will be faster than EA Studio, at least for now. However, for sure, it will consume less memory, also for now.

The biggest advantage of Express Generator is the possibility to automate the workflow.
Of course, we still need FSB Pro or EA Studio to look at the strategies and export Expert Advisors.

I hope to be able to provide a beta version for testing tomorrow at the meeting.

What is achieved for now:
- most of the Generator options are implemented
- account settings
- most common Acceptance Criteria
- all of the Strategy properties are implemented
- the Backtester calculates the swaps, the spread, and the account exchange rate
- 24 of the 37 EA Studio indicators are implemented
- we have a 100% backtest match between Express Generator and EA Studio
- a possibility to control the Generator by settings files and command options
- importing EA Studio collections and exporting collections for EA Studio and FSB Pro

What to be done:
- 100% indicators support
- the possibility of choosing which indicators to be used in the Generator
- Data Horizon, Optimizer, Normalizer, Monte Carlo
- Out of Sample
- calculating commissions
- sorting methods for the Collection
- Correlation analysis
...

Something I'll try to do until tomorrow:
- a script for downloading Premium Data,
- possibility to set Timezone offset
- updating the FX exchange rates

Re: Express Generator - Design

I'm glad to see that downloading Premium Data will be available shortly.  There was a time when I thought broker data was important -- but no longer.  The Premium Data is good enough and now I feel the more transparent and easy to download data then the simpler will be the work flow.

And from a purely selfish perspective, I would prioritize the "To Be Done" as follows:
- Data Horizon, Optimizer (I don't use Normalizer or Monte Carlo)
- Out of Sample
- 100% indicators support

As long as I can import a collection back into EA Studio then the other collection-related features are less important to me.  Also, choosing indicators seems kind of like overkill.  Does anyone really have any evidence that some indicators work better than others or that certain indicators should be avoided?  It seems to me that if the stats look good then it shouldn't concern the user which indicators are being used.  I understand why that would be important for FSB Pro because the quality and performance of Custom indicators can vary, but less so for EA Studio.

Popov -- this isn't intended to push you in any direction.  In the event that it is of interest to you to have another set of eyes priortize features from a user perspective, then I decided to take the opportunity to volunteer mine...  smile

I look forward to learning more tomorrow...

Re: Express Generator - Design

Premium Data import is read and work very well.

https://image-holder.forexsb.com/store/fetch-premium-data-thumb.png

Re: Express Generator - Design

Added Data Horizon


Data Horizon options

; Data Horizon
max_data_bars   = 100000

use_data_start  = true
data_start      = 01 Jan 2020 GMT

use_data_end    = true
data_end        = 01 May 2022 GMT

https://image-holder.forexsb.com/store/express-generator-data-horizon-thumb.png

A note regarding settings the dates.

If you set the date like this: "data_start = 01 Jan 2020", Node.js will use the Time zone of the local computer.

If you want to use GMT, you have to set the date as this: "data_start = 01 Jan 2020 GMT"

Acceptable format is also: "data_start = 01 Jan 2020 00:00 GMT". You can use also an exact hour if you want.

11 (edited by Popov 2022-08-26 19:17:32)

Re: Express Generator - Design

Very nice -- being able to control data horizon opens new possibilities.

Two questions:

1. Is there an "official" link to download the latest / greatest version of Express Generator?

2. To be clear -- this only affects the start / end dates of the data fetching.  It doesn't touch the OHLC time stamps, right?

Thank you, Popov...

Re: Express Generator - Design

> 1. Is there an "official" link to download the latest / greatest version of Express Generator?

Please use the link I shared in the meeting.

> 2. To be clear -- this only affects the start / end dates of the data fetching.  It doesn't touch the OHLC time stamps, right?

Data Horizon options work for the Generator only. It cuts the data according to the used options.
You can see how it works in EA Studio.

You can use "use-data-start" or "use-data-end" or both together.


https://forexsb.com/wiki/_media/eas-guide/eas-data-horizon.png

Re: Express Generator - Design

Very clear -- thanks for the graphic...

14 (edited by sammjacks 2022-08-27 00:37:40)

Re: Express Generator - Design

Hey Popov,

Getting express gen going and it is awesome!!

Got an issue trying to use data other than Premium though. When I try to validate using data from a different source i get the error shown in the screenshot

Post's attachments

express_gen_error1.PNG 39.78 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

Re: Express Generator - Design

Noticed that this text isnt in the file exported from my broker: "dataId":{"source":"Premium Data","symbol":"AUDNZD","period":15}

I could be using an old export script though so i will check if thats the cause

Re: Express Generator - Design

Hello sammjacks,

Would you consider sharing your my-settings1.ini file -- it would help me get a jump start on testing, as well.  And, if not, that is perfectly understandable...


sammjacks wrote:

Hey Popov,

Getting express gen going and it is awesome!!

Got an issue trying to use data other than Premium though. When I try to validate using data from a different source i get the error shown in the screenshot

Re: Express Generator - Design

Adding this text resolved the problem. Perhaps the quickest solve is to update the export script to include this text at the start of the json file?

Re: Express Generator - Design

Hey Sleythus, not sure how useful my files are as I am just starting to play around with setting things up myself but Ive sent them to you

Re: Express Generator - Design

> Noticed that this text isnt in the file exported from my broker: "dataId":

This issue is fixed. Thak you for the report!
Improved the work of `fetch --period`.

...

I found another problem when reading MT exported files.

undefined:1
��{
^

It happens because MT exports files with UTF BOM.
If you have this problem, please edit the Data Export script.

Replace line:

    int file_handle = FileOpen(fileName, FILE_WRITE|FILE_IS_TEXT);

with:

    int file_handle = FileOpen(fileName, FILE_WRITE|FILE_ANSI);

Re: Express Generator - Design

Great, cheers Popov.

I have a general question regarding time settings. They are not implemented in express gen yet, which would be really useful for me.

I usually set my time settings to 01:00 - 24:00 in order to avoid opening positions during the high spread time (where the spread can be around 100 which would destroy profits).

I am getting a lot of strategies that perform very well for when I let them trade 00:00 - 24:00 but then as soon as I change the settings 01:00 - 24:00 the equity curves change completely to usually perform very poorly.

It also obviously makes it much harder to find strategies with these time settings.

Any discussion on this topic or different perspective would be great as I am currently a bit unsure if my approach is sensible.

Re: Express Generator - Design

Also I have a question for sleytus.

You mention above that you use optimizer but not Monte Carlo or normalizer. Just wondering how you approach minimizing the risk of overfitting with these settings.

I avoid the optimizer completely but always use the Monte carlo, open to different perspectives though

Re: Express Generator - Design

Interestingly for a lot of the strategies (without a take profit), if I only let them open trades during 00:00 - 01:00 and set the spread to 100 they do really well.

23 (edited by sammjacks 2022-08-27 08:25:59)

Re: Express Generator - Design

Ive gotten out an absolute tonne of strategies today Popov, and lots of really good looking ones ones I have further filtered in EA studio :-). This is actually AMAZING!!!

Re: Express Generator - Design

sammjacks wrote:

Ive gotten out an absolute tonne of strategies today Popov, and lots of really good looking ones ones I have further filtered in EA studio :-). This is actually AMAZING!!!

I'm glad you find it useful!

Please make a video of how you use the program and with what settings. It will also be helpful for the other traders and me because I'll see your way and get ideas of how to make it even better.

Re: Express Generator - Design

Hey sammjacks...

Though your question was addressed to Popov I can't help adding my two cents...

With regards to time settings and time zones, I would tread carefully.  If your goal is to avoid trading at high spread times then the times to avoid must be based on your broker's server time.  And this could become much more complicated than what appears on the surface.

The OHLC data that you fetch may come from Premium or some other source -- like MetaTrader.  If it comes from MetaTrader it may come from the same broker that the portfolio EA trades on or, perhaps, another account.  Timestamps are embedded in the OHLC data -- and you certainly don't want to change those.  The military times that you wish to avoid will be different depending on whether you think in terms of the server's time, Premium Data's time, MetaTrader's terminal time.

My suggestion would be to **always** think in terms of the server's time.  If the data comes from a source who's server time is different than the server's time where the portfolio EA executes then you will need to compensate for that -- and you really need to put on your thinking cap to figure out exactly how you will do that.

A better solution is to take advantage of the MaxSpread setting under Data / Symbol Settings -- e.g. 40 points is a reasonable value.  If the current spread is greater than 40 points then a signal would be ignored.  Easy-peasy -- no need to worry about converting from one time zone to another...




sammjacks wrote:

Great, cheers Popov.

I have a general question regarding time settings. They are not implemented in express gen yet, which would be really useful for me.

I usually set my time settings to 01:00 - 24:00 in order to avoid opening positions during the high spread time (where the spread can be around 100 which would destroy profits).

I am getting a lot of strategies that perform very well for when I let them trade 00:00 - 24:00 but then as soon as I change the settings 01:00 - 24:00 the equity curves change completely to usually perform very poorly.

It also obviously makes it much harder to find strategies with these time settings.

Any discussion on this topic or different perspective would be great as I am currently a bit unsure if my approach is sensible.

Posts: 1 to 25 of 40

Pages 1 2 Next

You must login or register to post a reply

Forex Software → Express Generator → Express Generator - Design

Similar topics in this forum