forex software

Create and Test Forex Strategies

forex software

Skip to forum content

Forex Software

Create and Test Forex Strategies

You are not logged in. Please login or register.


(Page 1 of 3)

Forex Software → Express Generator → Express Generator - Wish List

Pages 1 2 3 Next

You must login or register to post a reply

RSS topic feed

Posts: 1 to 25 of 58

Topic: Express Generator - Wish List

Please post here your ideas for or improvements or new features

Re: Express Generator - Wish List

Hi Mr. Popov,

I just had another idea for people that run Express Generator 24/7. How about an auto-save feature when collection grows bigger than X? This could be done via a new setting:

auto_save_collection = true
auto_save_size = 100

With the above settings Express Generator would save the collection to disk once the collection reaches 100 strategies, reset the collection in memory to 0 and keeps on running. Once another 100 are reached, it does the same again. This way there would be no more intervention needed at all, no restarting of EG required and all we have to do is to collect the saved collections. Perfect for running EG 24/7 without any intervention. I hope this can be done, it would be heaven -:)

Re: Express Generator - Wish List

Mr Popov - this might have been covered elsewhere, but, do you have any future plans to incorporate a portfolio creation part of the workflow process and an Export to mt4/5 capability, directly in Express Generator?

I have everything scripted at the moment, from downloading the latest data from my broker, to generating strategies.  As soon as its exported from EAS, I have a script to add some code to the .mq5 file, and then I compile it on the fly, and push it out to my VPS.

The manual part at the moment is obviously loading it back into EAS and then exporting as a Portfolio.

Just curious...?

Thanks
Tim

4 (edited by geektrader 2022-10-21 08:17:48)

Re: Express Generator - Wish List

I´ve just tested the new CTRL+S feature to save the collection. It works, but it doesn´t clear the collection in memory after saving, which means that there are duplicate strategies on the hard drive after each saving.

  - Ascended: 1, Calculated: 108054, Time: 34:00, Coll: 1
Collection exported: ./collections/Coll_Unknown_GBPJPY_H1_2017-03-22_str_1_(1).json

  - Ascended: 1, Calculated: 109164, Time: 34:21, Coll: 1
Collection exported: ./collections/Coll_Unknown_GBPJPY_H1_2017-03-22_str_1_(2).json

^ After saving the first time, the already saved strategy still remains in memory and is saved again upon next time.

I would suggest clearing the strategies in memory after saving, to avoid duplicates on the hard drive, or adding an extra shortcut like CTRL+X, which saves AND clears the strategies in memory after saving, so that the next save will only have the new strategies that have been generated after the last save.

Otherwise, we still need to restart Express Generator after saving the collection, because we will always have duplicate strategies. The idea was to *not* have to do this, but keep it running 24/7 without having to close/restart it, and just save the collection for further processing, while Express Generator keeps on running all the time. If it doesn´t clear the strategies in memory after saving though, that won´t work and it needs to be restarted to clear the collection in memory.

If adding the auto-save all X strategies feature, it would be great if you could handle it the same there, possibly via an extra parameter like:

auto_save_collection = true //Auto-saving on/off
auto_save_threshold = 100 //After how many strategies should the collection be saved
auto_save_clear_collection_after_saving = true //Clear strategies in memory after saving, to avoid duplicates when saving again

5 (edited by geektrader 2022-10-20 08:55:52)

Re: Express Generator - Wish List

Mr. Popov, can you possibly add a keyboard shortcut for Express Generator that writes out the current collection to disk, empties the collection in memory, but doesn´t stop the strategy generation, and keeps on running? I want to run Express Generator 24 hours on a dedicated machine, but don´t want to cancel and relaunch the application just to save the current collection to disk. A simple keyboard shortcut like CTRL + S to save the current collection to disk, clear the collection in memory, and keep EG running would be absolutely great. Thanks.

Re: Express Generator - Wish List

> A simple keyboard shortcut like CTRL + S to save the current collection to disk but keep EG running would be absolutely great.

I'm not sure if this is possible for a console application. However, this is an interesting idea. If the OS sends a signal to the process on a keystroke, it may work.

If it works, it gives us a lot of opportunities. Thank you!

7 (edited by geektrader 2022-10-20 09:04:51)

Re: Express Generator - Wish List

It would be great if you could try this. The main problem I am currently having is that sometimes the CTRL+C hard stops Express Generator before it can actually write the collection to disk, so sometimes, collections don´t get saved at all. It doesn´t happen all the time but in 5% of the cases. I hope that a keyboard shortcut solution will be possible.

Edit: it is possible: https://thisdavej.com/making-interactiv … ss-events/

And this package looks very good: https://www.npmjs.com/package/hotkeys-js

Re: Express Generator - Wish List

> I´ve just tested the new CTRL+S feature to save the collection. It works, but it doesn´t clear the collection in memory after saving, which means that there are duplicate strategies on the hard drive after each saving.

This is intentional. If we clear the EG current collection, you will loose the effect of collection pruning and correlation analysis.

Clearing EG collection on save is absolutely equal to stopping and re-running it.

A possible solution to your request is exporting a temp collection.
An option for override the collection instead of adding a suffix will resolve the problem with duplicated collections.

I'm thinking to add an instance ID and a possibility to create Collections folders dynamically.

So, if we run 10 instances on the same market, each instance to maintain its own temp collection.
The temp export should happen at a predefined time. Let's say every minute.

9 (edited by geektrader 2022-10-21 09:27:30)

Re: Express Generator - Wish List

Popov wrote:

> I´ve just tested the new CTRL+S feature to save the collection. It works, but it doesn´t clear the collection in memory after saving, which means that there are duplicate strategies on the hard drive after each saving.

This is intentional. If we clear the EG current collection, you will loose the effect of collection pruning and correlation analysis.

Clearing EG collection on save is absolutely equal to stopping and re-running it.

A possible solution to your request is exporting a temp collection.
An option for override the collection instead of adding a suffix will resolve the problem with duplicated collections.

I'm thinking to add an instance ID and a possibility to create Collections folders dynamically.

So, if we run 10 instances on the same market, each instance to maintain its own temp collection.
The temp export should happen at a predefined time. Let's say every minute.


I totally understand that we are losing the correlation analysis if clearing the memory, but this can be done (rather quick) in EA Studio afterward still and will stop saving duplicates to disk, which is much more important to me. I would be happy to live without the correlation analysis, as my collections are just 10 to 15 strategies in 24 hours because of strict acceptance criteria, so it´s not a problem at all. I would rather like to have it clear the memory after saving. If you make it optional via an extra keyboard shortcut like CTRL+X (save and clear memory), and keep the CTRL+S with the current behavior, this would be great, so everyone can decide what they prefer to do. The same for the auto-saving feature, just make it optional to clear the collection in memory and everyone should be happy :-)

As for your idea with ID´s and temp files that are overwritten in the same file, I understand that too and it´s an idea as well, but that would still not fit my way of working either. What I really intend to do is simply let EG run 24/7 and save the collections to disk when I feel like it (via CTRL+S or via auto-save after X strategies are in the collection feature), then load the collections into EA Studio and do further processing. If EG would overwrite the same file, I again have duplicates that I already processed in EA Studio, and the only way to avoid it would be to restart EG after saving every time to clear the strategy memory, which I want to avoid.

Hence it would be great if you could add the save-and-clear feature still, it would ease my workflow a lot.

Re: Express Generator - Wish List

Something completely different: while we talk about the correlation analysis: my benchmarks seems to indicate that the correlation analysis is done for every strategy, even if it doesn´t pass the acceptance criteria. Is this correct? Because each time there is at least 1 strategy in the collection, the strategy generation slows down a little bit compared to instances that don´t have a strategy in the collection yet. I am not sure how you´ve implemented the correlation analysis, but it would make the most sense if it would just be done after a strategy has passed the acceptance criteria. It could be something else though, but it´s a pattern I am seeing that there is a slowdown as soon as there is 1 strategy in the collection compared to instances that don´t have a strategy in the collection yet and I wonder why that is.

Re: Express Generator - Wish List

> my benchmarks seems to indicate that the correlation analysis is done for every strategy, even if it doesn´t pass the acceptance criteria. Is this correct?

it is not true. Correlation analysis is computed after a strategy passes all the enabled acceptance criteria and is "pushed" to the Collection. Only after that the collection checks the new candidate record against all the others.

The effect you noticed sounds to me like R-squared. Express Generator uses one-pass backtesting algorithm, which also calculates all the stats parameters along the way except R-squared. R-squared needs 2 passes (or 3, depending how we count) and is slower. That's why I do it "only" if the other Acceptance Criteria are passed.
This is different (and more effective) than EA Studio. EA Studio calculates all stats params even if they are not used in Acceptance Criteria.

Express Generator is more effective and doesn't calculate R-squared if it is not used. Even if it is enabled, it is calculated only when the rest AC pass.

Most probably it happens that the Generator finds a strategy that pass all AC except R-squared. The Generator incrementally modifies it searching the next one and the new tries pass AC, which means they reach R-squared.

12 (edited by geektrader 2022-10-21 21:31:25)

Re: Express Generator - Wish List

Thanks for the explanation, that´s very smart.

All is good over here, have been enjoying EG all day:

Express Generator v1.4

Loaded: Unknown GBPJPY H1, bars: 189273
From: 1986-12-19 14:00, to: 2017-03-22 00:00

Generator started...

  - Ascended: 1, Calculated: 164725, Time: 01:52:17, Coll: 1
Collection exported: ./collections/Coll_Unknown_GBPJPY_H1_2017-03-22_str_1_(27).json

  - Ascended: 2, Calculated: 381589, Time: 04:03:10, Coll: 1
Collection exported: ./collections/Coll_Unknown_GBPJPY_H1_2017-03-22_str_1_(57).json

  - Ascended: 3, Calculated: 677292, Time: 07:00:45, Coll: 1
Collection exported: ./collections/Coll_Unknown_GBPJPY_H1_2017-03-22_str_1_(103).json

  - Ascended: 4, Calculated: 677293, Time: 07:00:45, Coll: 1
Collection exported: ./collections/Coll_Unknown_GBPJPY_H1_2017-03-22_str_1_(104).json

  - Ascended: 5, Calculated: 677771, Time: 07:01:03, Coll: 1
Collection exported: ./collections/Coll_Unknown_GBPJPY_H1_2017-03-22_str_1_(105).json

  - Ascended: 6, Calculated: 677772, Time: 07:01:03, Coll: 1
Collection exported: ./collections/Coll_Unknown_GBPJPY_H1_2017-03-22_str_1_(106).json

  - Ascended: 7, Calculated: 677773, Time: 07:01:03, Coll: 1
Collection exported: ./collections/Coll_Unknown_GBPJPY_H1_2017-03-22_str_1_(107).json

The auto-saving works perfectly, I´ve set it to "1", so each strategy goes into 1 JSON "collection". Now I can just leave it running 24/7, no restarting needed, and can just look at the new files that flow in in EA Studio for further processing. I am reaching 72 million raw strategies per day on 189000 bars of data, that´s just crazy and is the FASTEST generator I´ve ever used in my life. Thanks so much for creating Express Generator, it´s a game-changer!

Have a nice weekend :-)

Re: Express Generator - Wish List

i dont know how you do that but i never fetch so much bars

14 (edited by geektrader 2022-10-22 14:26:11)

Re: Express Generator - Wish List

It's my own data that I've collected, cleaned (spikes, holes, fat finger prices, etc.) and maintained over the last 12 years, I am not using the Premium Data that comes with EG / EAS.

Re: Express Generator - Wish List

geektrader wrote:

It's my own data that I've collected, cleaned (spikes, holes, fat finger prices, etc.) and maintained over the last 12 years, I am not using the Premium Data that comes with EG / EAS.

schick mal rüber die daten ;-)

Re: Express Generator - Wish List

I´ve sent you a PM.

17 (edited by geektrader 2022-10-24 16:54:44)

Re: Express Generator - Wish List

timelleston wrote:

Mr Popov - this might have been covered elsewhere, but, do you have any future plans to incorporate a portfolio creation part of the workflow process and an Export to mt4/5 capability, directly in Express Generator?

I have everything scripted at the moment, from downloading the latest data from my broker, to generating strategies.  As soon as its exported from EAS, I have a script to add some code to the .mq5 file, and then I compile it on the fly, and push it out to my VPS.

The manual part at the moment is obviously loading it back into EAS and then exporting as a Portfolio.

Just curious...?

Thanks
Tim

I would be absolutely interested in this as well. It could be an optional variable in the settings.ini to export MQ5/MQ4 code directly instead of JSON collections. Of course, it would be on a single strategy base, so each time EG finds a strategy, it would export it right away as MQ5/MQ4 code (user-selectable in which of the both formats). This way we have the strategies directly out of EG and can start trading them in Metatrader right away. They can still be loaded into EA Studio via the Validator if Mr. Popov includes the custom code at the end of each MQ4/MQ5 file for EA Studio as it is already the case now for every strategy exported from EA Studio (which is a GREAT feature that I always loved). Of course this only makes sense with strict acceptance criteria, as we otherwise would be swamped with hundreds of MQ5/MQ4 files. However, personally, I am working with extremely strict acceptance criteria and know that the strategies that EG finds with these, are pretty much ready for trading. So if EG would save them directly as MQ5/MQ4, that would automate things even more for a case like this.

Re: Express Generator - Wish List

Override indicators in my-XYZ setting file  ( something like preset indicators)

Re: Express Generator - Wish List

Is It applicable to scan more than price data file (eurusd15 , audusd15 , usdcad15 ) with one generator file and output one coll. file

What i mean " trading journal will contain same setup different pairs . it will simulate what happen in live trading .

Regards

20 (edited by geektrader 2022-10-26 00:15:27)

Re: Express Generator - Wish List

I´d like to see some advanced statistics about the data during generating. Currently it´s:

Loaded: Unknown GBPUSD H1, bars: 189224
From: 1986-12-23 14:00, to: 2017-03-22 00:00

But it would be great if it also would show the spread, commission, and swaps. This way we can make sure we haven´t messed up anything with these very important settings before possibly running the generator with wrong values for hours. So it could look like this:

Loaded: Unknown GBPUSD H1, bars: 189224
From: 1986-12-23 14:00, to: 2017-03-22 00:00
Spread: 10 Points, Commission: 11 Points, Swap Long: X Points, Swap Short: X Points

Thanks.

Re: Express Generator - Wish List

ahmedalhoseny wrote:

Override indicators in my-XYZ setting file  ( something like preset indicators)


Now the available option for indicators true or false .  is it applicable to be :


;; true , false , always use open logic , always use close logic , always use both logics

22 (edited by geektrader 2022-10-28 23:08:40)

Re: Express Generator - Wish List

Feature request: it would be great if we could specify the max period limit for indicators when generating. Right now (and just like in EA Studio), the max indicator periods seem to be limited to 200. From other platforms I´ve created strategies in, I can say that many of my best strategies use periods of up to 500. So it would be great if the max limit during random generating could be specified via the settings.ini.

For further processing in EA Studio, it would also mean that the 200 limit needs to be lifted there as well.

Re: Express Generator - Wish List

Feature request:
Is it possible to enable both re-validate and generate into the same output collection?

Case in mind:  I now have lots of previously generated collections, that I'd like to re-validate using the latest data file. So I use the --input parameter (as a folder) and then once it's re-validated them, start generating new strategies into that collection, overwriting as usual.

Thanks

Re: Express Generator - Wish List

timelleston wrote:

Feature request:
Is it possible to enable both re-validate and generate into the same output collection?

Case in mind:  I now have lots of previously generated collections, that I'd like to re-validate using the latest data file. So I use the --input parameter (as a folder) and then once it's re-validated them, start generating new strategies into that collection, overwriting as usual.

Thanks

I like this idea also, it would be good to be able to read a collection in and then generate on top of it to avoid correlation.

Seems like an option to the gen command to first upload a collection prior to generating would do the trick :-)

25 (edited by geektrader 2022-11-02 05:02:35)

Re: Express Generator - Wish List

Thanks, Mr. Popov, much appreciated.

I was wondering if you have any news on the max indicator periods as in my post here: https://forexsb.com/forum/post/71955/#p71955 . Seeing the strategies that Express Generator creates and looking at the Javascript code of EG, it seems like the max period for *any* indicator is currently being limited to 45 and 50 during random generating. That is a good bit too short from my experience because most of my best strategies use periods of nearly 200 and often more actually. Here is an example of how good such a strategy with higher periods / deviation values (another topic as Express Generator also just seems to go up to 2 for the randomization of  the deviation of the Bolinger Bands):

https://i.postimg.cc/rVybSb5H/Untitled.png

We are missing many strategies if the periods are limited to such low values during random generating and EG should at least allow randomization in the range of up to 200 for all indicator periods. It could also be made configurable per indicator, the same for the min/max ranges for indicators like Bolinger bands, standard deviation, etc. I am also wondering if Express Generator uses the auto-adjustment for the right range value like you´ve described here https://forexsb.com/forum/post/70341/#p70341 for EA Studio, or if it does not have this feature yet.

Posts: 1 to 25 of 58

Pages 1 2 3 Next

You must login or register to post a reply

Forex Software → Express Generator → Express Generator - Wish List

Similar topics in this forum