forex software

Create and Test Forex Strategies

forex software

Skip to forum content

Forex Software

Create and Test Forex Strategies

You are not logged in. Please login or register.


Forex Software → Express Generator → Removing most of data from datasets

Pages 1

You must login or register to post a reply

RSS topic feed

Posts: 16

Topic: Removing most of data from datasets

The Data Export script created for MT4 to produce a JSON file from broker data for use in express generator works beautifully. Is there any chance that you would be willing to add a user input variable in the script where the user could designate a specific intraday block of time desired and all other data would not be exported?

I have found that when I run my EA's only during the volatile periods intraday specific to a particular pair, the performance improves significantly. It seems to. logically flow that if the strategies were generated based on the same intraday blocks of time their performance might also improve.

I would be very interested in your thoughts.

Daniel

Re: Removing most of data from datasets

dlouisbriggs wrote:

The Data Export script created for MT4 to produce a JSON file from broker data for use in express generator works beautifully. Is there any chance that you would be willing to add a user input variable in the script where the user could designate a specific intraday block of time desired and all other data would not be exported?

I have found that when I run my EA's only during the volatile periods intraday specific to a particular pair, the performance improves significantly. It seems to. logically flow that if the strategies were generated based on the same intraday blocks of time their performance might also improve.

I would be very interested in your thoughts.

Daniel

You can do this in Express Generator or EA Studio by setting the open and close times of the trading session when creating your strategies, no need to limit the data export.

Re: Removing most of data from datasets

The setting that you are referring to doesn't seem to prohibit trading to certain blocks of time. The first screenshot shows the setting is set for 07:00 to 12:00 and the second screenshot showing the collection performance report which shows trades being entered outside of the permitted time block.

Post's attachments

Screenshot 2023-10-04 at 6.01.11 PM.png 92.97 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

Re: Removing most of data from datasets

dlouisbriggs wrote:

The setting that you are referring to doesn't seem to prohibit trading to certain blocks of time. The first screenshot shows the setting is set for 07:00 to 12:00 and the second screenshot showing the collection performance report which shows trades being entered outside of the permitted time block.

I know it works because I block out an hour before and after rollover and the EA never places any trades during that window, although trades could still close of course by hitting SL or TP. Did you recreate the EA after changing that setting?

Re: Removing most of data from datasets

If I understand the process correctly, strategy generation is the simulated running of an EA based on user-defined criteria and measured against performance metrics. If the EA simulation (strategy generation) includes trades out of the desired time block and includes the results of those trades in the performance analysis, the results will be skewed with respect to the EA applied to the MT4 chart in live trading.

Therefore, if the simulated EA does not respect the trading time block, I would need to modify the historical data to include only the time block in which I have an interest in running the EA simulation.

Post's attachments

Screenshot 2023-10-04 at 6.01.11 PM.png 92.97 kb, 1 downloads since 2023-10-05 

You don't have the permssions to download the attachments of this post.

Re: Removing most of data from datasets

I made it as a separate post/topic because this does not belong to wish list thread.

Daniel, you have to think before you start suggesting deleting data.

If trades get carried over your time limit, what happens then? A broken backtesting.

And most importantly, if you have deleted the data, on what should the indicator values be built on? Hopes and dreams?

Do as Aaron says, use trading session limits. If you have used Studio before, you'd know that after editing settings you have to recalculate the strats. So do that, please.

Re: Removing most of data from datasets

> could designate a specific intraday block of time desired and all other data would not be exported?

How are you gonna trade such a strategy on your broker?

Re: Removing most of data from datasets

aaronpriest wrote:
dlouisbriggs wrote:

The setting that you are referring to doesn't seem to prohibit trading to certain blocks of time. The first screenshot shows the setting is set for 07:00 to 12:00 and the second screenshot showing the collection performance report which shows trades being entered outside of the permitted time block.

I know it works because I block out an hour before and after rollover and the EA never places any trades during that window, although trades could still close of course by hitting SL or TP. Did you recreate the EA after changing that setting?

Exactly, I do the same. Works just fine.

Re: Removing most of data from datasets

Response for Popov post ... My primary use for Express Generator and EA Studio is to identify the technical indicators, their optimal parameter settings, and the optimal take profit and stop loss values for strategies (actually simulated EA trading).

Every experienced trader knows that price action for instruments varies across intraday blocks of time (e.g. EURUSD has 700% more volatility between its minimum and maximum intraday). Additionally, most experienced traders have come to the conclusion that particular time blocks produce the majority of the trader's profits and losses, so they limit their trading to these time blocks. This is not rocket science.

Why then would it make sense to backtest strategies that are skewed by the inclusion of price data for time blocks you were not intending to trade?

The process of filtering the MT4 data from the broker's server is actually quite simple. Simply use the regular expression replace function in Notepad++ (I use a separate iteration of MT4 to generate the filtered data set)

With respect to the creation of the EA for the purpose of trading, the EA produced by EA Studio does not have numerous trade management features that are commonly used by expereinced traders so I simply reverse-engineer the strategy settings and add the trade managment features in another software.

In this way I have the best of both worlds. I use the superior features of Express-Generator to find the optimal performance metric based strategy settings for the time blocks I intend to trade and also benefit from an EA that utilizes advanced trade management features.

There must at least one trader in the forum that agrees that this approach has validity.

Cheers
Daniel

Re: Removing most of data from datasets

Sorry, I forgot to mention that the intraday time block model that I use captures the London Pre-Opening through London Lunch and then resumes at New York Pre-Opening through London Close (peak volatility for EURUSD). The settings in EA Studio are functional for only one time block per day.

Re: Removing most of data from datasets

>script where the user could designate a specific intraday block of time desired, and all other data would not be exported?

There is no way such functionality can be native in Express Generator.
However, a data customization script may be helpful for experiments.

I'll make a demo script for that purpose (later).

Re: Removing most of data from datasets

Thank you sir for your willingness to write the script for exporting a JSON file that allows user input variables to designate the time blocks for the dataset to include. If possible it would be appreciated if you could provide for the capability of 3 time blocks.

Thank you again
Daniel

Re: Removing most of data from datasets

The take-away from this is simple: additional trading sessions would be great in Studio/Express. There is no need to overcomplicate things with export scripts with left-out data. That's potentially to break the backtest and indi values.

14 (edited by aaronpriest 2023-10-07 18:11:20)

Re: Removing most of data from datasets

You can create strategies that have different start and end times for various sessions, it's really easy with Express Generator because you can run multiple generators at the same time, each with a different time session. But pulling out data is a really bad idea for a number of reasons, particularly if you never closed every trade at the end of every trading session. If you always close every trade at 5pm every day (for example), then I suppose gaps in the data wouldn't matter so much. But too many indicators use previous candles, and if you cut out those candles, then you don't have valid strategies, because when they trade in real time, they are going to use that real time data.

Re: Removing most of data from datasets

I understand your recommendations to conform to the backtesting structure as it exists to avoid the potential of complicating the strategy generation effort. Unfortunately, conforming to an existing backtesting structure that doesn't optimize the two most significant factors in developing a trading edge (1) pattern recognition across unique time series segments and (2) trade execution and management, seems pointless.

There might be some psychological comfort in seeing hundreds of thousands of strategies being backtested but in the end only one strategy with an edge is what is needed to be profitable.

In either case, thank you for your thoughtful commentaries.

Re: Removing most of data from datasets

Reply to aaronpriest ... your point is well taken with respect to the indicators need for past bars to calculate their values. On the other hand, I don't want the indicator values from low-volatility time blocks influencing pattern recognition during the high-volatility time blocks. Fortunately, I see the generator shortening the bars back indicator parameters to produce the highest-performing strategies. I do appreciate your comments.

Posts: 16

Pages 1

You must login or register to post a reply

Forex Software → Express Generator → Removing most of data from datasets

Similar topics in this forum