1 (edited by geektrader 2024-06-24 12:21:04)

Topic: (Bug?) Extremely slow/"stuck" optimizations.

Hey everyone,

I wanted to bring attention to an issue I've been experiencing with Express Generator, particularly when using the optimizer alongside the strategy-building process. I've noticed that it occasionally gets "stuck" indefinitely during this phase, and I'd like to provide some more technical details to help pinpoint the problem.

First off, here are my (most important) settings, so that this can be recreated:

; It validates the "input" strategies and then continues generating new ones
validate_then_generate = false

; Prevent rising an error when an input file cannot be loaded
suppress_input_error = false

; Sort the Collection by one of the following metrics:
; NetBalance, Profit, ProfitFactor, ReturnToDrawdown, RSquared, Stagnation, WinLossRatio
sort_by = RSquared

; Correlation analysis threshold
correlation_threshold = 1.00

; Detect balance lines correlation
resolve_correlation = false

; Detect strategies with similar trading rules
resolve_similar_rules = false

; 0 means unlimited for all of the options
max_calculated_strategies = 0
max_ascended_strategies   = 0
max_collected_strategies  = 0
max_working_minutes       = 0

; Set 0 to ignore a criterion (except for min_profit)
max_consecutive_losses = 0
max_drawdown_percent   = 0
max_equity_drawdown    = 0
max_stagnation_days    = 0
max_stagnation_percent = 0
min_count_of_trades    = 2000
min_profit             = 0
min_profit_factor      = 0
min_profit_per_day     = 0
min_r_squared          = 35
min_return_to_drawdown = 2
min_win_loss_ratio     = 0

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Strategy properties      ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

entry_lots            = 1.00

; Trade direction: LongAndShort, LongOnly
trade_direction_mode  = LongAndShort

; Opposite entry signal: Ignore, Reverse, IgnoreOrReverse
opposite_entry_signal = Reverse

; Stop Loss: AlwaysUse, MayUse, DoNotUse
stop_loss_usage       = AlwaysUse
; Stop Loss type: Fixed, Trailing, FixedOrTrailing
stop_loss_type        = Fixed
stop_loss_range_min   = 28
stop_loss_range_max   = 120

; Take Profit: AlwaysUse, MayUse, DoNotUse
take_profit_usage         = MayUse
take_profit_range_min     = 28
take_profit_range_max       = 750
take_profit_gte_stop_loss = false

; Max number of indicators to use
max_entry_slots       = 4
max_exit_slots        = 1

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Forward testing          ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; Validate strategies on new unseen data.

use_forward_testing = false
preload_data_bars   = 0

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Out of sample            ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; Use these parameters to set the data range.
; The program cuts data after applying Data Horizon.

data_start_percent = 0
data_end_percent   = 100

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Trading Session          ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

session_open  = 01:00
session_close = 23:58
friday_close  = 23:58

trade_on_sunday = true

close_at_session_close = false
close_at_friday_close  = false

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Miscellaneous            ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; When enabled, the generator dynamically updates the best strategy on the console.
update_best = true

; How many strategies to print when finish
show_top = 0

; Prevent showing output in the console (except errors)
silent = false

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Indicators used by the Generator ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

ind_accelerator_oscillator       = true
ind_accumulation_distribution    = true
ind_adx                          = true
ind_alligator                    = true
ind_average_true_range           = true
ind_awesome_oscillator           = true
ind_bears_power                  = true
ind_bollinger_bands              = true
ind_bulls_power                  = true
ind_candle_color                 = true
ind_commodity_channel_index      = true
ind_demarker                     = true
ind_directional_indicators       = true
ind_do_not_exit                  = true
ind_donchian_channel             = true
ind_entry_time                   = true
ind_envelopes                    = true
ind_exit_time                    = true
ind_force_index                  = true
ind_long_or_short                = false
ind_macd                         = true
ind_macd_signal                  = true
ind_momentum                     = true
ind_money_flow_index             = true
ind_moving_average               = true
ind_moving_average_of_oscillator = true
ind_moving_averages_crossover    = true
ind_on_balance_volume            = true
ind_pin_bar                      = false
ind_rsi                          = true
ind_rvi                          = true
ind_rvi_signal                   = true
ind_standard_deviation           = true
ind_stochastic                   = true
ind_stochastic_signal            = true
ind_volumes                      = true
ind_williams_percent_range       = true

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Indicator options                ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; The following options are used by the Generator for composing new strategies.

; Randomize Moving Average Methods.
; When 'true', the Generator chooses between Simple, Exponential, Weighted and Smoothed.
randomize_ma_method = true

; Maximum period generated by the Generator. Recommended: 50. Recommended max 200.
max_indicator_period = 80

; Shift of the Moving Average indicator. When it is enabled, the Generator may use shift.
randomize_ma_shift = true

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Auto-save collection             ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; Auto-save is useful if you run Express Generator for long time on strict Acceptance Criteria
; Setting 0 disables an option. Choose one of the options only.

; Save the collection at a particular interval in minutes.
; It does not clear EG collection.
auto_save_at_minutes = 0

; Save the collection when it reaches the given count.
; It clears EG collection.
auto_save_at_collected = 1000

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Optimizer                        ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

enable_optimizer     = true
numeric_values_steps = 70
optimize_protections = true

; Search best strategy by:
; NetBalance, Profit, ProfitFactor, ReturnToDrawdown, RSquared, Stagnation, WinLossRatio
optimize_by = RSquared

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Monte Carlo                      ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

enable_monte_carlo            = false

From my observations, it appears that the "Calculated" count does not update once a strategy is passed/ascended to the Optimizer. This makes it challenging to determine whether the process is still running or has stalled. Without this crucial information, identifying and addressing the issue becomes more difficult.

To illustrate the problem, the screenshot below shows four Express Generator instances that were all initiated simultaneously. While three of them are running smoothly at the 6:44-hour mark, the instance processing GBPJPY is displaying a time of 6:12. This indicates that it spent over 30 minutes optimizing a single strategy, a significant deviation from the expected performance. Typically, my system can process around 10 strategies per second on the instances that are functioning correctly.

https://assets.forexhistorydata.com/images/various/eg-slow-optimization.webp

Are Strategies with High Trade Counts The Potential Culprit?

Based on my analysis, this issue seems to arise when the strategy undergoing optimization contains a substantial number of trades - >100.000. The high trade count drastically slows the optimization process, causing the software to "hang up" for extended periods during the optimization phase.

Proposing a Solution: A Timeout Setting / Amount of Trades Limit

To mitigate this recurring problem, I suggest implementing a new setting within Express Generator. This option would allow users to specify a maximum time limit for any strategy backtests and/or a maximum trade count (in addition to the minimum trade count). If a backtest exceeds these thresholds, Express Generator will simply skip further processing of that particular strategy. By providing users with the ability to set a timeout value and/or the max amount of trades, we could prevent strategies with excessively long runtimes from hindering the overall performance of Express Generator.

I believe this addition would greatly enhance the user experience and help maintain Express Generator's efficiency. I'm eager to hear your thoughts on this proposal and any insights you might have regarding the underlying cause of the issue.

Thank you,

Lorenz

Re: (Bug?) Extremely slow/"stuck" optimizations.

Good morning I also suffer from this and I found out that the error is in. Adding more than 40 steps in optimization, in fact I like to leave 100 steps, but it always gets stuck, especially when a strategy uses more than 4 indicators for entry and


To get around this you Hi that leave the steps of the tomizer in 20 And even then, depending on the amount of indicator the strategy has, it still crashes

Re: (Bug?) Extremely slow/"stuck" optimizations.

Thank you for the report!

I'll examine the issue and will propose a solution.

Re: (Bug?) Extremely slow/"stuck" optimizations.

At first glance, I confirm that stopping the progress update is weird.
I'll make it possible to see the Optimizer computations in some way.
It will show if the Optimizer freezes or still computes, having many cycles to do.
I may think of adding some time limitations for it. Let's say ''--max-optimization-time 3" to limit it to up to 3 minutes of work per strategy.

Re: (Bug?) Extremely slow/"stuck" optimizations.

Thanks for the confirmation and your insights on this issue.

After further analysis, I'm still leaning towards the idea that strategies with exceptionally high trade counts (think thousands or even 100,000+ trades) are the root cause of the slowdown, rather than the optimization step sizes or indicators. When Express Generator randomly generates strategies, it's entirely possible for some of these strategies to have an enormous number of trades. As you can imagine, this can bring even Express Generator to its knees, causing significant performance issues.

Now, when one of these resource-intensive strategies gets passed to the Optimizer, it's no surprise that it can take upwards of 30 minutes to optimize, whereas the typical optimization process usually completes within a few minutes at most. This aligns with the behavior I've observed in my own instances.

To tackle this issue head-on, I believe implementing a combination of two measures could be the most effective solution:

1. Backtest Time Limit: Introduce a setting that allows users to specify a maximum processing time for each backtest, measured in milliseconds. If a strategy's backtest exceeds this limit, Express Generator will automatically cancel its processing and discard the strategy entirely. This would prevent strategies with excessively long runtimes from bogging down the system.

2. Maximum Trade Count: Establish an upper limit for the number of trades allowed in a strategy. If a generated strategy surpasses this threshold, it will be immediately discarded. This measure would proactively filter out strategies that are likely to cause performance issues down the line.

By implementing these two safeguards, we can effectively mitigate the impact of strategies with abnormally high trade counts and ensure a smoother, more efficient optimization process.

Alternatively, setting a time limit for the entire optimization run, as you mentioned, could also be a viable approach. This would provide a global constraint on the optimization process, ensuring that it doesn't exceed a reasonable duration.

I believe that a combination of these measures, or at least one of them, would significantly alleviate the optimization hang-up issue.

Re: (Bug?) Extremely slow/"stuck" optimizations.

I say it again, allow me to wake up

I made one. Strategy with a maximum of 20 orders in 2008 to 2020 and placed 400 optimization steps guess what happened, it crashed for minutes This lock is closely linked to the optimization steps

7 (edited by Popov 2024-06-25 16:50:58)

Re: (Bug?) Extremely slow/"stuck" optimizations.

I don't know how. Send pictures here if not, I would send them lol for you to see:

https://image-holder.forexsb.com/store/IMG20240624220141-thumb.jpg

Re: (Bug?) Extremely slow/"stuck" optimizations.

Very interesting. Then I seriously hope that Popov can provide a fix.