#### Topic: (Bug?) Extremely slow/"stuck" optimizations.

Hey everyone,

I wanted to bring attention to an issue I've been experiencing with Express Generator, particularly when using the optimizer alongside the strategy-building process. I've noticed that it occasionally gets "stuck" indefinitely during this phase, and I'd like to provide some more technical details to help pinpoint the problem.

First off, here are my (most important) settings, so that this can be recreated:

```
; It validates the "input" strategies and then continues generating new ones
validate_then_generate = false
; Prevent rising an error when an input file cannot be loaded
suppress_input_error = false
; Sort the Collection by one of the following metrics:
; NetBalance, Profit, ProfitFactor, ReturnToDrawdown, RSquared, Stagnation, WinLossRatio
sort_by = RSquared
; Correlation analysis threshold
correlation_threshold = 1.00
; Detect balance lines correlation
resolve_correlation = false
; Detect strategies with similar trading rules
resolve_similar_rules = false
; 0 means unlimited for all of the options
max_calculated_strategies = 0
max_ascended_strategies = 0
max_collected_strategies = 0
max_working_minutes = 0
; Set 0 to ignore a criterion (except for min_profit)
max_consecutive_losses = 0
max_drawdown_percent = 0
max_equity_drawdown = 0
max_stagnation_days = 0
max_stagnation_percent = 0
min_count_of_trades = 2000
min_profit = 0
min_profit_factor = 0
min_profit_per_day = 0
min_r_squared = 35
min_return_to_drawdown = 2
min_win_loss_ratio = 0
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Strategy properties ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
entry_lots = 1.00
; Trade direction: LongAndShort, LongOnly
trade_direction_mode = LongAndShort
; Opposite entry signal: Ignore, Reverse, IgnoreOrReverse
opposite_entry_signal = Reverse
; Stop Loss: AlwaysUse, MayUse, DoNotUse
stop_loss_usage = AlwaysUse
; Stop Loss type: Fixed, Trailing, FixedOrTrailing
stop_loss_type = Fixed
stop_loss_range_min = 28
stop_loss_range_max = 120
; Take Profit: AlwaysUse, MayUse, DoNotUse
take_profit_usage = MayUse
take_profit_range_min = 28
take_profit_range_max = 750
take_profit_gte_stop_loss = false
; Max number of indicators to use
max_entry_slots = 4
max_exit_slots = 1
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Forward testing ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Validate strategies on new unseen data.
use_forward_testing = false
preload_data_bars = 0
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Out of sample ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Use these parameters to set the data range.
; The program cuts data after applying Data Horizon.
data_start_percent = 0
data_end_percent = 100
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Trading Session ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
session_open = 01:00
session_close = 23:58
friday_close = 23:58
trade_on_sunday = true
close_at_session_close = false
close_at_friday_close = false
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Miscellaneous ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; When enabled, the generator dynamically updates the best strategy on the console.
update_best = true
; How many strategies to print when finish
show_top = 0
; Prevent showing output in the console (except errors)
silent = false
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Indicators used by the Generator ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
ind_accelerator_oscillator = true
ind_accumulation_distribution = true
ind_adx = true
ind_alligator = true
ind_average_true_range = true
ind_awesome_oscillator = true
ind_bears_power = true
ind_bollinger_bands = true
ind_bulls_power = true
ind_candle_color = true
ind_commodity_channel_index = true
ind_demarker = true
ind_directional_indicators = true
ind_do_not_exit = true
ind_donchian_channel = true
ind_entry_time = true
ind_envelopes = true
ind_exit_time = true
ind_force_index = true
ind_long_or_short = false
ind_macd = true
ind_macd_signal = true
ind_momentum = true
ind_money_flow_index = true
ind_moving_average = true
ind_moving_average_of_oscillator = true
ind_moving_averages_crossover = true
ind_on_balance_volume = true
ind_pin_bar = false
ind_rsi = true
ind_rvi = true
ind_rvi_signal = true
ind_standard_deviation = true
ind_stochastic = true
ind_stochastic_signal = true
ind_volumes = true
ind_williams_percent_range = true
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Indicator options ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; The following options are used by the Generator for composing new strategies.
; Randomize Moving Average Methods.
; When 'true', the Generator chooses between Simple, Exponential, Weighted and Smoothed.
randomize_ma_method = true
; Maximum period generated by the Generator. Recommended: 50. Recommended max 200.
max_indicator_period = 80
; Shift of the Moving Average indicator. When it is enabled, the Generator may use shift.
randomize_ma_shift = true
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Auto-save collection ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Auto-save is useful if you run Express Generator for long time on strict Acceptance Criteria
; Setting 0 disables an option. Choose one of the options only.
; Save the collection at a particular interval in minutes.
; It does not clear EG collection.
auto_save_at_minutes = 0
; Save the collection when it reaches the given count.
; It clears EG collection.
auto_save_at_collected = 1000
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Optimizer ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
enable_optimizer = true
numeric_values_steps = 70
optimize_protections = true
; Search best strategy by:
; NetBalance, Profit, ProfitFactor, ReturnToDrawdown, RSquared, Stagnation, WinLossRatio
optimize_by = RSquared
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Monte Carlo ;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
enable_monte_carlo = false
```

From my observations, it appears that the "Calculated" count does not update once a strategy is passed/ascended to the Optimizer. This makes it challenging to determine whether the process is still running or has stalled. Without this crucial information, identifying and addressing the issue becomes more difficult.

To illustrate the problem, the screenshot below shows four Express Generator instances that were all initiated simultaneously. While three of them are running smoothly at the 6:44-hour mark, the instance processing GBPJPY is displaying a time of 6:12. This indicates that it spent over 30 minutes optimizing a single strategy, a significant deviation from the expected performance. Typically, my system can process around 10 strategies per second on the instances that are functioning correctly.

**Are Strategies with High Trade Counts The Potential Culprit?**

Based on my analysis, this issue seems to arise when the strategy undergoing optimization contains a substantial number of trades - >100.000. The high trade count drastically slows the optimization process, causing the software to "hang up" for extended periods during the optimization phase.

**Proposing a Solution: A Timeout Setting / Amount of Trades Limit**

To mitigate this recurring problem, I suggest implementing a new setting within Express Generator. This option would allow users to specify a maximum time limit for any strategy backtests and/or a maximum trade count (in addition to the minimum trade count). If a backtest exceeds these thresholds, Express Generator will simply skip further processing of that particular strategy. By providing users with the ability to set a timeout value and/or the max amount of trades, we could prevent strategies with excessively long runtimes from hindering the overall performance of Express Generator.

I believe this addition would greatly enhance the user experience and help maintain Express Generator's efficiency. I'm eager to hear your thoughts on this proposal and any insights you might have regarding the underlying cause of the issue.

Thank you,

Lorenz