Out of curiosity, how do you actually define “good metrics” in your workflow?

For example, do you have specific thresholds or ranges you expect to hold from build (backtest) to live (like SQN, drawdown, etc.), or is it more based on overall behavior and experience?

Just to make sure I understood you correctly.

When you say you’re satisfied with the outcome, it’s not mainly because of the absolute metrics, but because the strategy shows consistent behavior across all validation stages, from build, to extra out-of-sample, to Monte Carlo, and then also in live.

So in your view, that alignment across layers is more important than the standalone numbers at any given stage.

Did I get that right?

algotrader21 wrote:

Vincenzo wrote:

And are you satisfied with this outcome ?
Does it cover your expectations?

Yes, I’m satisfied with it.

Not because of the numbers alone, but because of the consistency across all layers:

build phase 
extra out-of-sample extension 
Monte Carlo under heavy stress 
and most importantly, real live performance over time 

For me, that combination is what defines whether a system meets expectations or not.

At that point it’s no longer about a single result, but about consistent behavior across different validation stages and real conditions.

Also, considering the metrics and the fact that this EA has been running across multiple live accounts with the same behavior, I think the outcome speaks for itself.

I never said standalone metrics are not important. If metrics were not important, I would not be posting screenshots of them in the first place.

For me, everything has to be good as a whole. Metrics matter a lot, because they already tell me whether a system is strong, mediocre, or just weak. But when I talk about robust systems, I’m looking at more than just numbers at one stage.

What matters to me is that the same quality keeps showing up across all layers: build phase, extra out-of-sample, Monte Carlo, demo performance, and then live performance over time.

And when I say it all has to be good as a whole, or that I focus a lot on structure, then metrics are obviously important as well. Without metrics, I cannot even judge performance properly, and I also cannot see whether the equity curve, drawdown, and overall behavior are actually strong or not.

So for me, it is not metrics versus alignment. It is the combination. Good metrics make a system interesting, but consistency across all validation stages is what tells me whether it is actually robust or not.

When you say you’re satisfied with the outcome, it’s not mainly because of the absolute metrics, but because the strategy shows consistent behavior across all validation stages, from build, to extra out-of-sample, to Monte Carlo, and then also in live.

So in your view, that alignment across layers is more important than the standalone numbers at any given stage.

Did I get that right?

And are you satisfied with this outcome ?
Does it cover your expectations?

algotrader21 wrote:

Vincenzo wrote:

Is this EA 165 the account you are referring to as outcome  ?

Yes, that is EA 165 from my main live account.

The symbol and magic number are visible in the screenshot as well.

Yes, I’m satisfied with it.

Not because of the numbers alone, but because of the consistency across all layers:

build phase 
extra out-of-sample extension 
Monte Carlo under heavy stress 
and most importantly, real live performance over time 

For me, that combination is what defines whether a system meets expectations or not.

At that point it’s no longer about a single result, but about consistent behavior across different validation stages and real conditions.

Also, considering the metrics and the fact that this EA has been running across multiple live accounts with the same behavior, I think the outcome speaks for itself.

Yup, you nailed it. They are very short lived and have to get replaced often, that is a huge downside. And quite likely some good strategies are replaced prematurely, but most short term ones are bad strategies in the long run. I've been running a mix of both short and long term strategies on demo accounts, and keeping track of their performance via magic numbers, swapping out short term ones frequently and only swapping out long term ones when they clearly break their pattern--that part takes more effort to track, but it's easy to see if one has been unprofitable for three months in a row for example to flag it for testing.

Yeah, honestly I think what you’re doing right now is probably one of the best ways to really understand what works and what doesn’t.

That hybrid approach combining short-term strategies with longer-data builds gives you a lot of insight through actual experience, not just theory.

For me personally, I lean more towards the long-data side. I build on larger datasets with 50% OOS, allow full optimization during the build phase (even quite heavy, like 40 steps), and then rely on Monte Carlo to validate whether the structure holds or breaks under stress. That’s exactly why I’m comfortable pushing optimization  because MC is there to test if it’s real or not.

But at the same time, I completely agree with you that experimenting like you’re doing is key. In the end, everyone has to go through that process and figure out what fits their own mindset and workflow.

Short-term strategies can definitely work, but like you said, they tend to require constant rotation and replacement. It becomes a much more active process. The longer-data approach is slower and honestly a bit more “boring”, but for me it fits better because it’s more focused on long-term behavior rather than constant switching.

For example, at this point I have a core group of around 12 strong systems that have been running live well for over a year, plus a few others that are still profitable but not top-tier. Because of that, my building frequency has dropped a lot. I mostly focus now on managing, validating, and slowly expanding what already works, rather than constantly generating new strategies.

On the execution side, I also try to keep things structured. Ideally, I want to see around 6 months on demo before moving a robot to live. But if a robot already reaches around 60 trades within 4 months and the metrics still look strong, that can also be enough for me. In the end, it always depends on the overall behavior and structure.

At the same time, I do agree with your point that after enough experience, you start recognizing patterns faster. You can often see earlier when something is off. But even then, I still try to balance that with giving strategies enough room, because sometimes it really depends on the market phase they start in.

That’s also something I’ve noticed a lot certain strategies just need the right conditions. For example, breakout systems can struggle in ranging periods at the start, even if they perform well over time.

So yeah, overall I think both approaches can work. It really comes down to what kind of process you want to run and what fits your mindset best.

Curious to see how your results evolve over time with that mix.

And yes, what you are doing can definitely work. Shorter data can work.

But one thing I’ve noticed with that approach is that it tends to push the process into a much faster and more rotational workflow.

You need to keep building new strategies regularly, because you need fresh ones for the next cycle.

And because of that constant rotation, everything becomes more reactive.

I also think one of the risks there is that some potentially good strategies might be rotated out too early, simply because they did not have enough time yet to show their behavior.

Strategies often need time to play out.

Yup, you nailed it. They are very short lived and have to get replaced often, that is a huge downside. And quite likely some good strategies are replaced prematurely, but most short term ones are bad strategies in the long run. I've been running a mix of both short and long term strategies on demo accounts, and keeping track of their performance via magic numbers, swapping out short term ones frequently and only swapping out long term ones when they clearly break their pattern--that part takes more effort to track, but it's easy to see if one has been unprofitable for three months in a row for example to flag it for testing.

Is this EA 165 the account you are referring to as outcome  ?

algotrader21 wrote:

Because this EA has already been running live for about a year, I decided to push the validation much further.

So this is not a random test.

This is a proven strategy being pushed to its limits.

I increased the stress deliberately:

history data heavily randomized (80% changed bars)
ATR variation increased to 40%
slippage increased up to 40 points
and 1500 Monte Carlo simulations

Not because I needed to “prove” it.

But because I already knew this strategy could handle it.

Why?

Because this is not a fresh build.

This is an EA that has already:

passed the build process
survived extra out-of-sample
and proven itself in live conditions

So at this stage, the goal is different.

Not discovery.

Confirmation under pressure.

Results

Even under heavy stress:

the structure holds
performance remains consistent
degradation stays controlled
SQN remains solid across confidence levels

This is exactly what you want to see when you push a proven system.

Key point

No changes were made to the strategy.

No reoptimization.
No adjustments.

Just pure stress testing.

At this level, you are no longer looking at a backtest.

You are looking at behavior under extreme conditions.

And it still holds.

Yes, that is EA 165 from my main live account.

The symbol and magic number are visible in the screenshot as well.

Thanks for your detailed response! It mostly mirrors my own experience. I've had many good strategies fail multi timeframe and multi symbol that still ran well for what it was designed for. I stick with H1 and higher, and occasionally M30. Multi broker gets a bit more nuanced, particularly if you can't get good data from a broker (missing candles, etc.), or enough history. This is less of a problem with MT5 if you have a good broker though.

If you don't mind me picking your brain some more... what is your opinion and experience on date range? I used to generate from 2010 (or as far back as my broker could go) with about 60/40 IS/OOS and heavy Monte Carlo, and run strategies that passed until they weren't profitable for three months in a row or so or broke a previous pattern. It was quite tedious to keep track of them. Lately I've been experimenting with generating on the last year of data (excluding the last 3 months for OOS) with optimizer and Monte Carlo, validating the last 3 months to see how it would have traded (also with Monte Carlo, but no optimizing), and then just throwing it straight on a demo account. I generate all week, then validate and swap out new EAs on weekends, so I'm always running new strategies for a week on recent data. I've got it automated to a point where I don't spend much time on a weekend setting up the next week. I don't have enough data yet to say if it's more profitable or not, but curious to hear other people's experience on this before I waste much more of my time testing the theory... Is it better to analyze many years of various markets that won't ever repeat the same anyway, or analyze and tune for more recent data? Either way, tomorrow won't continue the same as yesterday. LOL

I think I'm going back to a larger date range though because it's hard to find a strategy with enough trades on only 3 months of OOS data to be truly valid.

Thanks, and also respect for sharing your way of working so openly.

Yeah, exactly. I’m honestly glad to see someone who thinks like this, not just by theory, but by actually testing, by trial and error. That’s what this is really about in the end.

EA Studio offers a lot of functions, and in principle that makes sense because it belongs to algo building. More tools and more testing can give more answers in a certain way. But in my eyes, not every function adds real value when it comes to robustness testing.

Because like you also experienced yourself, robots can fail multi-pair and multi-timeframe tests, and still run well live on the exact pair and timeframe they were built for.

And the opposite also happens. Some strategies can pass those multi-pair or multi-timeframe checks inside EA Studio, and still fall apart later on demo or eventually live. Not all of them, of course, but you see it happen often enough.

That is also why, for me, EA Studio, Monte Carlo, and all those tools are still reference points, not final proof of robustness. As long as a robot has not proven itself through a long enough demo period and then a long enough live period, I do not really see it as a robust system yet.

For me, real robustness only starts to become clear after 6 months live, 1 year live, and ideally even longer. The more time and data a system survives, the more we actually know. I always come back to that same principle.

To me, that already says a lot.

It tells us that multi-pair and multi-timeframe are extra answers, maybe interesting later, but not something I see as core proof of robustness in the build phase.

If a system has already proven itself live, then yes, putting it on another pair or timeframe can make sense and can add diversification. But that is something else. Then you are expanding a proven system, not validating an unproven one.

About data, I’m on the long-data side, with a lot of OOS.

For me, data and OOS are key, plus Monte Carlo. Honestly, that is already enough.

That is also why I build from 2009 to 2022. Inside that build period I already use 50% OOS. The data I use is the premium data from Popov, and I’ve been using that from the beginning. I adjust that data inside EA Studio to match my broker specs, and because of that I get a correct match between EA Studio, MT4 backtests, demo, and live.

Good robots show the same structure everywhere.

So my process is like this:

I build from 2009 to 2022
with 50% OOS
and inside Reactor I optimize the full build period with 40 steps, also with 50% OOS

Then when that is done, I extend the end date forward to 2026.

That creates 4 extra years of pure, raw, unseen, non-optimized OOS data.

A strategy that survives that gets Monte Carlo tested.

After that I run an MT4 tick-data backtest from 2003 to 2026, mainly to see what the robot does on the years before my build period started. Since my build period starts in 2009, I want to see how it behaves on that extra OOS before 2009 as well, so roughly 2003 to 2009.

Can I be honest?

I can clearly see you are dedicated, and I respect that. I also see that you are not biased for long data or short data. You are clearly learning from your experiments and noticing patterns, and for that, real respect.

At the same time, I do think there is a downside to the shorter data approach you are currently testing.

And I’ll explain what I mean.

Right now you are building robots on one year of data, where three months of that is OOS inside the building process.

So basically, you are giving the robot around nine months to train on, plus three months as an OOS check.

That means the robot has only seen a relatively small portion of the market.

Even with OOS included, it is still trained on a limited amount of data.

For me personally, that is a bit too little to build strong confidence.

It is the same logic everywhere.

If someone wants to build an AI model, do they train it on as little data as possible, or on as much relevant data as possible?

Or if you choose someone to manage your capital, do you pick the one with 20 years of experience, or someone with only a short track record?

For me, it translates directly to algo building.

The more historical data a strategy has seen, the more market conditions it has been exposed to.

The more OOS inside the build phase, the better.

And on top of that, I like to add a wide extra OOS layer after the build.

If that is not possible inside Express generator, then building on 2009–2022 and testing in MT4 on 2009–2026 can give a similar effect, just a bit slower.

Of course, I don’t know exactly how Express Generator works, since I don’t use it.

And yes, what you are doing can definitely work. Shorter data can work.

But one thing I’ve noticed with that approach is that it tends to push the process into a much faster and more rotational workflow.

You need to keep building new strategies regularly, because you need fresh ones for the next cycle.

And because of that constant rotation, everything becomes more reactive.

I also think one of the risks there is that some potentially good strategies might be rotated out too early, simply because they did not have enough time yet to show their behavior.

Strategies often need time to play out.

That is one of the biggest differences for me.

With longer-data robust strategies, the process is slower, yes. It takes longer in building, longer in demo, and longer in live.

But it is much less rushed.

I don’t need to switch every week. I don’t need to build every day.

At this point I have around 12 robots with good metrics, plus maybe another 5 that are still profitable but where the metrics are not really top level. So those are not my best systems, but they still do their job.

And honestly, that’s also part of my view on this. We do not need a portfolio of 50 systems. Even in manual trading, one person with a good strategy and the right psychology can make serious money. For me it is the same with robots.

One strong, robust system that works for a long time can already make you serious money. So if you have a stack of around 10 robust robots that have already been running well live for a year, in my opinion you really do not need much more than that.

And yes, there are enough builders who can make shorter-data constant rotation work.

I’m not denying that.

It can work.

But for me personally, the goal is different.

I prefer building systems that are robust and can last over longer periods of time.

That is also where I see the main difference:

Long data is slower, but generally leads to more stable structures over time.

Shorter data is faster, but relies much more on constant rotation and replacement.

So yes, it moves faster.

You get robots faster.
You can move them faster from demo to live.

But you also tend to remove them faster again, because they depend more on staying in rotation.

That is why I personally prefer the long-data approach.

If you don't mind me picking your brain some more... what is your opinion and experience on date range? I used to generate from 2010 (or as far back as my broker could go) with about 60/40 IS/OOS and heavy Monte Carlo, and run strategies that passed until they weren't profitable for three months in a row or so or broke a previous pattern. It was quite tedious to keep track of them. Lately I've been experimenting with generating on the last year of data (excluding the last 3 months for OOS) with optimizer and Monte Carlo, validating the last 3 months to see how it would have traded (also with Monte Carlo, but no optimizing), and then just throwing it straight on a demo account. I generate all week, then validate and swap out new EAs on weekends, so I'm always running new strategies for a week on recent data. I've got it automated to a point where I don't spend much time on a weekend setting up the next week. I don't have enough data yet to say if it's more profitable or not, but curious to hear other people's experience on this before I waste much more of my time testing the theory... Is it better to analyze many years of various markets that won't ever repeat the same anyway, or analyze and tune for more recent data? Either way, tomorrow won't continue the same as yesterday. LOL

I think I'm going back to a larger date range though because it's hard to find a strategy with enough trades on only 3 months of OOS data to be truly valid.

Do you test multiple timeframes, multiple symbols, or multiple broker data? I ask because I'm using express generator and mm.js (multimarket) does not support the same flags that gen.js does (the generator) for OOS and data-end, etc.

Hi Aaron,

Thanks, good question.

I don’t test across multiple timeframes, multiple symbols, or different brokers, and I’ll explain why.

For me, a strategy does not need to work across everything to be considered good. What matters most is that it performs well on the specific market, timeframe, and data it was built for. I build strategies for a specific symbol and timeframe, and I validate them within that context.

Regarding timeframes, I don’t build on lower timeframes at all. There is too much noise there and, in my experience, a much lower chance of getting something truly robust. I mainly work from M30 and higher.

For symbols, I’m not a big believer in cross-pair validation as a main measure of robustness. Some strategies can work on multiple pairs, yes, but in my view that is rare. For me, the main question is not whether a robot can run on many pairs, but whether it is truly robust on the market and timeframe it was built for. And once a robot reaches demo or live, that is really what I focus on: the actual metrics and whether it is making money on the pair and timeframe it was designed for.

Another reason I don’t put much weight on multi-market or multi-timeframe validation during the build phase is that, in my view, we still cannot clearly tell what we are looking at.. Even if a robot shows good results across multiple markets or timeframes inside EA Studio, that still does not tell me clearly whether I’m looking at something truly robust or just something that is broadly overfit.

And if we then start putting that robot on demo across different pairs and timeframes, it can easily become a waste of time and resources, because the main question is still unanswered: is the robot actually robust or not?

For me, it makes much more sense to first go deeper on one market, one timeframe, and one environment, and only later, if the robot has already proven itself through forward or live results over a meaningful period, then maybe check whether it can be expanded to other pairs or timeframes.

The difference is big. Then we are not expanding with a robot that EA Studio simply approved. We are expanding with a robot that has already proven itself in real conditions.

Same idea for brokers.

I’m not building or selling EAs for general distribution, so I don’t test across different brokers. I focus on the broker I actually use for building, demo, and live trading.

In my view, if you are not selling robots and you already use one solid broker, then multi-broker testing does not add much value. I would rather use that time and energy to run more strategies on demo, expand within the same broker, and push more of them through real forward testing.

That gives me much more useful information than checking whether the same robot behaves slightly differently across several brokers.

If I ever want to expand beyond the maximum number of accounts with the same broker, I can always connect another broker through copy trading. I already do that with one account. Sometimes I see a difference of a few seconds in execution, not always, but that is normal. It can even happen between different live accounts at the same broker. That is simply part of trading reality and execution. But for me, that does not say much about the underlying robustness of the strategy.
The main point is that the robot needs to work properly on the broker and environment it was actually built for.

So for me, testing across multiple brokers is more about distribution and compatibility than about robustness. It can make sense if you sell robots and want to know whether they behave similarly everywhere, but for my own workflow it does not add much.

The “extra OOS” idea is, from a practical standpoint, one of the most solid approaches I’ve seen for reducing curve fitting.

Many traders rely on the standard OOS in EA Studio and assume it’s sufficient, but in reality the generator still “sees” the full data context. By isolating a completely unseen period and testing without any re-optimization, this method gets much closer to real live trading conditions.

What I find particularly valuable about this approach:

it enforces discipline (no tweaking after seeing results)
it filters out strategies that are “too good to be true”
it highlights true robustness rather than just performance

is probably the closest we can get to real-market conditions in a retail environment.

Great contribution,  this is the kind of method that actually improves how strategies are filtered, not just how they are generated.

Thanks Mentosan, appreciate it.

Good to see this perspective being recognized, not many people approach it this way.

Once the strategy is built and optimized, extending the end date without touching anything basically turns it into pure, untouched out-of-sample data.

That’s usually the point where the difference shows between strategies that only looked good during the build phase and those that actually hold up when pushed into completely unseen conditions.

A lot of strategies simply don’t survive that extension, even if they passed initial filtering and Monte Carlo.

At the same time, you’ll never filter everything out completely, but working with longer datasets and adding that extra OOS layer makes a big difference in what remains.

Like you said, it’s probably one of the closest things we have to real market conditions in a retail environment, especially when that oos extension covers multiple market regimes instead of just a short period.

That’s where the real filtering happens.

Many traders rely on the standard OOS in EA Studio and assume it’s sufficient, but in reality the generator still “sees” the full data context. By isolating a completely unseen period and testing without any re-optimization, this method gets much closer to real live trading conditions.

What I find particularly valuable about this approach:

it enforces discipline (no tweaking after seeing results)
it filters out strategies that are “too good to be true”
it highlights true robustness rather than just performance

is probably the closest we can get to real-market conditions in a retail environment.

Great contribution,  this is the kind of method that actually improves how strategies are filtered, not just how they are generated.