Yeah -- good question.  I don't know how to quantify and I'm not even sure whether it's possible or worth pursuing.  Also, consider this -- it could be that it's not the strategy's fault for poor performance but, rather, the market.  One day 33% of the strats perform well and another day a different 33% perform well.  So, if a different 33% perform well at different times then how are you going to figure that out?  Much too complicated...

I think a solution is to accept there will be non-winning days and focus on a workflow that provides an edge and yields an overall positive return.  In other words, instead of getting deeper into details, take a step back and look at the bigger picture.

Cool -- thank you...

Those are GMT times, right?  Where 09:00 is around London Open, right?

I've also occasionally seen that decline, but didn't know what to make of it.

Since I'm posting, here is another lesson I just now painfully learned.  I usually use broker data for live trading and Premium Data for testing (because it's quicker and I'm lazy).  And recently while testing I was seeing some crappy results that didn't make sense.  When I substituted broker data for Premium Data then the test results were much improved.  So, treat this as an FYI...   

We refer to "market conditions", "market changes", etc. -- how about we attempt to flush this out a bit so we are more clear about the meaning and implications.  I don't think there is anything we can do about market conditions, but I do think with a better understanding it would lead to a better understanding of why a portfolio EA performs as it does.

What market conditions is NOT:
1. Price direction -- i.e. trends
2. ???

What market conditions IS:
1. ADR (Average Daily Range)
2. Bar height
3. Swing length / interval between swings
4. ???

I would intentionally leave out economic news and other rare events that might cause spikes -- not because they don't affect the market but, rather, because they are rare events.  I'm more interested in those properties that routinely affect market conditions.

I've done enough testing and trading to see both good and bad days.  Bad days are usually when the portfolio EA treads water, and good days when profit grows gradually.  I don't think these have anything to do with the portfolio EA's algebra.  On a "bad" day I think most all portfolio EAs would trade poorly and on a "good" I think most all would perform well.  And the reason is simple:  On a "bad" day there are no Pips to be had, and on a "good" day there are available Pips.

So "market conditions" comes down to this:  Whether or not there are available Pips...  Some days Pips will be available and some days they won't.  And this is consistent with timelleston's Backward/Forward Excell charts...

QUES:  How does one know in advance whether Pips will be available in the near future?
ANS:    I have a friend in Chicago who will rent his time machine to anyone at a reasonable cost. 

I'm loving Express Generators ability to rapidly work through 2.8 million strategies in about 6 minutes, and ascend 100,000 strategies (my settings), into a collection of 10,000 strategies, which I can then upload into EAS, using some acceptance criteria, and then sort them, and add the top 50 of the best to a portfolio to export.

If EAS can generate 5,000 strategies in an hour, and keep 300, then being able to generate 100,000 > 10,000 > 300 > 50 surely = the creme de la creme of strategies for the next 24 hours.  In my book, thats an edge!

Running CADJPY 15m, on 900 bars now, in demo.

I'm getting lost here...

1. GD has returned to 20000 historical bars -- which has nothing to do with the **current** market.

2. timelleston's goal is millions of strategies with better stats -- even though we know that better stats don't necessarily translate to better real-time performance.  Is there no limit to the number of strategies one needs in order to achieve an edge?  Is the edge more strategies and prettier statistics?

Right -- I understand how you envision using Express Generator, though I may have exaggerated your perspective.

But even in your explanation above you again emphasize the goal being a "greater pool" of strategies to choose from, right?  And what criteria will you be using to select the top 50?  Might that be statistics?  And the prettier the statistics the more biased and over curve-fitted the settings.  And I'm referring to the aggregate statistics of the portfolio EA.

We currently have in our hands an amazing tool that generates thousands more strategies than we ever imaginged before introduction of EA Studio.  And that's not an edge?  You mean to say that instead of thousands we need to generate pools of millions from which to pick the top 50?

I'm not claiming that quality strategies aren't important -- rather, there are deeper issues to consider.  Your own charts provide evidence that different market conditions result in different performance -- even when your work flow is the same.  That is pretty strong proof that strategy generation is not what holds us back.

How's it going guys? Any update on these ideas from a month of testing?

Good point - haven't updated this thread in a while...so here goes.

I've had reasonable results during the non-volatile times, but got stuffed around due to the huge volatility that happened recently in CJ - to the point where I changed my approach slightly, and so far it seems to be working better...  I only do this on CJ M15 though.

I was running 480 bars in the market for a max 96 bars (24 hours), then changed it to 192 bars in market.  But it was opening quite a few trades and with the volatility I had a few days of good profit, then a couple of fairly big losses.  I'm trading this on a live account so I went back to the testing.  I found a few things...

I re-tested from 480 bars out to 2880 bars (4 days to 30 days) which incorporated the volatile markets recently as well as the quieter times.  My forward testing also used the same approach to see how long to keep the EA opening positions in market.

1440 bars (15 days) of history seemed to work better in both volatile and trending markets (slightly more history), and 384 bars (4 days) seemed to be the best length of time to let it open trades for - 74% of the last 40 days have resulted in profits when setting max trades to zero at 384 bars, trading a max of 0.01 lots with 10 open positions.

I'm also matching my live results to the forward-testing results, and they're pretty close - some minor slippage occurs in live - so I'm comfortable that the testing is correct.

So on a daily basis, on my live account, I generate strategies at 07:00 GMT, for CJ M15 using 1440 bars of data.  I generate for 45 minutes, then push live and start it trading at 08:00 GMT every day.  I leave it to open positions for a max of 384 bars (programmatically).

One of the most important things I found was the time of generating and pushing live.  It has to be consistent. And 07:00 worked best (for me at least).  If I left it an hour later, the results were very different.

1440 bars of history also seems to handle the volatility better.  I generate on 0.1 lots, but trade purely 0.01 lots, with a fixed 150 SL (min/max) and a "may use" TP.  I'll adjust my SL slightly based on ATR over the last 1440 bars - currently it's sitting around 110/115, so I'm just outside of the true range.

With the advent of a few new great features in Express Generator, I'm now able to re-validate previously generated collections as part of "todays" collection.  I'll use the previous 5 days collections and check them using the same generating criteria:

;; Generator stop settings
max_working_minutes      = 45
max_ascended_strategies  = 50000
collection_capacity      = 50

;; Acceptance criteria
min_win_loss_ratio     = 1.0
max_drawdown_percent   = 1
min_count_of_trades    = 5
min_r_squared          = 1.0

The min R-SQ of 1 was a late arrival - I found a few strategies that had weird r-sq values (negative) so I was using that to filter them out.  I tried using R-SQ of 80 or 85 on generating, but didn't get great results.

I also ensure that I've got correlation turned on (0.98) for both balance line and similar strategies - that seems to have worked well.  One thing I saw was that I was getting quite a few strategies in-market which were opening trades the same way at the same bar, which introduced far more risk.  I believe it was because they were similar strategies.  So the correlation settings have made that much better.

I also use the Monte Carlo testing in EG - but only loosely:

enable_monte_carlo        = true
count_of_tests            = 20
valid_tests_percent       = 100
spread_max                = 40
slippage_max              =  5
mc_min_count_of_trades    =  5
mc_max_drawdown_percent   =  1
mc_min_r_squared          =  1
mc_min_win_loss_ratio     =  1

Only about 45% of strategies ascended are passing the MC test as well - which is good to know.

I was also letting the EA open up to 50 positions as my EA contains 50 strategies each time.  However, I dropped it to 30, and now I'm down to 10 due to risk management - it has the potential to have open more than 40 positions at the same time (due to running 4 EAs opening positions for 384 bars and then letting them close naturally) and so I want to limit risk overall, which is tricky to do when you think of aggregate potential for loss.  Although it is comforting that the past 40 days has seen much volatility and this approach has performed well overall.

Obviously this is still a work in progress, but it continues to show promising results, both on forward testing, and in market.  The biggest thing for me was to manage risk better with this approach, which I think I've done now, so I'm happily letting it run on my live account and observing the results.

Attached is my settings file for reference.  Obviously my settings will suit me, not everyone.

Is there a setting in the EA to not open trades after x bars or did you add custom code to do that?

Sure, I'd love to see it if you don't mind. I was looking to see if there was a 3rd party EA to handle a feature like that, makes more sense to do it within the EA itself.

I also used to edit the EA to use a risk % per trade to ensure that none ever went over 1% risk for example for some prop firm rules, but that doesn't appear to work with recent updates. I haven't used EAS in over a year, and I'm eager to try out the new Express Generator. It's been fun reading up on the progress y'all have been making since I last visited the forums.

No worries - attached is a modified EA for Nov 2nd test in the spreadsheet.

My custom code is in two parts
Part 1 is from line 28 to 110 (inclusive)
Part 2 is from 380-400 (modified existing code - just compare the two trading functions OpenPosition and ClosePosition and you'll see the differences.

Originally I was using an account "safety" setting whereby it would close all open positions if the EA lost more than a specific amount of money (set as a parameter).  But I found it didn't work well so that's a bit of legacy code in there.

I basically use a spread check during the OpenPosition function.  Initially it checks the spread to see if it should open trades or not.  During that check I also calculate the number of bars since the launch date/time, and if thats less than my parameter (EALivesFor) then I say the spread is ok and max_open_positions remains at whatever was set, otherwise I set max_open_positions to zero and the spread is not ok, (piggybacking that function), and so the open new trade doesn't happen.

I admit I really need to clean that code up - apologies...but it works.

If you run this in MT5 strategy tester, the settings would be:

Entry lots: 0.01
Base Magic Number: 100
Max Open Positions: 10
Max Account balance loss: 1000
Don't Open Trade if Spread High: true
Max spread allowed: 40
Gracefully close after N bars: true
Time EA is set live: 2022.11.02 10:00:00
Number of bars EA opens: 384
(or if testing this using opt)
Number of bars EA opens: start: 4, step 4, stop 480

Thanks for sharing the spreadsheet!

It appears that your chart Total Profit / Loss by Number of Bars post launch could have passed an FTMO challenge very easily. That's promising! There was less than 250 draw down for 1400 profit. You could even increase the risk a little bit says .12 - .15 lots or so that you could hit the profit target in less than 30 days.

I'm in the USA so it's very difficult for me to put it on at 7 GMT as that's 2 AM for me.
Any suggestions for automating this process to happen while I sleep?

There's the Windows Events, which will run express-generator automatically.
I'm hung up on all the extra steps that would have to be done.
1. Export the EA in EA Studio
2. Add the additional source code correctly
3. Put the EA into data folder in MT4 / MT5.
4. Put the EA onto the chart.

Those are all things that would have to be automated for this to be realistic for me to even try.

Update adding the minimum count trades to 10 made my results far worse actually.