I think you will have to experiment as to what risk you feel comfortable with.
]]>Btw, I try to build my bots in a very conservative way. For example if my broker is typically a 2-pip spread on EURUSD, I tell FSB that it's 2.5. I also plugin a slippage number that is higher than typical. I want the Generator to be looking at worst-case scenarios if possible.
That's a very good practice. Even is the broker executes the orders perfectly on time, there is difference in the designs of FSB and FST that leads to slippage.
For example, FSB executes the orders at the exact level, if not slippage is intentionally applied.
FST works in slightly different way. It sends order when the market hits or jumps over the target price. In that way, we always have a small slippage on fast markets.
Generally speaking, I have been somewhat against backtesting with my discretionary strategies and more pro-forwardtesting.
Probably you can use backtest to filter out most of the strategies. If a strategy fails on the backtet, it will probably fail on the forward test also. This can save enormous amount of time, money and efforts.
I am not too concerned about over-optimization because of the OOS curve (it must look excellent for me to get excited about it).
OOS can be quite tricky if one does not understand what it means.
But if one takes a look at a chart from 20 years ago, it looks the same, doesn't it. A bar opens, and it has only 3 possible outcomes - it finishes higher, lower or stays at the same level as previous bar. It was like this, and it is like this now, I've been thinking about defining the "market change", cannot really give a good answer, can you?]]>What is your way of finding the true fitness of a strat? By true fitness I mean avoiding the over-optimization pitfall, a strat which has a greater probability being profitable in live trading (the opposite would be a great one in FSB, but a 25% DD generator in live).
Well, I haven't been doing this long enough to answer with conviction, but I will tell you my hypothesis on this:
Generally speaking, I have been somewhat against backtesting with my discretionary strategies and more pro-forwardtesting. I follow a similar philosophy with robotic trading. My theory is that markets are dynamic and change "too much" to make a backtest "highly relevant". My belief is that a backtest is useful to a degree, such as getting you in the ballpark. But the real test will be the forwardtest trading.
With FSB I'm focusing entirely on 1minute trading, at least for now. Especially in these turbulent times, I think it makes sense to trade shorter timeframes. In my discretionary trading I've been whipped around pretty regularly as of late in the medium-term (ie swing trade) timeframes.
As such, my belief is that an M1 strategy does not "need" a large amount of testing data. 50000 bars seems like a good bit to determine if a robot is "smart enough" to weather the storm. 1440 bars per day equates to 34 days, which is nearly a month-and-a-half in real life. In my view, if a robot can perform well on M1 for this long, including an excellent OOS 30%+ performance, then it should be ready for forward testing.
I am not too concerned about over-optimization because of the OOS curve (it must look excellent for me to get excited about it).
Also, I'm not comfortable with the 25% DD as my only metric to know when to stop trading a bot. My belief is that every bot will run its course and then become useless (due to macro market landscapes changing). I'd like to find a better (non-subjective) way to look at how a bot is performing and be able to say with conviction "this one is out of spec now". The 25% just happens to be one non-arbitrary way, since we know that the backtesting used by the Generator does not allow 25% DD (as configured). Therefore, if it hits 25% in forward/real performance then we know it is out-of-spec.
Btw, I try to build my bots in a very conservative way. For example if my broker is typically a 2-pip spread on EURUSD, I tell FSB that it's 2.5. I also plugin a slippage number that is higher than typical. I want the Generator to be looking at worst-case scenarios if possible.
PS: with regard to backtesting, I think OOS testing is one of the most valuable parts. I haven't figured out the best way to do this yet, but one thing I want to do is manually extract a subset of M1 data under varying turbulent conditions (ie, sideways ranging like we saw during the holidays, strong trend, etc) and then be able to use that as another OOS-validation step for prospective bots.
]]>What the Generator curve should look like
The curve should have a nice consistent angle, about 45 degrees. Several jaggy points along the way are preferred. Minimal major swings. Also preferable is that the balance/equity lines are hugging as closely as possible throughout the chart (in-trade drawdowns).
What the OOS portion of the curve should look like
The OOS portion of the plot should be a “contributor” to the overall plot quality. Specifically, the OOS portion should have an excellent consistent path upward, with a strong angle and multiple data points where balance/equity line are hugging.
What qualities should be used to differentiate the Top 10
Account Balance is important, but perhaps not the MOST important. A higher win/loss ratio means less risk in my view, as does a low max drawdown. Reduction of risk and/or preservation of capital is paramount -- more important than Account Balance to me. Ideally I want the best combination of: highest Account Balance, lowest win/loss ratio and prefer higher win/loss ratio.
When should a live tradebot be considered “out of spec” with regard to performance?
If the tradebot should generate an actual drawdown of 25%, it should be stopped immediately and sent back to the drawing board. During testing all strategies are scrapped that hit a 25% or greater drawdown, so this is a minimum expected performance standard. In this scenario, the first 25% account growth would be considered a “breakeven point” for the bot, because in its final life, it would be retired at 25% less than its peak.