Topic: Project To Determine Most Effective Acceptance Criteria Settings
Over the next few days, I'm going to make a rough estimate on the effectiveness of the various EA Studio Acceptance Criteria.
I hope to determine the relative importance of each element and logically compose a more powerful set Acceptance Criteria.
For this model, I am adopting Sleytus' approach to EA Studio strategy generation and adding an Out Of Sample element. (Kudos to Sleytus for talking me into trying out this approach, leading to this large-scale test! Also, Hannah told me that data from 2014 onward is in general of more use to backtesting, as that's when algorithmic trading took off.) I am hopeful that the way in which I use the Out Of Sample period will greatly improve strategy selection and filtering.
This will be done specifically for the following settings:
Currency Pair: EURUSD
Time Frame: M30
SL: 10-100 (Must Use)
TP: 10-100 (May Use)
In Sample Start Date: 10-22-2015
In Sample End Date: 01-04-2017
Total In Sample Bars: 15,000
Out Of Sample Start Date: 01-05-2017
Out Of Sample End Date: 08-11-2017
Total Out Of Sample Bars: 7,500
My generation method will be System Quality Number, as I believe this to be the best metric available. SQN maximizes the Number of Trades and average Profit-vs.-Risk of trades, and minimizes Standard Deviation. The latter is very important to create a more balanced and predictable Equity Curve and has the effect of lowering drawdown.
1. I will first generate exclusively over the In Sample period using only very basic Acceptance Criteria, those which first appear when you hit "Reset Acceptance Criteria". Max Ambiguous Bars will remain constant through every subsequent generation.
* Optimization will be used +/-5 steps (Search: SQN; In Sample).
>Monte Carlo tests will number 100
* requiring 90% "Validated Tests"
* using the default settings--with the exception that "Randomize Indicator Parameters" (10/10/20) will be used.
2. My first run will be six hours in length over eight instances. I will then tally up the following numbers for future comparison:
* Percent "Generator/Passed Validation"
* Percent "Monte Carlo/Passed Validation"
* The Collection will be sorted on SQN.
3. Next, I will run each Collection through the Validator over the Out Of Sample period.
* I will visually inspect the resulting Equity Curves and discard any I deem "unacceptable." This is highly subjective (denoted by use of quote marks below!), but I will attempt to eliminate those that exhibit any of the following characteristics:
> "Large" drawdown at any point
> "Excessive" Stagnation (including some that looks like stagnation, but is not technically stagnation)
> "Uneven" Equity Curve
> "Any curve that I just don't get along with"
* Additionally, I will record Out Of Sample stats of the top two or three strategies from each instance for future comparison.
4. I will record the percent of strategies I eliminate in Step #3 for future comparison.
5. Lastly, I will rerun the remaining strategies and take note of the lowest value of each stat. This can possibly be used to indicate the strength of the remaining strategies, as well as serve as a guide for what the minimum effective Acceptance Criteria values might be.
After the initial process above, I will add a single Acceptance Criterion at a time to the original Criteria and rerun the generation. These generations will not be nearly as long. I will run eight instances for each Criterion, with each instance set to a different value of the new Criterion.
For each Acceptance Criterion, I will record the same data as in the initial generation.
I will then analyze each Acceptance Criterion value, comparing its results to results from the baseline generation.
Using these comparisons, I will be able to rank the Acceptance Criteria into an order of relative effectiveness.
I will then start combining Acceptance Criteria to determine how they interact with each other.
My goal is to determine the values at which the weaker strategies are eliminated, disallowing them from entering the Collection, but ensure the Criteria's combined effect does not start removing too many potentially viable strategies (and certainly no top strategies). I imagine that I will end up with most values being somewhat low.
I expect Balance Line Deviation and Maximum Consecutive to possibly surprise me with their effectiveness.
When I'm done, I will post my basic results.
Please feel free to add your input as to what might make this process better. But hurry, I hope to have it all wrapped up in three days!