Topic: I'm Yet to Build the Best Backtester and Generator

Hello Traders and fellow followers,

I'm going to start designing and building a strategy tester and a generator from scratch.

After ten years of working on FSB and FSB Pro and six years on EA Studio, I'm more experienced and more critical to my previous work.

Let's say I'm not completely happy with my achievements so far.

The significant drawbacks on the current backtesting engines are:
- FSB Pro is very complex, making it slow, impossible to fix or expand, and impossible to export pretty looking MQL code.
- EA Studio addresses the FSB Pro issues but lacks some requested features.

So what to do next?

Here is an outline of my plans and rationals:

Application:
- the product will be web-based - The browser-based applications are as fast as the classical desktop solutions.

Strategy structure:
- a single entry point and a single exit point - The strategies will open position at bar's Open and close at next bar's Open (on the same bar's Close for TradingView).
- Stop Loss, Take Profit, Break Even implemented as indicators. It will make it possible to have variable values based on Standard Deviation, previous High or Low,  percentage of the price, percentage of the account (I know some traders die for that smile ). Of course, they will accept fixed Pip or Point values also.
- variable entry-amount - lots, dollar value, equity %, free margin %...

Computation:
- the indicators will reuse data and signals arrays to reduce the memory allocation and cleanup.
- separation of the calculation code from the code for generating MQL, PineScript, ...
- calculating strategies on pristine market conditions - this will allow us to reject strategies that fail early.
- applying spread, slippage, swaps, commissions, Monte Carlo distortions later. It will make the calculations faster and reusable. For example, Monte Carlo effects can be indicator signals instead of hard-coded within the backtester.
     Imagine the simulation of 'Randomly skip entries'. Now the backtester looks at it every time because it doesn't know if it is allowed to open a position or not. It will be much easier to make it an indicator with buy and sell signals where the entries are randomly distributed and take, let's say, 20% of the length — no unique code for that purpose in the backtester anymore.
- having pristine strategy results will allow us to collect them on a central repo and recalculate them later on new data with specific market conditions applied. It will even not need to recalculate the indicators.
- calculating backtesting stats and applying Acceptance Criteria gradually - it means why to calculate R-squared if the strategy fails on Drawdown. We can reject it and continue with the next one.


Work in progress

I'm already working on a new application now.

It is a Generator for creating strategies for TradingView and exporting PineScript. It is a perfect moment to start from scratch because TradingView works differently from MetaTrader. It closes positions at Bar CLose and has a different set of embedded indicators.

I'll test my ideas with this app. I'm planning it to be simple and oriented towards traders with no experience with the algotrading. I hope it will take me 4-6 months to finish it.

Later, I'll see what works best and optimize EA Studio whit whatever is possible without breaking its current workflow.

My final goal is to test and calculate such fast to make a standby mobile phone application possible to calculate real-time trading signals.

Please share your opinion.

Trade Safe!

Re: I'm Yet to Build the Best Backtester and Generator

It is a logical step forward in terms of re-working the areas of obvious limitations of the past and much needed optimization of the main concept. In my own development work I very much see the same - from a complex thing to a much simpler end-product, so to speak, with better efficiency and performance. Evolution, not revolution.

I'm not a fan of it being web-based, browser updates can make life difficult sometimes, and being web-dependent is dependency, which is a draw-back in itself.

What's the trade-off of having only one entry/exit point (bar open only, as I understand)? I feel it being a step back.

Re: I'm Yet to Build the Best Backtester and Generator

Additional stuff:

- Indicators - separating signals calculations from indicator values. It will reduce the resource required because when we generate a strategy, we don't care about the values necessary for the charts.

- Using TypedArray. it appears that the JS typed arrays are 2 times faster than the generic JS array. I just made a benchmark for initialising arrays with a real number: https://www.measurethat.net/Benchmarks/ … ult/269714. Actually, I'm using now generic arrays with `push`. It is 4 times slower.

Re: I'm Yet to Build the Best Backtester and Generator

Hello Footon,

Thank you for your comments!

>I'm not a fan of it being web-based, browser updates can make life difficult sometimes, and being web-dependent is dependency, which is a draw-back in itself.

I found out that web's development,  update, distribution and licensing is 10 times easier and faster. The UI is more fluid, faster and responsive.

The Windows software dependencies are heavier and the development requires third-party software.

90% of our future users will use phones and they all have a browser.
Even Microsoft consider selling Windows (or stopping the development).
Microsoft switched to Linux and make their money from Azzure.

I can upload (and I'm doing it) 20 updates per day.

With my recent framework (tested in https://coinrang.com/) the user's app is absolutely disconnected from the server.

> What's the trade-off of having only one entry/exit point (bar open only, as I understand)?

It gurantes a precise backtest.

If we have entries within a bar, we need various interpolation methods. FSB Pro provides the "Pessimistic" one, which is safe, but this is not the case in any other testing software - MT, or TradingView.

MT interpolation is a joke. They also don't provide a Close event or price.
Trading View opens only at Open and closes at Close.

If we make software for live trading or signals, what I want to do, must fetch ticks all the time in order to calculate a signal. As opposed to that, if you have 20 strategies in M5, I'l require only one ping every 5 minutes to calculate the signals. It can be a background process on a phone.

Re: I'm Yet to Build the Best Backtester and Generator

> I'm not a fan of it being web-based

To be honest, I would like to program in C on Linux.

My second backtester was in C. Simply set a data file and run it. It showed status on the console and saved a file with the strats.

Another preferable option is .NET Core.

The problem is that except few geeks, no one will run it.

Another problem is that desktop products are easier to hack. For that reason, 80% of my "clients" are from Russia, Syria and Iran. I like these people and wish them luck and success, but they even don't say Thank you.

And another significant reason:
- my web-based products are easier to re-brand and integrate into third-party websites. Now my partners sell EA Studio and I only care for the development.
Here are two examples:
   Algo-Camp
   Sapphire Markets

I generally don't care about the number of sales, but it is much easier to focus on the development if someone other makes marketing and answers hundreds of support questions such as "I started my EA but id doesn't work. 15 minutes passed and it still doesn't trade".

6 (edited by Popov 2022-02-22 18:05:50)

Re: I'm Yet to Build the Best Backtester and Generator

Testing new code fragments.
The current EA Studio code calculates 10 times MA on 100 000 bars for  11 milliseconds.
The new code does that for 3 milliseconds (3+ times faster)

Current code:

public static simpleMA(source: number[], period: number, shift: number): number[] {
    const length: number  = source.length
    const lastBar: number = Math.min(length, length - shift)
    const ma: number[]    = IndicatorBase.initMA(source, period, shift)

    for (let bar: number = period; bar < lastBar; bar++) {
        ma[bar + shift] = ma[bar + shift - 1] + (source[bar] - source[bar - period]) / period
    }

    return ma
}


private static initMA(source: number[], period: number, shift: number): number[] {
    const length: number  = source.length
    const lastBar: number = Math.min(length, length - shift)
    const ma: number[]    = new Array(length)

    for (let bar: number = 0; bar < period + shift - 1; bar++) {
        ma[bar] = 0.0
    }

    let sum: number = 0.0
    for (let bar: number = 0; bar < period; bar++) {
        sum += source[bar]
    }

    ma[period + shift - 1] = sum / period

    for (let bar: number = lastBar; bar < length; bar++) {
        ma[bar] = 0.0
    }

    return ma
}


New code:

/**
 * Simple Moving Average
 *
 * @param {Float64Array} source - the data source
 * @param {Float64Array} maRef  - reference to ma. maRef.length == source.length
 * @param {number}       period - period >= 1
 * @param {number}       shift  - shift  >= 0 && period + shift < source.length
 *
 * @return {void}
 */
public sma(source: Float64Array, maRef: Float64Array, period: number, shift: number): void {
    let sum = 0
    for (let bar = 0; bar < period; bar++)
        sum += source[bar]

    maRef[period + shift - 1] = sum / period

    for (let bar = period, lastBar = source.length - shift; bar < lastBar; bar++)
        maRef[bar + shift] = maRef[bar + shift - 1] + (source[bar] - source[bar - period]) / period
}

There are four differences between the code fragments.
- the first one uses a separate initMA function It is used by all other types of MAs. The necessary initialization is optimized and embedded in the second code variant.
- the first code uses generic arrays `number[]`. The second one uses typed arrays `Float64array`.
- the first code creates and returns a new MA array. It is semantically correct. However, it allocates a new array and the old one must be destroyed by the Garbage collector. On the other hand, the new code populates a given array by reference. It doesn't create garbage.
- the new code doesn't zero the start and the end of the MA because we provide it as `new Float64Array(source.length)`. It automatically initializes each element with 0.

The biggest performance improvement comes from the fact that the MA array is reused.

If I optimize the complete engine, it is feasible to have it 2 times faster or even more.

Re: I'm Yet to Build the Best Backtester and Generator

Wow smile 2- fold speed increase is a statement!

Re: I'm Yet to Build the Best Backtester and Generator

The simplest route is probably the best, you can backtrack if desired later in the project.

I prefer to see a few more indicators if that is feasible

Re: I'm Yet to Build the Best Backtester and Generator

> I prefer to see a few more indicators if that is feasible

Now we have the indicators that are both in MT4 and MT5.

Another option is to make a standard pack of MT indicators and to add it to MetaTrader before exporting experts.

10 (edited by geektrader 2022-02-25 23:05:24)

Re: I'm Yet to Build the Best Backtester and Generator

Very nice, go for it and I am happy to hear this is coming to EA Studio too. My most requested features would be:

- Break-even option

- Exit after bars (that´s one of the most effective exits that all the other platforms seem to steadily generate when searching for strategies - sure thing because exiting if a trade does not work out is the best option indeed)

- A trailing stop with a configurable starting point and trailing distance (in pips and ATR based)

- Just as you say, being able to use indicator-based levels (the ones that return price values like Fibonacci levels, MA´s, Parabolic SAR, etc.) and ATR-based levels for SL / TP / BE / Trailing Stop would be super nice.

- Merged multi-symbol EAs, so that we can export a portfolio of different strategies on different symbols into one EA that works not just on one symbol (currently have to do this manually and it works nice but would be great to have that as a standard feature).

- More indicators, already merged into the exported code, so that we don´t have to mess around with external indicator files.

- Weighted fitness (one of my most important requests as I use that a lot in other platforms to get exactly the strategies I want without having to mess around with multiple optimizations). E.g., instead of just going for the best R-Squared or the best Ret/DD ratio, or lowest stagnation, we can weight these goals - say 70% focus on best R-Squared, 25% on Ret/DD ratio, and 5% on lowest stagnation.

Any speed improvement is always welcome as well and it´s amazing to see that you can make this even faster - holy moly wink EA Studio, on a single thread, is already ~x20 faster than another platform I am using that runs on 32 threads. Amazing work Mr. Popov!

Thanks for all your efforts - it´s so much appreciated.

Re: I'm Yet to Build the Best Backtester and Generator

Popov wrote:

> I prefer to see a few more indicators if that is feasible

Now we have the indicators that are both in MT4 and MT5.

Another option is to make a standard pack of MT indicators and to add it to MetaTrader before exporting experts.


I think adding a pack of indicators to Metatrader is ideal, especially  for developments such as advocated by geektrader.

Re: I'm Yet to Build the Best Backtester and Generator

Great news! Hope to see another cutting-edge software for traders soon! TradingView is very popular between traders in Brazil too! Hope you the best

Re: I'm Yet to Build the Best Backtester and Generator

The new generator already works.

See my first comparison of performance between the new generator and EA Studio.

14 (edited by zenoni 2022-08-06 02:12:27)

Re: I'm Yet to Build the Best Backtester and Generator

The new software version looks very good. I'm a bit worried about what will happen to the FSB Pro software, whether it will be supported or retired?

Re: I'm Yet to Build the Best Backtester and Generator

>  I'm a bit worried about what will happen to the FSB Pro software, whether it will be supported or retired?

I'll release FSB Pro v4.00 within 2 weeks.
It will have easier data import (as EA Studio) and will be able to import collections from the new generator.
I'll think of import directly from EA Studio (if found a way how to fit position close at next Bar Open to the current FSB Pro backtester)

Ideally it may have someday Express Generator integrated and Express Experts - simple experts as the ones from EA Studio.