The importance of backtesting

In this article we will discuss one of the most important and surprisingly most overlooked step in building a trading system – regardless of the kind of market it is developed for. Backtesting is the process of simulating trades caused by rules of the trading system on past (historical) data. All the process of developing a trading system is based on the suggestion that if it consistently worked in the past, then it will work in the future. Thus, backtesting is the only reliable way of confirming the trading system’s profitability – or rejecting it. Maybe it’s not the best way, but still we have nothing else.

However curious it might seem, but many traders – especially novice – tend to ignore backtesting or underestimate its importance. Partly this opinion is based on books they read to learn technical trading where some ready-to-use recipes are usually published. Thus it might seem apparent that you only have to stay strict about these rules and remain profitable. However real life very often shows results different from expectations – and, unfortunately, too often opposite to them. When looking back at his failures such a trader tends to blame himself for allowing too much freedom to his trading and neglecting the system he acquired, or misinterpreting its rules. This is quite possible, but nevertheless there might be another, not so obvious reason that might cause general failure: the system itself might be losing from the very beginning. Since a novice trader usually doesn’t have any means to confirm if the system is indeed profitable, he is left to just take the rules by faith. Yet an accurate check of a system’s rules on historical data could have possibly saved a lot of ruined deposits.

This article is the first of a series dedicated to developing a trading system. Here we will consider basic starting points for a trading system’s suitability for backtesting and some requirements it should meet to be successfully verified and then used in real trading.

Before applying a backtesting technique you should have clear answers to three major groups of questions:

  1. Can you follow the rules of the trading system you’re going to backtest strictly? Does it leave any space for guessing or discretionary trading of any sort? Are you able to formalize the rules?

  2. Which amount of data should be taken into consideration? One week, several months or dozens of years?

  3. Which definition is suitable for backtesting a particular strategy? End-of-day, or a week is enough, or maybe a minute looks too long an interval?

Let’s closely consider all of these three.

Do you understand the system?

This is how all the questions of the first group might be summarized. First of all you should honestly answer this question to yourself. And this answer should not be based just on your desire to start trading a new promising system, but on your weighted analysis of the rules.

The easiest way of answering this question is simply performing a manual executability check of system’s rules. This means acting as a broker, not as a trader, and see if a rule can generate an order which in turn can be executed.

Let’s consider an example. Say, a rule tells us to place a buy limit order at the previous high if a new low is lower than the previous. This rule is clear, and surely can generate an order which in turn can be executed.

Now another example. Say, a rule assumes buying at market if closing price is very far from its moving average with certain period. Smart eyes have undoubtedly noticed the weak point of this rule: “very far” is different for different markets, in different situations and – worst of all – for different traders. Now imagine yourself a broker – if you receive such an order from a client, what would you do? Obviously, you reject this order since you don’t have an idea when and where to place it.

Another example of a rule with zero executability is not so obvious. Suppose many of us have seen anywhere something like “buy at market after 2 or 3 bars” or “place a sell stop order at the lowest of the last 3 – 5 bars”. Especially when accompanied by astounding chart examples, such a rule may seem very reliable. But if you look at the rule a bit closer – again from a broker’s point of view – you’ll see that again this rule cannot generate orders by itself – only with human discretion. For it’s completely uncertain if one should take 2, 3, 5, or maybe some other quantity of bars into consideration.

Similar and not so obvious example of an ambiguous rule could be an attempt to use terminology like “find the last important extreme”: such a formula leaves too much space for guesswork. What seems important for some, doesn’t seem important for others. What seems important when you look at just the latest 20-30 bars on a chart may look quite different if you look at 100-200 bars — and so on.

This point is very wide-spread, and too frequently not taken into account. An explanation to this phenomenon might lie in human ability to watch important details without being consciously aware of them. If this is the case, then you might want to look for these details more intently because it is them which determine the exact action. For example, if the original rule recommends an entry “after 2 or 3 bars after breakout” it may turn reasonable to enter when volatility fades – and this solves the problem of counting indefinite number of bars.

Note that not only chart and its derivatives (indicators) data could be used as triggers for trading rules construction. One may successfully use fundamental factors, or news, or anything else. But in any case there still should be no room left for discretion. For example, a rule which sets up a buy entry at the highest high of 5 days in case the FED discount rate is above 3% is fine. A rule which suggest sell short a share if the company reports net losses over 1 million is just fine as well. But a rule which assumes buying or selling or exiting when, say, a news report is “positive” or “negative” is a mere nonsense. You can never tell a positive news from a negative one unless you have a precise method of distinguishing them based on figures, not emotions. And if you do have such a method then you don’t need to use those terms anymore.

The worst trading rules in my opinion are those based on trend lines (although Elliott theory seem to be even worse in this sense) since the most authors draw them ad hoc, and hence chances of unambiguously interpreting them for placing real orders are virtually nil.

If you build your own trading system you also need to pay particular attention to transparency of all its rules and avoid ambiguities.

The bottom line is: always check each and every rule of the trading system if it is consistent enough so that it would not require any additional decision-making to generate an executable order.

Is your system long or short-lived?

This is not an easy question to answer. Usually a trading system is considered to remain stable for a period of at least some years. Anyway you’re going to have consistent profits out of it, aren’t you? Thus speaking, you seem to need some years of data to backtest the system. But some systems are able to work only in very special market conditions which seldom happen and are mostly very short-lived. In this case you need only data pertaining to that special period of history.

The period of history and subsequently the amount of data required is strictly dependent on the idea which underlies the trading rules. One of the most common mistakes here is that entry and exit rules in the overwhelming majority of systems I saw are based on some setups seen on chart patterns and indicator formations (the notorious moving averages crossover being the most well-known). Usually these rules do not have any explanations why price should go up or down or anywhere after such a setup. Thus such a rule does not have any trading idea behind it. We will thoroughly consider this point in further articles, but right now you may want to keep in mind that the trading idea is what determines everything, including the period of history to backtest.

To make vague things clear, let’s have a look at EURUSD pair from 2003 to summer 2008. This is a nice example of long and steady uptrend caused by macro-economical reasons – so global, that it required a world crisis to crash it. So, it was wise to enter long-side only all the way for many years using any substantial correction as a good entry point. This is the trading idea, and it has nothing in common with chart formations and even with any particular time frame, because a correction can be defined in terms of price only, regardless of time. And only to make trading rules formal one may want to use indicators such as moving averages, stochastics, or other. This way indicators should serve the idea, not vice versa. Then it was reasonable to backtest such a system on at least some years of history.

On the other hand when Russian market was in deep trouble late 2008, and liquidity was thin, the government started to inject liquidity to it, allowing for frontrunning huge orders and in fact scalping for good chunks of money. Of course, this situation could not last long, and in the beginning of 2009 it returned to normal conditions. So, if you for some reason want to backtest your trading idea based on extreme lack of liquidity, you don’t need the whole history – you just look at these brief periods and backtest your ideas in hope hard times will come back one day.

Also, the amount of data required for correct backtesting depends also on time frame which is supposed to use. If you have an end-of-day strategy then you likely need at least 10 years of historical data, while a tick-based strategy might require as little as some months. That’s why questions of the third group are also very important.

This is all about the preliminary questions and answers about a trading system to be backtested. In the next article we will consider some common and unusual methods of finding valuable trading ideas and building trading rules based on them. Meanwhile I hope that the importance of first understanding and then backtesting a trading system is clear now, and you won’t make a fatal mistake of jumping into the pool without checking if there’s water in it first.

Leave a Reply