Truth in Backtesting (Part 1)

I came across an advertisement for a trading service that provides a good educational point about backtesting.  The lesson to learn is that I must always attempt to trade like I backtest.  This is the “truth in backtesting” mantra, and if this is not possible then I can expect my live trading performance to be much worse than backtested models.

The first thing to understand is that I should always expect live trading performance to be somewhat worse than backtesting would suggest.  This can occur for many different reasons:  transaction fees (slippage), curve-fitting, loss of system effectiveness over time, etc.   Even if I can steer clear of any unintended factors that might exaggerate backtesting results, I should still expect the live trading results to disappoint.

This also suggests I should avoid any system that generates marginal profit in backtesting.  These systems may not profit at all in live trading.

One common way to generate exaggerated backtesting results is to couple trade signals and trade execution.  The advertisement mentioned above is for a trading system package that allegedly performed as follows over the last three years:

Each row represents performance of a different trading system (10 total).  Trades are triggered by end-of-day signals.  The first set of numbers shows performance when buy/sell signals are taken at the very same close.  The second set of numbers shows performance when buy/sell signals are taken at the following open.

Is it realistic to couple trading signals with immediate trade execution at the closing price as shown in the first set of data, above?

I will continue this discussion in the next post.