Walking it Forward with System Validation (Part 1)

As discussed in http://www.optionfanatic.com/2012/10/29/drive-the-monte-carlo-to-consistent-trading-profits, Monte Carlo analysis is a way to get a broader statistical view, or a range of what to expect from my trading system.  According to Howard Bandy, who has written many books on AmiBroker and System Development, the process is not complete until it passes validation in the form of Walk-Forward Analysis (WFA).  WFA combats the tendency of system developers to curve-fit.

In order to illustrate this, I will present four examples.

My hypothetical system enters long (short) at the next open if closing price is above (below) a simple moving average (SMA).  Trades are held until a reverse signal appears.  One market will be traded and only one position will be held at a time.

For the first example, suppose I backtest this system over the last 15 years using a 20-period SMA.  The equity curve looks great and the subjective function value is high.  Thinking I have found the Holy Grail, I start trading this tomorrow.  This is how things might look for many people who find trading strategies to backtest in books or on web sites.

Contrast this with a second example where I backtest over the last 15 years and optimize by varying the SMA length from 5 to 100 in five-day increments.  With 20 potential SMA lengths, I am testing 20 different systems.  I choose the best performing system to trade live starting tomorrow.

While this example is probably the epitome of curve fitting, I would consider it better than the first because I can see how the system performs with neighboring SMA lengths.  As discussed in http://www.optionfanatic.com/2012/09/28/trading-system-1-spy-vix-part-1, through the optimization process I have more data that enables me to determine whether my impressive performance is a spike peak on the graph (fluke) or part of a high plateau (more robust).  In the first example, I am flying completely blind.

I will continue with more examples in the next post.