Walking it Forward with System Validation (Part 2)

Last time, I began to build up to an example illustrating the process of Walk-Forward Analysis (WFA).  I introduced two examples of system development with the second including an optimization step. This prevents “flying blind,” to which the first example would be subject, and therefore makes the first example a more risky undertaking.

Just because a system traded well in the past, what reason do I have to think it will trade well in the future?  Robustness in terms of neighboring parameter values generating profitable backtesting returns is encouraging, but I still don’t have any future comparison to look at.  If I start to trade live without future comparison then I will eventually have the answer in terms of real performance, but this is another form of “flying blind” and seems a bit like gambling to me.

For this reason, it seems logical to divide up historical data into in-sample (IS) and out-of-sample (OOS) data.  IS data is that used to optimize the trading system and is known as the “training set” of data.  OOS data is that used to test the findings from the IS optimization and is known as the “test set” of data.  The key is to avoid use of the OOS data until the time of testing.  Do not use any OOS data to optimize variables during the IS period.

My third example will divide up the 15 years of historical data into the first 13 years, which I designate as IS, and the last two years, which I designate as OOS.  This time, I optimize over the first 13 years of data to find what SMA length works best.  I also check neighboring values of SMA length to make sure the optimal performance is not a fluke.  I will then backtest this system on the OOS data to see how performance compares.

I will continue the discussion in my next post.