Walking it Forward with System Validation (Part 3)

In http://www.optionfanatic.com/2013/01/30/walking-it-forward-with-system-validation-part-2/, I introduced the concept of in-sample (IS) and out-of-sample (OOS) data.  I will continue today by going into more detail about the third example.

In this example, I have divided the 15 years of historical data into 13 years of IS data and two years of OOS data.  What I can now do is test the system established using the first 13 years of data on the last two years.  This is an improvement over the second example where optimization was performed over all 15 years of historical data.  Only real money was available to test that system’s effectiveness, which is why I said it was somewhat of a gamble.

If the system does not perform well on the OOS data then I would not trade it with real money.  From my perspective, this is like bringing my wife along to purchase new clothes.  Neither of us are fashion mavens.  However, if I pick something out that she doesn’t like then an increased probability exists that other people won’t like it.  If I pick something out that she does like then an increased probability exists that other people will like it.  In both cases, one validation is better than none.

I believe it important to stress that neither one validation nor one rejection is any guarantee of future success or failure.  If the system performs well on the OOS data then it is not guaranteed to profit in live trading.  Similarly, if the system fails to perform well on the OOS data then it is not guaranteed to lose in live trading.  The goal of system development is simply to generate enough confidence in a system to consistently trade it with real money.  This premise guides my decision making.

In my next post, I will take 1-step validation to the next level.