Lingering Quandaries about System Development (Part 8)

In http://www.optionfanatic.com/2013/02/05/lingering-quandaries-about-system-development-part-7/, I introduced the third paradox encountered thus far in my System Development studies–this one having to do with walk-forward analysis (WFA).

As I suggested, something about the validation process Howard Bandy describes seems like curve fitting.  A few months ago, a reader asked Bandy in a forum post about the proper time ratio of IS:OOS data to be used in WFA.  Bandy did affirm that some time ratios may produce acceptable OOS performance where others may not.  His solution was to use whatever works.  To me, that sounds like cherry picking the right combination, which is “curve fitting:”  the four-letter word of System Development.

Just the other day, I once again directed this question to Bandy on his blog.  His response:

> Yes, in-sample and out-of-sample time periods are parameters of the system and they do need to
> be chosen. My recommendation is to choose them (particularly the length of the in-sample period)
> early in the development process, then keep them fixed from that point on… Keep in mind that
> every decision to adjust any component of a system based on examination of out-of-sample results
> reduces the out-of-sampleness of that data and increases the degree that the system is curve-fit to
> the specific data.

As a parameter of the system itself, choosing set values for IS and OOS periods is like Example 1 from http://www.optionfanatic.com/2013/01/29/walking-it-forward-with-system-validation-part-1/.  This fails to take into account the shape of the parameter space.  I want to see high plateaus of performance rather than peaks.  In taking Bandy’s suggestion, I would never study the neighboring values.

I will conclude this discussion in the next post.