Lingering Quandaries about System Development (Part 9)

http://www.optionfanatic.com/2013/02/06/lingering-quandaries-about-system-development-part-8/ continued discussion of a third System Development paradox that I have been trying to sort through:  Howard Bandy’s handling of WFA.  Bandy said to choose periods for IS and OOS data early in the process and then stick with them throughout development.

The more you vary these system parameters, Bandy said, the more the OOS data loses its “out-of-sampleness,” which is why they ought not to be changed.  I flat-out disagree with this statement.  Varying the time ratio does not change or determine parameter values involved with the trading rules.  Varying the time ratio is solely to verify that neighboring values also validate the system.  This will remove suspicion of system validation as a fluke occurrence.

One issue that remains unresolved for me is how to conceptualize “varying the time ratios” in a systematic manner that may be plotted.  I consider myself spatially challenged so I probably just need to cram into my brain the need to specify a minimum and maximum value for both IS and OOS periods and an increment by which to vary them.  I can then make a two-dimensional plot of the subjective function (e.g. RAR/MDD) and perhaps color code the values.  This way, I can see if an area of outperformance stands out.

One other issue I wonder about is whether OOS period might be limited by sample size considerations.  The OOS period will already be short relative to the IS period.  Might I need to be concerned about it being short enough to allow for any trades?  The shorter the period, the more total periods I will have in the WFA but if it is too small to allow for trades in any one iteration then I wonder if the entire WFA risks insufficiency.

I will only discover the answer to this question when I start attempting WFA myself.

In my next post, I will summarize the new-and-improved approach to WFA.