Lingering Quandaries about System Development (Part 7)

I left off this series in http://www.optionfanatic.com/2013/01/28/lingering-quandaries-about-system-development-part-6/ talking about paradoxical views on robustness.  This along with a subjective function paradox described in http://www.optionfanatic.com/2013/01/18/lingering-quandries-about-system-development-part-1/ have stunted my progress in making sense of System Development.  Today I will introduce a third paradox within my understanding having to do with walk-forward analysis (WFA).

I finally added WFA, or validation, to this blog in a series of posts ending with http://www.optionfanatic.com/2013/02/04/walking-it-forward-with-system-validation-part-5/.  In that fourth example, I used two years of IS data followed by one year of OOS data.  This was based on:

> Assuming that I can test and optimize the trading system on as little as two years of data… [1]

What does this actually mean?

I can optimize and test the trading system on an infinite number of time ratios.  A few examples include:  three years IS to six months OOS, 30 months IS to nine months OOS, 18 months IS to three months OOS, etc.  What I am most interested in is the overall equity curve formed by piecing together (also known as concatenating) the results of each OOS testing.  Total number of trades could be a limiting factor because as the OOS time interval decreases, fewer trades may be generated.  If the concatenated equity curve does not have at least 50-60 trades then I am likely to consider it fluke and less meaningful.

Sample size concerns aside, what seems logical is that some time ratios will generate acceptable concatenated equity curves and others will not.   Perhaps [1] should be rewritten:

> Assuming the specified time ratio generates solid OOS performance…

This, however, is circular reasoning because it suggests choosing the time ratio based on whether the concatenated equity curve is good.  A main reason to employ WFA in the first place is to validate whether the concatenated equity curve will be good.

Put another way, this reeks of curve-fitting WFA–the very tool being used to prevent curve-fitting!

Let’s sleep on this, shall we?