Walking it Forward with System Validation (Part 6)

My blog series “Lingering Quandaries about System Development” concluded by discussing a paradox with regard to Howard Bandy’s WFA discussion.  In http://www.optionfanatic.com/2013/02/07/lingering-quandaries-about-system-development-part-9/, I concluded by suggesting a new and improved WFA that can achieve system development goals.

I recommend not following Bandy’s advice to select periods for IS and OOS data early in the process and retaining them throughout development.  Besides effectively burying your head in the sand as described by Example 1 (http://www.optionfanatic.com/2013/01/29/walking-it-forward-with-system-validation-part-1/), you really can’t know what values may or may not work until you actually perform the WFA.  Validation is the final step of system development.

Rather, look to perform WFA by optimizing the periods and studying the entire parameter space.  Perhaps I will vary IS period from one year to three years by increments of two months.  Perhaps I will vary OOS period from one month to six months by increments of one month.  I then need to plot values of the subjective function in a three-dimensional space (or in two dimensions with color coding to represent subjective function ranges) to get a feel for where the high plateaus exist.  I should then select the time ratio to coincide with the middle of a high plateau and use trading parameters coincident with that combination.

Every time I do a WF iteration, I am selecting WF parameter values and subsequently trading parameter values.  The two differ, in effect, by an order of differentiation.  That is, the subjective function for the WF optimization coincides with a value for the concatenated equity curve to date.  The subjective function for the iteration coincides with a value for the equity curve of the preceding IS period only.

WFA serves to both validate a trading system and to direct trading at the right edge of a chart.

Lingering Quandaries about System Development (Part 9)

http://www.optionfanatic.com/2013/02/06/lingering-quandaries-about-system-development-part-8/ continued discussion of a third System Development paradox that I have been trying to sort through:  Howard Bandy’s handling of WFA.  Bandy said to choose periods for IS and OOS data early in the process and then stick with them throughout development.

The more you vary these system parameters, Bandy said, the more the OOS data loses its “out-of-sampleness,” which is why they ought not to be changed.  I flat-out disagree with this statement.  Varying the time ratio does not change or determine parameter values involved with the trading rules.  Varying the time ratio is solely to verify that neighboring values also validate the system.  This will remove suspicion of system validation as a fluke occurrence.

One issue that remains unresolved for me is how to conceptualize “varying the time ratios” in a systematic manner that may be plotted.  I consider myself spatially challenged so I probably just need to cram into my brain the need to specify a minimum and maximum value for both IS and OOS periods and an increment by which to vary them.  I can then make a two-dimensional plot of the subjective function (e.g. RAR/MDD) and perhaps color code the values.  This way, I can see if an area of outperformance stands out.

One other issue I wonder about is whether OOS period might be limited by sample size considerations.  The OOS period will already be short relative to the IS period.  Might I need to be concerned about it being short enough to allow for any trades?  The shorter the period, the more total periods I will have in the WFA but if it is too small to allow for trades in any one iteration then I wonder if the entire WFA risks insufficiency.

I will only discover the answer to this question when I start attempting WFA myself.

In my next post, I will summarize the new-and-improved approach to WFA.

Lingering Quandaries about System Development (Part 8)

In http://www.optionfanatic.com/2013/02/05/lingering-quandaries-about-system-development-part-7/, I introduced the third paradox encountered thus far in my System Development studies–this one having to do with walk-forward analysis (WFA).

As I suggested, something about the validation process Howard Bandy describes seems like curve fitting.  A few months ago, a reader asked Bandy in a forum post about the proper time ratio of IS:OOS data to be used in WFA.  Bandy did affirm that some time ratios may produce acceptable OOS performance where others may not.  His solution was to use whatever works.  To me, that sounds like cherry picking the right combination, which is “curve fitting:”  the four-letter word of System Development.

Just the other day, I once again directed this question to Bandy on his blog.  His response:

> Yes, in-sample and out-of-sample time periods are parameters of the system and they do need to
> be chosen. My recommendation is to choose them (particularly the length of the in-sample period)
> early in the development process, then keep them fixed from that point on… Keep in mind that
> every decision to adjust any component of a system based on examination of out-of-sample results
> reduces the out-of-sampleness of that data and increases the degree that the system is curve-fit to
> the specific data.

As a parameter of the system itself, choosing set values for IS and OOS periods is like Example 1 from http://www.optionfanatic.com/2013/01/29/walking-it-forward-with-system-validation-part-1/.  This fails to take into account the shape of the parameter space.  I want to see high plateaus of performance rather than peaks.  In taking Bandy’s suggestion, I would never study the neighboring values.

I will conclude this discussion in the next post.

Lingering Quandaries about System Development (Part 7)

I left off this series in http://www.optionfanatic.com/2013/01/28/lingering-quandaries-about-system-development-part-6/ talking about paradoxical views on robustness.  This along with a subjective function paradox described in http://www.optionfanatic.com/2013/01/18/lingering-quandries-about-system-development-part-1/ have stunted my progress in making sense of System Development.  Today I will introduce a third paradox within my understanding having to do with walk-forward analysis (WFA).

I finally added WFA, or validation, to this blog in a series of posts ending with http://www.optionfanatic.com/2013/02/04/walking-it-forward-with-system-validation-part-5/.  In that fourth example, I used two years of IS data followed by one year of OOS data.  This was based on:

> Assuming that I can test and optimize the trading system on as little as two years of data… [1]

What does this actually mean?

I can optimize and test the trading system on an infinite number of time ratios.  A few examples include:  three years IS to six months OOS, 30 months IS to nine months OOS, 18 months IS to three months OOS, etc.  What I am most interested in is the overall equity curve formed by piecing together (also known as concatenating) the results of each OOS testing.  Total number of trades could be a limiting factor because as the OOS time interval decreases, fewer trades may be generated.  If the concatenated equity curve does not have at least 50-60 trades then I am likely to consider it fluke and less meaningful.

Sample size concerns aside, what seems logical is that some time ratios will generate acceptable concatenated equity curves and others will not.   Perhaps [1] should be rewritten:

> Assuming the specified time ratio generates solid OOS performance…

This, however, is circular reasoning because it suggests choosing the time ratio based on whether the concatenated equity curve is good.  A main reason to employ WFA in the first place is to validate whether the concatenated equity curve will be good.

Put another way, this reeks of curve-fitting WFA–the very tool being used to prevent curve-fitting!

Let’s sleep on this, shall we?

Walking it Forward with System Validation (Part 5)

In http://www.optionfanatic.com/2013/02/01/walking-it-forward-with-system-validation-part-4/, I introduced the process of Walk-Forward Analysis (WFA).

The pictorial representation of the WFA process for Example #4 is as follows:

Note how WFA achieves 13 full years of OOS testing and validation.

Howard Bandy considers WFA to be the gold standard of trading system validation, and I see multiple reasons to support his claim.  First, the largest risk for curve-fitting has been eliminated by using OOS data to test a system developed using IS data.  To find the best combination of trading parameters and then to advertise said system to potential customers is unrealistic at best and criminal at worst.  Second, with WFA my system will better adapt to changes in market behavior over time.  Change in market behavior is responsible for systems working well until they don’t.  WFA provides the means to adapt.  Provided my OOS data is extensive enough to sample all market environments, a WFA equity curve sufficient to meet my personal criteria (i.e. subjective function) should give me the confidence necessary to trade the system live.  WFA is not just a nifty backtesting tool; it offers a process that may be done at any time to resync trading parameters with recent market activity.

In this blog series, I have studied four general approaches to system development.  In Example #1, I backtested one set of system parameters to trade live if results impressed.  In Example #2, I optimized a trading system over historical data to subsequently trade live.  In Example #3, I optimized a trading system over 13 of 15 years of historical data and used the last two years to validate the system.  In Example #4, I used WFA to generate 13 full years of OOS validation on a trading system that periodically aligns itself to recent market activity.

Because the system parameters may adapt, WFA results in a dynamic trading system that is qualitatively different from what most people conceptualize when discussing system development.

This writer believes it makes a lot of sense.

Walking it Forward with System Validation (Part 4)

In http://www.optionfanatic.com/2013/01/31/walking-it-forward-with-system-validation-part-3/, I described validation in the third example that allowed me to test the optimized system on some new (OOS) data before risking real money.  Today I want to go one step further and introduce walk-forward analysis (WFA).

The problem with the third example is that only the last two years of data are used to validate a system optimized over the previous 13.  What would be nicer would be to generate a longer equity curve composed solely of OOS data.  This is precisely what WFA allows me to do.

Assuming that I can test and optimize the trading system on as little as two years of data, suppose for my fourth example that I begin by optimizing trading parameters over the first two years and then testing the system over the following 12 months.  The precise ratio of IS:OOS lengths is somewhat arbitrary (a point I will refocus on later) but for this example I will stick with two years:one year.  I now have one year of validation, which is less than the two years I had in Example #3.

What happens next is the key ingredient:  I slide the two-year data window forward by a year and repeat the analysis.  What results is one additional year of validation, which brings me to the end of data year #4.

In this manner, I will continue walking forward and optimizing each rolling two-year data period followed by a one-year test of efficacy.  When the data finally runs out, I will have amassed 13 years of OOS validation–potentially with different trading parameters every 12 months.  The system performance for these 13 OOS years is a much better indication of how a system will perform in real time than the performance of any single time period used for optimization.

I will continue discussion of WFA in the next post.