Trading System Development 101 (Part 4)

Back in Part 3 of this mini-series, I drilled down into some details about feasibility testing. I left off with ways to increase number of trades in order to avoid small sample sizes.

Requiring a decent sample size in feasibility has the controversial consequence of eliminating strategies with long average hold times. Some people would be happy with a strategy that generates, for example, less than one trade every two months if their subjective function criteria are met. This is a matter of personal preference. I think if I am going to test trades that have few trades in feasibility periods, then I need to go straight to the full dataset and test. In doing this, I need to be careful to avoid curve fitting: seeing the results, tweaking the strategy, retesting, and repeating.

Another situation where few/zero trades may be generated in feasibility is when testing hedge strategies. Recall the VIX filter previously discussed (second-to-last paragraph here). How many instances do we have of VIX over 20 – 50 by increment of 3 in the last 12 years? Not many, and when we do they are largely clustered in time. Most 2-year feasibility periods have zero trades, which makes preliminary assessment difficult. In this case, I would probably scrap feasibility testing altogether and look for more than a small sample size when testing over the whole dataset to avoid curve fitting.

Going back to the Eurostat excerpt, empirical evidence based on OS forecast performance is generally considered more trustworthy than evidence based on IS performance. The latter is more sensitive to extremes and data mining. Because the strategy has not yet tested on OS data, OS forecasts better reflect the information available to the forecaster in live trading (i.e. strategy has not been tested on future data either).

For this reason, the next phase after I find a strategy with 70% iterations profitable is walk-forward analysis (WFA). I described WFA here and included a pictorial representation here. WFO (optimization) is the same thing as WFA except it places emphasis on the fact that specific parameters used for OS are determined by an exhaustive optimization over IS.

If a strategy passes WFO, then the next phase of development is Monte Carlo (MC) simulation, which I discussed here and here. For each simulation, I will compute a ratio of average annualized return to maximum drawdown. I want to see a ratio above a pre-determined threshold to advance the strategy to the next phase.

The final phase of development is incubation. Here, I will paper trade the strategy (i.e. trade on “sim”). If performance looks to be “within normal limits” based on WFA and MC, then I can start trading it live.

Next time, I will make some comments about this walk-forward approach to trading system development.