Backtesting Methodology (Part 1)

I don’t feel the backtesting statistics do the data justice without some explanation of the study methodology.

One of the most common approaches I have seen to option backtesting is to start trades on a particular day. I have seen people start backtested trades with 45 days to expiration, 36 days to expiration, or on the first trading day of every month.

One potential problem I see here is insufficient sample size. Traditionally, people could go back a number of years with a backtest and only have a sample size equating to 12 times the number of years. More recently with weekly options, sample sizes can be 4+ times greater but only if the strategy is a weekly strategy. Because Weeklys^SM are considered by many to carry excessive risk, I believe monthly trades are much more common.

Exactly what is an adequate sample size, then? I’ll go with: 1. I don’t know; 2. It depends (on the particular situation). I really want to lean toward the latter and say “whatever sample size is sufficient for statistical significance.” The question would then be whether a statistically significant result is practically significant.

Based on my experience, traders are notorious for using insufficient sample sizes in backtesting and live trading. I’ve seen plenty of people backtest a monthly option trade for 1-3 years. With 12-36 occurrences, this hardly strikes me as sufficient! Perhaps my favorite example comes from traders trying to apply technical analysis (TA). For example, this comes courtesy of the Twittosphere (names blacked out to protect the “innocent”):

Obviously (according to TA) the long trade is “likely good” here because each of the previous times the stock has pulled back to the positively-sloped green line, it has moved higher. How many times was that? Look closely to make sure you count them all (circled): one, two… TWO!

Oh wait… only two?

How many days does it take to make a habit: 21? 30? 66? Certainly not two… you get the idea.

But wait: there’s more! I will continue in the next post.