General Theories on System Development (Part 1)

I have a lot of loose ends in this blog. Some of them you see (most recently here). Some of them, which take the form of unpublished drafts, you don’t. What follows (italicized) are unpublished drafts from December 2012. I thought these might be especially interesting to revisit in the midst of my recent algorithmic trading experience.

In this post, long-only outperformance seen with SPY and QQQ did not hold with IWM.  Because I approach system development with a healthy dose of critical analysis [this hasn’t changed!], I tend to question whether the pattern is real when I see something that selectively applies.  This suggests two different approaches to system development.

Let me point out one nuance about terminology. More recently, I have been using “approach” to describe the how of trading system development: walk forward (Part 1 through Part 4) or data mining. With the 2012 posts, “approach” pertains to the what of trading system development: one or multiple markets being tested (using either walk forward or data mining, presumably). Hopefully that allays any potential confusion.

The first approach is to backtest many trading rules [or strategies as I call them in 2020] on one ticker in search of the trading rule(s) that generates widespread and consistent profits when being used to trade that ticker.  This approach implies that different tickers have different personalities.  This may be a reflection of what technical analysis is being used by the largest institutions involved.  For example, suppose institutions accounting for 60% of a ticker’s volume use MACD signals.  I could then expect MACD strategies to work well with said ticker.

…if I test enough rules on any given ticker then some rules will show significant profits just by chance alone (e.g. one in 100 at the 0.01 level of significance).  How do I know if I have stumbled upon a true gem or a chance finding?  In case of the latter, profitable results seen in backtesting are unlikely to persist into the future.

This final point is a caution not to buy into the theory too heavily. I can never prove institutions are responsible for a strategy that works. I should never be so confident in that belief that I stop monitoring for signs of a broken system.

Next time, I will discuss an alternative theory about trading systems.