Trading System Development 101 (Part 1)

Today is the day I start talking more specifically about the trading system development process.

Numerous articles on trading system development methodology can be found on the internet. For this reason, I won’t go into extreme detail. I do hope to delve into a few deeper theoretical issues.

As discussed here, I am trying to come up with a model that has a high SNR. The Eurostat website says:

     > Statistical tests of a model’s forecast performance are commonly conducted
     > by splitting a given data set into an in-sample [IS] period, used for the initial
     > parameter estimation and model selection, and an out-of-sample [OS] period,
     > used to evaluate forecasting performance.

I previously showed how I can use IS data to make a backtested strategy look really good. This is what I want to avoid. Ultimately, the only thing I care about is that the strategy perform well on OS data, which has yet to be seen.

I expect a model that trains on some data (IS) to test well on that data (IS). I need to find out whether this correlates with how it will test on future data (OS). To do this, I train (develop) the model on IS data and test the model on OS data.

I start with feasibility study, which will be limited to a small percentage of the total data available. Maybe I designate two out of 10 total years of data to be used for feasibility testing. For each variable in the strategy, I then want to define ranges and test strategies across those ranges. I discussed this in the second-to-last paragraph here. I will then designate a strategy worthy of further consideration if at least 70% of the iterations are profitable (using net income as my subjective function).

Understand that feasibility study includes arbitrary features that should be determined ahead of time by the individual developer. How many years should be used for feasibility testing? I said “small percentage;” there is no right answer. What percentage of iterations must test well in order to pass feasibility? I said 70%; there is no right answer. What fitness function to use? No right answer. What small percentage just mentioned will be selected to be used for feasibility testing? I will discuss this one in further detail later, but here’s a sneak peak: no right answer.

The whole notion of feasibility study is not without critique due to the possibility of false negatives. No strategy performs well at all times, and I will miss out on a viable strategy if the feasibility study happens to be a portion of time where the strategy lags. I’m playing the probabilities* here. A viable strategy is more likely to perform well during more periods. Given this, a viable strategy is more likely to pass feasibility. None of this is ever a guarantee.

I will continue discussion of other unknowns in the next post.

* — The best we can ever hope for when it comes to trading and investing.