Trading System Development 101 (Part 3)

Choice of subjective function is just the tip of the iceberg. As the last two posts have made clear, trading system development can be a very individual process.

Granularity of the variable grid is another important detail that has no right answer. A coarse (less granular) grid has fewer potential variable values whereas a fine (more granular) grid will have more. Testing from 10-30 by 10 gives me three values to test (10, 20, 30) whereas testing from 10-30 by 2 gives me 11 values to test. A more granular grid will result in more iterations (third-to-last paragraph here) and more processing time.

If I want to differentiate between spike and high plateau regions, then I really need increased granularity.* Recall the diagram in the last blog post linked. If one [variable] value tests well, then a high plateau region means an adjacent one should too. This makes more sense for 10 or 14 relative to 12 than it does for 10 or 30 relative to 20. Ten or 14 are only 17% away from 12 whereas 10 or 30 are 50% away from 20. Qualitatively, 17% away may be “similar” while 50% away is apples and oranges rendering exploration of the surrounding space impossible. Signal can fall right through the cracks with a coarse grid.

Feasibility testing is where I can tweak the strategy to get a good result. This is curve-fitting on a small percentage of the total data. If this results in an overfit model, then the strategy will be subsequently rejected when I test on a larger portion of data.

I found number of trades to be a recurring challenge with feasibility testing. Basic statistics (see third-to-last paragraph here) dictates preference of larger sample sizes when evaluating test results. I don’t feel confident in the outcome when one or two routine trade results can drastically skew the result. I’m not looking for an enormous number of trades since this is just a 2-year feasibility test, but if I get too few then I’ll be concerned. What minimum number to accept—all together now!—has no right answer.

I have a few ideas on how to increase number of trades. A strategy that closes or reverses direction when the opposite trigger occurs is fine as long as the triggers occur with some degree of regularity. If only 1-2 triggers occur per year then over two years I may get a tiny number of trades. I’ve tried implementing profit targets, stop losses, and m-day exits to increase number of trades. Feasibility testing is the time to do this; if I get a tiny sample size of trades over the full dataset then I must junk the strategy and move on to avoid curve fitting.

* — Of course, if subsequent steps in the development process don’t do this then why be concerned at all?

Trading System Development 101 (Part 2)

Today I am discussing unknowns in the feasibility testing phase.

Variable range selection can significantly affect results and may have no correct answer. If I have a short-term signal and I test a strategy over a short-term range (e.g. 10-30), then I am more likely to hit the critical value of 70% profitable than if I test over a mixed range (e.g. 10-90) or a longer-term range (e.g. 30-90).

Number of iterations can also significantly affect results and may have no correct answer. In terms of granularity percentage, success or failure of one iteration contributes 2% of the total for 50 iterations vs. 0.2% of the total for 500 iterations (target = 70%).

What segment of the data to use for feasibility is another important detail that may have no right answer. Doing feasibility testing on one 2-year period that represents a particular market environment may generate significantly different results than feasibility testing on another 2-year period. In trying to find a strategy for any given futures market, feasibility testing over multiple environments would be ideal.

Testing over multiple market environments first requires a listing of said environments. This is a subjective task (vulnerable to hindsight bias) that also has no right answer. Were I to pursue this, then I should also determine how often the different environments occur. This might feed back to help determine whether this is worth doing at all (e.g. is it worth testing on exceedingly rare market conditions?).

In my testing thus far, I have eschewed all this and simply chosen to rotate the 2-year feasibility period. How I do the rotation probably doesn’t matter as long as I do it to give strategies suited to different market environments a chance. If I end up testing 100 strategies on a futures market, then maybe I test 10 on each 2-year period within the full 10 years. I will have false negatives, as I discussed in Part 1, but such remains an inevitable reality of this approach.

I have to be a bit lucky to get a strategy to pass feasibility testing, which brings to mind two possibilities. First, it’s okay to rely on some luck and incur some false negatives since I have an infinite number of potential strategies to test. Because feasibility failure decreases the possibility that I’m dealing with a viable strategy (see last long paragraph in Part 1), I should feel good about moving on to the next candidate and minimizing wasted time.

Alternatively, perhaps some method exists that eliminates false negatives (and any reliance on luck) by testing everything. I think this would require an enormous amount of processing power (and programming), though. I already encounter processing delays with my mediocre hardware: a 10-year backtest with 70 iterations takes up to 25 minutes. Many people backtest with hundreds (or even thousands) of iterations, which would take my computer all night to run.

To feasibility test or not to feasibility test differ in how time will be spent. With feasibility testing, extra time will be spent testing additional strategies since viable strategies are inadvertently dismissed. Without feasibility testing, extra time will be spent testing strategies that would have otherwise been dismissed in feasibility over a much longer period.

I will continue next time.

Trading System Development 101 (Part 1)

Today is the day I start talking more specifically about the trading system development process.

Numerous articles on trading system development methodology can be found on the internet. For this reason, I won’t go into extreme detail. I do hope to delve into a few deeper theoretical issues.

As discussed here, I am trying to come up with a model that has a high SNR. The Eurostat website says:

     > Statistical tests of a model’s forecast performance are commonly conducted
     > by splitting a given data set into an in-sample [IS] period, used for the initial
     > parameter estimation and model selection, and an out-of-sample [OS] period,
     > used to evaluate forecasting performance.

I previously showed how I can use IS data to make a backtested strategy look really good. This is what I want to avoid. Ultimately, the only thing I care about is that the strategy perform well on OS data, which has yet to be seen.

I expect a model that trains on some data (IS) to test well on that data (IS). I need to find out whether this correlates with how it will test on future data (OS). To do this, I train (develop) the model on IS data and test the model on OS data.

I start with feasibility study, which will be limited to a small percentage of the total data available. Maybe I designate two out of 10 total years of data to be used for feasibility testing. For each variable in the strategy, I then want to define ranges and test strategies across those ranges. I discussed this in the second-to-last paragraph here. I will then designate a strategy worthy of further consideration if at least 70% of the iterations are profitable (using net income as my subjective function).

Understand that feasibility study includes arbitrary features that should be determined ahead of time by the individual developer. How many years should be used for feasibility testing? I said “small percentage;” there is no right answer. What percentage of iterations must test well in order to pass feasibility? I said 70%; there is no right answer. What fitness function to use? No right answer. What small percentage just mentioned will be selected to be used for feasibility testing? I will discuss this one in further detail later, but here’s a sneak peak: no right answer.

The whole notion of feasibility study is not without critique due to the possibility of false negatives. No strategy performs well at all times, and I will miss out on a viable strategy if the feasibility study happens to be a portion of time where the strategy lags. I’m playing the probabilities* here. A viable strategy is more likely to perform well during more periods. Given this, a viable strategy is more likely to pass feasibility. None of this is ever a guarantee.

I will continue discussion of other unknowns in the next post.

* — The best we can ever hope for when it comes to trading and investing.

Fire and Fortitude in Algorithmic Trading (Part 2)

In order to be an algorithmic trader, I think one must have a burning desire (fire) to beat the market as well as the fortitude to plod through significant failure with regard to strategy development.

I left off with mention of KD’s approach to algorithmic trading. He also says discovery of the first viable strategy may take an extra long time with subsequent viable strategies becoming easier.

I think one of the biggest challenges to trading system development is sticking with the process and maintaining the drive to continue despite serial failure. I’m so here to tell you: getting knocked over the head so many times is absolutely brutal. As a pharmacist, most everything I did either accomplished something or got me one step closer. With trading system development, I need to reprogram myself to get positive reinforcement from failure. I need to regard every rejected strategy as being one step closer to viability.

In addition to serial failure, I think the specter of scam combines to create a harsh double whammy that few people can overcome. Until I find a viable strategy on my own, I have only others to trust. As failure mounts, a belief that the process is somehow flawed becomes harder to resist. Maybe the talk of viable strategies is all just a good marketing story used to pad the pockets of people like KD. How do we know he has actually found success himself? This is consistent with the comments from [11]. I have reached out to almost 15 algorithmic traders to share experiences and have only gotten two responses. I would not be surprised if most spin their wheels for a time and subsequently exit the arena.

I am not saying KD is a con artist, but like the broader financial industry, he does have a strong underlying motive to get us to believe. If nobody believes, then his business selling trading education will struggle. For the broader financial industry, when people stop believing and sell everything, investment advisors experience a severe decline in assets under management and generate less revenue. They therefore have an inherent conflict of interest when recommending clients “stay the course” and ride out market turbulence.

Fire and Fortitude in Algorithmic Trading (Part 1)

Today I want to review where I’ve been (starting here) in this discussion of algorithmic trading and then move forward.

I have three main critiques of technical analysis (TA). First, it’s not always objective (as discussed here and here). Second, it’s rarely backed by supporting evidence. Finally, it seems really difficult to find viable trading strategies.

With regard to critique #3, I feel like I want to climb to the nearest mountaintop and yell “nothing works, folks!” An infinite number of possible strategies await to be tested, though, as mentioned in [1]. If I have 40 different indicators then I can test them on 20 different markets, throw in daily + 4h + hourly + 5m time frames, and voila: 3,200 strategies. What I have tested in my first three months is tiny in comparison.

Seldom have I experienced failure like I have in my brief stint as a quant. One by one, I saw hallmarks of popular TA fall by the wayside: momentum, moving averages, Bollinger Bands, RSI, Williams %R, trend-following, mean-reversion, etc. None of these have worked sufficiently well over a long period of time, and many are losers. I went through books with each setup presented accompanied by [deceptive?] claims of live-trading success on different markets. None of these worked either. Having immersed myself in the financial domain for a while, I’m not surprised to see an indicator or two not work very well. I have read books, articles, and sat through countless presentations these indicators over the years, though, and failure to find a single viable strategy, is quite the shock.

My discipline and commitment have eroded over time. I started by testing one strategy per day. As I learned the process better, I increased to two, five, and eventually 10/day. I started to run lean of strategy ideas. It’s very important to have this in strong supply because once my time had to be devoted to finding strategies, my daily testing decreased and I felt less productive as a result. Up until that point, I was able to achieve my daily goal, but as I watched the tally of total strategies tested continue to mount, my frustration and fatigue got real.

What I experienced is to be expected according to algorithmic trading guru KD. KD is a vendor of algorithmic trading education. He has my respect (until he doesn’t, and I’ll let you know if that happens). He says we can expect to find one viable strategy for every 100 – 200 strategies tested. He also says to break for a few weeks if you start to get frustrated and impatient.

In other words, those who undertake this trading approach will need to settle in for a very long haul…