Truth in Backtesting (Part 10)
Posted by Mark on December 28, 2012 at 05:36 | Last modified: December 11, 2012 05:21In http://www.optionfanatic.com/2012/12/27/truth-in-backtesting-part-9/, I discovered that trading the CDC on S&P 500 stocks may require holding up to 235-250 stock positions at any one time. With this discovery, I am finally starting to gather some truth in backtesting.
To be prepared for a more extreme future, I want to budget for 300. At some point the markets will present a more extreme case than anything seen in the 33 years of backtesting–it’s just a matter of when. Truth in backtesting: if I cannot manage 300 open positions then this system is untradable for me.
The next question regards what initial account equity I need to preserve system performance. In #21478, I cut initial equity to $3M and only got 41,477 trades rather than 41,710. That’s too large of a cut. In #21479, I went up to $30M and got the 41,710 trades back. I was able to preserve performance with initial equity as low as $10M. Truth in backtesting: if my account isn’t at least $10M in size then I should not expect to see the kind of performance recorded in this backtest.
Perhaps I can further cut initial account equity by taking only long trades. Out of 41,710 trades, 26,711 trades are long. Cutting max open positions doesn’t help. Cutting initial equity to $5M also reduces total trades so that doesn’t help.
Can I cut initial account equity by decreasing position size? In #21497 I reduce position size from $100K to $10K, which not only decreases total trades but also PF from 1.48 to 1.36. With initial equity of $5M and a $50K position size, the long-only PF is 1.46–still a bit lower than 1.48.
I’ll make some concluding remarks in my next post.
Categories: Backtesting, System Development | Comments (1) | PermalinkTruth in Backtesting (Part 9)
Posted by Mark on December 27, 2012 at 03:50 | Last modified: December 7, 2012 18:41I have been describing a layperson’s approach to trading the CDC system on S&P 500 stocks. http://www.optionfanatic.com/2012/12/26/truth-in-backtesting-part-8/ got me to the point where I was encouraged by the system. Now, I need to better understand system exposure.
The truth in backtesting (or lack thereof) exists in the consistency between live trading and tacit backtesting assumptions. Although the backtesting says trading this system could generate a PF of 1.48 for long trades, as written the system requires an account size of $50M and the ability to manage 500 open positions. These are no-can-do for me. Were I to try and execute this system right now then I could expect great difference in live trading performance from that seen in backtesting. Better or worse? Probably worse. This is how markets work and I would be foolish to take a gamble on it.
To pin down actual exposure, I first must determine how many open positions the system may require. Were I more skilled at programming then I could code AmiBroker to chart the number of open positions over time. Since I am not, my solution is to decrease the maximum number of allowable open positions in subsequent backtests and watch to see when the performance statistics begin to change. Here are the baseline performance statistics for backtest #21471, which includes the 200-MA filter but not the BB filter:
In backtests #21472 and #21473, I reduced the max number of open positions to 400 and 300 with no change in performance statistics. This tells me no more than 300 open positions were ever seen in the backtest. I was subsequently able to narrow this down between 235-250.
I will continue the analysis in my next post.
Categories: Backtesting, System Development | Comments (1) | PermalinkTruth in Backtesting (Part 8)
Posted by Mark on December 26, 2012 at 05:47 | Last modified: December 7, 2012 16:09In the last couple of posts, I have been taking a layperson’s approach to development of the CDC trading system with S&P 500 stocks. I started with initial backtests in http://www.optionfanatic.com/2012/12/21/truth-in-backtesting-part-6/ and have continued to explore one or two modifications at a time.
My next step is to exclude the 200-MA filter (or 50-MA filter, which was used in #21468) and see how the system performs with just the BB filter. This is backtest #21469 and should be contrasted to #21467 from http://www.optionfanatic.com/2012/12/24/truth-in-backtesting-part-7/ where just the 200-MA filter was active:
The results are very similar, which means adding either the BB filter or the 200-MA filter similarly improves the efficiency of the system.
What if the BB filter is modified to make the overbought/oversold condition even stronger? I will now require a close outside the previous BB for the last two bars rather than one. This is backtest #21470:
Compared to #21466 above, the short trades performed slightly better. The other numbers are roughly similar, though. The total number of trades is significantly reduced, which is to be expected, and the long trades still significantly outperform the short trades by a factor of almost three based on average trade %.
By this point I’m somewhat convinced that this system has promise. With tens of thousands of trades over nearly 33 years, this is not a surgical-strike system that trades infrequently and generates gobs of profit on average. This is a system that trades frequently and grinds out smaller profits with each swing.
The next step involves better understanding system exposure. I have been backtesting up to 500 open positions at once and an initial account equity of $50M. Neither of these is realistic because my account is smaller.
Categories: Backtesting | Comments (0) | PermalinkTruth in Backtesting (Part 7)
Posted by Mark on December 24, 2012 at 04:49 | Last modified: December 7, 2012 13:36In http://www.optionfanatic.com/2012/12/21/truth-in-backtesting-part-6/, I hypothetically wiped the slate clean of all accumulated book knowledge to illustrate some other misleading applications of the system development process. I am currently applying the CDC system to S&P 500 stocks the way an average Jane might approach system development with analytical software.
Backtest #21464 was an improvement over the one before it. The next step is to assess the impact of the Bollinger Band (BB) breakdown/breakout filter and 200-SMA filter by removing them. I also reset CDCs back to four. The following is backtest #21466 and should be compared to backtest #21463 (http://www.optionfanatic.com/2012/12/21/truth-in-backtesting-part-6/):
Net profit % and # trades are much larger, which suggests the filters had a dramatic effect. PF is down from 1.46 to 1.28 but for long trades, PF is 1.49 compared to 1.58 with the BB filter. That is not a big difference. In fact, everything about long trades here is much better than short trades: better RAR%, much better average trade %, much better subjective function, PF, and SR.
Since I made two changes to get these results, I will next reinstate the 200-MA filter. This is backtest #21467:
As with backtest #21466, long trades perform much better than short trades: average trade % is double, the subjective function is better than triple, PF and SR are all much improved.
What if the 200-MA filter is shortened such that long (short) trades are only taken when price closes above (below) the 50-MA instead? This is backtest #21468:
This decreased the number of trades and net profit % compared to #21467. That makes sense since the 50-MA will track closer to price than the 200-MA and therefore may be more restrictive. The efficiency of this system is comparable to the 200-MA system, however, based on net RAR%, the subjective function, PF, and SR.
I will continue this analysis with my next post.
Categories: Backtesting | Comments (0) | PermalinkTruth in Backtesting (Part 6)
Posted by Mark on December 21, 2012 at 02:47 | Last modified: December 7, 2012 13:24So far in this series (e.g. http://www.optionfanatic.com/2012/12/04/truth-in-backtesting-part-5/), I have discussed the misleading nature of EOD backtesting when trade delays are not used. Today I want to use the Consecutive Directional Close (CDC) system to start discussing another misleading aspect of backtesting: multiple positions.
Consider this the beginning of a new day. I wake up bright and early (well not so bright because I really do like to wake up extremely early), jump out of bed, and shake off all book knowledge pertaining to system development that I have accumulated over the past couple of years. My mind reverts to old patterns of knowledge. I just got AmiBroker and now I’m ready to play.
I start by coding the CDC system and setting the filter to S&P 500 stocks. I set initial equity to $50,000,000, maximum number of open positions to 500, number of CDCs to four, and length of trades to five. I set position size to $100,000 and I include a couple other filters in the buy and short conditions: Bollinger Band breakdown or breakout and price above or below the 200-SMA for long and short trades, respectively. I include a liquidity filter to make sure the position size divided by closing price is less than 1% of the average daily volume over the past 50 days. Backtesting dates are 1/1/1980 through 11/30/2012. Commissions are assessed at $8/trade. Buys and sells are executed at the next open.
The results for backtest #21463 are as follows:
RAR = risk-adjusted return (i.e. performance if the system were in the market 100% of the time)
Payoff Ratio = average winning trade / average losing trade
RAR/MDD = the subjective function
I will use PF and SR as abbreviations for Profit Factor and Sharpe Ratio, respectively.
These results aren’t bad, but I want to see if I can make the system more efficient. I increase CDC to five. Here are the results of backtest #21464:
These results are better. While net profit is lower, the system has become more efficient (PF is higher along with payoff ratio, subjective function, and SR).
I will continue this analysis in my next post.
Categories: Backtesting, System Development | Comments (1) | PermalinkSystem Development for Individuals (Part 4)
Posted by Mark on December 20, 2012 at 05:08 | Last modified: November 16, 2012 06:50My previous posts on this topic have led me to understand System Development as an endeavor people must learn on their own based in part on individual trading experience.
Understanding System Development as an individualistic pursuit also means your potential takeaway from my blog is limited. Even if I develop worthy systems to trade, many potential factors may prevent you from trading them–not the least of which is the fact that you might be foolish to trade what I claim to be profitable without replicating the work yourself as a matter of due diligence.
What you can do is study my process and use that as a starting point after which to pattern your own work. This is what books have done for me. Only I can bring myself the rest of the way.
It was not my intent to make any grandiose claims with this mini-series but rather to summarize my experience to date. I originally learned about systematic trading and backtesting as a contrast to discretionary trading. I then learned about trading strategies vs. trading systems and system development. I purchased what I believed to be the most comprehensive and inexpensive software package and then started to read books. It was at this point that I found different books to have limited content overlap.
Once I set out on my own system development journey, I edged closer to the holes in understanding that I grapple with today. I believe research awaits to better understand how patterns I see in backtesting correlate with live trading performance, but this will be personal science only applicable to my trading concept and to my trading preferences.
Categories: System Development | Comments (0) | PermalinkSystem Development for Individuals (Part 3)
Posted by Mark on December 19, 2012 at 04:45 | Last modified: November 16, 2012 06:35In this mini-series, I have been challenging the notion that System Development exists as an objective discipline applicable to a widespread audience of traders. Understanding system development as a wholly individualistic pursuit makes me better understand why past attempts to organize a system development group have failed.
Of the limited number of people who trade, I was looking for people with a significant amount of time to devote to system development. This may exclude people with separate full-time jobs. I was then looking for people in/near my city.
By now, the field of potential collaborators was probably narrowed to a handful and many selection criteria still remained. How can I find these people? I tried www.meetup.com without success. Are they on Craigslist, Facebook, or LinkedIn? Each of these is a different avenue. It’s probable that some candidates were ill-suited to work with me because I demanded a degree of open-mindedness, willingness to suspend judgment, and substantial freedom from commercial influence (e.g. trader education companies, affiliate programs, etc.). I also demanded they use my preferred software package since I had already spent 18 months arduously working to learn it.
Additional selection criteria make this seem like another instance of the knowledge-specificity monster where at the end of the road, nothing remains (see http://www.optionfanatic.com/2012/12/17/system-development-for-individuals-part-1). Candidates would have needed to use the same or comparable brokerage platform (else one or more of us might have to change), shared desired markets and time frames to trade, perhaps shared attitudes toward fundamentals or other potential means to generate trading candidates, etc. The greater the number of selection criteria, the more restrictive the overall requirements and the fewer people available to meet them all.
Categories: System Development | Comments (1) | PermalinkSystem Development for Individuals (Part 2)
Posted by Mark on December 18, 2012 at 06:44 | Last modified: November 16, 2012 06:07In http://www.optionfanatic.com/2012/12/17/system-development-for-individuals-part-1 , I started to develop the thesis that System Development as an objective discipline may not exist because what it is to each person is too specific for global application.
With book content of limited use because of macro factors like software package, the amount of usable information shrinks further due to personal factors. If my brokerage differs from the author’s then I may not be able to execute the same kinds of orders or use the same kinds of alerts. I may prefer to monitor different time frames or enter different orders–maybe use use limit orders with a larger offset or no offset at all just waiting for the fill. What if I don’t get filled? These are unables that may differentially affect system results. Do the author and I treat these the same? What about slippage?
Perhaps most damning for the prospect of System Development is the observation that most everything discussed in the last paragraph is hardly addressed in most writings. Partial education from books must now be supplemented with personal experience, which means System Development will be different for everybody.
Also damning for the prospect of System Development as an objective discipline is the likelihood that a complete education may require some gambling. The personal experience just mentioned–how can I even attain that before having a complete understanding of system development? “Cart before the horse” comes to mind. I cannot attain it through backtesting or paper trading, which have limited resemblance to live trading. I am therefore left to learn from my live trading performance itself. Without having a developed system, I may or may not have an edge.
Who wants to gamble when they can participate in other forms of discretionary trading that can allegedly be taught?
Categories: System Development | Comments (0) | PermalinkSystem Development for Individuals (Part 1)
Posted by Mark on December 17, 2012 at 03:08 | Last modified: November 16, 2012 05:46I would love to see a “Dummies” book on system development, but I’m not sure such a book is plausible. What system development is and how to approach it may vary so much from one person to the next that System Development for Dummies may never come to fruition.
I addressed subjectivity and system development in six blog posts beginning with http://www.optionfanatic.com/2012/10/04/the-subjective-function-part-1. Howard Bandy’s books define and discuss the objective function. I renamed this “subjective function” upon realizing the irony that where system development and backtesting sound like scientifically-oriented endeavors, if the measure of success varies from one person to the next then perhaps there’s nothing objective about them at all.
Some of the subjectivity is related to specificity of knowledge. If I want to know something very specific then I can Google until I’m blue in the face without finding what I need. Family Law is one example. As a citizen trying to learn and prepare for a case, I almost need to get reviews on a particular lawyer (since different lawyers will define and strategize differently) in a particular county of a particular state. This is going to be very difficult information to find because every case is different. The only place I may be able to get related information is on a particular attorney’s web site, which is not objective enough for my taste.
Education about system development starts with what I can read in books. Authors of different books use different software packages, though. Use of one software package limits what I can take away from an author who uses another because anything about the programming language, backtesting capabilities, and even system development definitions may be different from what I know using different software.
I will continue to explicate the knowledge filter in my next post.
Categories: System Development | Comments (1) | PermalinkFlushing Out the Arbitrary
Posted by Mark on December 12, 2012 at 07:00 | Last modified: November 26, 2012 07:16In the journey to succeed with System Development or anything trading-related, critical elements in the category “what I don’t know I don’t know” must become illuminated, become analyzed, and become known. Today I will attempt to illustrate this by describing my initial concept and the present concept System Development has become.
The reason I bought AmiBroker was to backtest systems I read about in books and on-line to verify whether the purported results were accurate. I have talked about some initial claims (see http://www.optionfanatic.com/2012/12/05/trading-system-2-consecutive-directional-close-part-11/). I have coded them into the software and run the backtesting to verify those claims. I originally believed this was the entirety of due diligence required in order to obtain tradable systems.
Not even close.
Perhaps the biggest missing detail is optimization, which I also discussed in http://www.optionfanatic.com/2012/10/18/laziness-dissected. If a trading system holds for W days (or uses X as a critical value, or Y and Z as indicator periods, etc.), how do I know whether its success was just lucky? While the reading is inspirational and alluring (think “trader porn”), if the profits are unlikely to persist then it is simply another useless Best Seller in the minefield that is Wall Street.
Optimization forces me to flush out the arbitrary by identifying variable parameters. The total number of potential trading systems is the product of potential values for each parameter. I must plot the subjective function vs. values of the parameters, identify plateau vs. spike regions, and identify the viable trading system candidates.
Next up is Monte Carlo simulation to better understand potential DDs (see http://www.optionfanatic.com/2012/12/11/trading-system-3-naked-puts-part-4/ ), position sizing, and Walk-Forward Analysis.
For this trader working to learn System Development, flushing out the arbitrary has been the biggest component to date of “what I don’t know I don’t know.” With this block now destroyed, hopefully personal and professional growth can occur at an accelerated pace.
Categories: System Development | Comments (2) | Permalink