Mining for Trading Strategies (Part 5)

Today I want to present and discuss results from my initial testing of 2-rule versus 4-rule strategies.

The methodology is given in the third-to-last paragraph here.

I basically developed the strategies with random entry signals and simple exits from 2011-2015 and tested OOS from 2007-2011. I sorted by best performers over the entire 8-year period. I took the top (+/-) 60 (two simulations of ~30 each) and tested from 2015-2019. As the focus of my study, I then tested these strategies from 2015-2019 (incubation).

I applied $25 plus $3.50 / contract-trade to account for transaction fees.

The analysis consisted of T-tests assuming unequal variance in Microsoft Excel. I chose unequal because I had no reason to assume equal variance. I tested on five parameters: PNL, PNLDD, number of trades during incubation, average trade, and PF. Number of trades is not a performance metric and could probably have been left out.

Since I did five statistical comparisons, I adjusted the significance value from 0.05 per Holm Method. I think Holm is better than Bonferroni in this instance because the hypotheses are not independent: all measures of four rules versus two.

Here are the results:

No differences were significant after application of the Holm method.

As I reviewed the study, I found that I made a few mistakes. First, the starting date was 2001 instead of 2007. Second, I had OOS set to 40% of the data instead of 50%. Third, I had cash set to be invested in the Biotech index instead of nothing. The latter probably affected strategies across the board and is nothing to be concerned about. It may have differentially affected the groups if one tended toward more/less time in the market during a period when Biotechs performed significantly better/worse than large-cap US equities.

I would rather do the study correctly before making too much of it. I will look to build on this with later studies.

Mining for Trading Strategies (Part 4)

Today I resume this blog mini-series with a slightly different approach.

I communicate with an algo trader who has recently gone live with his first strategy. If you have read about or watched algorithmic trading videos for any period of time, then you have seen pictures like his near-perfect equity curve. Also:

• Backtested over 19+ years
• PF > 9
• Total time in market < 30 months
• Maximum drawdown < 9% of initial capital
• Made money every calendar year
• Out of the market entirely for three consecutive years
• Passes Randomized OOS
• Beautiful Monte Carlo analysis
• Monte Carlo DD < backtested DD (see sixth paragraph here)
• Noise Test looks pretty good (which may not mean much)

I started discussing my attempt to validate Randomized OOS and he interrupted to say that I’m using lousy strategies. I was looking at simple, 2-rule strategies. He said I should be using at least four including signal exits. Even then, he said only 0.001% of strategies I find will actually “be good” (I assume by this he meant to be “viable” per this second paragraph).

Realizing his response was personal opinion not based on actual data, a couple things came to mind—the first one being whether I can ever validate a stress test if he is correct. I took one day to go through ~120 strategies; what would it take to show the stress test adds value if only one out of 100,000 strategies are good enough to be viable anyway?

A Chi Square calculator can answer this question. Even one in 2,000 is significantly better than one in 100,000, but that would take at least 17 days of backtesting.

I’m pretty sure 0.001% is way too low (see second paragraph here). I think one in 1,000 (0.1%), if not one in 200 – 500, is more like it. I don’t have actual data on this so I’m going to table it for now.

The second question that came to mind is whether 4-rule strategies are any better than 2-rule strategies?

I figured I could do this study by using the software to build random 2- and 4-rule strategies for long ES over 2007-2015. I ran the simulation for a couple minutes and then stopped. I took the best page of overall results (IS + OOS) and tested those on 2015-2019. I collected data on PNL, PNLDD, total number of trades, average trade, and PF.

I didn’t really have an opinion as to which group I thought would do better. The more rules, the greater the risk of overfitting would mean good performance from 2007-2015 and worse performance thereafter. However, 50% of the former period is OOS, which is not fit to anything. If the strategy is poor OOS then it’s probably not going to be one of the best. These are logical, contradictory ideas and as a result, I approached the results without bias and looked at two-tailed statistics.

I will continue next time.

Random Simulation 1 (Part 2)

Today I will conclude analysis of my first random simulation with the new software.

The more strategies I analyze, the more I hope to learn about what components are correlated with viability and profitability. I will also get a better feel for how representative particular results are relative to all strategies that I’ve studied.

Continuing on with the stress testing, the delay test doesn’t look so great:

I’m not sure how concerning this is. This delays entries and exits by x bars. When the strategy exits at the first profitable close or 15 days—whichever comes first—any delay in terms of 1+ bars could be an unrealistically large amount.

Factoring in worse execution might be more relevant and along these lines, the slippage test looks better:

Like the delay test, the liquidity test looks poor (my notes Screenshot 16), but I would pose the same questions. If profitable trades are being held for one day and entry is delayed by a large part of that period, then most of the profit potential evaporates immediately. Significantly poorer performance would be no surprise.

The noise test looks poor:

I included up to 25% noise here for OHLC. The actual backtest is pretty much the best equity curve on the graph. At least all the other simulations are profitable, but this suggests I shouldn’t necessarily expect such good performance if traded live.

For me, the second paragraph above especially resonates with something like the noise test. I would think altering the data series in this way would dramatically worsen any trading strategy that is better than random. I expect to better understand the importance and variety of noise test results as I get closer to studying a large sample size of strategies.

Randomized OS graph looks encouraging:

On the OS graph, the actual equity curve is right in the middle of the randomized simulations.

Overall, my initial impression is cautiously optimistic with this strategy. The OS curve looks solid. Performance versus random is solid. Monte Carlo is solid. MFE/MAE looks solid. Slippage test looks solid. The variance test seems pretty solid. Delay, liquidity, and noise tests don’t look so great. The randomized OS simulation looks ideal.

Further directions could include:

• Modifying the simulation dates
• Modifying [canned] exit criteria
• Toggling OS between beginning/end
• Changing fitness function
• Verifying the backtest in TS and studying performance report
• Checking on unseen data (i.e. incubation)

Another exploration would be to vary OS percentage with a caveat. While I think varying OS % may get me different strategies, with a minimum number of trades required, increasing OS % will exclude otherwise-qualifying strategies.

Random Simulation 1 (Part 1)

As I enter the second stage of what I hope will be a long career in trading system development, this is my documentation of steps taken and my software learning process.

As a first attempt, I selected a desired market and direction along with random symbols (my notes Screenshot 1). I selected exit criteria as shown (Screenshot 2). I made sure to select at least 50 trades each and I selected 50% in-sample data. I entered max holding time (range with increment 5) and profitable closes (range with increment 2). I entered transaction fees (charge) and max number of two rules to minimize processing time.

Once the simulation had run, I was left with 500 strategies (Screenshot 3). The fitness results were eye-popping until I realized that I was looking at IS results.

I toggled to “both” (OS too) and turned on the equity curve thumbprint. I started scrolling down the list in search of good OS performance. This would be a curve that roughly matches IS performance, slopes flat/up, and does not have any large drawdowns. I was able to dismiss most strategies very quickly. When I later sorted OS results, I found only a few strategies that were even profitable (no wonder most curves looked poor!).

Then I got to #482:

Performance was astounding versus B&H. Performance was pretty good versus random, too:

MFE/MAE looked really strong for days:

Lowering win rate by up to 10% and simulating 100 trades x1000:

This shows an 18.5% chance of losing $10,000 with ~5% of drawdowns exceeding $40K. Average drawdown is just over $13K compared to $5,200 for the original backtest. PnL/DD remains over 2, which seems pretty solid. I could base position sizing off this, but being stressed conditions I don’t have context for reliability. In other words, I will run this test on all strategies but the amount of stress may differ depending on win rate, the impact of stressing may differ across strategies, etc.

I will continue next time.