Mining for Trading Strategies (Part 1)

On the heels of my validation work with the Noise Test and Randomized OOS, I am going to proceed with a new methodology to develop trading systems.

I built today’s strategies in the following manner:

• Random 2-rule long CL with basic exit criteria (see Mining 1)
• Test period 2007 – 2015
• Second half of test period designated as OOS
• Strategies sorted by OOS PNLDD
• Searched on first page of simulation results for strategies that passed Randomized OOS
• Retested passing strategies from 2015 – 2019 (“incubation”)
• Looked for strategies with 4-year PNLDD > 2 and PF > 1.30 during incubation

The incubation criteria are nothing magical. I found a couple handfuls of decent-looking strategies and settled upon these numbers after seeing the first few (the numbers were actually lowered somewhat by the end). Also, more than anything else at this point I am trying to gauge whether Randomized OOS is at all helpful to screen for new strategies; a specific critical value will hopefully be determined in the future.

Of the top 28 strategies (all had PNLDD > 3.3 OOS and PF > 1.48 OOS), 21 passed Randomized OOS. Many of these satisfied DV #2 for the IS portion as well, but I did not require that in order to pass.

Nine of 21 strategies met the lowered incubation criteria with PNLDD > 1.68 and PF > 1.28.

On the Monte Carlo Analysis, I look for average drawdown (DD) to be less than the backtested strategy DD. This is mentioned by the software developers as a metric to provide confidence that performance statistics are not artificially inflated due to luck. I have not yet tested this, but I am monitoring it.

In the current simulation, zero of 10 strategies that met the lowered incubation criteria had Monte Carlo DDs less than backtested. None of the other 11 strategies that passed Randomized OOS did either.

I have no major takeaways right now since I am early in the data collection stage. What percentage of strategies pass Randomized OOS? What percentage of strategies have MC DDs less than backtested? What percentage of strategies go on to pass incubation? What kind of performance deterioration can I expect going from IS to OOS to incubation?? How often will I find a strategy that does not follow this pattern?

The software is advertised to come standard with an arsenal of tools capable of stress testing strategies. If passing those stress tests is not correlated with profitable strategies, then we will have an ugly disconnect.

For now, though, all I need are more simulations, more samples, and more data.