Lingering Quandaries about System Development (Part 5)

I am using this series of blog posts to articulate seeming details about System Development and my cognitive framework that just do not get along.  I identified the first one in http://www.optionfanatic.com/2013/01/18/lingering-quandries-about-system-development-part-1/, which was difficulty interpreting RAR/MDD.  The second conflict was uncovered in http://www.optionfanatic.com/2013/01/24/lingering-quandaries-about-system-development-part-4/.

To review, I have defined a trading system that works well with one ticker but does not trade frequently enough.  I have expanded from one ticker to a 5-ETF basket that also backtests well.  Before I trade this live, however, I must screen for selection bias to ensure the positive results may be attributed to the trading system rather than a lucky choice of ETF basket.

Suppose I therefore backtest the system on 2.54 billion 5-ETF baskets and find the results from my original basket to be within the average of all 5-ETF baskets.  Great!  There is no selection bias and I may therefore proceed with trading the system live.

On another hand, suppose I backtest the system on 2.54 billion 5-ETF baskets and find the results from my original basket to be more than 2 SD better than the average of all 5-ETF baskets.  The probability of this occurring is under 5% so perhaps I should not trade the system live because the outperformance of my original basket may have been the result of a lucky selection of five ETFs–a fluke, if you will.

But wait… who says that with particular rules, some markets can’t trade better than others?  Each market will, to some degree, reflect the cumulative personality of its largest institutional traders and it certainly seems possible that if many of them follow certain criteria then those criteria may carry some edge.  By this line of reasoning, I should not only be encouraged to see the original basket perform significantly better than all baskets–I should demand it!

Yes, I should demand what–according to the former viewpoint–I absolutely did not want to see and would not proceed to trade live.

So which is it?

I don’t know, I don’t think I can know, and I don’t think there is any statistical method by which I can possibly know.  The problem is too multivariate and complex.

Take that, Debbie Downer!