Statistical Manipulation

Peter Berezin, chief strategist for BCA Research, wrote an article for the September 2016 issue of AAII Journal. Near the end he made some comments about statistics:

     > Statisticians like to say that if you torture
     > the data long enough, it will confess to
     > anything. This old adage is especially relevant
     > to the study of stock market anomalies. First,
     > there is the risk that any anomaly that is
     > unearthed will simply end up being the product
     > of data mining. Second, even if an anomaly
     > turns out to be genuine, there is a risk that
     > it will be arbitraged away once the investment
     > community becomes aware of it.

I argued for an increased use of inferential statistics here and I later relayed the opinion of a financial adviser as to why inferential statistics may be relatively uncommon.

I still believe inferential statistics are useful to offer an apples-to-apples comparison but Berezin makes a good point that statistics may be used to manipulate. We can never be sure of an investigator’s underlying motives and unless we do the research ourselves, we also cannot be sure the statistics were correctly computed.

I do believe we can do a couple things to avoid these statistical issues. Data mining involves searching a large collection of data with the purpose of finding significant results. This should be avoided. Give me an indicator and enough data and I can find a snippet of price action for which the indicator works fabulously (fallacy of the well-chosen example). This is unlikely to be profitable in live trading, however. One way to avoid this involves searching the surrounding parameter space for a high-plateau rather than a spike region of profit.

With regard to market edge being arbitraged out over time, I need to monitor my system and have criteria indicating when it might be broken. Walk-forward analysis can help to keep a strategy current thereby increasing the probability it will work with live trading. I may also monitor total profit/loss and stop trading the system when this value falls below the equity moving average. This should be developed through proper validation methodology.