Randomization: Not the Silver Bullet

Random samples are often a research requirement but I believe trading system development requires something more.

Deborah J. Rumsey, author of Statistics for Dummies (2011), writes:

     > How do you select a statistical sample in a way
     > that avoids bias? The key word is random. A
     > random sample is a sample selected by equal
     > opportunity; that is, every possible sample of
     > the same size as yours had an equal chance to
     > be selected from the population. What random
     > really means is that no subset of the
     > population is favored in or excluded from the
     > selection process.
     >
     > Non-random (in other words bad) samples are
     > samples that were selected in such a way that
     > some type of favoritism and/or automatic
     > exclusion of a part of the population was
     > involved, whether intentional or not.

Randomized controlled trials have been said to be the “gold standard” in biomedical research but I do not believe randomization is good enough for trading system development. Yes it would be good to avoid selection bias but this is not sufficient. I wrote about this here. The only way I can know my results are not fluke is to optimize and test the surrounding parameter space. Evaluating the surface will reveal whether a random parameter set corresponds to a spike peak or the middle of a plateau.

This perspective on randomization concurs with my last post. Perry Kaufman selected one and only one parameter set to test and for that reason I took him to task.

As Rumsey’s quote suggests, statistical bias is never a good thing. E-mail correspondence suggests Kaufman did not pick any of the values in his particular parameter set at random. He selected them based on his experience and knowledge of market tendencies. My gut response to this is “just do the work and test them all.”

While this may or may not be feasible depending on how multivariate the system, it brings to light the main objective of trading system development. I will discuss this next time.