Short Premium Research Dissection (Part 41)

I e-mailed our author giving some general feedback about the report.

The first paragraph was the focus of my discussion from last time:

     > I’ve gone over it extensively and there’s a lot I have to
     > say. I don’t have solutions for some of the concerns and
     > it’s possible that these particulars have no correct answers
     > at all. It’s a complicated project with many permutations.
     >
     > I would like to see a consistent set of statistics provided
     > after every single backtest. The “hypothetical portfolio
     > growth” graph template is consistent. The statistics vary
     > widely. I often wanted more than what you provided.

Part of this standard battery should have been PnL per day, which our author did not really discuss.

     > I would also like to see complete methodology given for
     > every backtest. The methodology should allow me to
     > replicate your study and get the same/similar results.
     >
     > No explanation was given for the disappearance of
     > 2007-9 data in Sct 5… It really should be in the report…
     > because it could otherwise be construed as curve fitting…
     > 2008 provided one of the great market shocks of all
     > time and we could really benefit by seeing how the
     > final trading system performed during that time.

I have since learned the data was lost because she switched from ETF to index data. The latter was only available from 2010 onward. If it meant losing 2007-9, then I think she should have stuck with the ETF.

     > I wondered why some components of Scts 3-4 were not
     > in Sct 5 and vice versa. How would time and delta stops
     > have fared in Sct 3? How would a VIX filter and rolling
     > up the put performed in Sct 5? It’s hard to compare
     > Sct 5 with Scts 3-4 because of these key differences
     > (along with the missing 2007-9 data).

She does not include transaction fees in the backtesting. This is a fault. I mentioned this in the second-to-last paragraph of Part 36 and the fourth paragraph of Part 38.

On several occasions (e.g. paragraph after sixth excerpt here and paragraph after first table here), I wrote “when something changes without explanation, the critical analyst should ask why.” She should be ahead of this and check herself for such inconsistencies throughout. A proofreader could help.

I have mentioned proofreading a couple times (e.g. Parts 36 and 38) and sloppiness many times in this review. Absence of that “hypothetical computer simulated performance” disclaimer was sloppy and plagued me throughout much of the report. A proofreader should have caught this.

A proofreader educated about trading system development could have flagged sloppiness suggestive of curve fitting (e.g. second paragraph below table here) and future leaks (e.g. this footnote and third paragraph below last excerpt here).

Sample sizes should always be given and the report (e.g. third paragraph below third excerpt here and first bullet point below table here) would be better with inferential statistics [testing] to identify the real differences (e.g. third paragraph below first table here and second paragraph below final excerpt here). Criteria for adopting trade guidelines should be detailed at the top. Control group performance would also be useful (e.g. paragraph below graph here and third paragraph following table here).

In conclusion, I think the report would be better described as a trading strategy than a fully developed system. With regard to the latter, though, the report is a great educational piece and a valuable springboard for further discussion.