Key takeaways
- Validation is a reading discipline, not a tool.
- Data, assumptions, costs, slippage, out-of-sample, and sensitivity are the six pillars.
- Always ask 'what would invalidate this study?'.
- A study that cannot be invalidated by any observation is not a study.
Why independent validation matters
The most useful skill for an independent learner is the ability to read someone else's study and stress it without taking the author's word. This note describes a neutral checklist for doing so. It works for any study, not only ours.
Data source questions
Where did the data come from? Is the time-stamping consistent? Are adjusted or unadjusted prices used, and is that documented? Are corporate actions handled correctly? For options, are end-of-day or intraday snapshots used, and what is the source for implied volatility?
Assumptions review
Is the execution model written down? Is the cost stack itemised? Is slippage modelled and is it scaled by liquidity? Is the strike universe filtered? Is position sizing fixed or adaptive, and if adaptive, on what?
Out-of-sample review
Is there an explicit out-of-sample window? Was it chosen before tuning began? Are the out-of-sample metrics reported separately, or only the aggregate?
Sensitivity testing
Does the rule survive ±25% perturbation of each parameter? Does it survive doubled slippage? Does it survive removal of the single worst and single best observations? If any of these collapses the result, the headline depends on a knife-edge.
What would invalidate the study?
Every research study should be able to answer this question. If the author cannot name an observation, a window, or a stress test that would invalidate the result, the study is decorative rather than scientific.
Reproducibility
Is the rule defined precisely enough that an independent reader could in principle reproduce it? If not, the result is not reproducible and the reader is being asked to trust the author.
Common mistakes
- Accepting summary metrics without inspecting the assumptions section.
- Skipping the limitations and what-can-go-wrong sections.
- Comparing only to cash returns and ignoring a passive benchmark.
- Treating a study with no failure conditions as a strong study.
How this appears in OptionScience reports
Every OptionScience report includes an Independent Validation Checklist in the protected report viewer so the reader has a structured way to scrutinise it.
Practical educational example
Checklist
- Is each assumption stated and reasonable?
- Are limitations and risk observations documented by the author?
- Does the rule survive stress on slippage and parameters?
- Does the rule outperform a relevant passive benchmark net of costs?
- Can the author name what would invalidate the study?