Key takeaways
- Overfitting captures noise as if it were signal.
- Every additional free parameter increases the risk.
- Out-of-sample evaluation and parameter perturbation are the standard defences.
- If a rule's edge disappears under ±25% parameter changes, the original setting was probably overfit.
What overfitting is
Overfitting happens when a rule has been tuned so tightly to one historical window that it captures noise as if it were signal. The result looks excellent in-sample and disintegrates everywhere else. It is the most common failure mode in published quantitative research.
Why it happens so easily
Given enough free parameters, any historical window can be made to look excellent. With three parameters and a small sample, the search space is already large enough that some combination will look strong by chance. With eight parameters, you can almost always produce a clean equity curve.
The problem is rarely intentional. A researcher tries a parameter, sees a slightly worse result, tries another, and stops when the curve looks right. Each individual decision is reasonable; the cumulative process is parameter mining.
Signs of overfitting
An unusually smooth equity curve. A small number of trades supporting an aggressive CAGR. Many free parameters relative to the sample. No out-of-sample window. Reported performance that collapses when parameters are perturbed even slightly. Filters that exclude specific months or specific instruments without a structural reason.
Defences
Reduce free parameters. Use an explicit out-of-sample window the researcher has not looked at while tuning. Report parameter sensitivity: perturb each parameter ±25% and re-report. Use walk-forward windows so the choice of in-sample is not itself a degree of freedom.
None of these defences make a rule certain. They reduce the probability that the reported result is artefact rather than signal.
Common mistakes
- Reporting in-sample results without an out-of-sample window.
- Adding parameters until the equity curve looks clean.
- Ignoring parameter sensitivity around the chosen values.
- Excluding specific months or instruments without a structural reason.
How this appears in OptionScience reports
OptionScience reports include a Robustness section in the Performance dashboard showing walk-forward and parameter-sensitivity behaviour.
Practical educational example
Checklist
- How many free parameters does the rule have?
- Is there an explicit out-of-sample window?
- Does the rule survive ±25% parameter perturbation?
- Is the number of trades large enough to support the reported metrics?