Common backtesting mistakes

Backtests fail predictably. They use information the rule could not have had. They ignore costs. They optimise on the same window they report on. They survive only because losing instruments quietly fell out of the dataset. Each of these flaws inflates headline metrics in the same direction — upward — which is why almost every amateur backtest looks attractive.

Once you know the failure modes, you can scan any published study in five minutes and rank how much of the result is method and how much is artefact.

Lookahead is the use of information that the rule would not have had at the moment of decision. The textbook case is reading a bar's close to decide an action 'at' that close — in reality the close is the moment the bar ends, and the order goes on the next bar.

Subtler versions include using adjusted prices that fold in future corporate actions, using revised macro data that was not yet published at the historical date, and using indicators whose calculation window includes the current bar when the rule fires intrabar.

Survivorship bias arises when the universe used in the backtest is the set of instruments that exist today, instead of the set that existed at each historical point. Delisted stocks, expired contracts, and brands that vanished after a corporate event quietly disappear, biasing the surviving sample upward.

For Indian markets, this also covers index reconstitutions: NIFTY 50 and NIFTY Bank constituents change over time, and using today's constituents to backtest historical signals produces an unrealistic universe.

An honest Indian options backtest models brokerage, STT, exchange transaction charges, SEBI turnover fees, GST on charges, and stamp duty separately, and applies a slippage assumption that scales with strike liquidity and timeframe. Modelling only brokerage is the most common shortcut and the most expensive one — STT alone can flip the sign of a high-turnover study.

Optimisation bias appears when parameters are chosen on the same window used to report results. Given enough free parameters, any window can be made to look excellent. The fix is to choose parameters on one window (in-sample) and report on a different window the author has never looked at (out-of-sample), then to disclose both.

When you scan a published backtest, look for: a single equity curve presented as proof, no assumptions section, no slippage model, no out-of-sample window, no parameter-sensitivity panel, no limitations section, and no comparison to a relevant passive benchmark. The more of these are missing, the more likely the headline is artefact.