Backtests, Luck and Overfitting
Test enough ideas and some will look brilliant by pure chance. These are the checks that separate a real pattern from a lucky one.
- About 1 in 20 random ideas passes a simple 5% significance test by luck alone.
- Count how many ideas you tried, and demand more than luck can explain.
- Check that the idea works on data it was not built on, and on related markets.
Why great backtests disappoint
If you try many rules on the same data (different averages, stop sizes, holding periods), the best one will look good partly because it happened to fit the noise of that period. This is called overfitting or data mining. When the rule meets new data, the luck part disappears.
The checks
- Sample size: 30 trades say much more than 5. Ten years of monthly data is only about 10 examples of each month.
- Both halves: does the idea work in the first half of the data and the second half?
- Related markets: does a rule that works on Nifty also work on Bank Nifty or Nifty 500?
- Robustness: do nearby settings work too, or only one exact setting?
- Multiple testing: if you tried 100 ideas, a 1-in-20 result is expected about 5 times by chance.
What the data on this site shows
The Edge Scanner tests about 1,000 event patterns on NSE indices. At a 3-month horizon, about as many looked special as luck alone would produce, and none passed every check. That is a normal and useful result: it shows how easily patterns appear in prices.
Common mistakes
- Tuning a strategy until the backtest looks perfect.
- Ignoring costs, slippage and gaps.
- Trusting a strategy because of a single great year.
Common questions
What is overfitting in trading?
Fitting a strategy so closely to past data that it captures random noise. It looks excellent in the backtest and usually disappoints on new data.
How many trades does a backtest need?
There is no fixed number, but results from fewer than about 30 trades are easily explained by luck. More trades, a longer period and different market conditions all help.
What is out-of-sample testing?
Building a rule on one part of the data and checking it on another part it never saw. A real edge should still show up.