Walk-Forward Backtesting: Testing a Model Honestly
The problem it solves
A model fitted to a season and then scored on that same season always looks good, because it has seen the answers. The same happens if a stat quietly includes information from after the game, such as the starting quarterback of that day when only last week's starter was knowable. That is look-ahead bias, and it is why most backtests look better than the live results.
How a walk-forward test runs
- Go through the games in date order. Each game is priced using only games played before it.
- Tune settings on one stretch of seasons, the tuning window.
- Freeze the settings. Then score a later stretch, the test window, that was not used for tuning. Look at it once.
- Keep a change only if it beats the baseline on the tuning window and still does on the test window. Report a null result as a null result.
Example years only. The actual windows differ by sport.
What it does not prove
A good walk-forward result shows a method held up on games it had not seen. It does not guarantee the next season will look the same. That is why live picks are locked before the start and graded in public: the live record, with its calibration, is the final check. Probetrics does not publish the exact inputs, weights or settings, or what was tried and rejected.
Questions and answers
- What is walk-forward testing?
- A backtest that moves through time, pricing each game only from earlier games, tuning on one window and scoring a later, untouched window.
- What is look-ahead bias?
- Using information in a test that would not have been known at the time, such as a final box score or a starter announced after the pick. It makes results look better than they could be live.
- Why freeze the settings before testing?
- Because adjusting them after seeing the test result quietly turns the test window into a tuning window, and the result no longer measures anything the model has not seen.
Read next
- CalibrationCalibration is whether stated probabilities match what happens: of the picks a model gave about 70%, about 70% should win.
- Market-anchored probabilityA market-anchored probability starts from the sportsbook's no-vig price for a game and lets a model change it only where testing has shown the model adds information.
- Hit rateA hit rate is the share of a player's games that finished on one side of a prop line, and over a small sample it is a wide range rather than a single number, which a Wilson interval makes visible.
- No-vig oddsNo-vig odds, also called fair odds, are a two-sided price with the sportsbook's margin (the vig, or juice) taken out, so the two sides' chances add up to 100%.
This page explains arithmetic and vocabulary. It does not recommend a bet, a side, an amount or a sportsbook, and the examples are illustrations. For information only, not betting advice. 21+ only. Gambling problem? Call or text 1-800-GAMBLER.