Silver appeared on ESPN before the divisional round and predicted a Seahawks-Patriots Super Bowl. The Seahawks lost that weekend, in a close game against the Falcons. The Patriots went down the following week, to the Ravens. To be fair to Nate, he was merely reciting the Football Outsiders odds at this time.
Football Outsiders and NFL Advanced Stats, which are at the forefront of NFL analytics, had fine first rounds, Football Outsiders only missing the Bengals-Texans game and NFL Advanced Stats going perfect. In the division round, both mis-predicted the Seahawks vs Falcons and Ravens vs Broncos games. They both had the 49ers winning in the championship round, but had the Patriots winning too. Combine it all and Football Outsiders picks picked six out of 11, or were correct 55% of the time; NFL Advanced Stats was right seven out of 11, or 64%.
Vegas did better. They got all four games in the first round and all but the Broncos vs Ravens in the second round, then finished it off by going one for three in the championship round and the Super Bowl, missing the Ravens' victories. That's a 73% accuracy rating overall in postseason.
The Accuscore system, which appears on ESPN, did equally as well. It forecasted all four first-round games, missed Ravens vs Broncos and Packers vs 49ers in the second round and the Ravens vs Patriots in the AFC Championship Game, and then got the Super Bowl right.
So was this year just bad luck? It turns out that there is really nothing unique about it. Accuscore has gone 6-5 or 7-4 in the last three postseasons. Football Outsiders was within a game of this year's 6-5 record the past few postseasons as well.
Is it just that the postseason is unpredictable? No, picks during the regular season haven't been any better. AccuScore's accuracy has been bouncing within a few percentage points of 65% over the last three seasons. That happens to do a pretty good job matching Vegas as well. Basically, you would expect for the average system to do just a little better than get seven out of every 11 games correctly predicted – just like this postseason. Sometimes they do slightly better, like 8-3 in a short game span, and slightly worse, like 6-5.
The confidence trick
The key is to look at the confidence in the predictions. Take the AFC divisional round's Ravens vs Broncos match-up. NFL Advanced Stats had the Broncos as a 78% to 22% favorite. That's slightly worse than 5/1. To put that in perspective, Barack Obama had as good a chance on the final day before the election of winning Colorado and Virginia. Most experts regarded the states as toss-ups, and well-known electoral analyst Larry Sabato actually had Mitt Romney taking Virginia.
Even in the Super Bowl, confidence was not particularly high. Football Outsiders said the 49ers were "slight favorites" at 58%. NFL Advanced Stats had the 49ers to win at 62%. That's about equal to what Obama's chance was to win Florida on the day of the election. The state went back and forth and wasn't called by the networks until days after the election.
It seems to me these experts are doing a fairly good job at reflecting the uncertainty in their predictions, and it generally matches the 7-out-of-11 odds seen this postseason.
The only place where this might not be true is in game probabilities. These might be based on old data that is no longer as good as the odds suggest. NFL Advanced Stats had the Ravens as a 97% favorite when they were up 28-6 in the Super Bowl. They dropped to 46% late in the fourth quarter. That sort of turnaround is statistically unusual, in my mind. I wouldn't find that to be a big deal, except NFL Advanced Stats had the Falcons as 98% favorite in the third quarter during their divisional game with the Seahawks, which then reversed to 8% when they fell behind late in the fourth quarter. The 49ers vs Falcons game featured a similar turnaround, though in this case the initial favorite didn't come back at the end.
The NFL is not like baseball, in which we can usually predict who will win and lose. Baseball is a game of large sample sizes and is really an individual's game of batter against pitcher, even if it is a team sport of nine players on each side. The NFL is a team sport where the linebacker's actions affect the cornerbacks and the right tackle makes or misses a block for the running back. If all the parts don't work together, it all falls apart.
The NFL playoffs come down to one game. It's quite possible the 49ers might have won the Super Bowl more times than not, if the game was played over and over again. We can't project football individual game winners in a way that should give us a lot confidence. The good news is that we mostly know our confidence level. It is, in the words of Donald Rumsfeld, a known unknown.
guardian.co.uk © Guardian News and Media Limited 2010
image: © rmtip21