College Playoffs Are This Week. Did the Right Teams Get In?

December 28, 2016

Did the 2015 College Football Playoff Selection Committee get it right? Will the best four teams be competing for the NCAA College Football Playoff National Championship? We leave it up to you to decide. But before you decide, consider how Decision Lens might approach this question.

We used the Decision Lens prioritization platform to create a decision model to rank the top teams against a limited set of criteria that generally follow the committee guidelines: victories, strength of schedule, conference championships, wins over ranked opponents, losses to non-ranked opponents, and voter judgment. We then gathered the data and tallied the results.

Our initial rankings completed on December 4—before conference championship games—offered some surprises, mainly a pair of two-loss teams, Stanford and Notre Dame, edging out two undefeated teams (Iowa and Clemson) for playoff spots alongside Alabama and Oklahoma.

Decision Lens College Playoff Rankings – All Criteria (excluding Conference Championships)

December 4, 2015

Criteria: Voter Ranking FBS Wins Bad Losses Quality Wins Strength of Schedule


This initial result was largely based on the strength of schedule of both Stanford and Notre Dame and the number of quality wins amassed by the top teams.

As expected, the conference championship games changed the equation as it presented opportunities for highly ranked teams to bolster their resumes by adding conference titles and “quality” wins to their resumes. Could conference champions Clemson and Michigan State gain a spot in the top four?

In our final rankings, Notre Dame is replaced by Clemson, saving us the embarrassment of arguing that the nation’s only undefeated Power 5 team should be excluded from the playoff. But Michigan State, despite impressive wins over Iowa and Ohio State, remains a playoff bridesmaid by a relatively slim margin behind Stanford.

Decision Lens College Playoff Rankings

All Criteria Equally Weighted

December 18, 2015

Criteria: Voter Ranking FBS Wins Bad Losses Quality Wins Strength of Schedule Conference Champ


Did Decision Lens get it right? Let’s try to poke a few holes in our assumptions.

First, examine the large gap in score between Michigan State and Notre Dame in the exhibit above. Notre Dame, an independent, does not participate in a conference championship game. Considering that conference champions gain additional points in other model criteria, including Strength of Schedule, FBS Wins, and Quality Wins, does it even make sense to have an additional criterion for winning a conference title? And if not, what would be the model results if we exclude conference championships as a criterion? Did we stack the deck against Notre Dame?

Luckily we can test this assumption very quickly in Decision Lens. The exhibit below illustrates the results of excluding Conference Championship as a criterion. Under this scenario, the top four playoff qualifiers do not change: Alabama, Oklahoma, Clemson, and Stanford. Notre Dame gets promoted, but only above Michigan State to number 5 overall, which is probably small consolation for fans of the Fighting Irish. However, given that Stanford executed a last second field goal to beat Notre Dame, perhaps there is some justice here.

Still, we have a two-loss conference champion edging out a one-loss conference champion for the final spot. How can this be? The exhibit below includes three critical metrics that we used to help inform our ranking model: Strength of Schedule (using Sagarin rankings); Quality Wins (over ranked opponents); and “Bad” Losses (to unranked opponents). These metrics largely explain the results.

Of our three “bubble” teams competing for the fourth playoff spot (Stanford, Notre Dame, and Michigan State), Stanford played the toughest schedule to edge Notre Dame, and amassed enough quality wins without bad losses to put sufficient distance between itself and Michigan State. The Big Ten in general is penalized for a weak Strength of Schedule, and so despite Michigan State defeating Iowa, Ohio State, Michigan, and CFP-ranked Oregon, our model remains insufficiently impressed. Sorry, Spartan Fans.

Decision Lens College Playoff Rankings

All Criteria Excluding Conference Championship, Equally Weighted

December 18, 2015

Criteria: Voter Ranking FBS Wins Bad Losses Quality Wins Strength of Schedule


Is there any scenario at all that rewards Michigan State? There is. As part of our exercise, we asked our voters to help prioritize our decision critiera using the Pairwise process. In prioritizing the model criteria, our voters de-emphasized the only subjectively-informed Voter Ranking criterion, giving it only 10% weight. Are they sending a message to the CFP Selection Committee?

Our voters instead opted to emphasize Quality Wins (29%) and Strength of Schedule (20%), leaving their own judgment a marginal influence at best. The results, illustrated below, elevate Michigan State while dropping Clemson from the playoff group, with the pivotal difference being Quality Wins.

Decision Lens College Playoff Rankings

All Criteria, Weighted by Voters

December 18, 2015

Criteria: Voter Ranking FBS Wins Bad Losses Quality Wins Strength of Schedule Conference Champ


The interesting part of this scenario is that if we ignore Conference Championship as a criterion, Notre Dame is elevated into the fourth spot behind Stanford, just ahead of Michigan State, and substantially ahead of Clemson. This outcome is illustrated below.


Finally, what did our Decision Lens’ voters decide when left completely to their own opinions? A quick scenario analysis that weights Voter Ranking at 100% produces a result very similar to what the CFP decided, with Alabama facing Michigan State and Clemson facing Oklahoma in the first round. Our voters were not as impressed with Stanford’s resume as our model was, knocking the Cardinal down to #9 in the rankings. Perhaps it’s an East Coast thing.

Decision Lens College Playoff Rankings

Voter Ranking Only

December 18, 2015


In closing, our rankings model suggest an outcome that is not unlike what we often encounter with out clients. We identify a few “no-brainers”—choices that clearly merit top positioning regardless of what assumptions might apply. In this case, Alabama and Oklahoma have amassed very robust college football playoff credentials.

There is also a long list of “also rans”—viable options that offer selective merits, but in general are not competitive across the full spectrum of criteria.

This leaves the players that reside at what we like to call the “margin” of decision. This is the hard part, and frankly where we look to stakeholders to decide, weighing different scenarios in the model. Our model and analysis builds a strong case for Clemson and Stanford, with some debate about whether Michigan State could replace either team under certain assumptions, or whether Notre Dame should be given the fourth spot behind Stanford. In effect, that is the stakeholder decision: Four teams competing for two spots.

In case you missed it, here is the first post in this college football series: