The RPI vs. all

As we move through the final couple weeks of college basketball season, the selection committee and teams on the bubble will begin to take over what is important in basketball. And this year, as those potential selections have started to draw our attention, I've noticed that the RPI seems to have taken more shots than usual. Eamonn Brennan of ESPN.com has written about his disdain for the RPI and Scott Van Pelt delivered a diatribe (that made about 30 good points) on the matter.

Then yesterday I read an excellent piece from John Gasaway on the history of the RPI from Basketball Prospectus (it's a long article, but if you've got the time, check it out). Gasaway attended the NCAA's mock selection committee and came away searching for more information. Some of the more interesting tidbits to be found in this article:

"Van Valkenburg and his staff looked at using margin of victory in their new rating system, but rejected the idea for two reasons: it could encourage coaches to run up the score, and final scores "oftentimes do not reflect the closeness of games." Instead Van Valkenburg and his colleagues formulated a rating system that adjusted a team's Division I winning percentage based on strength of schedule and game location. The name "Ratings Percentage Index" apparently came about because it did not include the taboo word "Ranking" and because it was thought "RPI" formed a serviceable acronym."
Among the largest complaints about the RPI is that it does not take margin of victory into account. Many of the newer metrics for measuring teams lists margin of victory as an essential component to whether or not a team is actually good (KenPom, LRMC, ESPN's new BPI, John Hollinger's NBA rankings, the list goes on). And there is some truth to that. A team that is 10-0 with all its wins by three or less points is probably not as good as one that is 8-2 with seven wins by double digits, and two losses by five or less. But what is more interesting from that quote is why the RPI creators chose not to use margin of victory. They believed it would encourage coaches to have their teams run up the score. And think about it: for the past decade or so, haven't we heard about teams doing what they can to satisfy the RPI? It started out by teams just trying to secure as high a winning percentage as possible, then when the RPI was adjusted, the fad became scheduling as many top-100 RPI teams as they can to improve their RPI ranking with wins. So if margin of victory was included in the initial RPI, wouldn't coaches have just left their starters in the game in early season games and tried to win by 50? Would that have helped to tell us anything about which of those teams was best? Interestingly, as Gasaway notes in the Prospectus piece, the exclusion of margin of victory in the RPI has "created the perfect conditions under which scoring margin, adjusted for strength of schedule, could and has become a surpassingly powerful evaluative tool. But those of us who use that tool should acknowledge the analytic free ride we've been given, courtesy of Van Valkenburg. The stats we're able to deploy in 2012 are great at evaluating teams -- but the only teams we've ever been able to evaluate have been ones striving to look good to Jim Van Valkenburg, and not to us."

Also of note from the article: "Today when Van Valkenburg's invention is faulted on the grounds that it doesn't perform as well as other rating systems that benefit from more recent advances, an oddly prevalent response is that the NCAA "isn't trying to predict anything." Gasaway goes on to say that it seems odd to him the NCAA wouldn't want to improve upon its own formulas between seasons, but I couldn't help but be struck by the above quote, because I agree with it wholeheartedly. I hate the RPI, and have since I've been old enough to grasp what it is. But when selecting teams for the NCAA Tournament, I don't think the NCAA should be "trying to predict anything". It should be selecting for at-large bids the 38 teams that have earned it more than others. Of course, who has "earned" it most is weak language because it could be interpreted in many ways, but I think first and foremost to "earning" a berth in the tournament should be winning. And so it occurred to me that while the Jeff Sagarins and Ken Pomeroys of the world may have better systems for showing us which teams are the best in the country (I believe they do), their systems may not be any better than the RPI for selecting tournament teams, because they are meant to be predictive. I don't care how well the numbers think a team like Minnesota will perform if given a shot, I care whether the team's results from the season warrant being given a shot. So I decided to do a comparison between the ratings systems.

There are 38 at-large bids given out in the NCAA Tournament. So I looked at the top 38 teams in the RPI (as according to realtimerpi.com), Sagarin and Kenpom ratings. Of course, several of those teams will win their conference tournaments and receive automatic bids. To account for those teams, I removed any team that was in first place in its conference as of games played Feb. 22, and used the top 38 excluding those teams. Here were the final 15 teams to make the bracket from each rating system:

From looking that list over, one thing becomes abundantly clear quickly: the RPI is grossly overrating the Conference-USA. In the RPI, the C-USA would be getting four teams into the tournament (Southern Miss is rated 11th! and Memphis is 27th), including a 17-10 Marshall squad whose best win this season is a 65-64 triumph at home over UCF. Um, no. Also, the RPI has St. Joseph's making the tournament, an 18-11 team out of the A-10 with essentially the same resume' as Dayton, but less quality wins. Clearly, the RPI is not a very good system for placing these teams in the tournament.

But a quick glance at the Kenpom final 15 doesn't exactly set one's mind at ease. Two names jump off the page from the Pomeroy final 15: La Salle and South Dakota St. Pomeroy has been championing the Explorers out of the A-10 as one of the nation's most underrated teams, but they are just 18-10 (7-6 in the A-10) with losses to Robert Morris, Delaware, Dayton, Richmond and St. Joe's. Doesn't look like a tournament team to me. And South Dakota State has gone 22-7 on the season, but they play in the Summit League. They have lost to 12-14 North Dakota by 19 points, 15-14 Oakland, and (wait for it) a 9-17 South Dakota team. I'm sure the battle for South Dakota is as intense as any rivalry in the country, but South Dakota is 4-12 in the Summit this year. Maybe if SDSU's record were more like 26-3, you could forgive them a loss like that. But 22-7? I don't think so.

In Sagarin's final 15, the largest disturbance for me is in the five consecutive Big East teams near the top. In addition to those, Sagarin has four rated above them receiving bids. His nine Big East teams do not include South Florida, a team that sits third in the conference standings and in the eyes of the committee is probably ahead of at least a couple of those teams in the final 15. So according to those rankings, NINE OR TEN BIG EAST TEAMS SHOULD MAKE THE TOURNAMENT THIS YEAR. THE BIG EAST IS NOT VERY GOOD THIS YEAR! Come on. Looking at the Big East standings doesn't do anything to assuage the discomfort. The team currently sitting 10th is West Virginia (17-11, 7-8 Big East). Perfect resume' for sneaking into a 10 or 11 seed. Ugh. The other issue I see in Sagarin's is one that pops up in Kenpom as well. San Diego State is rated by both of them as barely making the tournament. In Kenpom, the Aztecs are actually the last team to make the field. Sure, SDSU has suffered some bad losses of late and fallen off a bit, but that is still insane! They are so far superior to the teams surrounding them in each of those ratings (Northwestern, South Dakota State, Minnesota, Southern Miss), that this almost looks like a joke.

In addition to all this, if Murray State were to lose its conference tournament, it would not receive an at-large bid according to Kenpom's numbers (currently rated 56, San Diego St. is 57), and would be right on the borderline for Sagarin (currently 46, Illinois is 55). That is ridiculous. I don't think there is much doubt the Racers have proven they belong comfortably in the tournament, not on the bubble (let alone off it). Of course, the RPI would have three Pac-12 teams making the field (absurd), and would be setting Colorado State as a six or seven seed. The Rams (17-9, 6-5) have had a couple great wins in the Mountain West, but should not be considered a tournament team yet.

Interesting to me is that all three systems have Miami of Florida making the field relatively comfortably (a nine or 10 seed). The U certainly qualifies as a bubble team at the moment, but at 16-10 overall and 7-6 in the ACC with exactly zero good wins on its resume' (best so far are wins over Clemson and Virginia Tech), the Hurricanes should not be considered an easy choice right now (as they seem to be by all three ratings systems). Maybe Miami actually is better than it seems, though, if three different methods of looking at them come out relatively positive.

I realize that neither Jim Van Valkenberg, Ken Pomeroy nor Jeff Sagarin would say the best way to make tournament selections would be to simply run down their list and pick them in order. Pomeroy has said as much and Van Valkenberg's son says the same in the Prospectus piece. I know am pretty sure all those men would believe that Murray State and San Diego State are easy tournament calls.

But that wasn't the point of this post. Yes, the RPI is horrible. But when it comes to its supposed principal use, selecting tournament teams, all of our advanced metrics from today aren't much better at doing the job.