Division I Computer Rankings: Final for 2003-2004 Season

[LakersFan]

The final computer rankings for the season are (finally) available at:

http://vortex.bd.psu.edu/~rutter/WomensHockey/WomensDIHockeyRankings.html

These are the final rankings because things are being decided on the ice this weekend. The true national champ is number 1, not what some computer program spits out.

Mease Top 5:

1. Harvard
2. Minnesota
3. Dartmouth
4. Wisconsin
5. St. Lawrence

First, Harvard barely slid into the number one slot (1.509 vs. 1.496) after winning the ECAC. So according to my computer, Harvard is the slim favorite in the Frozen Four. Also, according to both Mease and RKRACH, Wisconsin should be the fourth team in the Frozen Four. Send all complaints to mbrand(at)ncca.org.

Also, for you stat fiends out there, I included the unrestricted KRACH rankings as well. When you remove the bounds, the only ranking difference is a switch between St. Lawrence and UMD. Also, the KRACH score for Minnesota is 2709, not the 5000 under RKRACH. This is due to another version of the "Union effect." When I set upper and lower bounds for RKRACH, it forced Union and Sacred Heart to be 2, and adjusted everything else accordingly. It is interesting to compare the results.

This summer will allow me the time to compare Mease vs. KRACH vs. RKRACH so that I can have a modified system in place for next year. I believe there are ways to tweak Mease (the .5 variance assumption, for example) to better suit the system for college hockey. I also want to explore things like home ice advantage (short answer: there is one, but it has no effect on the ranking) and the effect of the prior distribution. Stay tuned to this message board for details.

[ARM]

Thanks for all of the time you have put into providing us with the computer ratings this season. It is interesting to have yet another way to compare the teams.

You mentioned that the limit you selected for the bottom teams changed the score for the top teams. I think this illustrates a flaw with any of the computer models and trying to apply the numbers to the teams' real life performance. In practice, the top teams are not going to lose to programs at the level of a Sacred Heart or a Union. In the men's game, the top teams can lose to the bottom teams. A "smart" system which could learn from year to year would almost work better. Such a ranking might provide a better handle on the matchup for which an upset is a realistic possibility, not just a mathematical one.

[Liam3851]

Frankly these sorts of changes are why I still favor pure KRACH or KARA (home-away corrected KRACH) over any form of restriction. It still seems too hacked to me. And honestly, if a team makes it through without a tie, they really are either the best or worst team. After the first few weeks of the season this usually encompasses just a couple of teams anyway, and RRWP can still give you a relative number, if not betting odds.
</rant>

That said-- thanks a bunch, Lakers, for introducing me (and briefly, some KRACH-heads on the men's board when I posted the concept over there) to Mease. I'm looking forward to seeing if experimental variance changes things any. Are you considering just trying to find a better constant across all teams, or trying to evaluate a team-by-team variance? Team-by-team would be probably difficult to calculate given the few data points in any season, but would I think be pretty instructive. You could certainly end up with situations where team A had lots of talent but was inconsistent, having less chance of beating a poorer team C than a team B with a slightly lower average rating but playing more consistently. In particular, it could be argued that Wisconsin-SLU is a good candidate for this measure-- Wisconsin with the better average performances, but a few extremely uncharacteristic ones (e.g. Northeastern).

As for ARM's comments about learning from year to year, it's interesting to think about. Any such changes would probably disqualify the system as an NCAA tournament measurement, but might make it more accurate as a predictor. *ponder*

[ARM]

Originally posted by Liam3851
As for ARM's comments about learning from year to year, it's interesting to think about.
My "learning" suggestion was not related to a specific team, since obviously even though teams have the same name from year to year, they are not the same team. Rather, the model could be adjusted for the likelihood of a bad team beating a good team. For example, if a team that is 5-22 never wins over a team that is 25-2, but the model thinks the probability is 2%, then the model is giving the low-ranked team too much credit against the powerful one.

For instance, who has Harvard lost to over the last two years? Dartmouth, UMD, Minnesota, Princeton. Only teams ranked in the top 12. So for Bemidji to have even a 2% chance of beating them might be too high, if it doesn't happen. By "them", I don't mean Harvard, but the team with the top record and a similar score in the ratings. And maybe considering factors like score of games would be a better indicator.

[LakersFan]

In my definition of a perfect ranking system, if two teams are 20-0-0, then the team with the harder schedule should be ranked higher. Under KRACH, both teams would have an infinite KRACH rating, and be tied for first. Now, RKRACH doesn't solve that, as evidence by the tie between Minn and Harvard all season. Mease, however, would rank one of those teams as number one, due to the likelihood penalty. This is the "add a win and a loss to an average team" situation, which can be added to KRACH. Yeah, RKRACH is hacked, but I have a problem with a system that fails to converge to solutions under observable conditions (a team with a perfect record). The nice thing is that we can run both systems, and see if there is a difference. (For once, I am glad there are only 30 or so teams.)

One thing I have discovered tinkering with the variances is that it doesn't change the ranking, but it does effect the Mease number, and therefore the likelihood that team A beats team B. In theory, you should be able to tweak the variances so that the predicted likelihood of victory matches the observed likelihood of victory. I have thought about individual team variances, but I haven't tried to implement it yet. There may be enough data points, but estimating variance is always hit or miss. Using a bayesian prior to say the mean variance is .35 (or whatever), and allow it to very a little for each team is probably the best answer.