Here I am compiling a list of test cases that can be used to help evaluate the quality of any set of computer (or human/voter) rankings. There is nothing special about these teams, other than they present useful cases I've noticed over time studying scores, upsets, and strange-looking rankings. In each case several of the computers out there violate what is sensible. I think these are all clear-cut cases, and if a system violates most or all of them, we should be skeptical of it. But don't take my word for it - do the comparisons yourself.

2005

• Fresno State should be above Boise State. Fresno beat Boise 27-7. The two had similar success in their conference. Out of conference, we see this: Oregon State beat Boise State by 3. Oregon beat Fresno State by 3. Oregon whipped Oregon State 56-14.

2004

• Wisconsin (9-3) should be ranked above Ohio State (8-4). Wisconsin won 24-13 at Ohio State. Both had one loss to a bad team. Ohio State lost to two teams that Wisconsin beat, while Wisconsin only lost to one team Ohio State beat.

• Utah State should be above Arkansas State. Utah State beat Arkansas State 49-0. Both had three wins over Div.-1A teams. Arkansas State's were by 1, 3, and 4 points over weak teams. Utah State's wins were by 49, 21, and 18 points! Both beat UL-Monroe: Arkansas State won by 3, Utah State won by 18. I think this is probably the most useful comparison on this page. A system simply cannot violate this and have any credibility in my eyes...
• Alabama should be near TCU. Alabama beat Southern Miss by 14. TCU lost to Southern Miss by 12. Alabama beat South Florida by 23. TCU beat South Florida by 3. Alabama was competitive with several Top 25 teams (two Top 5). TCU won by only 3 over several teams ranked in the #70-95 range. It's hard to set an exact rule here, but I assert that Alabama should be within 10 ranking positions of TCU. The BCS ratings having TCU 25th and Alabama 68th are clearly wrong!

2002

• TCU (10-2) should be above Colorado State (10-4). TCU beat Colorado State 17-3 in a bowl game. TCU's losses were both close games on the road. Colorado State lost a home game to #85 UNLV. TCU won at Louisville by 14. Colorado State beat Louisville by only 3 at home.

• Washington should be above BYU (Washington 29, BYU 17). BYU struggled with two mediocre teams (beat New Mexico by 3, beat Texas A&M by 4). Washington only lost by 3 at #5 Arizona State, and their other two losses were to Top 20 teams (Colorado & Notre Dame). One might argue that the loss to Notre Dame was a blow-out and therefore Washington needs to come down a few notches. However, BYU simply has no saving arguments - they didn't beat any teams better than Washington. Their best wins were over #17 Kansas State (by only 4) and #29 Rice. Washington played a stronger schedule, won more impressively, and most importantly, they beat BYU. Mainly I have a problem with rankings like the AP Poll having BYU 5th and Washington 16th. That is patently WRONG. If someone has BYU 10th and Washington 11th, so be it. Personally, I think Entropy has it about right: Washington 10th, BYU 17th...
  
• Note to BYU fans - I like the BYU program and wish it well. Please consider all seasons; I think you'll see B.Y.U. treated quite fairly. Entropy's 1984 ranking of BYU at #3 is higher than many computers I've seen. Most place them 4th-5th.

• Denver (9-0) was not a top 20 team. The best team they beat was Colorado - a team that lost 23-0 to Utah State. Utah State destroyed Wyoming and Colorado State, both of whom were common opponents of Denver and Colorado. So, it appears Utah State proved an undeniable dominance over the Colorado teams and Wyoming. Based on scores, the Montana and Washington teams were considerably better than Utah State. All the comparisons I have considered point to Denver and Colorado not being very good. Denver only beat Tulsa by 1 point. Tulsa was one of the worst teams in the country (0-8-1 record). This is one of the best cases I've found where an undefeated team is not a great team. None of Denver's opponents were very good and Denver barely beat a few. Rankings systems that rely too much on records will be deceived...
• Keep in mind, various rankings systems will have different numbers of teams listed for years this far in the past. Others have 90-100 teams included, while Entropy has 272. If I listed only 95 teams, I would have Denver around #43. As it is, Denver is #122.

Note: The final 2003 BCS ratings are guilty on both counts! They had:

• 25. TCU
• 68. Alabama
• 102. Arkansas State
• 107. Utah State

The BCS ratings almost fail the 2004 test, having Wisconsin 19th and Ohio State 20th. I haven't seen any composite listings of the 2002 BCS rankings, but I do know that a majority of their systems ranked Colorado State above TCU, and one (NY Times) ranked South Florida impossibly high at 14th, and 7 spots above the Arkansas team that beat them 42-3!

{back}