Welcome to the 2nd annual NCAA PT Awards. The purpose of these
awards is to help raise the publicity for those systems that
have been superior in various qualities of interest.
Most awards will be based entirely on the numbers I've
monitored with my prediction tracker web pages. For the 2001
season I followed the weekly performance of 34 computer rating
systems.
BEST STRAIGHT UP WINNERS (Entire Season)
Winner: Massey Ratings, Kenneth Massey
The simplest and possibly most important way of measuring the
predictive ability of a system is to count the number of games where
the winner is correctly predicted. I collected predictions on the 652
games in 2001 involving two 1A teams.
The system that correctly predicted the most winners over the
course of the 2001 season was Kenneth Massey's Massey Ratings.
That is his original ratings. Not the version that is used in
the BCS. Massey's ratings correctly predicted the winner in 76.38
percent of the games (498-154). This was two games better than the
predictions that is defined to be the average of all system
predictions. The Vegas line was fives games back.
Kenneth Massey in 2001 and Stewart Huckaby in 2000 are the only
systems to ever outperform the vegas line (in the three years that
I have kept track). Last years winner, Stewart Huckaby's ARGH Power
ratings, fell to 7th place this year even though his percentage was
almost the same as a year ago.
2001 Winner: Massey Ratings. Kenneth Massey
2000 Winner: ARGH Power Ratings, Stewart Huckaby
1999 Winner: Vegas line
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)
Winner: The Vegas line
Deviation from actual game results is another way of measuring
the predictive ability of a computer system. Deviation from the game
score is the difference between the game prediction and the actual
result. A value of zero would mean the score difference is predicted
exactly. So one property of a good system would be to minimize
the systems average game deviation.
This season the lowest average game deviation was found in the
Vegas line. So for the third year in a row this award goes to
Roxy and the vegas oddsmakers. This year the Vegas line's average
deviation from the actual score was 12.25 points. That is an improvement
over last year's 12.37.
I will give an honorable mention to the runner up, Kenneth Massey's
Massey Ratings, the winner of the best prediction record. Massey's
average deviation was about a quarter of a point behind the Vegas Line.
Since the Vegas line has won this award three season's running that
means that I have never seen a single computer rating system that
has been able to beat the line in this category.
2001 Winner: Vegas Line
2000 Winner: Vegas Line
1999 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Entire Season)
Winner: Flyman Performance
Bias is a little different from deviation. Deviation measures the
distance between a prediction and the actual result. Bias combines
distance and location of the prediction. (Deviation is the
absolute value of the bias) Bias is measuring whether the
predictions are too high or too low. So if a sytem had an
average bias of zero that would mean that on average the system
predicts the margin exactly. This can often be used as a statistic
to help measure home field advantage.
For the entire season the lowest bias was observed in the Flyman
Ratings. He had an average bias of about -.06, meaning that on average
Flymans predictions were 0.06 points lower than the game result.
The average bias of the systems was around -0.60, meaning the computers
did not give enough points to the home team.
I will give an honorable mention to Ray Waits of YourLinx and Steve Wrathell's
CPA Rankings. Both came in a very close 2nd at -0.09.
2001 Winner: Flyman
2000 Winner: ARGH Power Ratings
1999 Winner: Jeff Sagarin
BEST AGAINST THE SPREAD (Entire Season)
Winner: BMC Picks, Brent Craig
Beating the spread is not the number one goal for most computer
rating systems. At least not considering ever possible game, as
one reader pointed out last season But it is something that the
average person considers. Will a system help you win your office pool.
The top system against the spread for the entire season was the
Brent Craig's BMC Picks. BMC was 324-283, (53.38%) against the
spread. As with last year, a method of doing better than the
best system would be to take opposite the worst system, The Harmon
Forcast. The Dunkel Inxed was the winner last season at 52.3%.
2001 Winner: BMC Picks
2000 Winner: Dunkel Index
1999 Winner: Average across all systems
MOST ACCURATE PREDICTOR (Entire Season)
Winner: The Vegas line
This award is based on mean square error, which takes into account
both deviation and bias.
The vegas line has the smallest mean square error at 248.25.
Honorable mention goes to the Kenneth Massey for being the best
among the computer systems. Sonny Moore was also right behind Massey.
2001 Winner: The Vegas Line
2001 Winner: The Vegas Line
1999 Winner: The Vegas Line
BEST STRAIGHT UP WINNERS (Second Half of Season)
Winner: Chris Montgomery
It is debatable how meaningful rating systems are in the early
parts of the season. How can you rate the teams in the first
week of the season when no one has played a game yet? The BCS
waits until mid October before releasing it's first ranking
of teams. Some systems chose to wait until around this time
before being made public. Looking at only the second half of the
season also gives an estimate of how well a system does based
only(or mostly) on data from the current year.
So to measure this I looked at all 1A games starting with with
week eight of the season. Out of these 348 games in the second
half of the season the system predicting the most winners belonged
to Chris Montgomery, 251-97 (72.1%). This was three games better
than Kenneth Massey, the system average, and the Harmon Forcast.
2001 Winner: Chris Montgomery
2000 Winner: Geoff Freeze
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season)
Winner: The Vegas line
As with the regular season the Vegas line also wins this award
for having the smallest deviation from the actual game scores
over the second half of the season. An honorable mention here
also goes to Jeff Bihl for being the best among the computer systems.
Average deviations were actually higher during the second half
of the season. This is the opposite of what happened in 2001.
2001 Winner: Vegas Line
2000 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Second Half of Season)
Winner: PerformanZ Ratings, Todd Beck
The lowest game bias in the second half of the season goes to me with my
PerformanZ Ratings. I'm surprised. I did not realize this until just now
looking at the results. On average, PerformanZ's predictions were only .02
above the actual result. This is an extremely accurate result but it sure
didn't carry over to the other categories. The second best score here
goes to the Gupta Power Ratings at -0.10.
2001 Winner: PerformanZ Ratings
2001 Winner: Darryl Marsee's Rankings
MOST ACCURATE PREDICTOR (Second Half of Season)
Winner: The Vegas line
The vegas line also takes this award for having the smallest mean
square error overt he second half of the season. For the second
half results honorable mention goes to Ed Kambour who had the best
results among the computer systems.
For what it is worth the values for 2001 were about 2 points worse
than they were the previous season.
2001 Winner: The Vegas Line
2001 Winner: The Vegas Line
BEST AGAINST THE SPREAD (Second Half of Season)
Winner: BMC Picks, Brent Craig
For the second half of the season the system that did the best against
the spread was the same as the winner for the full season, Brent Craig's
BMC Picks, 242-103 (54.2%). In general, the systoms did worse in the
second half of the season but BMC actually improved. Only 8 out of 34
systems were better than 50%. Here again, a better system would have
actually been to play the opposite of the Harmon Forcast which had a very
bad 43.5%
2001 Winner: BMC Picks
2001 Winner: The Buck System
BEST PREDICTIVE SYSTEM IN 2001
Winner: Massey Ratings, Kenneth Massey
Kenneth Massey's non-BCS ratings was clearly the best overall predictive
system this year. Just like ARGH last year, Massey was the winner in wins
and second to the Vegas line in deviation. But Massey tops it off this
year by also being near the top in the other categories as well.
2001 Winner: Kenneth Massey (non-BCS)
2000 Winner: ARGH Power Ratings
RETRODICTION AWARDS
Retrodiction refers to 'retrodicting' the previous results, rather
than predicting future games. It is possible that a computer rating
system can put more emphasis on explaining past results than attempting
to predict future results. So these systems may not neccesarily be
the best predictive systems but can still be very good at their
main objective. And some people will argue that explaining the
season to date is the only thing that matters in ranking teams.
So the retrodiction results on my page come from taking the final
ratings and using them to retrodict the entire season.
MOST RETRODICTIVE WINS
Winner: CPA Rankings, Steve Wrathell
The best retrodiction win/loss record for the 2001 season goes
to Steve Wrathell's CPA Rankings with a 562-90 (86.2%) record.
CPA also has a retro ranking that finished second. These systems
had 13 and 15 more wins than the next closest finisher.
This is the second year in a row for CPA to finish first. This year
improves on last years record of 83.9%.
2001 Winner: CPA Rankings
2001 Winner: CPA Rankings
SMALLEST RETRODICTIVE ABSOLUTE ERROR
Winner: Edward Kambour Football Ratings, Edward Kambour
This year we don't have the same person winning every retrodiction category.
The system with the smallest average retrodiction error was Edward Kambour
with his NCAA Football Ratings. Kambour's average error was 10.152.
Last year CPA Rankings led with a value of 9.62.
Kambour led by a fairly significant margin. The second best was the average
of the systems at 10.379 and the closest independent system was Kenneth
Massey (his original system again) at 10.425
2001 Winner: Edward Kambour Football Ratings
2001 Winner: CPA Rankings
SMALLEST RETRODICTIVE BIAS
Winner: CPA Rankings, Steve Wrathell
As with most wins Steve Wrathell's two CPA Rankings came in first and
second for smallest retrodictive bias with values of -0.0067 and
-0.0279. CPA Rankings also lead this category last year with a mean
bias of -0.0057. For what it is worth, the group as a whole did much
better in this category this year compared to last year.
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST ACCURATE RETRODICTIVE RATING
Winner: Edward Kambour Football Ratings, Edward Kambour
Edward Kambour's Football Ratings picks up his second retrodictive
award. This one for being the 'most accurate' at retrodicting the
scores determined by having the smallest mean square error.
Kambours mean square error was 169.03.
Before you ask. It does not have to be the case that the system
with the smallest absolute error will have the smallest square error.
Although it does happen here and in the predictive awards. But
note that the second place systems are not the same, neither here or
above. Here the second place goes to Jeff Sagarin at 179.17
2001 Winner: Edward Kambour FOotball Ratings
2000 Winner: Although not computed I'm quite certain CPA Rankings would have been the leader.
BEST RETRODICTIVE SYSTEM 2001
Winner: CPA Rankings, Steve Wrathell
The two obvious contenders for this are CPA Rankings and Kambour Ratings.
To choose between the two look at how good they did in the categories
they did not win. The difference is that Kambour does really bad
in retrodicting win/loss. Next to last in fact. I think this is a
strange and interesting result but it is enough to give CPA Rankings the nod.
Of note is that of all the systems only CPA, Sagarin, Rothman, and PerformanZ
are above average in every category.
2001 Winner: CPA Rankings
2001 Winner: CPA Rankings