Welcome to the 3rd annual NCAA PT Awards. The purpose of these
awards is to help raise the publicity for those systems that
have been superior in various qualities of interest.
Most awards will be based entirely on the numbers I've
monitored with my prediction tracker web pages. For the 2002
season I followed the weekly performance of 42 computer rating
systems. This is up from 34 last season.
MOST STRAIGHT UP WINNERS (Entire Season)
Winner: ARGH Power Ratings, Stewart Huckaby
The simplest and possibly most important way of measuring the
predictive ability of a system is to count the number of games where
the winner is correctly predicted. I collected predictions on the 706
games in 2001 involving two 1A teams.
The system that correctly predicted the most winners over the
course of the 2002 season was Stewart Huckaby's ARGH Power Ratings.
ARGH correctly predicted the winner in 75.78 percent of the games
(535-171). A couple of special honorable mentions have to be given
to Brent Craig and Ed Kambour. Kambour actually had the highest
winning percentage 76.48 (517-159), but he does not make predictions
for bowl games. Craig likely would have tied with ARGH if he had made
picks on just four more games. But since ARGH had the most winners
he takes the award for the second time in the last three seasons.
Kambour's percentage of 76.48% is the highest of all time.
(where all time means in the four years I have kept track).
This eclipses Kenneth Massey's mark of 76.38% last season.
As of last year only two systems had ever outperformed the Vegas
line, Kenneth Massey in 2001 and ARGH in 2000, but this year eight
different systems picked more winners than the line. Is this a sign
that the ratings are getting better or does it just mean Vegas had
an off season? Hopefully it is a sign of rating improvements.
2002 Winner: ARGH Power Ratings, Stewart Huckaby
2001 Winner: Massey Ratings. Kenneth Massey
2000 Winner: ARGH Power Ratings, Stewart Huckaby
1999 Winner: Vegas line
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)
Winner: The Vegas line
Deviation from actual game results is another way of measuring
the predictive ability of a computer system. Deviation from the game
score is the difference between the game prediction and the actual
result. A value of zero would mean the score difference is predicted
exactly. So one property of a good system would be to minimize
the system's average game deviation.
This season the lowest average game deviation was found in the
Vegas line. So for the fourth year in a row this award goes to
the vegas oddsmakers. This year the Vegas line's average deviation
from the actual score was 12.86 points. That is half a point worse
than last year's 12.25.
Honorable mention here goes to the runner up, the system average.
The system average is the average prediction of all the computer system
predictions. In fact, the system average's mean error of of 12.88 was
only 0.0284 points behind the line. By far the closest anything has
ever come to catching the Vegas line. The best mark by an individual
this season was Sonny Moore who was also very close behind the line at
12.90.
I have been asked why I don't give this award to the highest individual
rather than to the line every year. I believe that if we can't beat the
line then we don't deserve any credit. We need to find ways of improving
this aspect of predicting game outcomes.
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
1999 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Entire Season)
Winner: Gupta Power Ratings, Bapi Gupta
Bias is a little different from deviation. Deviation measures the
distance between a prediction and the actual result. Bias combines
distance and location of the prediction. Bias measures whether the
predictions are too high or too low. So if a sytem has an
average bias of +0.25 that means that on average the system
gives 0.25 points too many to the home team. This statistic can
be used to help guage home field advantage.
For the entire season the lowest bias was observed in the Gupta Power
Ratings. Gupta had an average bias of 0.00691. This was actually a very
good year for low bias. Last year Flyman ratings had the lowest bias
at 0.06. This season, eight systems counting Gupta, were lower than 0.05.
Another sign of possible improvement in the rating systems.
2002 Winner: Gupta Power Ratings
2001 Winner: Flyman
2000 Winner: ARGH Power Ratings
1999 Winner: Jeff Sagarin
BEST AGAINST THE SPREAD (Entire Season)
Winner: Dunkel Index
Beating the spread is not the number one goal for most computer
rating systems. At least not considering ever possible game, as
some readers like to point out each season. But it is something
that the average person likes to look at. Will a system help you
win your office pool?
The top system against the spread for the entire season was the
Dunkel Index. Dunkel was 357-280, (56.04%). Had someone placed
a $10 bet on every game using Dunkel they would have earned only $44.54
over the course of the year, or a 0.70% return on their money.
The Dunkel Index has now won this award two out of the last three
seasons. It was also a good year for systems to beat the spread.
Last year's winner was Brent Craig at only 53.38% and the year before
that Dunkel led with 52.3%
2002 Winner: Dunkel Index
2001 Winner: BMC Picks
2000 Winner: Dunkel Index
1999 Winner: Average across all systems
MOST ACCURATE PREDICTOR (Entire Season)
Winner: The Vegas line
This award is based on mean square error. Mean square error takes
into account both deviation and bias and is the most commonly
used measure of evaluating estimators.
The vegas line has the smallest mean square error at 261.966.
Honorable mention again goes to the system average (166.091) for coming
the closest any system has ever come to the line. Kenneth Massey and
Sonny Moore were the best individuals this year as they also were last
year.
2002 Winner: The Vegas Line
2001 Winner: The Vegas Line
2001 Winner: The Vegas Line
1999 Winner: The Vegas Line
------------------------
MOST STRAIGHT UP WINNERS (Second Half of Season)
Winner: Born Power Index ,William Born
It is debatable how meaningful rating systems are in the early
parts of the season. How can you rate the teams in the first
week of the season when no teams have played a game yet? The BCS
waits until mid October before releasing it's first ranking
of teams. Some systems chose to wait until around this time
before being made public. Looking at only the second half of the
season also gives an estimate of how well a system does based
only(or mostly) on data from the current year.
So to measure this I looked at all 1A games starting with with
week nine of the season. Out of the 368 games in the second
half of the season. the system predicting the most winners was the
Born Power Index, 278-90 (75.54). This was two games better
than ARGH, the full season leader. As he did for the full season,
Ed Kambour actually had the best winning percentage, 76.33%, but did
not have the most wins due to not making predictions for bowl games.
Both Born and Kambour's winning percentages in the second half were
higher than we have ever seen before.
I also just want to make a note here about the New York Times ratings.
They often have often gotten trashed in the past for their ranking
of teams. Over the second half of the season only three systems picked
more winners than the New York Times. I think their reinclusion in
the BCS this season turned out to be a good decision.
2002 Winner: Born Power Index
2001 Winner: Chris Montgomery
2000 Winner: Geoff Freeze
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season)
Winner: The Vegas line
As with the regular season, the Vegas line also wins this award
for having the smallest deviation from the actual game scores
over the second half of the season. An honorable mention here
also goes to Ed Kambour for being the best among the computer
systems.
The numbers here have gotten worse every year. I believe systems
have tried to become too politically correct and now attempt to maximize
retro wins rather than retro error.
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Second Half of Season)
Winner: Massey BCS, Kenneth Massey
The lowest game bias in the second half of the season goes to Kenneth Massey's
BCS Ratings. On average, Massey's BCS rating predictions were only 0.12
above the actual result. Overall, the numbers here did not look very good.
The majority of the systems were high, meaning they gave too many points
to the home team.
I want to make a note about the predictions of Massey's BCS system and
some of the other BCS systems. They do not actually make predictions themselves.
Because I want to be able to include them in my results I have had to
translate them to a different scale. The ordering of the teams is not changed
at all, the rating is simply transformed to a scaling that results in
suitable predictions.
2002 Winner: Massey BCS
2001 Winner: PerformanZ Ratings
2000 Winner: Darryl Marsee's Rankings
BEST AGAINST THE SPREAD (Second Half of Season)
Winner: The Sports Report
For the second half of the season the system that did the best against
the spread was The Sports Report, 191-154 (55.36%). This was only
0.0015 percentage points better than the Born Power Index.
As is often the case, the best way to beat the spread is often to pick
opposite of what a system says. This year that system would be picking
opposite of either of two BCS systems, either Jeff Sagarin's or Peter Wolfe's.
Those systems were both 42.9% against the spread.
2002 Winner: The Sports Report
2001 Winner: BMC Picks
2000 Winner: The Buck System
MOST ACCURATE PREDICTOR (Second Half of Season)
Winner: The Vegas line
The vegas line also takes this award for having the smallest mean
square error over he second half of the season. For the second
half results honorable mention goes to Ed Kambour who had the best
results among the computer systems. It is the second year in a row
that Kambour has been the closest to the line. If the gap continues
to get narrow at the pace it is going we may actually see a system
overtake the line in the next year or two.
2002 Winner: The Vegas Line
2001 Winner: The Vegas Line
2000 Winner: The Vegas Line
BEST PREDICTIVE SYSTEM IN 2001
Winner: 'Edward Kambour'
To come up with the best overall predictive system I give each system
points for how well they do in all of the above categories. I then sum up
the points as the system with the highest total is dubbed predictive system
of the year. So this award goes to the system that is the most well rounded
category. One flaw can totally take a system out of the running.
This season I give this award to Ed Kambour. First runner up is the Vegas line
and third place goes to Sonny Moore.
2002 Winner: Ed Kambour
2001 Winner: Kenneth Massey (non-BCS)
2000 Winner: ARGH Power Ratings
-------------------
RETRODICTION AWARDS
Retrodiction refers to 'retrodicting' the previous results, rather
than predicting future games. It is possible that a computer rating
system can put more emphasis on explaining past results than attempting
to predict future results. So these systems may not neccesarily be
the best predictive systems but can still be very good at their
main objective.
The retrodiction results on my page come from taking the final
ratings and using them to repredict the entire season.
MOST RETRODICTIVE WINS
Winner: The Sports Report - SLOTS
A new system from The Sports Report made a debut this year
with some very impressive numbers. SLOTS had a retro record of
638-69, (90.24%). This was a whopping 33 games better than the
second place system. The second place finisher, CPA Rankings,
which has won for the past two seasons also deserves special
recognition for having the best record among what I will call the
pure rankings. His 85.57% is down from his winning percentage
of 86.25 a year ago.
I also want to make a special note of another new system that did
extremely well in this category, Logistic Regression. This system ties
with CPA Retro Rankings for third place. This was my test of a system
that uses nothing but home/road and win/loss.
2002 Winner: The Sports Report - SLOTS
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
SMALLEST RETRODICTIVE MEAN ERROR
Winner: The Average of all systems
This category was extremely close this season. Three systems finished
vitually tied with the smallest mean error but the one with the lowest is
the system that is found by taking the average of all the other systems.
The average had a mean error of 10.5716. CPA Rankings finished very
close behind 10.5722 and was followed closely by Least Squares regression
at 10.5750.
The winning mean error has gotten worse each season. The best mark was
9.62 by CPA Rankings in 2000. Last season Ed Kambour led with 10.152.
2002 Winner: The average of all systems.
2001 Winner: Edward Kambour Football Ratings
2000 Winner: CPA Rankings
SMALLEST RETRODICTIVE BIAS
Winner: CPA Rankings, Steve Wrathell
It wouldn't seem like retro awards without giving something to CPA Rankings.
CPA had the smallest retro bias, -0.007. Kenneth Massey's BCS system was
really the only other competitor in this category this year.
The winning bias has also gotten worse each season. The lowest bias of .005 was
CPA Rankings in 2000.
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST ACCURATE RETRODICTIVE RATING
Winner: CPA Rankings, Steve Wrathell
The system with the smallest mean square error was actually least squares
regression, but since I did not include it until the season was over I will
give this award to CPA Rankings which had the next best score. Least squares
had a mean square error of 174.969. CPA Rankings was 2nd with 178.605.
The record for this category was 169.03 by Ed Kambour in 2001.
2002 Winner: CPA Rankings
2001 Winner: Edward Kambour FOotball Ratings
2000 Winner: Although not computed I'm quite certain CPA Rankings would have been the leader.
BEST RETRODICTIVE SYSTEM 2003
Winner: CPA Rankings, Steve Wrathell
Like best predictive system this award is found by giving each system a
score for each category. The system that has the highest total is the retroditive
system of the year.
CPA Rankings wins this award for the third year in a row. It really wasn't
that close either. CPA Retro ratings came in second place and the system average
in third place.
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings