Welcome to the 5th annual NCAA PT Awards. The purpose of these
awards is to help raise the publicity for those systems that
have been superior in various qualities of interest.
Most awards will be based entirely on the numbers I've
monitored with my prediction tracker web pages. For the 2005
season I followed the weekly performance of 56 computer rating
systems. This is up from 50 last season.
MOST STRAIGHT UP WINNERS (Entire Season)
Winner: Pigskin Index, Gary Hawley
This year I collected predictions on 662 games involving two 1A teams.
The season was fairly closely contested with the best record going
to Gary Hawley's Pigskin Index, 488-174 (73.7%). This is the lowest
winning winning percentage we have seen since 1999. The record for best
prediction record was set by Ed Kambour in 2002 at 76.48%.
This year 6 out of 50+ ratings managed to finish better than the Vegas line.
2005 Winner: Pigskin Index
2004 Winner: Jeff Sagarin's Overall Rating
2003 Winner: Edward Kambour & Jeff Sagarin's Overall Rating
2002 Winner: ARGH Power Ratings
2001 Winner: Massey Ratings
2000 Winner: ARGH Power Ratings
1999 Winner: Vegas line
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)
Winner: The Vegas line
Deviation from the game score is the difference between the game prediction
and the actual result. A value of zero would mean the score difference is predicted
exactly. One property of a good system would be to minimize the system's average
game deviation.
The fact that the Vegas line wins this category for the 7th straight year is not
surprising. However, it is interesting that in a year with very low winning percentages
compared to previous years the line sets a new record for smallest absolute error with
a mark of 11.93, breaking the 12 point level for the first time ever. The previous record
was 12.25 back in 2001. So this year beats that by a very large .32 points. The was
a poor season for predicting against the spread, perhaps this was a result of the lines
being closer to the real outcomes than we have ever seen.
This was the first full year using the computer adjusted line. I had hoped that this would
be able to beat the line. It did't but came in second only 0.03 points behind.
The best manned system this year was the Dunkel Index at 12.17. That was good enough to
break the old record but not enough to win the season.
2005 Winner: Vegas Line
2004 Winner: Vegas Line
2003 Winner: Vegas Line
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
1999 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Entire Season)
Winner: Computer Adjusted Line
Bias is a little different from deviation. Deviation measures the
distance between a prediction and the actual result. Bias combines
distance and location of the prediction. Bias measures whether the
predictions are too high or too low. So if a sytem has an
average bias of +0.25 that means that on average the system
gives 0.25 points too many to the home team.
There were a few very low scores this year, including the line, but the
lowest average bias was 0.011 points by the Computer Adjusted Line. The line
had a value of 0.02. Several systems finished less than 0.10. Least squares
regression actually had the lowest mark of 0.004 but it starts up at mid season.
If you want to know how the computer adjusted line works here it is: I wanted
to try something similar to the line which tries to balance the books. So I start
with the line and if a certain majority of the computer systems are all on one side
of the line I shift it that direction a fraction, if a larger majority are on one
side I shift it a larger fraction.
2005 Winner: Computer Adjusted Line
2004 Winner: CPA Rankings
2003 Winner: Edward Kambour
2002 Winner: Gupta Power Ratings
2001 Winner: Flyman
2000 Winner: ARGH Power Ratings
1999 Winner: Jeff Sagarin
BEST AGAINST THE SPREAD (Entire Season)
Winner: Keeper's Ratings K.D. Keep
Beating the spread is not the number one goal for most computer
rating systems. But it is something that the average person likes
to look at and probably the most popular subject that I get emails
about. I know that looking at all the games is probably not the
most meaningful of measurements but it is what I have been reporting.
As I mentioned above, I believe that the fact that the Vegas line had
an all time low average error and also an extremely small bias lead to
some poor records against the spread. There was a handful of systems
that were able to beat the spread more than 50% of the time but the majority
of us this year were in the 47-49% range. The best record was 342-298
(53.43%) by Keeper's College Football Ratings. A curious result this year
is that the best way to beat the line was actually to pick against the system
that had won this category two years in a row. If you picked against Kambour's
system you would have been right 54.8% of the time. Does that mean he should win
for the third year in a row? The record for this category was set in 2002 at
56.04% by the Dunkel Index.
People always ask me how well would a given system do if you only looked at
games significantly different from the line. Well, I have looked at that for
this season and the results do not change very much at all. I have looked at
games that are 3 points or more from the line and even games 7 points or more
from the line and the numbers do not really improve that much if at all.
Counting predictions 3 points or more from the line, Keeper still has the
highest percentage at 54.04.
2005 Winner: Keeper's College Football Ratings
2004 Winner: Edward Kambour
2003 Winner: Edward Kambour
2002 Winner: Dunkel Index
2001 Winner: BMC Picks
2000 Winner: Dunkel Index
1999 Winner: Average across all systems
MOST ACCURATE PREDICTOR (Entire Season)
Winner: The Vegas line
This award is based on mean square error. Mean square error takes
into account both deviation and bias and is perhaps the most commonly
used measure of evaluating estimators.
The Vegas line has the smallest mean square error at 234.541. This is
also a new record. The previous record was 248.249 set in 2001. The
line had it's most accurate season ever points wise even though it was
reasonably low win/loss wise. The most accurate manned system this year
was Dunkel Index at 241.902.
2005 Winner: The Vegas Line
2004 Winner: The Vegas Line
2003 Winner: The Vegas Line
2002 Winner: The Vegas Line
2001 Winner: The Vegas Line
2000 Winner: The Vegas Line
1999 Winner: The Vegas Line
------------------------
It is debatable how meaningful rating systems are in the early
parts of the season. How can you rate the teams in the first
week of the season when no teams have played a game yet? The BCS
waits until mid October before releasing it's first ranking
of teams. Some systems choose to wait until around this time
before being made public. Looking at only the second half of the
season also gives an estimate of how well a system does based
only(or mostly) on data from the current year.
MOST STRAIGHT UP WINNERS (Second Half of Season)
Winner: Dunkel Index
The second half data started with week 8 and consisted of 323 games
between two division 1A teams. The system with the best record over the
second half of the season was the Dunkel Index, 234-89, 72.45%. The winning
winning percentage has declined each of the last two years since the record
of 76.7% set two years ago by Born and Kambour. My PerformanZ Rating finished
in second place, one game back. My ratings also finished first in Div 1AA over the
second half.
2005 Winner: Dunkel Index
2004 Winner: Jeff Sagain, Wesley Colley
2003 Winner: Born Power Index & Edward Kambour
2002 Winner: Born Power Index
2001 Winner: Chris Montgomery
2000 Winner: Geoff Freeze
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season)
Winner: Vegas Line
The updated Vegas line returns to the top spot after a two year absense with an average
error of 12.1646 points. This is the lowest we have seen since the line had a score of
12.11 back in 2000. The automated computer adjusted line and the opening line came in
second and third. The top manned system was the Dunkel Index at 12.300.
2005 Winner: Vegas Line
2004 Winner: Ken Ashby AccuRatings
2003 Winner: Vegas Line (opening)
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Second Half of Season)
Winner: Brent Moore
The lowest game bias in the second half of the season goes to Brent Moore
Moore's average bias over the second half of the season was -0.1056. The numbers were
substantially worse for the second half of the season as they were for the full season.
My PerformanZ Ratings again finished in second place.
2005 Winner: Brent Moore
2004 Winner: Least Squares Regression
2003 Winner: Darrly Marsee
2002 Winner: Massey BCS
2001 Winner: PerformanZ Ratings
2000 Winner: Darryl Marsee's Rankings
BEST AGAINST THE SPREAD (Second Half of Season)
Winner: The Sports Report
For the second half of the season the system that did the best against
the spread was The Sports Report, 168-144 (53.85). The Sports Reports submits
three rating systems, this is their predictive ratings which also won this award
three years ago. The numbers were a little better in the second half that they
were for the season as a whole, but this winning mark is down considerably from
the 59% that StatFox had last season.
2005 Winner: The Sports Report
2004 Winner: StatFox
2003 Winner: Edward Kambour
2002 Winner: The Sports Report
2001 Winner: BMC Picks
2000 Winner: The Buck System
MOST ACCURATE PREDICTOR (Second Half of Season)
Winner: Vegas Line
Predictably, the Vegas line gets the award for most accurate predictor over
the second half of the season. The computer adjusted line and the opening line
came in second and third. The best manned system was the Dunkel Index.
As good as the line was this year it still had a mean square error higher
than Ashby Accuratings had last season.
2005 Winner: Vegas Line
2004 Winner: Ashby Accuratings
2003 Winner: Vegas Line
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
BEST PREDICTIVE SYSTEM IN 2005
Winner: Dunkel Index
To come up with the best overall predictive system I give each system
points for how well they do in all of the above categories. I then sum up
the points and the system with the highest total is dubbed predictive system
of the year. So this award goes to the system that is the most well rounded
One flaw can totally take a system out of the running.
Looking over the various awards above there is no clear cut favorite on who
would be the best overall predictive system. You might guess the Vegas line.
And actually I don't remember if I typcially exclude the line or not, but regardless
they did not come in first place this year anyway. The award this year was won
in the second half were the Dunkel Index rose to the top in all areas. If I include
the line in the calculations it would have come in second. Lee Burdorf and Jeff Bihl
also had good overall ratings this year.
2005 Winner: Dunkel Index
2004 Winner: Ken Ashby AccuRatings
2003 Winner: Ed Kambour
2002 Winner: Ed Kambour
2001 Winner: Kenneth Massey (non-BCS)
2000 Winner: ARGH Power Ratings
RETRODICTION AWARDS
Retrodiction refers to 'retrodicting' the previous results, rather
than predicting future games. It is possible that a computer rating
system can put more emphasis on explaining past results than attempting
to predict future results. So these systems may not neccesarily be
the best predictive systems but can still be very good at their
main objective.
The retrodiction results on my page come from taking the final
ratings and using them to repredict the entire season.
Rather than list some honorable mentions I am going to break the
retrodictive categories into what I will call large and small systems.
A small system will be any system that estimates only a global home
field advantage. Systems that estimate individual home field advantages
are always going to end up winning these awards so I am opening up new
categories to the more traditional systems.
MOST RETRODICTIVE WINS - Large System
Winner: The Sports Report - SLOTS
The Sports Report SLOTS system had the best retrodictive record
for the fourth year in a row with a record of 615-48 (92.76%).
This is pretty much identical to last years record.
2005 Winner: The Sports Report - SLOTS
2004 Winner: The Sports Report - SLOTS
2003 Winner: The Sports Report - SLOTS
2002 Winner: The Sports Report - SLOTS
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST RETRODICTIVE WINS - Small System
Winner: Jeff Sagarin
Winner: Warren Claassen
Among the small systems the best retrodictive winning percentage was a
tie between Jeff Sagarin and Warren Claassen, both were 551-112, 83.11%.
Several of the BCS ratings were the runner ups, Sagarin, Wolfe,
Colley and Massey were all within 3 games of the lead.
The record was 86.02% set by logistic regression in 2004.
2005 Winner: Jeff Sagarin, Warren Claassen
2004 Winner: Logistic Regression
2003 Winner: Anderson/Hester
2002 Winner: Logistic Regression
2001 WInner: System Average
SMALLEST RETRODICTIVE MEAN ERROR - Large System
Winner: Least Square Regression with team HFA
Least squares regression with individula team HFA finished 0.60 points
better than the second best system for the second year in a row.
Being a 'large' system did not give much of an advantage this year as the
2nd, 3rd, and 4th place systems were all simple systems.
The winning mean error this year was 9.3118, which is a little worse that
the record of 9.206 set last season.
2005 Winner: Least Square Regression with team HFA
2004 Winner: Least Square Regression with team HFA
2003 Winner: The Sports Report - SLOTS
2002 Winner: The average of all systems.
2001 Winner: Edward Kambour Football Ratings
2000 Winner: CPA Rankings
SMALLEST RETRODICTIVE MEAN ERROR - Small System
Winner: Jeff Sagarin
The winner of the small class ratings is Jeff Sagarin's points or predictive ratings,
with a mean error of 10.0129. This breaks least square regressions two year reign
and sets a new record.
2005 Winner: Jeff Sagarin Predictive
2004 Winner: Least Squares Regression
2003 Winner: Least Squares Regression
SMALLEST RETRODICTIVE BIAS
Winner: The Sports Report - SLOTS
I am not breaking this one down into sub categories. The system that had the smallest
mean bias was The Sports Report SLOTS system. They had a bias of 0.0141.
The numbers weren't that good in this category this year. Other systems with good scores
were CPA Rankins and Peter Wolfe.
2005 Winner: Sports Report - SLOTS
2004 Winner: Anderson/Hester
2003 Winner: Anderson/Hester
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST ACCURATE RETRODICTIVE RATING - Large System
Winner: Least Square Regression with team HFA
The system that had the smallest mean square error this season
was my new entry Least Square Regression with team HFA. As has been
typical in the NFL it wins this category easily with a mean square error
of 137.965. This virtually ties the old record set last season
The next closest was also least square regression but with a global home
field advantage, the winner of the small system category.
2005 Winner: Least Square Regression with team HFA
2004 Winner: Least Square Regression with team HFA
2003 Winner: The Sports Report - SLOTS
2002 Winner: CPA Rankings
2001 Winner: Edward Kambour FOotball Ratings
MOST ACCURATE RETRODICTIVE RATING - Small System
Winner: Least Squares Regression
It should not be too surprising to see least squares regression do well
here since that is what it does, minimize the square errors. Least squeres
ended with a mean square error of 166.988, which is 1 point better than last
year. Sagarin Predictive was the closest runner up.
2005 Winner: Least Squares Regression.
2004 Winner: Least Squares Regression.
2003 Winner: Least Squares Regression.
BEST RETRODICTIVE SYSTEM 2004 - Large System
Winner: The Sports Report - SLOTS
Like the award for best predictive system this award is found by giving each system a
score for each category. The system that has the highest total is the retroditive
system of the year.
Sports Report Slots takes an easy win this year due in large part to such an excellent win/loss record.
Second and third place go to Least Squares with team HFA, and CPA Retro.
2005 WInner: TSR SLOTS
2004 Winner: CPA Rankings
2003 WInner: TSR SLOTS
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
BEST RETRODICTIVE SYSTEM 2004 - Small system
Winner: Peter Wolfe
In this class, Wolfe's BCS system is the best retrodictive system of the year, and 6th place overall.
THe system Median and Martien Maas were second and third place finishers.
2005 Winner: Peter Wolfe
2004 Winner: Frank Alder
2003 Winner: Anderson & Hester