Welcome to the 6th annual NCAA PT Awards. The purpose of these
awards is to help raise the publicity for those systems that
have been superior in various qualities of interest.
Most awards will be based entirely on the numbers I've
monitored with my prediction tracker web pages. For the 2006
season I followed the weekly performance of 56 computer rating
systems. For the first time the number of systems did not go up.
It stayed the same from the previous year.
MOST STRAIGHT UP WINNERS (Entire Season)
Winner: Thompson SPRS , Ted Thompson
Sagarin Points, Jeff Sagarin
Sagarin Overall, Jeff Sagarin
System Average, everyone
With the 12th week on the schedule this season the total number of games played
(between two IA teams) was up to 714 this season. This season the top of the
chart was as competitive and crowded as ever and we ended up with a four way tie
between Thompson SPRS, Sagarin Points, Sagarin Rating, and the average of all the
systems. They each finished with a record of 543-171 (76.05%). This was the highest
winning percentage we have seen since the record of 76.48% was set by Ken Massey in the
2001 season. The Vegas line had a below average 11th place finish this season.
I think something needs to be said about the consistency of Jeff Sagarin's Rating.
He has now won this category three out of the last four seasons and is really starting
to stand out as the top rating system for predicting straight up winners.
2006 Winner: Thompson SPRS, system average, Sagarin predictive, & Sagarin overall
2005 Winner: Pigskin Index
2004 Winner: Jeff Sagarin's Overall Rating
2003 Winner: Edward Kambour & Jeff Sagarin's Overall Rating
2002 Winner: ARGH Power Ratings
2001 Winner: Massey Ratings
2000 Winner: ARGH Power Ratings
1999 Winner: Vegas line
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)
Winner: Computer Adjusted Line
Deviation from the game score is the difference between the game prediction
and the actual result. A value of zero would mean the score difference is predicted
exactly. One property of a good system would be to minimize the system's average
game deviation.
The Vegas line has finally fallen. For the first time we have a new winner.
Although the winner, the computer adjusted line, is actually highly related to
the Vegas line. I came up with this two years ago to see if I could use the
other 50 rating to fine tune the line to be more accurate. It showed promise in
testing, finished a fraction of a point behind last year and now wins this season.
It also sets a new record low of 11.81 points. This beats the record of 11.03 that
was set last season. The line was only good enough for fourth place this season so you
have to wonder what was up with that. Off hand I'm not seeing an obvious reason other
than the computer ratings just had a good year. The other ratings systems to have a lower
deviation than the line were Thompson SPRS and Hank Trexler.
2006 Winner: Computer Adjusted Line
2005 Winner: Vegas Line
2004 Winner: Vegas Line
2003 Winner: Vegas Line
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
1999 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Entire Season)
Winner: Congrove Rankings, Dave Congrove
Bias is a little different from deviation. Deviation measures the
distance between a prediction and the actual result. Bias combines
distance and location of the prediction. Bias measures whether the
predictions are too high or too low. So if a system has an
average bias of +0.25 that means that on average the system
gives 0.25 points too many to the home team.
In general the bias scores were not extremely good this year. The
winner was Congrove Computer Rankings with a mean bias of -0.03. This
was the only score below 0.05.
2006 Winner: Congrove Computer Rankings
2005 Winner: Computer Adjusted Line
2004 Winner: CPA Rankings
2003 Winner: Edward Kambour
2002 Winner: Gupta Power Ratings
2001 Winner: Flyman
2000 Winner: ARGH Power Ratings
1999 Winner: Jeff Sagarin
BEST AGAINST THE SPREAD (Entire Season)
Winner: Harmon Forcast
Beating the spread is not the number one goal for most computer
rating systems. But it is something that the average person likes
to look at and probably the most popular subject that I get emails
about. I know that looking at all the games is probably not the
most meaningful of measurements but it is what I have been reporting.
With the line being a little off this year there were more systems
that were able to beat the spread more than 50% this year than we normally
see. At least half were better than 50% this season. It is generally much
lower than this, maybe a dozen. This best record against the spread this
season was the Harmon Forecast, 375-290, 56.4% This is a new record for
this category, breaking hte record of 56.04% set by the Dunkel Index back
in 2002. A couple of honorable mentions go to Ed Bemiss and the computer
adjusted line which both had higher percentages but for fewer games.
Bemiss because he started midseason and the computer adjusted line because
it is only different from the line for about one third of the games.
2006 Winner: Harmon Forecast
2005 Winner: Keeper's College Football Ratings
2004 Winner: Edward Kambour
2003 Winner: Edward Kambour
2002 Winner: Dunkel Index
2001 Winner: BMC Picks
2000 Winner: Dunkel Index
1999 Winner: Average across all systems
MOST ACCURATE PREDICTOR (Entire Season)
Winner: The Vegas line
This award is based on mean square error. Mean square error takes
into account both deviation and bias and is perhaps the most commonly
used measure of evaluating estimators.
The Vegas line has the smallest mean square error at 220.556. This is
also a new record for the second year in a row. Last year the line won
with a mse of 234.541. So even though the line was off this year in
absolute error it still takes the prize for most accurate.
The best mse by a manned system, depending on what you call a manned system,
either computer adjusted line, 220.623, Thompson SPRS 223.492 or Pigskin
Index 226.284
2006 Winner: The Vegas Line
2005 Winner: The Vegas Line
2004 Winner: The Vegas Line
2003 Winner: The Vegas Line
2002 Winner: The Vegas Line
2001 Winner: The Vegas Line
2000 Winner: The Vegas Line
1999 Winner: The Vegas Line
------------------------
It is debatable how meaningful rating systems are in the early
parts of the season. How can you rate the teams in the first
week of the season when no teams have played a game yet? The BCS
waits until mid October before releasing it's first ranking
of teams. Some systems choose to wait until around this time
before being made public. Looking at only the second half of the
season also gives an estimate of how well a system does based
only(or mostly) on data from the current year.
MOST STRAIGHT UP WINNERS (Second Half of Season)
Winner: NationalSportsRankings.co, Ed Bemiss
The second half data started with week 8 and consisted of 364 games
between two division 1A teams. This year there was an extra, or 12th week.
The system with the best record over the second half of the season was
Ed Bemiss of NationalSportsrankings.com, 275-89, 75.55%. This was a good
record for the second half but a little short of the record 76.7%
set in 2003 by Born and Kambour. Sagarin's predictive and overall rating
finished tied for second two games back
2006 Winner: Ed Bemiss
2005 Winner: Dunkel Index
2004 Winner: Jeff Sagain, Wesley Colley
2003 Winner: Born Power Index & Edward Kambour
2002 Winner: Born Power Index
2001 Winner: Chris Montgomery
2000 Winner: Geoff Freeze
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season)
Winner: Computer Adjusted Line
As also seen for the full season the computer adjusted line managed to have the smallest
average game deviation during the second half of the season. It was off by an average of
11.7344 points. This shatters the previous record of 12.11 set way back in 2000. There were
21 scores this year that bettered the previous record. It was a good year for predicting
spreads. The computer adjusted line only narrowly finished ahead of Thompson SPRS, 11.7555
and the line, 11.7582
2006 Winner: Computer Adjusted Line
2005 Winner: Vegas Line
2004 Winner: Ken Ashby AccuRatings
2003 Winner: Vegas Line (opening)
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Second Half of Season)
Winner: Vegas Line
The lowest game bias in the second half of the season goes to the Vegas line.
The line's average bias over the second half of the season was 0.00549. The numbers were
substantially different for the second half of the season as they were for the full season.
For the full season the tended to be positive while they tended to be negative for the
second half, and also relative large in magnitude.
2006 Winner: Vegas Line
2005 Winner: Brent Moore
2004 Winner: Least Squares Regression
2003 Winner: Darrly Marsee
2002 Winner: Massey BCS
2001 Winner: PerformanZ Ratings
2000 Winner: Darryl Marsee's Rankings
BEST AGAINST THE SPREAD (Second Half of Season)
Winner: NationalSportsRankings.co, Ed Bemiss
For the second half of the season the system that did the best against
the spread was Ed Bemiss of nationalsportsrankins.com, 206-151 (57.70). You
know you are having a good year when you have both the best straight up record
and best record against the spread. The computer adjusted line deserved mention
here as well with a 61.61% rate picking about one third of the games.
The record for the second half was 59% set by Statfox in 2004.
2006 Winner: Ed Bemiss
2005 Winner: The Sports Report
2004 Winner: StatFox
2003 Winner: Edward Kambour
2002 Winner: The Sports Report
2001 Winner: BMC Picks
2000 Winner: The Buck System
MOST ACCURATE PREDICTOR (Second Half of Season)
Winner: Vegas Line
The Vegas line gets the award for most accurate predictor over
the second half of the season. The computer adjusted line came close
and Harmon Forecast also had a good year. This year's winning mark
of 215.552 was easily the highest seen to date.
2006 Winner: Vegas Line
2005 Winner: Vegas Line
2004 Winner: Ashby Accuratings
2003 Winner: Vegas Line
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
BEST PREDICTIVE SYSTEM IN 2006
Winner: Computer Adjusted Line
To come up with the best overall predictive system I give each system
points for how well they do in all of the above categories. I then sum up
the points and the system with the highest total is dubbed predictive system
of the year. So this award goes to the system that is the most well rounded
One flaw can totally take a system out of the running.
This year I am giving the best predictive system of the year award to the
computer adjusted line. It excelled in all categories both in the full season
and in the second half. Ed Bemiss was also a contender but not joining until
midseason pushed him down. Other strong all around systems for 2006 were
the Harmon Forecast and Thompson SPRS.
2006 Winner: Computer Adjusted Line
2005 Winner: Dunkel Index
2004 Winner: Ken Ashby AccuRatings
2003 Winner: Ed Kambour
2002 Winner: Ed Kambour
2001 Winner: Kenneth Massey (non-BCS)
2000 Winner: ARGH Power Ratings
RETRODICTION AWARDS
Retrodiction refers to 'retrodicting' the previous results, rather
than predicting future games. It is possible that a computer rating
system can put more emphasis on explaining past results than attempting
to predict future results. So these systems may not necessarily be
the best predictive systems but can still be very good at their
main objective.
The retrodiction results on my page come from taking the final
ratings and using them to re-predict the entire season.
Rather than list some honorable mentions I am going to break the
retrodictive categories into what I will call large and small systems.
A small system will be any system that estimates only a global home
field advantage. Systems that estimate individual home field advantages
are always going to end up winning these awards so I am opening up new
categories to the more traditional systems.
MOST RETRODICTIVE WINS - Large System
Winner: The Sports Report - SLOTS
The Sports Report SLOTS system had the best retrodictive record
for the fifth consecutive season with a record of 652-62 (91.32%).
This is down from the last couple of years. The record is 92.78
set in 2004.
2006 Winner: The Sports Report - SLOTS
2005 Winner: The Sports Report - SLOTS
2004 Winner: The Sports Report - SLOTS
2003 Winner: The Sports Report - SLOTS
2002 Winner: The Sports Report - SLOTS
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST RETRODICTIVE WINS - Small System
Winner: Superlist Waits and others
Among the small systems the best retrodictive winning percentage was
The Superlist, which is a composite of 9 systems. It had a record of
609-105, 85.29%. The record was 86.02% set by logistic regression in 2004.
2006 Winner: The SuperList
2005 Winner: Jeff Sagarin, Warren Claassen
2004 Winner: Logistic Regression
2003 Winner: Anderson/Hester
2002 Winner: Logistic Regression
2001 Winner: System Average
SMALLEST RETRODICTIVE MEAN ERROR - Large System
Winner: Least Square Regression with team HFA
Least squares regression with individual team HFA finished with the smallest
average deviation for the third year in a row. This year's mark of 9.1615
breaks the record of 9.206 set in 2004.
2005 Winner: Least Square Regression with team HFA
2005 Winner: Least Square Regression with team HFA
2004 Winner: Least Square Regression with team HFA
2003 Winner: The Sports Report - SLOTS
2002 Winner: The average of all systems.
2001 Winner: Edward Kambour Football Ratings
2000 Winner: CPA Rankings
SMALLEST RETRODICTIVE MEAN ERROR - Small System
Winner: Jeff Sagarin
The winner of the small class ratings is Jeff Sagarin's points or predictive ratings,
with a mean error of 10.0155. This is just a fraction behind the record of 10.0129 set
last year.
2006 Winner: Jeff Sagarin Predictive
2005 Winner: Jeff Sagarin Predictive
2004 Winner: Least Squares Regression
2003 Winner: Least Squares Regression
SMALLEST RETRODICTIVE BIAS
Winner: Warren Claassen
I am not breaking this one down into sub categories. The system that had the smallest
mean bias was Warren Claassen. He had a bias of 0.0151. A couple other systems,
TSR SLOTS and the system average were extremely close at 0.0158
2005 Winner: Sports Report - SLOTS
2004 Winner: Anderson/Hester
2003 Winner: Anderson/Hester
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST ACCURATE RETRODICTIVE RATING - Large System
Winner: Least Square Regression with team HFA
The system that had the smallest mean square error this season
was Least Square Regression with team HFA. As has been
typical the past few years, it wins this category easily with a mean square error
of 130.437. This breaks the previous record by 7 points.
2005 Winner: Least Square Regression with team HFA
2005 Winner: Least Square Regression with team HFA
2004 Winner: Least Square Regression with team HFA
2003 Winner: The Sports Report - SLOTS
2002 Winner: CPA Rankings
2001 Winner: Edward Kambour Football Ratings
MOST ACCURATE RETRODICTIVE RATING - Small System
Winner: Least Squares Regression
It should not be too surprising to see least squares regression do well
here since that is what it does, minimize the square errors. Least squares
ended with a mean square error of 157.303. Also a new record.
2006 Winner: Least Squares Regression.
2005 Winner: Least Squares Regression.
2004 Winner: Least Squares Regression.
2003 Winner: Least Squares Regression.
BEST RETRODICTIVE SYSTEM 2004 - Large System
Winner: The Sports Report - SLOTS
Like the award for best predictive system this award is found by giving each system a
score for each category. The system that has the highest total is the retrodictive
system of the year.
2005 Winner: TSR SLOTS
2004 Winner: CPA Rankings
2003 Winner: TSR SLOTS
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
BEST RETRODICTIVE SYSTEM 2004 - Small system
Winner: Warren Claassen
In this class, Warren Claassen is the best retrodictive system of the year, and 3rd place overall.
2006 Winner: Warren Claassen
2005 Winner: Peter Wolfe
2004 Winner: Frank Alder
2003 Winner: Anderson & Hester