Welcome to the 4th annual NCAA PT Awards. The purpose of these
awards is to help raise the publicity for those systems that
have been superior in various qualities of interest.
Most awards will be based entirely on the numbers I've
monitored with my prediction tracker web pages. For the 2004
season I followed the weekly performance of 50 computer rating
systems. This is up from 49 last season.
MOST STRAIGHT UP WINNERS (Entire Season)
Winner: Jeff Sagarin
This year I collected predictions on 651 games involving two 1A teams.
(I am not counting games for Florida Atlantic or Florida International
because not all systems included theseteams).
For the first time we have a system win for consecutive seasons.
Jeff Sagarin's ratings won by a big 6 game margin. Sagarin finished
495-168, (76.0%). This is up slightly from last years 75.5%
The record for best prediction record was set by Ed Kambour in 2002 at
76.48%. This year a whopping 16 systems outperformed the Vegas Line. The
line had it's worst season to date. The line generally finished right around
75% and near the top of the list, this year it dropped down to 72.8%
2004 Winner: Jeff Sagarin's Overall Rating
2003 Winner: Edward Kambour & Jeff Sagarin's Overall Rating
2002 Winner: ARGH Power Ratings, Stewart Huckaby
2001 Winner: Massey Ratings. Kenneth Massey
2000 Winner: ARGH Power Ratings, Stewart Huckaby
1999 Winner: Vegas line
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)
Winner: The Vegas line
Deviation from the game score is the difference between the game prediction
and the actual result. A value of zero would mean the score difference is predicted
exactly. One property of a good system would be to minimize the system's average
game deviation.
Even thought the line was lousy at predicting winners this season it still wins
the award for smallest average deviation for the 6th year in a row. This year the
winning mark was 12.56. This is about a 0.30 point improvement over the past two
years. The record is 12.25 in 2001.
The best manned system this year was the Pigskin Index, 12.71. This continues to
be a weak category for computer rankings. I have developed a computer adjusted line
that was able to outperform the line, so look for it to possibly surplant the Vegas
line next season.
2004 Winner: Vegas Line
2003 Winner: Vegas Line
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
1999 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Entire Season)
Winner: CPA Rankings, Steve Wrathell
Bias is a little different from deviation. Deviation measures the
distance between a prediction and the actual result. Bias combines
distance and location of the prediction. Bias measures whether the
predictions are too high or too low. So if a sytem has an
average bias of +0.25 that means that on average the system
gives 0.25 points too many to the home team. This statistic can
be used to help guage home field advantage.
Wow, was this a bad year for this category. Just by looking over the
results I would say the average bias this year was 2.5 to 3 points in favor
of the road team. It is interesting that home field was often running as
high as 6 points per team for some systems this year. Only one system that
was followed all season had a value under 1, Actually it was two systems,
both CPA Rankings and CPA Retrorankings. They had average biases of -0.40 and
-0.44. Usually there is always a system very close to zero.
2004 Winner: CPA Rankings
2003 Winner: Edward Kambour
2002 Winner: Gupta Power Ratings
2001 Winner: Flyman
2000 Winner: ARGH Power Ratings
1999 Winner: Jeff Sagarin
BEST AGAINST THE SPREAD (Entire Season)
Winner: Edward Kambour
Beating the spread is not the number one goal for most computer
rating systems. But it is something that the average person likes
to look at and probably the most popular subject that I get emails
about. I know that looking at all the games is probably not the
most meaningful of measurements but it is what I have been reporting.
We have another two time winner, the top system against the spread
for the entire season was Edward Kambour. Kambour was 345-298, (53.66%).
It was a mediocre season agains the spread, just about everyone finished
in the 48-52% range.
The record for this category was set in 2002 at 56.04% by the Dunkel Index.
2004 Winner: Edward Kambour
2003 Winner: Edward Kambour
2002 Winner: Dunkel Index
2001 Winner: BMC Picks
2000 Winner: Dunkel Index
1999 Winner: Average across all systems
MOST ACCURATE PREDICTOR (Entire Season)
Winner: The Vegas line
This award is based on mean square error. Mean square error takes
into account both deviation and bias and is perhaps the most commonly
used measure of evaluating estimators.
The vegas line has the smallest mean square error at 254.532. This is a
few points of the record set in 2001 but better than the previous two seasons.
Honorable mention this year goes to Hank Trexler, having the best mean square
error among the comuter systesm at 260.76
2004 Winner: The Vegas Line
2002 Winner: The Vegas Line
2001 Winner: The Vegas Line
2000 Winner: The Vegas Line
1999 Winner: The Vegas Line
------------------------
It is debatable how meaningful rating systems are in the early
parts of the season. How can you rate the teams in the first
week of the season when no teams have played a game yet? The BCS
waits until mid October before releasing it's first ranking
of teams. Some systems choose to wait until around this time
before being made public. Looking at only the second half of the
season also gives an estimate of how well a system does based
only(or mostly) on data from the current year.
MOST STRAIGHT UP WINNERS (Second Half of Season)
Winner: Jeff Sagarin
Winner: Colley Rankings
The second half data started with week 9 and consisted of 314 games
between two division 1A teams. Something very interesting has happened
this year. Not that we have a two way tie but that one of the systems in
the tie is a BCS system. Colley's Rankings have tied with Jeff Sagarin.
Note that it is the overall Sagarin rating and not his BCS system.
Billingsly and Anderson/Hester only finished one game back.
Sagarin and Hester tied with a record of 234-80, 74.5%. This was
down 2% from the winners last season, which was a record high.
2004 Winner: Jeff Sagain, Wesley Colley
2003 Winner: Born Power Index & Edward Kambour
2002 Winner: Born Power Index
2001 Winner: Chris Montgomery
2000 Winner: Geoff Freeze
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season)
Winner: Ashby Accuratings
We have another very interesting result. The Vegas line, which had
won this category four straight years fell all the way to 7th place this year.
The new winner of this award is Ken Ashby's AccuRatings. Ashby's winning
average deviation was 12.37 points. This is the highest winning score since
the line in 2000. The numbers had been getting worse and worse each year until
this year.
I'll give honorable mention to the others that also beat the line for the first time:
StatFox and Edward Kambour, plus the average and median predictions, as well as the
computer adjusted line.
2004 Winner: Ken Ashby AccuRatings
2003 Winner: Vegas Line (opening)
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Second Half of Season)
Winner: Least Squares Regression
The lowest game bias in the second half of the season goes to Least Squares Regression.
This is a system that I opperate. L2's average bias over the second half of the season
was very small, .0087. The numbers here were a little better than for the season as a whole,
maybe this is an indication that people started to increase their home field advantages as
the season progressed.
2004 Winner: Least Squares Regression
2003 Winner: Darrly Marsee
2002 Winner: Massey BCS
2001 Winner: PerformanZ Ratings
2000 Winner: Darryl Marsee's Rankings
BEST AGAINST THE SPREAD (Second Half of Season)
Winner: Stat Fox
For the second half of the season the system that did the best against
the spread was StatFox, 177-122 (59.2%). For the whole season the numbers
were average but for the second half the systems seemed to do pretty well again
the spread. StatFox's 59% is excellent, a couple others, Ashby AccuRatings and
Edward Kambour, last year's winner, finished at 57%.
2004 Winner: StatFox
2003 Winner: Edward Kambour
2002 Winner: The Sports Report
2001 Winner: BMC Picks
2000 Winner: The Buck System
MOST ACCURATE PREDICTOR (Second Half of Season)
Winner: Winner: Ashby Accuratings
I guess Ken Ashby accurately named his ratings, his Accuratings get the award
for most accurate predictor for the second half of the season. It also
marks the first time in 5 seasons that the Vegas line did not win.
StatFox finishes second. These are now the first two systems to outperform
the Vegas Line. My computer adjusted line also finished ahead of the line.
The line came in 40th place picking winnings over the second half, but still
manages a 4th place finish here.
BEST PREDICTIVE SYSTEM IN 2003
Winner: Winner: Ashby Accuratings
To come up with the best overall predictive system I give each system
points for how well they do in all of the above categories. I then sum up
the points and the system with the highest total is dubbed predictive system
of the year. So this award goes to the system that is the most well rounded
One flaw can totally take a system out of the running.
This year the race for best overall predictive system was wide open. If you
just look at the variety of names on the individual awards you see that no one
system dominated this year. It came down to a three man race between Frank Alder,
Ashby AccuRatings and CPA Rankings. Alder wasn't included long enough to really
be considered. And so by a slim margin, 85 to 82, the best overall predictive system
award goes to a new comer this year, Ken Ashby AccuRatings.
2004 Winner: Ken Ashby AccuRatings
2003 Winner: Ed Kambour
2002 Winner: Ed Kambour
2001 Winner: Kenneth Massey (non-BCS)
2000 Winner: ARGH Power Ratings
RETRODICTION AWARDS
Retrodiction refers to 'retrodicting' the previous results, rather
than predicting future games. It is possible that a computer rating
system can put more emphasis on explaining past results than attempting
to predict future results. So these systems may not neccesarily be
the best predictive systems but can still be very good at their
main objective.
The retrodiction results on my page come from taking the final
ratings and using them to repredict the entire season.
Rather than list some honorable mentions I am going to break the
retrodictive categories into what I will call large and small systems.
A small system will be any system that estimates only a global home
field advantage. Systems that estimate individual home field advantages
are always going to end up winning these awards so I am opening up new
categories to the more traditional systems.
MOST RETRODICTIVE WINS - Large System
Winner: The Sports Report - SLOTS
The Sports Report SLOTS system had the best retrodictive record
for the Third year in a row with a record of 604-47 (92.78%).
This improves on the previous record of 90.24% set in 2002.
2004 Winner: The Sports Report - SLOTS
2003 Winner: The Sports Report - SLOTS
2002 Winner: The Sports Report - SLOTS
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST RETRODICTIVE WINS - Small System
Winner: Anderson/Hester
Among the small systems the best retrodictive winning percentage goes
to Logistic Regression. There were actually very few 'large' systems this
year so I considered not seperating the awards but I went ahead with it. The
most retrodictive wins category among the regular systems was quite
competive this year. Logistic Regression led the way with a record of
560-91, 86.02%. The Colley, Massey, and Sagarin BCS ratings all finished
just one game behind. Hmmm BCS ratings finished in places 3 through 6 while
my simple logistic regression model did better than all of them. Logistic
Regression sets a new record in this category.
2004 Winner: Logistic Regression
2003 Winner: Anderson/Hester
2002 Winner: Logistic Regression
2001 WInner: System Average
SMALLEST RETRODICTIVE MEAN ERROR - Large System
Winner: Least Square Regression with team HFA
The overall top finisher in mean error was least squares regression using
individual team home field advantages. This system has won this award every
year in the NFL so I added for college this year to see how good it would do.
It finished 0.60 points better than the next best, last year's winner TSR Slots.
And a full point better than the third best, simple least squares regression.
The winning mark this year was 9.206. And it also happens to be the best
score ever recorded to date.
2004 Winner: Least Square Regression with team HFA
2003 Winner: The Sports Report - SLOTS
2002 Winner: The average of all systems.
2001 Winner: Edward Kambour Football Ratings
2000 Winner: CPA Rankings
SMALLEST RETRODICTIVE MEAN ERROR - Small System
Winner: Least Squares Regression
The winner of the small class ratings is Least Squares Regression (L2)
with a mean error of 10.2409. L2 wins this award for the second year in
a row. THe next closest and 4th overall was Sagarin points.
2004 Winner: Least Squares Regression
2003 Winner: Least Squares Regression
SMALLEST RETRODICTIVE BIAS
Winner: Anderson/Hester
I am not breaking this one down into sub categories. The system that had the smallest
mean bias was Anderson & Hester. They had a bias of 0.01183. This deserves a little bit
of an asterisk because for this and other BCS systems I am the person actually setting
the home field advantage for these systems. So I am not sure if they should not be considered
or if I should get half the credit. But I don't really want to award systems that weren't
smart enough to raise their home fields this year. Home field was extremely high this year,
running nearly 6 points on average. Both CPA systems which are always strong in this category
finish right behind Anderson & Hester.
2003 Winner: Anderson/Hester
2003 Winner: Anderson/Hester
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST ACCURATE RETRODICTIVE RATING - Large System
Winner: Least Square Regression with team HFA
The system that had the smallest mean square error this season
was my new entry Least Square Regression with team HFA. As has been
typical in the NFL it wins this category easily with a mean square error
of 137.739. This easily breaks the old record of 169.03 set by Ed Kambour
back in 2001. The next closest was last year's winner TSR SLOTS.
2004 Winner: Least Square Regression with team HFA
2003 Winner: The Sports Report - SLOTS
2002 Winner: CPA Rankings
2001 Winner: Edward Kambour FOotball Ratings
MOST ACCURATE RETRODICTIVE RATING - Small System
Winner: Least Squares Regression
It should not be too surprising to see least squares regression do well
here since that is what it does, minimize the square errors. Least squeres
ended with a mean square error of 167.977, which is 20 points better than last
year when it also won. The next best was Sagarin Points.
2004 Winner: Least Squares Regression.
2003 Winner: Least Squares Regression.
BEST RETRODICTIVE SYSTEM 2004 - Large System
Winner: CPA Rankings, Steve Wrathell
Like the award for best predictive system this award is found by giving each system a
score for each category. The system that has the highest total is the retroditive
system of the year.
This is actually a hard category to even give an award this year. The system that
clearly ends up with the most points is TSR SLOTS, but since I only had their ratings
for roughly 25% of the season it is clearly not enough to win a major award. The
clear second place system would have been Least Squares with team HFA. But I also
did not start this system until late in the season so I don't think it is fair to
give it this award either. So I have to end up dropping all the way to 7th place
overall to come up with the large system that was available all year. This
is CPA Rankings.
2004 Winner: CPA Rankings
2003 WInner: TSR SLOTS
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
BEST RETRODICTIVE SYSTEM 2004 - Small system
Winner: Anderson/Hester
Unfortunately there was also not a system at the top in the small class that was available for
the entire season. The leaders were all systems that started up at the second half. Frank Alder
comes in with the 3rd most points. But since the top two are the two large systems mentioned above
that don't qualify I actually consider Frank Alder's system to be the Retrodictive system of the year.
He beats Anderson & Hester by only a fraction of a point.
2004 Winner: Frank Alder
2003 Winner: Anderson & Hester