Welcome to the 4th annual NCAA PT Awards. The purpose of these
awards is to help raise the publicity for those systems that
have been superior in various qualities of interest.
Most awards will be based entirely on the numbers I've
monitored with my prediction tracker web pages. For the 2003
season I followed the weekly performance of 49 computer rating
systems. This is up from 42 last season.
I start by using the predictions from the entire season. As you
will see, the systems performed very poorly this season.
MOST STRAIGHT UP WINNERS (Entire Season)
Winner: Edward Kambour
Winner: Jeff Sagarin
The simplest and possibly most important way of measuring the
predictive ability of a system is to count the number of games where
the winner is correctly predicted. I collected predictions on 698
games in 2003 involving two 1A teams.
This year two systems tied for the best overall season record, Edward
Kambour and Jeff Sagarin's overall rating. Both of these systems finished
with a record of 527-171, (75.5%). This winning percentage is slightly
down from ARGH's winning mark of 75.78% last season.
The record for best prediction record was set by Kambour last year at
76.48%. This year only three systems outperformed the Vegas line: Kambour,
Sagarin and the Bassett Model.
2003 Winner: Edward Kambour & Jeff Sagarin's Overall Rating
2002 Winner: ARGH Power Ratings, Stewart Huckaby
2001 Winner: Massey Ratings. Kenneth Massey
2000 Winner: ARGH Power Ratings, Stewart Huckaby
1999 Winner: Vegas line
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)
Winner: The Vegas line
Deviation from actual game results is another way of measuring
the predictive ability of a computer system. Deviation from the game
score is the difference between the game prediction and the actual
result. A value of zero would mean the score difference is predicted
exactly. So one property of a good system would be to minimize
the system's average game deviation.
For the fifth year in a row the lowest average game deviation was
found in the Vegas line. This year the Vegas line's average deviation
from the actual score was 12.88 points. That is very close to 12.86 which
the odds makers had last year. The record is 12.25 in 2001.
Honorable mention here goes to the runner up, Edward Kambour.
Kambour had an average game deviation of 13.17. I have been asked why I don't
give this award to the highest individual rather than to the line every year.
I believe that if we can't beat the line then we don't deserve the credit.
We need to find ways of improving this aspect of predicting game outcomes.
This year we really didn't even come close to matching the Vegas line.
2003 Winner: Vegas Line
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
1999 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Entire Season)
Winner: Edward Kambour
Bias is a little different from deviation. Deviation measures the
distance between a prediction and the actual result. Bias combines
distance and location of the prediction. Bias measures whether the
predictions are too high or too low. So if a sytem has an
average bias of +0.25 that means that on average the system
gives 0.25 points too many to the home team. This statistic can
be used to help guage home field advantage.
Each year there is usually one system that had a great season
that dominates a lot of the awards. This year the person to do
that is Ed kambour. He had the best record, 2nd best average deviation
and the smallest average bias. Kambour's average bias this season was
-0.075 points. So on average his predictions were slightly lower than
the actual result. We also did pretty bad in this category as a whole.
StatFox comes in a close second at +0.083, but other than that only four
other systems were within half a point of the truth. Last year Gupta's
ratings had an average bias of 0.00691. So we really fell off this year.
2003 Winner: Edward Kambour
2002 Winner: Gupta Power Ratings
2001 Winner: Flyman
2000 Winner: ARGH Power Ratings
1999 Winner: Jeff Sagarin
BEST AGAINST THE SPREAD (Entire Season)
Winner: Edward Kambour
Beating the spread is not the number one goal for most computer
rating systems. But it is something that the average person likes
to look at and probably the most popular subject that I get emails
about. I know that looking at all the games is probably not the
most meaningful of measurements but it is what I have been reporting.
The top system against the spread for the entire season was
Edward Kambour. Kambour was 361-323, (52.78%). As a group we
also did a poor job against the spread this season. Only 22 out
of 49 systems got more games right than wrong. You would have
done better than Kambour's record this year by picking opposite
of several of the systems. I believe the record for this category
was set last year at 56.04% by the Dunkel Index.
2003 Winner: Edward Kambour
2002 Winner: Dunkel Index
2001 Winner: BMC Picks
2000 Winner: Dunkel Index
1999 Winner: Average across all systems
MOST ACCURATE PREDICTOR (Entire Season)
Winner: The Vegas line
This award is based on mean square error. Mean square error takes
into account both deviation and bias and is perhaps most commonly
used measure of evaluating estimators.
The vegas line has the smallest mean square error at 266.958, down a few
points from last years 261.966. Honorable mention again goes to Edward
Kambour for having the smallest mean square error among the systems.
Kabour's mark was about 275. Several systems did better than this last
season.
2003 Winner: The Vegas Line
2002 Winner: The Vegas Line
2001 Winner: The Vegas Line
2001 Winner: The Vegas Line
1999 Winner: The Vegas Line
------------------------
It is debatable how meaningful rating systems are in the early
parts of the season. How can you rate the teams in the first
week of the season when no teams have played a game yet? The BCS
waits until mid October before releasing it's first ranking
of teams. Some systems choose to wait until around this time
before being made public. Looking at only the second half of the
season also gives an estimate of how well a system does based
only(or mostly) on data from the current year.
MOST STRAIGHT UP WINNERS (Second Half of Season)
Winner: Born Power Index ,William Born
Winner: Edward Kambour
The second half data started with wee 9 and consisted of 369 games
between two division 1A teams. Two systems tied for the most straight
up wins in the second half. The full season winner, Ed Kambour, and
the person that won this category last season, William Born. Both
Kambour and Born had records of 283-86 (76.69%). These two guys also
set the best ever winning percentage in this category. The previous
best was 76.33% set by Kambour last season. So both Kambour and Born
have a great record of being strong in the second half of the season.
This season was marked by the controversial BCS matchup. It is
interesting to note that of the 7 BCS systems the best finish was
#24 by Billingsly. None of the other even finished in top 35. The
BCS needs to add some predictie systems.
2003 Winner: Born Power Index & Edward Kambour
2002 Winner: Born Power Index
2001 Winner: Chris Montgomery
2000 Winner: Geoff Freeze
SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season)
Winner: The Vegas line (opening line)
As with the regular season, the Vegas line also wins this award
for having the smallest deviation from the actual game scores
over the second half of the season. But interestingly enough the
opening line narrowly beats the updated line. The opening line
had an average game deviation of 13.13 following by the updated
line at 13.18.
I will give an honorable mention for Ed Kambour for having the
smallest deviation among the systems. Kambour wasn't that far behind
at 13.28. The winning total has gotten worse for 4 consecutive seasons.
2003 Winner: Vegas Line (opening)
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
SMALLEST AVERAGE GAME BIAS (Second Half of Season)
Winner: Darryl Marsee
The lowest game bias in the second half of the season goes to Darryl Marsee.
On average, Marsee's rating predictions were 0.35 under the actual result.
Overall, the numbers here did not look very good.
The majority of the systems were way too low, meaning they did not give
enough points to the home team.
2003 Winner: Darrly Marsee
2002 Winner: Massey BCS
2001 Winner: PerformanZ Ratings
2000 Winner: Darryl Marsee's Rankings
BEST AGAINST THE SPREAD (Second Half of Season)
Winner: The Sports Report
For the second half of the season the system that did the best against
the spread was again Ed Kambour, 194-166 (53.89%). This was slightly ahead
of my PerformanZ Ratings. Those are the only two systems worth mentioning
as only 12 out of 48 systems broke the .500 mark.
As is often the case, the best way to beat the spread is often to pick
opposite of what a system says. This year that system would be picking
opposite of Chris Montgomery's system, which was only right 43% of the time.
2003 Winner: Edward Kambour
2002 Winner: The Sports Report
2001 Winner: BMC Picks
2000 Winner: The Buck System
MOST ACCURATE PREDICTOR (Second Half of Season)
Winner: The Vegas line (opening)
The opening vegas line also takes this award for having the smallest mean
square error over he second half of the season. For the second
half results honorable mention goes to Ed Kambour who had the best
results among the computer systems. It is the third year in a row
that Kambour has been the closest to the line. Kambour is slowly getting
closer and closer to matching the line. He has close the game from 13 points
in 2001 to 4 points in 2002 to half a point in 2003.
2003 Winner: The Vegas Line (opening)
2002 Winner: The Vegas Line
2001 Winner: The Vegas Line
2000 Winner: The Vegas Line
BEST PREDICTIVE SYSTEM IN 2003
Winner: Edward Kambour
To come up with the best overall predictive system I give each system
points for how well they do in all of the above categories. I then sum up
the points as the system with the highest total is dubbed predictive system
of the year. So this award goes to the system that is the most well rounded
One flaw can totally take a system out of the running.
This season I give this award to Ed Kambour. Kambour was an easy winner this
year. He becomes the first person to win this award twice, winning two years
in a row. Other contenders this year were the Vegas line and Gene Bassett.
2003 Winner: Ed Kambour
2002 Winner: Ed Kambour
2001 Winner: Kenneth Massey (non-BCS)
2000 Winner: ARGH Power Ratings
-------------------
RETRODICTION AWARDS
Retrodiction refers to 'retrodicting' the previous results, rather
than predicting future games. It is possible that a computer rating
system can put more emphasis on explaining past results than attempting
to predict future results. So these systems may not neccesarily be
the best predictive systems but can still be very good at their
main objective.
The retrodiction results on my page come from taking the final
ratings and using them to repredict the entire season.
Rather than list some honorable mentions I am going to break the
retrodictive categories into what I will call large and small systems.
A small system will be any system that estimates only a global home
field advantage. Systems that estimate individual home field advantages
are always going to end up winning these awards so I am opening up new
categories to the more traditional systems.
MOST RETRODICTIVE WINS - Large System
Winner: The Sports Report - SLOTS
The Sports Report SLOTS system had the best retrodictive record
for the second year in a row with a record of 584-80 (87.95%).
This is down from last year's record of 90.24%.
2003 Winner: The Sports Report - SLOTS
2002 Winner: The Sports Report - SLOTS
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST RETRODICTIVE WINS - Small System
Winner: Anderson/Hester
Among the small systems the best retrodictive winning percentage goes
to Anderson & Hester. This system had a competitive record even against
the large systems. Anderson & Hester's record was 561-103 (84.49%).
This is a good time to make the the note that Anderson & Hester are one of
the BCS computers that do not make predictions on their own. Because
I want to include the BCS systems in my comparison I translate each of
these systems onto a new scale that is suitable for making predictions.
I don't really understand why everyone doesn't do that in the first place.
And since this is a retrodictive category I am giving awards retroactively
to the ones that would have won this in the past.
2003 Winner: Anderson/Hester
2002 Winner: Logistic Regression
2001 WInner: System Average
SMALLEST RETRODICTIVE MEAN ERROR - Large System
Winner: The Sports Report - SLOTS
The top three finishers in mean error were all large systems. The one
with the smallest mean error was The Sports Report SLOTS. SLOTS had
a mean error of 10.5391. This is just slightly better than the winner last
year. This category continues to get worse with time. With the exception
of SLOTS the numbers were very high this year. The record for this category
was 10.152 by Ed Kambour in 2001.
2003 Winner: The Sports Report - SLOTS
2002 Winner: The average of all systems.
2001 Winner: Edward Kambour Football Ratings
2000 Winner: CPA Rankings
SMALLEST RETRODICTIVE MEAN ERROR - Small System
Winner: Least Squares Regression
The winner of the small class ratings is Least Squares Regression
with a mean error of 10.9216. That is almost half a point worse than
the large system winner.
2003 Winner: Least Squares Regression
SMALLEST RETRODICTIVE BIAS
Winner: Anderson/Hester
I am not breaking this one down into sub categories. The system that had the smallest
mean bias was Anderson & Hester. They also set the record for this category with
a average bias of 0.00295 points towards the road team. This year actually reverses
a downward trend in this category. Three time previous winner CPA Rankings came in a
very close 2nd at -0.00314. A couple other systems also came in under the previous
record of 0.005.
2003 Winner: Anderson/Hester
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
MOST ACCURATE RETRODICTIVE RATING - Large System
Winner: The Sports Report - SLOTS
The system that had the smallest mean square error this season
was The Sports Report SLOTS. SLOTS' mean square error was 184.131.
I am a little surprised that this number is so high. Higher thant CPA Rankings
had last year quite a bit off the record set by Ed Kambour of 169.03 back in 2001.
2003 Winner: The Sports Report - SLOTS
2002 Winner: CPA Rankings
2001 Winner: Edward Kambour FOotball Ratings
MOST ACCURATE RETRODICTIVE RATING - Small System
Winner: Least Squares Regression
It should not be too surprising to see least squares regression do well
here since that is what it does, minimize the square errors. Least squeres
ended with a mean square error of 186.493, which is just slightly behind
the large class winner that uses twice as many parameters.
2003 Winner: Least Squares Regression.
BEST RETRODICTIVE SYSTEM 2003 - Large system
Winner: The Sports Report - SLOTS
Like the award for best predictive system this award is found by giving each system a
score for each category. The system that has the highest total is the retroditive
system of the year.
For the first time we have a new winner this year. TSR SLOTS won most of the awards
so this should not come as a surprise. It wasn't that close but the only one within
sight of SLOTS was the previous three time winner CPA Rankings.
2003 WInner: TSR SLOTS
2002 Winner: CPA Rankings
2001 Winner: CPA Rankings
2000 Winner: CPA Rankings
BEST RETRODICTIVE SYSTEM 2003 - Small system
Winner: Anderson/Hester
I have never been a fan of Anderson & Hester's ratings before. But this year their retro
number held up. Looking at previous years, it appears held up is the appropriate term.
Anderson & Hester's overall numbers stayed about the same from last year. But for
what ever reason just about everyone else declined year to year.
Anderson & Hester barely beat out Matthews' Grid. And both of these are were a
long way from challenging the large systems.
2003 WInner: Anderson & Hester