Welcome to the first NCAA PT Awards.  The purpose of these
awards is to help raise the publicity for those systems that
have been superior in various qualities of interest.
Most awards will be based entirely on the numbers I've
monitored with my prediction tracker web pages.  I have
followed the weekly performance of 25+ computer ratings systems
over the course of the past two seasons.  Since I didn't give
any awards last year I will mention last year's winners when
applicable.



  BEST OVERALL NCAA 1A SYSTEM (2000)

Winner:  ARGH Power Ratings, Stewart Huckaby
Winner:  CPA Rankings, Steve Wrathell

 In the 2000 NCAA season there were two systems that clearly
outperformed the rest of the pack.  And because they each excelled
in different areas I really don't have much choice but to
name Stewart Huckaby's ARGH Power Ratings and Steve Wrathell's
CPA Rankings as co-systems of the year.  ARGH was the predictive
system of the year and CPA the retrodictive system of the year.
But each of them also performed very well in the category
they didn't win.  These two systems are at or near the top of the
charts in all categories. Read the individual category awards
for more detail on the accomplishments of these two systems.


  BEST STRAIGHT UP WINNERS (Entire Season)

Winner:  ARGH Power Ratings, Stewart Huckaby

 The simplest and possibly most important way of measuring the
predictive ability of a system is to count the number of games where
the winner is correctly predicted.  My prediction Tracker gathered
predictions on the 641 games in 2000 involving two 1A teams.
The system that correctly predicted the most winners over the
course of the entire season was Stewart Huckaby's ARGH Power
Ratings. ARGH correctly predicted the winner in 75.5% of this
season's games. This was a full 7 games better than any other
system.
 Last season the Vegas line was the best at picking winners at 73.1%.
And since the line was second best this year that makes this year's
ARGH Power Ratings as the only system (that I have tracked) to ever
have a better season picking winners than the Vegas Line.  That
is a nice accomplishment.


  SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)

Winner:  The Vegas line

 Deviation from actual game results is another way of measuring
the predictive ability of a computer system. Deviation from the game
score is the difference between the game prediction and the actual
result.  A value of zero would mean the score difference is predicted
exactly.  So one property of a good system would be to minimize
the systems average game deviation.
 This season the lowest average game deviation was found in the
Vegas line.  So this award goes out to Roxy and the oddsmakers.
The line was almost half a point better than the next best system.
Honorable mention goes out to Steve Wrathell's CPA Rankings who
had the smallest average deviation among the computer ratings.
 The line also had the smallest deviation last season followed
by the average system prediction, which was a close 3rd this
season. So make a note that this is an area where we as raters need to
find ways of improving.  No one was able to beat the line and
the very best system still was off by an average of 12.7 points
per game.


  SMALLEST AVERAGE GAME BIAS (Entire Season)

Winner:  ARGH Power Ratings, Stewart Huckaby

 Bias is a little different from deviation.  Deviation measures the
distance between a prediction and the actual result.  Bias combines
distance and location of the prediction. (Deviation is the
absolute value of the bias) Bias is measuring whether the
predictions are too high or too low.  So if a sytem had an
average bias of zero that would mean that on average the system
predicts the margin exactly.  This can often be used as a statistic
to help measure home field advantage.
 For the entire season the lowest bias was observed in Stewart
Hackaby's ARGH Power Ratings.  He had an average bias of about +.10,
meaning that on average ARGH's predictions were .10 points higher
than the game result.  The majority of the systems were typically
around -1, meaning they did not give enough points to the home team.
I will give an honorable mention to the Harmon Forcast which came in
a very close 2nd at +.12 and Jeff Sagarin in 3rd place with a +.19.
I also want to make a note that Jeff Sagarin has the smallest bias
last year in the 1999 season with a value of -.62.
 I would also like to share a statistical use of these individual
game biases.  In an 1991 paper in the American Statistician, Hal
Stern showed that the individual game biases of the vegas line
of NFL games between 1981 and 1984 were approxamately a normal
distribution with a mean of zero.  So you could estimate the probability
of an X point favorite would win the game simply by looking up
X /(standard deviation) in a normal distribution table.  This gives
results such as an even game being 50/50, a 3 point favorite has about
a 59% probability of winning and a 7 point favorite a 70% probability
of winning.  The results page of the prediction tracker presents
each systems average bias and the standard deviations, so rough
probabilities could be calculated for any of these systems.
I say rough because you would be assuming normality.


  BEST AGAINST THE SPREAD (Entire Season)

Winner:  Dunkel Index 

 Beating the spread is not the number one goal for most computer
rating systems.  But it is something that the average person considers.
Will a system help you win your office pool.
The top system against the spread for the entire season was the
Dunkel Index.  Dunkel was only 52.3% against the spread and only 7
out of 25 were better than 50%.  If you wanted to get technical
the best was to pick against the spread would be to pick opposite
of what the logistic elo picked.  That would have given you 57%.
But in general all the systems were very close to the 50% mark.





  BEST STRAIGHT UP WINNERS (Second Half of Season)

Winner:   Freeze Ratings, Geoff Freeze

 It is debatable how meaningful rating systems are in the early
parts of the season.  How can you make predictions on the first
week of the season when no one has played a game yet?  The BCS
actually waits until mid October before releasing it's first ranking
of teams.  Other systems chose to wait until around this time
before being made public. Looking at only the second half of the
season also gives an estimate of how well a system does based
only(or mostly) on data from the current year.
 So to measure this I looked at all 1A games starting with with
week nine of the season.  Out of these 253 games in the second
half of the season the system predicting the most winners was
Geoff Freeze's Freeze Ratings at 74.3%.  During the second half
the systems were very competitive.  The top 9 systems all finished
within 4 games of each other. Freeze finished 1 game better
than 4 other systems.
 The Vegas line falls all the way to 16th, down from 2nd, during
the second half of the season.  This gives some evidence that as
the season progresses the computer systems are becoming much more
accurate as the amount of data is accumulating.


  SMALLEST DEVIATION FROM ACTUAL GAME SCORES  (Second Half of Season)

Winner:  The Vegas line

 As with the regular season the Vegas line also wins this award
for having the smallest deviation from the actual game scores
over the second half of the season.
 An honorable mention here also goes to Steven Wrathell's CPA Rankings
for being the best among the computer systems.  Average deviations
were lower for the second half of the season.  The line went down
to 12.11 from 12.37 and CPA went from 12.70 down to 12.21.  So again
the computer systems are getting better as the season progresses
but the closest to the final result is still the Vegas line.


  SMALLEST AVERAGE GAME BIAS  (Second Half of Season)

Winner:    Marsee Power Ratings, Darryl Marsee

The lowest game bias in the second half of the season goes to Darryl
Marsee's power ratings.  On average, his predictions were only .01 below
the actual result.  This is an extremely accurate result. Marsee also
had one of the better average deviations in the second half of the
season.  In a rather interesting result the biases of most system
really didn't improve as the season progressed.  They stayed about
the same.  This may suggest that some people might have done
better had they varied their home field advantage factors as the
season went along rather than using a constant number such a 3 points.


  BEST AGAINST THE SPREAD  (Second Half of Season)

Winner:  The Buck System

 For the second half of the season the system that did the best against
the spread was the Buck System at 53%.  As with the entire season
a better results could actually be seen by taking the opposite of what
the systems say.  Here only 8 of 25 were better than 50%.



  BEST PREDICTIVE SYSTEM IN 2000

Winner:  ARGH Power Ratings, Stewart Huckaby

 Stewart Huckaby's ARGH Power Ratings is the fairly obvious winner of
the predictive system of the year.  He was the leader in two categories
for the entire season and was right up there for the 2nd half categories
as well.




           RETRODICTION AWARDS

 Retrodiction (I'm not sure who originally coined the term) refers
to 'retrodicting' the previous results, rather than predicting
future games.  It is possible that a computer rating system can
put more emphasis on explaining past results than attempting to
predict future results.  So these systems may not neccesarily be
the best predictive systems but can still be very good at their
main objective.  And some people will argue that explaining the
season to date is the only thing that matters in ranking teams.
So the retrodiction results on my page come from taking the final
ratings and using them to retrodict the entire season.


   BEST RETRODICTIVE SYSTEM

Winner:  CPA Rankings, Steve Wrathell

 I was going to give awards for each category for retrodiction
but I would just have been repeating myself each time because
Steve Wrathell's CPA Rankings were the best in every retrodictive
category I measured.  He has the best retrodictive percentage at
83.9%.  Compare this to the best predictive percentage of 75.5%
and you see that while hindsite isn't perfect it is easier
than predicting the future.  In fact the worst retrodiction
record is better than the best prediction record.
 The CPA Rankings also had the smallest absolute error and the
smallest retrodictive bias.  The absolute error was the only system
under 10 points and the bias was almost zero.
There really isn't any other system that did well at everything.
Scripps Howard had a good record but high error and bias.
Dwiggens, which uses a least squares based method which minimizes
the square error, did predicitably well in absolute error category,
second to CPA, but had a poor record and an average bias.
Another interesting note is that the Dunkel Index is a highly
predictive rating system.  From what I have seen the BCS is
fairly balanced.  Dunkel is predictive, Scripps Howard is more
retrodictive and Massey and Sagarin are balanced between the
two.