Welcome to the first NCAA PT Awards. The purpose of these awards is to help raise the publicity for those systems that have been superior in various qualities of interest. Most awards will be based entirely on the numbers I've monitored with my prediction tracker web pages. I have followed the weekly performance of 25+ computer ratings systems over the course of the past two seasons. Since I didn't give any awards last year I will mention last year's winners when applicable. BEST OVERALL NCAA 1A SYSTEM (2000) Winner: ARGH Power Ratings, Stewart Huckaby Winner: CPA Rankings, Steve Wrathell In the 2000 NCAA season there were two systems that clearly outperformed the rest of the pack. And because they each excelled in different areas I really don't have much choice but to name Stewart Huckaby's ARGH Power Ratings and Steve Wrathell's CPA Rankings as co-systems of the year. ARGH was the predictive system of the year and CPA the retrodictive system of the year. But each of them also performed very well in the category they didn't win. These two systems are at or near the top of the charts in all categories. Read the individual category awards for more detail on the accomplishments of these two systems. BEST STRAIGHT UP WINNERS (Entire Season) Winner: ARGH Power Ratings, Stewart Huckaby The simplest and possibly most important way of measuring the predictive ability of a system is to count the number of games where the winner is correctly predicted. My prediction Tracker gathered predictions on the 641 games in 2000 involving two 1A teams. The system that correctly predicted the most winners over the course of the entire season was Stewart Huckaby's ARGH Power Ratings. ARGH correctly predicted the winner in 75.5% of this season's games. This was a full 7 games better than any other system. Last season the Vegas line was the best at picking winners at 73.1%. And since the line was second best this year that makes this year's ARGH Power Ratings as the only system (that I have tracked) to ever have a better season picking winners than the Vegas Line. That is a nice accomplishment. SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season) Winner: The Vegas line Deviation from actual game results is another way of measuring the predictive ability of a computer system. Deviation from the game score is the difference between the game prediction and the actual result. A value of zero would mean the score difference is predicted exactly. So one property of a good system would be to minimize the systems average game deviation. This season the lowest average game deviation was found in the Vegas line. So this award goes out to Roxy and the oddsmakers. The line was almost half a point better than the next best system. Honorable mention goes out to Steve Wrathell's CPA Rankings who had the smallest average deviation among the computer ratings. The line also had the smallest deviation last season followed by the average system prediction, which was a close 3rd this season. So make a note that this is an area where we as raters need to find ways of improving. No one was able to beat the line and the very best system still was off by an average of 12.7 points per game. SMALLEST AVERAGE GAME BIAS (Entire Season) Winner: ARGH Power Ratings, Stewart Huckaby Bias is a little different from deviation. Deviation measures the distance between a prediction and the actual result. Bias combines distance and location of the prediction. (Deviation is the absolute value of the bias) Bias is measuring whether the predictions are too high or too low. So if a sytem had an average bias of zero that would mean that on average the system predicts the margin exactly. This can often be used as a statistic to help measure home field advantage. For the entire season the lowest bias was observed in Stewart Hackaby's ARGH Power Ratings. He had an average bias of about +.10, meaning that on average ARGH's predictions were .10 points higher than the game result. The majority of the systems were typically around -1, meaning they did not give enough points to the home team. I will give an honorable mention to the Harmon Forcast which came in a very close 2nd at +.12 and Jeff Sagarin in 3rd place with a +.19. I also want to make a note that Jeff Sagarin has the smallest bias last year in the 1999 season with a value of -.62. I would also like to share a statistical use of these individual game biases. In an 1991 paper in the American Statistician, Hal Stern showed that the individual game biases of the vegas line of NFL games between 1981 and 1984 were approxamately a normal distribution with a mean of zero. So you could estimate the probability of an X point favorite would win the game simply by looking up X /(standard deviation) in a normal distribution table. This gives results such as an even game being 50/50, a 3 point favorite has about a 59% probability of winning and a 7 point favorite a 70% probability of winning. The results page of the prediction tracker presents each systems average bias and the standard deviations, so rough probabilities could be calculated for any of these systems. I say rough because you would be assuming normality. BEST AGAINST THE SPREAD (Entire Season) Winner: Dunkel Index Beating the spread is not the number one goal for most computer rating systems. But it is something that the average person considers. Will a system help you win your office pool. The top system against the spread for the entire season was the Dunkel Index. Dunkel was only 52.3% against the spread and only 7 out of 25 were better than 50%. If you wanted to get technical the best was to pick against the spread would be to pick opposite of what the logistic elo picked. That would have given you 57%. But in general all the systems were very close to the 50% mark. BEST STRAIGHT UP WINNERS (Second Half of Season) Winner: Freeze Ratings, Geoff Freeze It is debatable how meaningful rating systems are in the early parts of the season. How can you make predictions on the first week of the season when no one has played a game yet? The BCS actually waits until mid October before releasing it's first ranking of teams. Other systems chose to wait until around this time before being made public. Looking at only the second half of the season also gives an estimate of how well a system does based only(or mostly) on data from the current year. So to measure this I looked at all 1A games starting with with week nine of the season. Out of these 253 games in the second half of the season the system predicting the most winners was Geoff Freeze's Freeze Ratings at 74.3%. During the second half the systems were very competitive. The top 9 systems all finished within 4 games of each other. Freeze finished 1 game better than 4 other systems. The Vegas line falls all the way to 16th, down from 2nd, during the second half of the season. This gives some evidence that as the season progresses the computer systems are becoming much more accurate as the amount of data is accumulating. SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season) Winner: The Vegas line As with the regular season the Vegas line also wins this award for having the smallest deviation from the actual game scores over the second half of the season. An honorable mention here also goes to Steven Wrathell's CPA Rankings for being the best among the computer systems. Average deviations were lower for the second half of the season. The line went down to 12.11 from 12.37 and CPA went from 12.70 down to 12.21. So again the computer systems are getting better as the season progresses but the closest to the final result is still the Vegas line. SMALLEST AVERAGE GAME BIAS (Second Half of Season) Winner: Marsee Power Ratings, Darryl Marsee The lowest game bias in the second half of the season goes to Darryl Marsee's power ratings. On average, his predictions were only .01 below the actual result. This is an extremely accurate result. Marsee also had one of the better average deviations in the second half of the season. In a rather interesting result the biases of most system really didn't improve as the season progressed. They stayed about the same. This may suggest that some people might have done better had they varied their home field advantage factors as the season went along rather than using a constant number such a 3 points. BEST AGAINST THE SPREAD (Second Half of Season) Winner: The Buck System For the second half of the season the system that did the best against the spread was the Buck System at 53%. As with the entire season a better results could actually be seen by taking the opposite of what the systems say. Here only 8 of 25 were better than 50%. BEST PREDICTIVE SYSTEM IN 2000 Winner: ARGH Power Ratings, Stewart Huckaby Stewart Huckaby's ARGH Power Ratings is the fairly obvious winner of the predictive system of the year. He was the leader in two categories for the entire season and was right up there for the 2nd half categories as well. RETRODICTION AWARDS Retrodiction (I'm not sure who originally coined the term) refers to 'retrodicting' the previous results, rather than predicting future games. It is possible that a computer rating system can put more emphasis on explaining past results than attempting to predict future results. So these systems may not neccesarily be the best predictive systems but can still be very good at their main objective. And some people will argue that explaining the season to date is the only thing that matters in ranking teams. So the retrodiction results on my page come from taking the final ratings and using them to retrodict the entire season. BEST RETRODICTIVE SYSTEM Winner: CPA Rankings, Steve Wrathell I was going to give awards for each category for retrodiction but I would just have been repeating myself each time because Steve Wrathell's CPA Rankings were the best in every retrodictive category I measured. He has the best retrodictive percentage at 83.9%. Compare this to the best predictive percentage of 75.5% and you see that while hindsite isn't perfect it is easier than predicting the future. In fact the worst retrodiction record is better than the best prediction record. The CPA Rankings also had the smallest absolute error and the smallest retrodictive bias. The absolute error was the only system under 10 points and the bias was almost zero. There really isn't any other system that did well at everything. Scripps Howard had a good record but high error and bias. Dwiggens, which uses a least squares based method which minimizes the square error, did predicitably well in absolute error category, second to CPA, but had a poor record and an average bias. Another interesting note is that the Dunkel Index is a highly predictive rating system. From what I have seen the BCS is fairly balanced. Dunkel is predictive, Scripps Howard is more retrodictive and Massey and Sagarin are balanced between the two.