Welcome to the 2nd annual NCAA PT Awards. The purpose of these awards is to help raise the publicity for those systems that have been superior in various qualities of interest. Most awards will be based entirely on the numbers I've monitored with my prediction tracker web pages. For the 2001 season I followed the weekly performance of 34 computer rating systems. BEST STRAIGHT UP WINNERS (Entire Season) Winner: Massey Ratings, Kenneth Massey The simplest and possibly most important way of measuring the predictive ability of a system is to count the number of games where the winner is correctly predicted. I collected predictions on the 652 games in 2001 involving two 1A teams. The system that correctly predicted the most winners over the course of the 2001 season was Kenneth Massey's Massey Ratings. That is his original ratings. Not the version that is used in the BCS. Massey's ratings correctly predicted the winner in 76.38 percent of the games (498-154). This was two games better than the predictions that is defined to be the average of all system predictions. The Vegas line was fives games back. Kenneth Massey in 2001 and Stewart Huckaby in 2000 are the only systems to ever outperform the vegas line (in the three years that I have kept track). Last years winner, Stewart Huckaby's ARGH Power ratings, fell to 7th place this year even though his percentage was almost the same as a year ago. 2001 Winner: Massey Ratings. Kenneth Massey 2000 Winner: ARGH Power Ratings, Stewart Huckaby 1999 Winner: Vegas line SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season) Winner: The Vegas line Deviation from actual game results is another way of measuring the predictive ability of a computer system. Deviation from the game score is the difference between the game prediction and the actual result. A value of zero would mean the score difference is predicted exactly. So one property of a good system would be to minimize the systems average game deviation. This season the lowest average game deviation was found in the Vegas line. So for the third year in a row this award goes to Roxy and the vegas oddsmakers. This year the Vegas line's average deviation from the actual score was 12.25 points. That is an improvement over last year's 12.37. I will give an honorable mention to the runner up, Kenneth Massey's Massey Ratings, the winner of the best prediction record. Massey's average deviation was about a quarter of a point behind the Vegas Line. Since the Vegas line has won this award three season's running that means that I have never seen a single computer rating system that has been able to beat the line in this category. 2001 Winner: Vegas Line 2000 Winner: Vegas Line 1999 Winner: Vegas Line SMALLEST AVERAGE GAME BIAS (Entire Season) Winner: Flyman Performance Bias is a little different from deviation. Deviation measures the distance between a prediction and the actual result. Bias combines distance and location of the prediction. (Deviation is the absolute value of the bias) Bias is measuring whether the predictions are too high or too low. So if a sytem had an average bias of zero that would mean that on average the system predicts the margin exactly. This can often be used as a statistic to help measure home field advantage. For the entire season the lowest bias was observed in the Flyman Ratings. He had an average bias of about -.06, meaning that on average Flymans predictions were 0.06 points lower than the game result. The average bias of the systems was around -0.60, meaning the computers did not give enough points to the home team. I will give an honorable mention to Ray Waits of YourLinx and Steve Wrathell's CPA Rankings. Both came in a very close 2nd at -0.09. 2001 Winner: Flyman 2000 Winner: ARGH Power Ratings 1999 Winner: Jeff Sagarin BEST AGAINST THE SPREAD (Entire Season) Winner: BMC Picks, Brent Craig Beating the spread is not the number one goal for most computer rating systems. At least not considering ever possible game, as one reader pointed out last season But it is something that the average person considers. Will a system help you win your office pool. The top system against the spread for the entire season was the Brent Craig's BMC Picks. BMC was 324-283, (53.38%) against the spread. As with last year, a method of doing better than the best system would be to take opposite the worst system, The Harmon Forcast. The Dunkel Inxed was the winner last season at 52.3%. 2001 Winner: BMC Picks 2000 Winner: Dunkel Index 1999 Winner: Average across all systems MOST ACCURATE PREDICTOR (Entire Season) Winner: The Vegas line This award is based on mean square error, which takes into account both deviation and bias. The vegas line has the smallest mean square error at 248.25. Honorable mention goes to the Kenneth Massey for being the best among the computer systems. Sonny Moore was also right behind Massey. 2001 Winner: The Vegas Line 2001 Winner: The Vegas Line 1999 Winner: The Vegas Line BEST STRAIGHT UP WINNERS (Second Half of Season) Winner: Chris Montgomery It is debatable how meaningful rating systems are in the early parts of the season. How can you rate the teams in the first week of the season when no one has played a game yet? The BCS waits until mid October before releasing it's first ranking of teams. Some systems chose to wait until around this time before being made public. Looking at only the second half of the season also gives an estimate of how well a system does based only(or mostly) on data from the current year. So to measure this I looked at all 1A games starting with with week eight of the season. Out of these 348 games in the second half of the season the system predicting the most winners belonged to Chris Montgomery, 251-97 (72.1%). This was three games better than Kenneth Massey, the system average, and the Harmon Forcast. 2001 Winner: Chris Montgomery 2000 Winner: Geoff Freeze SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season) Winner: The Vegas line As with the regular season the Vegas line also wins this award for having the smallest deviation from the actual game scores over the second half of the season. An honorable mention here also goes to Jeff Bihl for being the best among the computer systems. Average deviations were actually higher during the second half of the season. This is the opposite of what happened in 2001. 2001 Winner: Vegas Line 2000 Winner: Vegas Line SMALLEST AVERAGE GAME BIAS (Second Half of Season) Winner: PerformanZ Ratings, Todd Beck The lowest game bias in the second half of the season goes to me with my PerformanZ Ratings. I'm surprised. I did not realize this until just now looking at the results. On average, PerformanZ's predictions were only .02 above the actual result. This is an extremely accurate result but it sure didn't carry over to the other categories. The second best score here goes to the Gupta Power Ratings at -0.10. 2001 Winner: PerformanZ Ratings 2001 Winner: Darryl Marsee's Rankings MOST ACCURATE PREDICTOR (Second Half of Season) Winner: The Vegas line The vegas line also takes this award for having the smallest mean square error overt he second half of the season. For the second half results honorable mention goes to Ed Kambour who had the best results among the computer systems. For what it is worth the values for 2001 were about 2 points worse than they were the previous season. 2001 Winner: The Vegas Line 2001 Winner: The Vegas Line BEST AGAINST THE SPREAD (Second Half of Season) Winner: BMC Picks, Brent Craig For the second half of the season the system that did the best against the spread was the same as the winner for the full season, Brent Craig's BMC Picks, 242-103 (54.2%). In general, the systoms did worse in the second half of the season but BMC actually improved. Only 8 out of 34 systems were better than 50%. Here again, a better system would have actually been to play the opposite of the Harmon Forcast which had a very bad 43.5% 2001 Winner: BMC Picks 2001 Winner: The Buck System BEST PREDICTIVE SYSTEM IN 2001 Winner: Massey Ratings, Kenneth Massey Kenneth Massey's non-BCS ratings was clearly the best overall predictive system this year. Just like ARGH last year, Massey was the winner in wins and second to the Vegas line in deviation. But Massey tops it off this year by also being near the top in the other categories as well. 2001 Winner: Kenneth Massey (non-BCS) 2000 Winner: ARGH Power Ratings RETRODICTION AWARDS Retrodiction refers to 'retrodicting' the previous results, rather than predicting future games. It is possible that a computer rating system can put more emphasis on explaining past results than attempting to predict future results. So these systems may not neccesarily be the best predictive systems but can still be very good at their main objective. And some people will argue that explaining the season to date is the only thing that matters in ranking teams. So the retrodiction results on my page come from taking the final ratings and using them to retrodict the entire season. MOST RETRODICTIVE WINS Winner: CPA Rankings, Steve Wrathell The best retrodiction win/loss record for the 2001 season goes to Steve Wrathell's CPA Rankings with a 562-90 (86.2%) record. CPA also has a retro ranking that finished second. These systems had 13 and 15 more wins than the next closest finisher. This is the second year in a row for CPA to finish first. This year improves on last years record of 83.9%. 2001 Winner: CPA Rankings 2001 Winner: CPA Rankings SMALLEST RETRODICTIVE ABSOLUTE ERROR Winner: Edward Kambour Football Ratings, Edward Kambour This year we don't have the same person winning every retrodiction category. The system with the smallest average retrodiction error was Edward Kambour with his NCAA Football Ratings. Kambour's average error was 10.152. Last year CPA Rankings led with a value of 9.62. Kambour led by a fairly significant margin. The second best was the average of the systems at 10.379 and the closest independent system was Kenneth Massey (his original system again) at 10.425 2001 Winner: Edward Kambour Football Ratings 2001 Winner: CPA Rankings SMALLEST RETRODICTIVE BIAS Winner: CPA Rankings, Steve Wrathell As with most wins Steve Wrathell's two CPA Rankings came in first and second for smallest retrodictive bias with values of -0.0067 and -0.0279. CPA Rankings also lead this category last year with a mean bias of -0.0057. For what it is worth, the group as a whole did much better in this category this year compared to last year. 2001 Winner: CPA Rankings 2000 Winner: CPA Rankings MOST ACCURATE RETRODICTIVE RATING Winner: Edward Kambour Football Ratings, Edward Kambour Edward Kambour's Football Ratings picks up his second retrodictive award. This one for being the 'most accurate' at retrodicting the scores determined by having the smallest mean square error. Kambours mean square error was 169.03. Before you ask. It does not have to be the case that the system with the smallest absolute error will have the smallest square error. Although it does happen here and in the predictive awards. But note that the second place systems are not the same, neither here or above. Here the second place goes to Jeff Sagarin at 179.17 2001 Winner: Edward Kambour FOotball Ratings 2000 Winner: Although not computed I'm quite certain CPA Rankings would have been the leader. BEST RETRODICTIVE SYSTEM 2001 Winner: CPA Rankings, Steve Wrathell The two obvious contenders for this are CPA Rankings and Kambour Ratings. To choose between the two look at how good they did in the categories they did not win. The difference is that Kambour does really bad in retrodicting win/loss. Next to last in fact. I think this is a strange and interesting result but it is enough to give CPA Rankings the nod. Of note is that of all the systems only CPA, Sagarin, Rothman, and PerformanZ are above average in every category. 2001 Winner: CPA Rankings 2001 Winner: CPA Rankings