Welcome to the 4th annual NCAA PT Awards. The purpose of these awards is to help raise the publicity for those systems that have been superior in various qualities of interest. Most awards will be based entirely on the numbers I've monitored with my prediction tracker web pages. For the 2003 season I followed the weekly performance of 49 computer rating systems. This is up from 42 last season. I start by using the predictions from the entire season. As you will see, the systems performed very poorly this season.MOST STRAIGHT UP WINNERS (Entire Season)Winner: Edward Kambour Winner: Jeff Sagarin The simplest and possibly most important way of measuring the predictive ability of a system is to count the number of games where the winner is correctly predicted. I collected predictions on 698 games in 2003 involving two 1A teams. This year two systems tied for the best overall season record, Edward Kambour and Jeff Sagarin's overall rating. Both of these systems finished with a record of 527-171, (75.5%). This winning percentage is slightly down from ARGH's winning mark of 75.78% last season. The record for best prediction record was set by Kambour last year at 76.48%. This year only three systems outperformed the Vegas line: Kambour, Sagarin and the Bassett Model. 2003 Winner: Edward Kambour & Jeff Sagarin's Overall Rating 2002 Winner: ARGH Power Ratings, Stewart Huckaby 2001 Winner: Massey Ratings. Kenneth Massey 2000 Winner: ARGH Power Ratings, Stewart Huckaby 1999 Winner: Vegas lineSMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)Winner: The Vegas line Deviation from actual game results is another way of measuring the predictive ability of a computer system. Deviation from the game score is the difference between the game prediction and the actual result. A value of zero would mean the score difference is predicted exactly. So one property of a good system would be to minimize the system's average game deviation. For the fifth year in a row the lowest average game deviation was found in the Vegas line. This year the Vegas line's average deviation from the actual score was 12.88 points. That is very close to 12.86 which the odds makers had last year. The record is 12.25 in 2001. Honorable mention here goes to the runner up, Edward Kambour. Kambour had an average game deviation of 13.17. I have been asked why I don't give this award to the highest individual rather than to the line every year. I believe that if we can't beat the line then we don't deserve the credit. We need to find ways of improving this aspect of predicting game outcomes. This year we really didn't even come close to matching the Vegas line. 2003 Winner: Vegas Line 2002 Winner: Vegas Line 2001 Winner: Vegas Line 2000 Winner: Vegas Line 1999 Winner: Vegas LineSMALLEST AVERAGE GAME BIAS (Entire Season)Winner: Edward Kambour Bias is a little different from deviation. Deviation measures the distance between a prediction and the actual result. Bias combines distance and location of the prediction. Bias measures whether the predictions are too high or too low. So if a sytem has an average bias of +0.25 that means that on average the system gives 0.25 points too many to the home team. This statistic can be used to help guage home field advantage. Each year there is usually one system that had a great season that dominates a lot of the awards. This year the person to do that is Ed kambour. He had the best record, 2nd best average deviation and the smallest average bias. Kambour's average bias this season was -0.075 points. So on average his predictions were slightly lower than the actual result. We also did pretty bad in this category as a whole. StatFox comes in a close second at +0.083, but other than that only four other systems were within half a point of the truth. Last year Gupta's ratings had an average bias of 0.00691. So we really fell off this year. 2003 Winner: Edward Kambour 2002 Winner: Gupta Power Ratings 2001 Winner: Flyman 2000 Winner: ARGH Power Ratings 1999 Winner: Jeff SagarinBEST AGAINST THE SPREAD (Entire Season)Winner: Edward Kambour Beating the spread is not the number one goal for most computer rating systems. But it is something that the average person likes to look at and probably the most popular subject that I get emails about. I know that looking at all the games is probably not the most meaningful of measurements but it is what I have been reporting. The top system against the spread for the entire season was Edward Kambour. Kambour was 361-323, (52.78%). As a group we also did a poor job against the spread this season. Only 22 out of 49 systems got more games right than wrong. You would have done better than Kambour's record this year by picking opposite of several of the systems. I believe the record for this category was set last year at 56.04% by the Dunkel Index. 2003 Winner: Edward Kambour 2002 Winner: Dunkel Index 2001 Winner: BMC Picks 2000 Winner: Dunkel Index 1999 Winner: Average across all systemsMOST ACCURATE PREDICTOR (Entire Season)Winner: The Vegas line This award is based on mean square error. Mean square error takes into account both deviation and bias and is perhaps most commonly used measure of evaluating estimators. The vegas line has the smallest mean square error at 266.958, down a few points from last years 261.966. Honorable mention again goes to Edward Kambour for having the smallest mean square error among the systems. Kabour's mark was about 275. Several systems did better than this last season. 2003 Winner: The Vegas Line 2002 Winner: The Vegas Line 2001 Winner: The Vegas Line 2001 Winner: The Vegas Line 1999 Winner: The Vegas Line ------------------------ It is debatable how meaningful rating systems are in the early parts of the season. How can you rate the teams in the first week of the season when no teams have played a game yet? The BCS waits until mid October before releasing it's first ranking of teams. Some systems choose to wait until around this time before being made public. Looking at only the second half of the season also gives an estimate of how well a system does based only(or mostly) on data from the current year.MOST STRAIGHT UP WINNERS (Second Half of Season)Winner: Born Power Index ,William Born Winner: Edward Kambour The second half data started with wee 9 and consisted of 369 games between two division 1A teams. Two systems tied for the most straight up wins in the second half. The full season winner, Ed Kambour, and the person that won this category last season, William Born. Both Kambour and Born had records of 283-86 (76.69%). These two guys also set the best ever winning percentage in this category. The previous best was 76.33% set by Kambour last season. So both Kambour and Born have a great record of being strong in the second half of the season. This season was marked by the controversial BCS matchup. It is interesting to note that of the 7 BCS systems the best finish was #24 by Billingsly. None of the other even finished in top 35. The BCS needs to add some predictie systems. 2003 Winner: Born Power Index & Edward Kambour 2002 Winner: Born Power Index 2001 Winner: Chris Montgomery 2000 Winner: Geoff FreezeSMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season)Winner: The Vegas line (opening line) As with the regular season, the Vegas line also wins this award for having the smallest deviation from the actual game scores over the second half of the season. But interestingly enough the opening line narrowly beats the updated line. The opening line had an average game deviation of 13.13 following by the updated line at 13.18. I will give an honorable mention for Ed Kambour for having the smallest deviation among the systems. Kambour wasn't that far behind at 13.28. The winning total has gotten worse for 4 consecutive seasons. 2003 Winner: Vegas Line (opening) 2002 Winner: Vegas Line 2001 Winner: Vegas Line 2000 Winner: Vegas Line SMALLEST AVERAGE GAME BIAS (Second Half of Season) Winner: Darryl Marsee The lowest game bias in the second half of the season goes to Darryl Marsee. On average, Marsee's rating predictions were 0.35 under the actual result. Overall, the numbers here did not look very good. The majority of the systems were way too low, meaning they did not give enough points to the home team. 2003 Winner: Darrly Marsee 2002 Winner: Massey BCS 2001 Winner: PerformanZ Ratings 2000 Winner: Darryl Marsee's RankingsBEST AGAINST THE SPREAD (Second Half of Season)Winner: The Sports Report For the second half of the season the system that did the best against the spread was again Ed Kambour, 194-166 (53.89%). This was slightly ahead of my PerformanZ Ratings. Those are the only two systems worth mentioning as only 12 out of 48 systems broke the .500 mark. As is often the case, the best way to beat the spread is often to pick opposite of what a system says. This year that system would be picking opposite of Chris Montgomery's system, which was only right 43% of the time. 2003 Winner: Edward Kambour 2002 Winner: The Sports Report 2001 Winner: BMC Picks 2000 Winner: The Buck SystemMOST ACCURATE PREDICTOR (Second Half of Season)Winner: The Vegas line (opening) The opening vegas line also takes this award for having the smallest mean square error over he second half of the season. For the second half results honorable mention goes to Ed Kambour who had the best results among the computer systems. It is the third year in a row that Kambour has been the closest to the line. Kambour is slowly getting closer and closer to matching the line. He has close the game from 13 points in 2001 to 4 points in 2002 to half a point in 2003. 2003 Winner: The Vegas Line (opening) 2002 Winner: The Vegas Line 2001 Winner: The Vegas Line 2000 Winner: The Vegas LineBEST PREDICTIVE SYSTEM IN 2003Winner: Edward Kambour To come up with the best overall predictive system I give each system points for how well they do in all of the above categories. I then sum up the points as the system with the highest total is dubbed predictive system of the year. So this award goes to the system that is the most well rounded One flaw can totally take a system out of the running. This season I give this award to Ed Kambour. Kambour was an easy winner this year. He becomes the first person to win this award twice, winning two years in a row. Other contenders this year were the Vegas line and Gene Bassett. 2003 Winner: Ed Kambour 2002 Winner: Ed Kambour 2001 Winner: Kenneth Massey (non-BCS) 2000 Winner: ARGH Power Ratings -------------------RETRODICTION AWARDSRetrodiction refers to 'retrodicting' the previous results, rather than predicting future games. It is possible that a computer rating system can put more emphasis on explaining past results than attempting to predict future results. So these systems may not neccesarily be the best predictive systems but can still be very good at their main objective. The retrodiction results on my page come from taking the final ratings and using them to repredict the entire season. Rather than list some honorable mentions I am going to break the retrodictive categories into what I will call large and small systems. A small system will be any system that estimates only a global home field advantage. Systems that estimate individual home field advantages are always going to end up winning these awards so I am opening up new categories to the more traditional systems.MOST RETRODICTIVE WINS - Large SystemWinner: The Sports Report - SLOTS The Sports Report SLOTS system had the best retrodictive record for the second year in a row with a record of 584-80 (87.95%). This is down from last year's record of 90.24%. 2003 Winner: The Sports Report - SLOTS 2002 Winner: The Sports Report - SLOTS 2001 Winner: CPA Rankings 2000 Winner: CPA RankingsMOST RETRODICTIVE WINS - Small SystemWinner: Anderson/Hester Among the small systems the best retrodictive winning percentage goes to Anderson & Hester. This system had a competitive record even against the large systems. Anderson & Hester's record was 561-103 (84.49%). This is a good time to make the the note that Anderson & Hester are one of the BCS computers that do not make predictions on their own. Because I want to include the BCS systems in my comparison I translate each of these systems onto a new scale that is suitable for making predictions. I don't really understand why everyone doesn't do that in the first place. And since this is a retrodictive category I am giving awards retroactively to the ones that would have won this in the past. 2003 Winner: Anderson/Hester 2002 Winner: Logistic Regression 2001 WInner: System AverageSMALLEST RETRODICTIVE MEAN ERROR - Large SystemWinner: The Sports Report - SLOTS The top three finishers in mean error were all large systems. The one with the smallest mean error was The Sports Report SLOTS. SLOTS had a mean error of 10.5391. This is just slightly better than the winner last year. This category continues to get worse with time. With the exception of SLOTS the numbers were very high this year. The record for this category was 10.152 by Ed Kambour in 2001. 2003 Winner: The Sports Report - SLOTS 2002 Winner: The average of all systems. 2001 Winner: Edward Kambour Football Ratings 2000 Winner: CPA RankingsSMALLEST RETRODICTIVE MEAN ERROR - Small SystemWinner: Least Squares Regression The winner of the small class ratings is Least Squares Regression with a mean error of 10.9216. That is almost half a point worse than the large system winner. 2003 Winner: Least Squares RegressionSMALLEST RETRODICTIVE BIASWinner: Anderson/Hester I am not breaking this one down into sub categories. The system that had the smallest mean bias was Anderson & Hester. They also set the record for this category with a average bias of 0.00295 points towards the road team. This year actually reverses a downward trend in this category. Three time previous winner CPA Rankings came in a very close 2nd at -0.00314. A couple other systems also came in under the previous record of 0.005. 2003 Winner: Anderson/Hester 2002 Winner: CPA Rankings 2001 Winner: CPA Rankings 2000 Winner: CPA RankingsMOST ACCURATE RETRODICTIVE RATING - Large SystemWinner: The Sports Report - SLOTS The system that had the smallest mean square error this season was The Sports Report SLOTS. SLOTS' mean square error was 184.131. I am a little surprised that this number is so high. Higher thant CPA Rankings had last year quite a bit off the record set by Ed Kambour of 169.03 back in 2001. 2003 Winner: The Sports Report - SLOTS 2002 Winner: CPA Rankings 2001 Winner: Edward Kambour FOotball RatingsMOST ACCURATE RETRODICTIVE RATING - Small SystemWinner: Least Squares Regression It should not be too surprising to see least squares regression do well here since that is what it does, minimize the square errors. Least squeres ended with a mean square error of 186.493, which is just slightly behind the large class winner that uses twice as many parameters. 2003 Winner: Least Squares Regression.BEST RETRODICTIVE SYSTEM 2003 - Large systemWinner: The Sports Report - SLOTS Like the award for best predictive system this award is found by giving each system a score for each category. The system that has the highest total is the retroditive system of the year. For the first time we have a new winner this year. TSR SLOTS won most of the awards so this should not come as a surprise. It wasn't that close but the only one within sight of SLOTS was the previous three time winner CPA Rankings. 2003 WInner: TSR SLOTS 2002 Winner: CPA Rankings 2001 Winner: CPA Rankings 2000 Winner: CPA RankingsBEST RETRODICTIVE SYSTEM 2003 - Small systemWinner: Anderson/Hester I have never been a fan of Anderson & Hester's ratings before. But this year their retro number held up. Looking at previous years, it appears held up is the appropriate term. Anderson & Hester's overall numbers stayed about the same from last year. But for what ever reason just about everyone else declined year to year. Anderson & Hester barely beat out Matthews' Grid. And both of these are were a long way from challenging the large systems. 2003 WInner: Anderson & Hester