Welcome to the 4th annual NCAA PT Awards. The purpose of these awards is to help raise the publicity for those systems that have been superior in various qualities of interest. Most awards will be based entirely on the numbers I've monitored with my prediction tracker web pages. For the 2004 season I followed the weekly performance of 50 computer rating systems. This is up from 49 last season.MOST STRAIGHT UP WINNERS (Entire Season)Winner: Jeff Sagarin This year I collected predictions on 651 games involving two 1A teams. (I am not counting games for Florida Atlantic or Florida International because not all systems included theseteams). For the first time we have a system win for consecutive seasons. Jeff Sagarin's ratings won by a big 6 game margin. Sagarin finished 495-168, (76.0%). This is up slightly from last years 75.5% The record for best prediction record was set by Ed Kambour in 2002 at 76.48%. This year a whopping 16 systems outperformed the Vegas Line. The line had it's worst season to date. The line generally finished right around 75% and near the top of the list, this year it dropped down to 72.8% 2004 Winner: Jeff Sagarin's Overall Rating 2003 Winner: Edward Kambour & Jeff Sagarin's Overall Rating 2002 Winner: ARGH Power Ratings, Stewart Huckaby 2001 Winner: Massey Ratings. Kenneth Massey 2000 Winner: ARGH Power Ratings, Stewart Huckaby 1999 Winner: Vegas lineSMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)Winner: The Vegas line Deviation from the game score is the difference between the game prediction and the actual result. A value of zero would mean the score difference is predicted exactly. One property of a good system would be to minimize the system's average game deviation. Even thought the line was lousy at predicting winners this season it still wins the award for smallest average deviation for the 6th year in a row. This year the winning mark was 12.56. This is about a 0.30 point improvement over the past two years. The record is 12.25 in 2001. The best manned system this year was the Pigskin Index, 12.71. This continues to be a weak category for computer rankings. I have developed a computer adjusted line that was able to outperform the line, so look for it to possibly surplant the Vegas line next season. 2004 Winner: Vegas Line 2003 Winner: Vegas Line 2002 Winner: Vegas Line 2001 Winner: Vegas Line 2000 Winner: Vegas Line 1999 Winner: Vegas LineSMALLEST AVERAGE GAME BIAS (Entire Season)Winner: CPA Rankings, Steve Wrathell Bias is a little different from deviation. Deviation measures the distance between a prediction and the actual result. Bias combines distance and location of the prediction. Bias measures whether the predictions are too high or too low. So if a sytem has an average bias of +0.25 that means that on average the system gives 0.25 points too many to the home team. This statistic can be used to help guage home field advantage. Wow, was this a bad year for this category. Just by looking over the results I would say the average bias this year was 2.5 to 3 points in favor of the road team. It is interesting that home field was often running as high as 6 points per team for some systems this year. Only one system that was followed all season had a value under 1, Actually it was two systems, both CPA Rankings and CPA Retrorankings. They had average biases of -0.40 and -0.44. Usually there is always a system very close to zero. 2004 Winner: CPA Rankings 2003 Winner: Edward Kambour 2002 Winner: Gupta Power Ratings 2001 Winner: Flyman 2000 Winner: ARGH Power Ratings 1999 Winner: Jeff SagarinBEST AGAINST THE SPREAD (Entire Season)Winner: Edward Kambour Beating the spread is not the number one goal for most computer rating systems. But it is something that the average person likes to look at and probably the most popular subject that I get emails about. I know that looking at all the games is probably not the most meaningful of measurements but it is what I have been reporting. We have another two time winner, the top system against the spread for the entire season was Edward Kambour. Kambour was 345-298, (53.66%). It was a mediocre season agains the spread, just about everyone finished in the 48-52% range. The record for this category was set in 2002 at 56.04% by the Dunkel Index. 2004 Winner: Edward Kambour 2003 Winner: Edward Kambour 2002 Winner: Dunkel Index 2001 Winner: BMC Picks 2000 Winner: Dunkel Index 1999 Winner: Average across all systemsMOST ACCURATE PREDICTOR (Entire Season)Winner: The Vegas line This award is based on mean square error. Mean square error takes into account both deviation and bias and is perhaps the most commonly used measure of evaluating estimators. The vegas line has the smallest mean square error at 254.532. This is a few points of the record set in 2001 but better than the previous two seasons. Honorable mention this year goes to Hank Trexler, having the best mean square error among the comuter systesm at 260.76 2004 Winner: The Vegas Line 2002 Winner: The Vegas Line 2001 Winner: The Vegas Line 2000 Winner: The Vegas Line 1999 Winner: The Vegas Line ------------------------ It is debatable how meaningful rating systems are in the early parts of the season. How can you rate the teams in the first week of the season when no teams have played a game yet? The BCS waits until mid October before releasing it's first ranking of teams. Some systems choose to wait until around this time before being made public. Looking at only the second half of the season also gives an estimate of how well a system does based only(or mostly) on data from the current year.MOST STRAIGHT UP WINNERS (Second Half of Season)Winner: Jeff Sagarin Winner: Colley Rankings The second half data started with week 9 and consisted of 314 games between two division 1A teams. Something very interesting has happened this year. Not that we have a two way tie but that one of the systems in the tie is a BCS system. Colley's Rankings have tied with Jeff Sagarin. Note that it is the overall Sagarin rating and not his BCS system. Billingsly and Anderson/Hester only finished one game back. Sagarin and Hester tied with a record of 234-80, 74.5%. This was down 2% from the winners last season, which was a record high. 2004 Winner: Jeff Sagain, Wesley Colley 2003 Winner: Born Power Index & Edward Kambour 2002 Winner: Born Power Index 2001 Winner: Chris Montgomery 2000 Winner: Geoff FreezeSMALLEST DEVIATION FROM ACTUAL GAME SCORES (Second Half of Season)Winner: Ashby Accuratings We have another very interesting result. The Vegas line, which had won this category four straight years fell all the way to 7th place this year. The new winner of this award is Ken Ashby's AccuRatings. Ashby's winning average deviation was 12.37 points. This is the highest winning score since the line in 2000. The numbers had been getting worse and worse each year until this year. I'll give honorable mention to the others that also beat the line for the first time: StatFox and Edward Kambour, plus the average and median predictions, as well as the computer adjusted line. 2004 Winner: Ken Ashby AccuRatings 2003 Winner: Vegas Line (opening) 2002 Winner: Vegas Line 2001 Winner: Vegas Line 2000 Winner: Vegas Line SMALLEST AVERAGE GAME BIAS (Second Half of Season) Winner: Least Squares Regression The lowest game bias in the second half of the season goes to Least Squares Regression. This is a system that I opperate. L2's average bias over the second half of the season was very small, .0087. The numbers here were a little better than for the season as a whole, maybe this is an indication that people started to increase their home field advantages as the season progressed. 2004 Winner: Least Squares Regression 2003 Winner: Darrly Marsee 2002 Winner: Massey BCS 2001 Winner: PerformanZ Ratings 2000 Winner: Darryl Marsee's RankingsBEST AGAINST THE SPREAD (Second Half of Season)Winner: Stat Fox For the second half of the season the system that did the best against the spread was StatFox, 177-122 (59.2%). For the whole season the numbers were average but for the second half the systems seemed to do pretty well again the spread. StatFox's 59% is excellent, a couple others, Ashby AccuRatings and Edward Kambour, last year's winner, finished at 57%. 2004 Winner: StatFox 2003 Winner: Edward Kambour 2002 Winner: The Sports Report 2001 Winner: BMC Picks 2000 Winner: The Buck SystemMOST ACCURATE PREDICTOR (Second Half of Season)Winner: Winner: Ashby Accuratings I guess Ken Ashby accurately named his ratings, his Accuratings get the award for most accurate predictor for the second half of the season. It also marks the first time in 5 seasons that the Vegas line did not win. StatFox finishes second. These are now the first two systems to outperform the Vegas Line. My computer adjusted line also finished ahead of the line. The line came in 40th place picking winnings over the second half, but still manages a 4th place finish here.BEST PREDICTIVE SYSTEM IN 2003Winner: Winner: Ashby Accuratings To come up with the best overall predictive system I give each system points for how well they do in all of the above categories. I then sum up the points and the system with the highest total is dubbed predictive system of the year. So this award goes to the system that is the most well rounded One flaw can totally take a system out of the running. This year the race for best overall predictive system was wide open. If you just look at the variety of names on the individual awards you see that no one system dominated this year. It came down to a three man race between Frank Alder, Ashby AccuRatings and CPA Rankings. Alder wasn't included long enough to really be considered. And so by a slim margin, 85 to 82, the best overall predictive system award goes to a new comer this year, Ken Ashby AccuRatings. 2004 Winner: Ken Ashby AccuRatings 2003 Winner: Ed Kambour 2002 Winner: Ed Kambour 2001 Winner: Kenneth Massey (non-BCS) 2000 Winner: ARGH Power RatingsRETRODICTION AWARDSRetrodiction refers to 'retrodicting' the previous results, rather than predicting future games. It is possible that a computer rating system can put more emphasis on explaining past results than attempting to predict future results. So these systems may not neccesarily be the best predictive systems but can still be very good at their main objective. The retrodiction results on my page come from taking the final ratings and using them to repredict the entire season. Rather than list some honorable mentions I am going to break the retrodictive categories into what I will call large and small systems. A small system will be any system that estimates only a global home field advantage. Systems that estimate individual home field advantages are always going to end up winning these awards so I am opening up new categories to the more traditional systems.MOST RETRODICTIVE WINS - Large SystemWinner: The Sports Report - SLOTS The Sports Report SLOTS system had the best retrodictive record for the Third year in a row with a record of 604-47 (92.78%). This improves on the previous record of 90.24% set in 2002. 2004 Winner: The Sports Report - SLOTS 2003 Winner: The Sports Report - SLOTS 2002 Winner: The Sports Report - SLOTS 2001 Winner: CPA Rankings 2000 Winner: CPA RankingsMOST RETRODICTIVE WINS - Small SystemWinner: Anderson/Hester Among the small systems the best retrodictive winning percentage goes to Logistic Regression. There were actually very few 'large' systems this year so I considered not seperating the awards but I went ahead with it. The most retrodictive wins category among the regular systems was quite competive this year. Logistic Regression led the way with a record of 560-91, 86.02%. The Colley, Massey, and Sagarin BCS ratings all finished just one game behind. Hmmm BCS ratings finished in places 3 through 6 while my simple logistic regression model did better than all of them. Logistic Regression sets a new record in this category. 2004 Winner: Logistic Regression 2003 Winner: Anderson/Hester 2002 Winner: Logistic Regression 2001 WInner: System AverageSMALLEST RETRODICTIVE MEAN ERROR - Large SystemWinner: Least Square Regression with team HFA The overall top finisher in mean error was least squares regression using individual team home field advantages. This system has won this award every year in the NFL so I added for college this year to see how good it would do. It finished 0.60 points better than the next best, last year's winner TSR Slots. And a full point better than the third best, simple least squares regression. The winning mark this year was 9.206. And it also happens to be the best score ever recorded to date. 2004 Winner: Least Square Regression with team HFA 2003 Winner: The Sports Report - SLOTS 2002 Winner: The average of all systems. 2001 Winner: Edward Kambour Football Ratings 2000 Winner: CPA RankingsSMALLEST RETRODICTIVE MEAN ERROR - Small SystemWinner: Least Squares Regression The winner of the small class ratings is Least Squares Regression (L2) with a mean error of 10.2409. L2 wins this award for the second year in a row. THe next closest and 4th overall was Sagarin points. 2004 Winner: Least Squares Regression 2003 Winner: Least Squares RegressionSMALLEST RETRODICTIVE BIASWinner: Anderson/Hester I am not breaking this one down into sub categories. The system that had the smallest mean bias was Anderson & Hester. They had a bias of 0.01183. This deserves a little bit of an asterisk because for this and other BCS systems I am the person actually setting the home field advantage for these systems. So I am not sure if they should not be considered or if I should get half the credit. But I don't really want to award systems that weren't smart enough to raise their home fields this year. Home field was extremely high this year, running nearly 6 points on average. Both CPA systems which are always strong in this category finish right behind Anderson & Hester. 2003 Winner: Anderson/Hester 2003 Winner: Anderson/Hester 2002 Winner: CPA Rankings 2001 Winner: CPA Rankings 2000 Winner: CPA RankingsMOST ACCURATE RETRODICTIVE RATING - Large SystemWinner: Least Square Regression with team HFA The system that had the smallest mean square error this season was my new entry Least Square Regression with team HFA. As has been typical in the NFL it wins this category easily with a mean square error of 137.739. This easily breaks the old record of 169.03 set by Ed Kambour back in 2001. The next closest was last year's winner TSR SLOTS. 2004 Winner: Least Square Regression with team HFA 2003 Winner: The Sports Report - SLOTS 2002 Winner: CPA Rankings 2001 Winner: Edward Kambour FOotball RatingsMOST ACCURATE RETRODICTIVE RATING - Small SystemWinner: Least Squares Regression It should not be too surprising to see least squares regression do well here since that is what it does, minimize the square errors. Least squeres ended with a mean square error of 167.977, which is 20 points better than last year when it also won. The next best was Sagarin Points. 2004 Winner: Least Squares Regression. 2003 Winner: Least Squares Regression.BEST RETRODICTIVE SYSTEM 2004 - Large SystemWinner: CPA Rankings, Steve Wrathell Like the award for best predictive system this award is found by giving each system a score for each category. The system that has the highest total is the retroditive system of the year. This is actually a hard category to even give an award this year. The system that clearly ends up with the most points is TSR SLOTS, but since I only had their ratings for roughly 25% of the season it is clearly not enough to win a major award. The clear second place system would have been Least Squares with team HFA. But I also did not start this system until late in the season so I don't think it is fair to give it this award either. So I have to end up dropping all the way to 7th place overall to come up with the large system that was available all year. This is CPA Rankings. 2004 Winner: CPA Rankings 2003 WInner: TSR SLOTS 2002 Winner: CPA Rankings 2001 Winner: CPA Rankings 2000 Winner: CPA RankingsBEST RETRODICTIVE SYSTEM 2004 - Small systemWinner: Anderson/Hester Unfortunately there was also not a system at the top in the small class that was available for the entire season. The leaders were all systems that started up at the second half. Frank Alder comes in with the 3rd most points. But since the top two are the two large systems mentioned above that don't qualify I actually consider Frank Alder's system to be the Retrodictive system of the year. He beats Anderson & Hester by only a fraction of a point. 2004 Winner: Frank Alder 2003 Winner: Anderson & Hester