I figured I would spend a few posts over the next few days describing some of the challenges to creating a model that will predict success of NFL draft picks. I’ve said before that draft evaluation eventually has to get to a place where you can “measure the measuring” or where you can assess the effectiveness of the ranks or ratings that players are assigned.
Most of the work I’ve done on this topic has been related to running backs and wide receivers. I have some thoughts on quarterbacks, but I don’t have anything concrete yet. For running backs I’ve sort of come down on the side of usage in terms of predicting the pro success of a running back. You can scout a back, or you can look at Speed Score, or you can look at college stats. But in the end I think the number one predictor of success is how much the guy gets the ball. That’s at least as far as fantasy football goes. Actual real football might be a little bit different.
For wide receivers I’ve spent quite a bit of time working on a model that uses Market Share of college team production to predict pro success. I use the term Market Share to mean a college receiver’s percent of team production. Using these variables has allowed me to increase the accuracy of the model when compared against a more simple one that looks only at draft position. But creating a model for wide receivers isn’t easy.
When you’re creating a model, you have to figure out what you’re trying to figure out. What are you trying to solve for? What are you trying to predict?
Are you trying to predict production as a rookie? Maybe you want to know what the player’s upside is during their alleged third year breakout? That’s the first challenge you have to get over. You have to figure out what you’re trying to figure out.
In my models I basically try to solve for the player’s average fantasy production for their first three years in the league. I do that because I think it’s a long enough time to give a player time to fully learn what they’re doing. After three years I feel like a player would have been labeled a bust or a success already.
But three years isn’t a perfect measurement period by any means. Andre Johnson is a good example. Johnson was a physical freak coming out of college. He weighed 230 pounds, ran a 4.4, and had a 40+ inch vertical leap. But he didn’t become Andre Johnson really until his fifth year in the league. He was good his rookie year and then better his second year. But in his third year he produced only 6 standard fantasy points per game.
Andre Johnson Fantasy Points/Game by Year
Johnson was probably a victim of situation. Houston wasn’t very good for much of the early part of Johnson’s career. In my model there are a number of other receivers whose first three years resulted in more fantasy points/game than Johnson. For instance, Lee Evans came into the league at about the same time and averaged slightly more fantasy points per game than Johnson did.
When we create a model to predict pro success, it doesn’t work to just say “I want to predict the odds that this player can be as successful as Andre Johnson”. It doesn’t work that way because we have to create an objective measure and Andre Johnson was very good during his first three years, although players who we now see an inferior receivers had better starts to their careers. The model is going to see Lee Evans as better than Johnson. To me that’s not a problem, but for some people I think it might be. To me it’s not a problem because In the end it shouldn’t matter if the other very good young wide receivers are more similar to Andre Johnson than they are to Lee Evans (which they are). The Larry Fitzgeralds and Calvin Johnson types who are more similar to Johnson than they are to Evans will improve the model
Also, even though Evans might not be a prototype in terms of size (197 pounds) for the receiver position, there were other things that were a part of Evans’ resume which can explain his success. He caught 40% of Wisconsin’s yards and almost a touchdown per game in his last year there.
So basically, even though Andre Johnson might not have had the best first three years ever, there were players like him who did have very good starts to their careers (Fitzgerald, Megatron) and the players who exceeded Johnson’s production like Lee Evans might be similar in every respect except size.
