I’ve been thinking a lot lately about an analogy that could be made between the NFL’s player personnel executives (I’ll just say scouts from here on out) and doctors. In the same way that doctors have to look at a patient’s symptoms and try to diagnose a disease, scouts have to look at a football player’s attributes and decide whether or not they will be a good pro. Doctors use all of the education and experience that is available to them, they conduct a mental search, and then land on what they consider to be the most likely diagnosis. Scouts do the same thing.
A potential problem with what doctors and scouts are doing is the mental search. The human brain is limited in the amount of information it can store and it often makes for a poor computer. Human bias makes its way into the work of doctors and I think it would be stupid to say that it doesn’t affect what scouts do as well.
One of the most reliable biases is overconfidence. As humans we are poor judges of how confident we should be. Doctors are no different. One survey found that 94% of all doctors considered themselves to be “above average”. But that’s impossible by definition. Only ~49% of doctors can be above average.
Other studies have found that if doctors are asked to diagnose the cause of death for a patient, and then to rate how confident they are in their answers, they are often wrong (40% of cases) even when they are absolutely confident of their assessment. One study asked dermatologists to diagnose potential cases of melanoma and then rate their level of confidence in their diagnosis. The study group was confident more than 50% of the time, but they still got 30% of all of the melanoma cases wrong.
Despite misdiagnoses and overconfidence that can be easily documented, most doctors when asked, can’t remember a single instance of having gotten a diagnosis wrong.
It’s probably worth mentioning at this point that doctors go to school for at least seven years to do what they do. In the study that asked doctors to diagnose the cause of death, that’s actually a difficult problem. But the problem was that the doctors didn’t approach it as a difficult problem where they might say “I’m not sure, it’s too difficult to say”. The problem was that there was no correlation between how sure they were, and the correctness of their diagnosis.
One of the potential ways to mitigate overconfidence is to engage in more regular feedback. Doctors should be intent on studying the results of their actions. Feedback is a way to confront yourself with your failures and try to avoid making them in the future, and at a minimum, to remove the issue of overconfidence from the equation.
This year’s NFL draft will put several teams in the difficult position of having to decide whether to pay a king’s ransom for RGIII. On the surface the decision might look easy. If you can get a guy like an Aaron Rodgers for instance, you pay whatever you have to. But it’s also worth remembering that the NFL isn’t really that good at figuring out which college players will be good pros. I’m not saying RGIII won’t be any good. I frankly don’t have any idea how good he’ll be. But overconfidence is a silent killer.
To illustrate how rampant overconfidence can be, look at the top 10 picks of the draft from 2005-2007. Or if you want to make a more specific comparison, look at the top 2 picks each year and try to think about how many of those guys you would want to start a franchise with. This is what I mean by engaging in regular feedback. Teams thinking about trading for RGIII should think about how likely they are to make the correct decision, and then decide what they can pay for that chance (maybe 30-40%) of hitting a homerun.
*20 out of the 30 picks haven’t made a pro-bowl.
** The 2005 draft was particularly terrible. None of the first 7 picks have delivered anything for the teams that drafted them.
*** Of the top 2 of each draft, only Mario Williams and Calvin Johnson have been legitimate successes. So about 4 out of those guys haven’t provided much value to their teams.
| Year | Pick | Player | Pos | Tm | All Pro | Pro Bowls | Years Starter | Games |
|---|---|---|---|---|---|---|---|---|
| 2007 | 1 | JaMarcus Russell | QB | OAK | 0 | 0 | 2 | 31 |
| 2007 | 2 | Calvin Johnson | WR | DET | 1 | 1 | 5 | 76 |
| 2007 | 3 | Joe Thomas | T | CLE | 3 | 4 | 5 | 80 |
| 2007 | 4 | Gaines Adams | DE | TAM | 0 | 0 | 1 | 47 |
| 2007 | 5 | Levi Brown | T | ARI | 0 | 0 | 5 | 77 |
| 2007 | 6 | LaRon Landry | DB | WAS | 0 | 0 | 4 | 64 |
| 2007 | 7 | Adrian Peterson | RB | MIN | 2 | 4 | 5 | 73 |
| 2007 | 8 | Jamaal Anderson | DE | ATL | 0 | 0 | 3 | 75 |
| 2007 | 9 | Ted Ginn Jr. | WR | MIA | 0 | 0 | 3 | 75 |
| 2007 | 10 | Amobi Okoye | DT | HOU | 0 | 0 | 4 | 78 |
| 2006 | 1 | Mario Williams | DE | HOU | 0 | 2 | 5 | 82 |
| 2006 | 2 | Reggie Bush | RB | NOR | 0 | 0 | 5 | 75 |
| 2006 | 3 | Vince Young | QB | TEN | 0 | 2 | 4 | 61 |
| 2006 | 4 | D’Brickashaw Ferguson | T | NYJ | 0 | 2 | 6 | 96 |
| 2006 | 5 | A.J. Hawk | LB | GNB | 0 | 0 | 6 | 94 |
| 2006 | 6 | Vernon Davis | TE | SFO | 0 | 1 | 6 | 88 |
| 2006 | 7 | Michael Huff | DB | OAK | 0 | 0 | 5 | 91 |
| 2006 | 8 | Donte Whitner | DB | BUF | 0 | 0 | 5 | 84 |
| 2006 | 9 | Ernie Sims | LB | DET | 0 | 0 | 4 | 87 |
| 2006 | 10 | Matt Leinart | QB | ARI | 0 | 0 | 1 | 31 |
| 2005 | 1 | Alex Smith | QB | SFO | 0 | 0 | 6 | 70 |
| 2005 | 2 | Ronnie Brown | RB | MIA | 0 | 1 | 6 | 92 |
| 2005 | 3 | Braylon Edwards | WR | CLE | 0 | 1 | 5 | 99 |
| 2005 | 4 | Cedric Benson | RB | CHI | 0 | 0 | 5 | 91 |
| 2005 | 5 | Cadillac Williams | RB | TAM | 0 | 0 | 4 | 82 |
| 2005 | 6 | Pacman Jones | DB | TEN | 0 | 0 | 2 | 52 |
| 2005 | 7 | Troy Williamson | WR | MIN | 0 | 0 | 1 | 49 |
| 2005 | 8 | Antrel Rolle | DB | ARI | 0 | 2 | 5 | 100 |
| 2005 | 9 | Carlos Rogers | DB | WAS | 0 | 0 | 5 | 94 |
| 2005 | 10 | Mike Williams | WR | DET | 0 | 0 | 2 | 56 |





