Daniel Kahneman is probably one of the most influential thinkers of our time. His work has made it into the thinking of people like Nassim Taleb and it causes all sorts of problems for economists because it contradicts the notion of human beings as rational actors. If you’re interested, Michael Lewis recently profiled Kahneman when his book Thinking, Fast and Slow was published.
I’m interested in Kahneman’s work insofar as he has covered the issue of experts vs. analytics. A number of studies have been undertaken to answer the question of whether experts can be more accurate than simple algorithms. This research has implications for the ways that NFL teams conduct their business.
NFL coaches and player personnel staff are experts of the same sort that Kahneman is interested in and you would have to think they are subject to the same limitations.
From Thinking, Fast and Slow:
Meehl reviewed the results of 20 studies that had analyzed whether clinical predictions based on the subjective impressions of trained professionals were more accurate than statistical predictions made by combining a few scores or ratings according to a rule. In a typical study, trained counselors predicted the grades of freshmen at the end of the school year. The counselors interviewed each student for forty-five minutes. They also had access to high school grades, several aptitude tests, and a four-page personal statement. The statistical algorithm used only a fraction of this information: high school grades and one aptitude test. Nevertheless, the formula was more accurate than 11 of the 14 counselors. Meehl reported generally similar results across a variety of other forecast outcomes, including violations of parole, success in pilot training, and criminal recidivism.
Not surprisingly, Meehl’s book provoked shock and disbelief among clinical psychologists, and the controversy it started has engendered a stream of research that is still flowing today, more than fifty years after its publication. The number of studies reporting comparisons of clinical and statistical predictions has increased to roughly two hundred, but the score in the contest between algorithms and humans has not changed. About 60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy, but a tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment. No exception has been convincingly documented.
Kahneman, Daniel (2011-10-25). Thinking, Fast and Slow (pp. 222-223). Macmillan. Kindle Edition.