Superforecasting: The Art and Science of Prediction, by Philip Tetlock and Dan Gardner
Overview
Overall, I thought this book was interesting. I learned a few things about Brier scores, Fermi questions, and the outside view. Otherwise, the main characteristics that epitomize superforecasters were not new to me. Superforecasters are intellectually curious, self-critical, and mentally agile. I’ve read many other books and listened to many other podcasts that discuss the importance of these characteristics. I think these traits are useful not only for forecasting, but also being successful in any job.
Some more details and thoughts
The author started investigating prediction capabilities with a personal project that he called “Expert Political Judgement” (EPJ). In this project, the participants were asked about political and economic predictions for the future. The project concluded that experts and non-experts alike are about as skilled at making predictions about the future as monkeys are at throwing darts. The ability of humans to predict future events is no more accurate than a dart-throwing monkey. Predictions are wild, scattered, and mostly inaccurate. However, the researchers did notice that a few individuals consistently made better predictions than others. Similarly, some participants consistently made worse predictions than others. The worst forecasters viewed the world through a single, idealistic lens, which consequently tinted their perceptions of events. They had one “big idea”, and all of the world events were shaped by this big idea. On the other hand, the best forecasters viewed the world from a variety of perspectives, and aggregated these perspectives into their predictions. I think this is a great example of how important it is to be open-minded, listen, gather opposing information, consider other perspectives, and learn to make your own judgements. This type of thinking can be applied not only to forecasting, but also religion, business, and politics. There were two memorable examples for me regarding this idea. In the first example, people guessed the weight of an ox. The average of all the guesses was different from the actual weight by only one pound! This shows the power of aggregating different perspectives. In the second example, people were asked to guess a number. The winner was the person whose guess was closest to 2/3 of the average guess. The actual winning guess was 13, which incorporated lots of different perspectives for looking at the problem.
To build on the EPJ, the “Good Judgement Project” (GJP) was organized next. The idea was similar – to test the ability of people to make future predictions about political and economic events. However, this time the questions targeted the forecasting sweet spot, which is just far enough in the future to be uncertain, but not so far that it’s comparable to a dart-throwing monkey. For example, predicting 10 years in the future is impossible and fruitless. Through the research of the GJP, researchers learned a lot about what makes some people exceptional at predicting future events, as measured by their Brier score, which measures the accuracy of probabilistic predictions. For example, there is a X% probability that event Y will occur. The book stated that the Brier score ranges from 0 to 2, but the standard Brier score that I’m familiar with ranges from 0 to 1. I wonder what calculation they used? One of the most significant traits that makes a superforecaster, that is, somebody with a low Brier score, is intellectual curiosity and self-reflection. Superforecasters also know how to incorporate the “outside view”, acknowledge the counterargument, learn from their mistakes, and incorporate new information. They are mentally agile.
Something that stood out to me was the idea of Fermi questions. Fermi was a physicist who asked brainteasers in order to judge how well people were at reasoning. For example, a famous Fermi question is, “How many piano tuners are in Chicago?” To answer this question, break it down into smaller sub-questions. By doing this, you can arrive at a reasonable and logical answer.
How many people are in Chicago?
How many people own pianos?
How many institutions own pianos?
How often does a piano need to be tuned?
How long does it take to tune a piano?
How many hours/year does a piano tuner work?