Notes on Superforecasting by Philip E. Tetlock & Dan Gardner

When the facts change, I change my mind. What do you do, sir? – John Maynard Keynes

The above quote kind of sums up the approach. A superforecaster constantly updates his or her estimates and is a news junkie. My summary of the key lessons and insights from the book is:

Superforecasters are good at combining inside and outside views. They don't fall (as easily) pray to availability and sampling biases. The outside view refers to the base rate. History rhymes, as they say, and uniqueness is a question of degree. In other words, there are usually some data you can use to give yourself a base rate of an event, such as a terrorist attack or a pandemic.
Superforecasters use more degrees of doubt. Your average person might use yes, maybe and no, but a superforecaster differ between a 62% probability and 63% probability, and the difference was meaningful in Tetlock's experiments. Think about it, a trader or a poker player who can separate a 51% probability of winning from a 49% probability wins.
Superforecasters are Bayesians. They frequently update their estimates in small increments when new evidence comes to light.
Superforecasters also make errors, but they are better at making the trade-off between prudence and decisiveness. They are able to control their error rate on both misses and false alarms.
Post-mortems are key to learn, but don't be captain hindsight. Superforecasters know when they got lucky and when they got unlucky, i.e., when a 20% prediction was the right one, even if the event still happened.
Predictions should be well-defined. For example, after the financial meltdown of 2007-2008 and the ensuing recession a lot of influential, smart people criticized the policy of quantitative easing. They said that money printing will lead to inflation and currency debasement. Who is right? We still don't really know. Testing forecasters needs forecasts to be precise. What is meant by inflation and what is the timeline? Plenty of the doomsayers hold that they are right, but it just hasn't happened yet, while others say stock prices are inflated, even if CPI is not increasing at inflatory rates. However, as any trader will tell you, you can be wrong two ways: wrong analysis and wrong timing. It doesn't help if only on of them is correct.

Take a look at this prediction graph: https://twitter.com/ManifoldMarkets/status/1589703623935565826

Marginalia

The Financial Times held a competition that invited readers to guess the closest to two-thirds the average of all the guesses. What is the best bet? Logic would say 0, while first-order thinking would give 33 as the right guess. In the end, the average was 18.91 and the winning guess was 13. In other words, a combination of the two views were needed and it is not obvious how to weigh the two views against each other.
There is an interesting anecdote in the book about John F. Kennedy's White House changed their decision-making process radically between the Bay of Pigs to the Cuban Missile Crisis. New advisors were brought in, Robert Kennedy played the devil's advocate, the president would occasionally leave the room, process and hierarchy were set aside (at least in theory; who knows in practice?). No longer any "cozy unanimity", but rather stressful, tense discussions.