Prediction accuracy needs a record before the outcome

The forecasting research

Mellers and colleagues examined drivers of accuracy in geopolitical forecasting. This setting permits predictions to be compared with later outcomes. It does not establish that a broad decision-practices questionnaire measures forecasting talent, nor that success on one kind of question transfers unchanged to every business decision.

A forecast revised after new evidence

Imagine a fictional team estimating whether a regulatory approval will arrive before a planned launch. They initially assign a moderate chance, then receive notice of an additional review stage. A useful process records the original estimate, the new information, and the revised estimate. If the team merely replaces the old number, later discussion may exaggerate how much was known from the start.

The record should also separate a forecast from a commitment. Predicting that an event is likely does not determine whether the organization should accept the consequences if it does not occur. The launch decision may require a contingency plan, a reversible investment, or a delay despite a relatively favorable forecast. Accuracy and decision quality are related questions, but the costs of different outcomes and the available alternatives also matter.

The role of Decision Practices

Machiavelli Decision Practices asks about stated habits of examining evidence, alternatives, confidence, and consequences. A high response index is not an accuracy score. A participant may understand the language of careful judgment while failing to update a particular belief. Another may make a sound domain-specific forecast without describing their method fluently. The report should ask for an episode whose timing and evidence can be reconstructed responsibly.

A prospective research design

An original study could collect predictions on clearly resolvable questions and preserve every update with a timestamp. Researchers should define scoring rules before outcomes arrive and examine the effect of domain knowledge, question difficulty, and repeated exposure. They could then ask whether the questionnaire adds information about observable updating behavior. A separate report study might examine whether readers become more willing to record uncertainty rather than simply more confident in their reasoning. The useful practical habit is an honest account of what was believed when, why it changed, and what decision followed. An assessment should support that record rather than substitute a personality explanation for it.

Preserving the earlier estimate also makes it possible to distinguish genuine updating from a story reconstructed after success.

Continue the reading

Confidence should be connected to evidence and feedback →

A good outcome does not prove a good decision process →

Testing MDP requires both process observations and appropriate outcomes →