Forecasting practice makes confidence inspectable

The tournament setting

Mellers and colleagues studied psychological strategies in a geopolitical forecasting tournament. The setting matters because predictions could be recorded and evaluated against defined outcomes. It does not establish that a general self-report about decision habits can measure forecasting skill or predict how well someone will make every organizational choice.

A forecast with a moving target

Consider a fictional operations manager who predicts that a supplier will deliver on time. After a delay, the manager explains that they meant delivery within the month rather than the contractual date. Without a recorded question, the forecast cannot be evaluated fairly. A more useful statement identifies the shipment, the deadline, and the evidence available when the estimate was made. It also records confidence before the outcome is known.

The process can improve the conversation even when the probability is rough. The manager can ask what information would change the estimate and when it should be updated. A colleague might identify a relevant historical delivery rate or a specific disruption. Disagreement can then focus on evidence and assumptions rather than on which person appears more decisive. The purpose is to expose the reasoning, not to make a numerical guess look scientific.

Connecting to Decision Practices

Machiavelli Decision Practices concerns stated habits such as checking evidence and revisiting confidence. Its scores are not forecast accuracy estimates. A person may endorse careful reasoning while performing poorly on a prediction task, or may perform well in a familiar domain without articulating a general method. The report should ask for a recorded decision episode and should distinguish confidence, accuracy, and the quality of the available information.

A research design with honest timing

An original study could collect predictions on questions with clear resolution rules, require time-stamped updates, and evaluate results only after outcomes are known. It should distinguish repeated related questions from genuinely independent opportunities and account for domain knowledge. Questionnaire responses could be examined alongside performance without using them to define success. Researchers should also study whether a report encourages appropriate uncertainty or merely encourages participants to add percentages to their language. The useful development target is a traceable process for updating beliefs. A polished confidence statement is not evidence of accurate judgment until it has faced a clearly specified outcome.

Continue the reading

Confidence should be connected to evidence and feedback →

Testing MDP requires both process observations and appropriate outcomes →