Reading a scientific disagreement about integrity tests

Recognize the source's role

Sackett and Schmitt's commentary responds to an integrity-test meta-analysis. The source belongs to a scientific discussion about how a body of evidence should be assembled and interpreted. A commentary is neither an independent replication nor a reason to dismiss the entire topic. Its value lies in making the assumptions behind a conclusion more visible to the reader.

A fictional evidence review

A fictional research committee compares two summaries of the same assessment literature. One emphasizes a modest relationship with job performance; another argues that methodological decisions make that estimate incomplete. A manager asks which number is the real one. The committee should resist choosing whichever estimate best supports a planned purchase. It needs to identify why the estimates differ: eligible studies, outcome definitions, corrections, or the question being answered. Two numbers can address different assumptions rather than representing a simple contest between an honest analysis and a misleading one.

Create an assumption ledger

For this exercise, list each disputed analytical decision in a separate row. Add the rationale offered for it, the likely direction of its effect, and what evidence could resolve the disagreement. Distinguish observed relationships from estimates intended to account for measurement or sampling features. Adjustments may answer a useful theoretical question while moving farther from the relationship a practitioner will observe in a particular setting. The fictional committee should preserve both the estimate and its conditions. A conclusion stripped of those conditions can travel into marketing copy as an apparently universal performance claim, even when the original authors were discussing a narrower issue.

Questions for reading the exchange

Ask whether the commentary disputes the included evidence, the statistical treatment, or the interpretation of results. Locate the exact claim it challenges rather than treating the tone of disagreement as evidence. What would both sides accept as a fair test of their explanation? For the fictional committee, which uncertainties matter to the intended use? A decision involving consequences for applicants requires direct role-relevant evidence and careful consideration of errors; an educational discussion has a different purpose. The same literature summary should not be used as a shortcut across those different decisions.

What this means for an original tool

Machiavelli Integrity Orientation is not one of the instruments evaluated in that exchange. Its original reflection questions cannot borrow either side's estimates. The useful lesson is methodological: disclose what has been studied, what remains uncertain, and which interpretation the evidence can support. Participants should use the form to examine habits and situations, without treating a score as proof of honesty or a basis for employment selection.

Continue the reading

What integrity-test evidence can and cannot establish →

Social desirability, memory, and the limits of self-report →