Self-report items, scenarios, and performance tests measure different things

Research context

Reviews of ethical decision models and intervention evaluation show that describing a model is different from demonstrating improved decisions in practice. MED’s item bank is explicitly self-report. It should not be presented as an objective test merely because its topic is ethical judgment.

Machiavelli development analysis

The sixty MED statements describe habits such as identifying absent stakeholders, considering a reversible option, and recording an uncertainty. Respondents indicate how well those statements fit their recent practice. That format can support reflection, but it does not test whether they can identify the best option in a standardized dilemma. It also does not observe whether they act consistently with their stated process. The report should explain this plainly, because the title Ethical Deliberation could otherwise imply a performance assessment that the questionnaire does not provide.

A scenario-based instrument would create additional problems to solve. Designers would need representative situations, adequate information, a defensible scoring rationale, and evidence that the task measures the intended process rather than reading skill or professional knowledge. Experts might disagree about acceptable options for legitimate reasons. A scoring key based on majority opinion would not automatically establish moral correctness. MED should not casually add a few dilemmas and call the result validated judgment measurement. Such an extension would require its own purpose, evidence, and version.

For the current item bank, content review should examine whether each statement belongs to the intended domain and whether it assumes authority or expertise the participant may lack. The wording about seeking advice should allow an appropriate route without rewarding indiscriminate sharing of confidential details. The wording about affected people should not require an impossible attempt to consult everyone. Reverse-keyed statements should describe a clear alternative habit, not rely on double negatives or ask participants to endorse obviously reprehensible identities.

An original cognitive-interview prompt could focus on the item about distinguishing a favorable outcome from sound reasoning. Participants might understand it as acknowledging luck, questioning success, or refusing to learn from positive results. The interviewer should ask for an example and clarify what the item is meant to capture: whether outcome information is considered alongside the quality of the original evidence. If interpretations vary substantially, the item needs revision. Statistical performance cannot compensate for unclear response meaning.

The initial sixty questions are therefore a development pool. Research may show that some domains overlap, some questions are too abstract, or the reference period is difficult to use. The framework should be revised rather than protected by its published page count. A final report should describe the score as an endorsed-process summary, include missingness and version information, and avoid language implying demonstrated ethical competence. The scientific question is whether the self-report has a useful, bounded interpretation, not whether the presentation can make it resemble a more objective test.