- Published
- Updated
When an endpoint relies on human judgement, different assessors should apply its rules consistently. A training attendance record alone does not show that assessors interpret and perform the measurement in the same way.
Define the question and assumptions
Identify judgement points that permit interpretation. Use reference examples to investigate disagreement rather than reporting only an overall agreement percentage. Discuss whether the agreement statistic fits the category structure and event distribution. Calibration should teach application of the measurement rules, not interpretation favouring a particular treatment outcome.
Hypothetical study example
In a hypothetical trial, two assessors assign different severity categories to identical records. Disagreement clusters around one category boundary. The team clarifies instructions and tests fresh examples; memorising answers from the original exercise is not treated as evidence that real-world agreement has improved.
Design review checks
- Include boundary examples and frequently confused categories in calibration exercises.
- Preserve initial ratings so the location and causes of disagreement remain visible.
- Plan suitable checks for drift in assessment practice during study conduct.
Record the decision and its limits
Acceptance relies on evidence of competent application for the specific task, not an assumed universal agreement threshold. Document retraining and reassessment. Improved agreement does not automatically establish clinical validity: accuracy, consistency and scientific meaning are different questions and should remain distinguishable when results are reported.
Frequently asked questions
- Does high agreement establish accuracy?
- No. Assessors can consistently apply the same incorrect interpretation.
- Is one training exercise enough for the whole study?
- Not always. Method changes and drift signals may justify reassessment.