.jpg){kind=link}
- Published
- Updated
Overall performance of an AI-enabled device may hide difficulties in underrepresented groups. Subgroup assessment can help, but unstable estimates from small samples should not be presented as definitive differences. Explain the purpose and limits of the assessment.
Define the assessment
Select groups through clinical and technical reasoning rather than every available variable. Users, input quality, hardware and clinical characteristics represent different questions. Assess privacy and permitted use of sensitive attributes separately, and disclose missing group information.
Worked example
For example, an imaging investigation has favourable overall results but few observations from one hardware group. The team does not present its failure proportion as a definitive device effect. It examines uncertainty, record quality and other differences, linking further evidence needs to the poorly represented conditions of use.
Practical checks
- Show the rationale for selecting important groups alongside their available sample sizes.
- Check differences in image quality and reference assessment between groups.
- Distinguish exploratory findings clearly from established performance claims.
Decision and limitations
The report should explain which groups contribute to the overall result and where evidence remains limited. Do not generalise a favourable subgroup finding to every unrepresented group. Qualified reviewers consider clinical relevance and further investigation; a ranking table alone does not establish equitable performance.
Frequently asked questions
- Does a poor result in a small group prove failure?
- No. Examine uncertainty, data quality and possible confounding before drawing definitive conclusions.
- Are subgroups unnecessary when overall results are favourable?
- No. They can help assess representation of intended users and use conditions.