Throughout my career as an educator, I have been challenged to help learners develop their clinical reasoning skills. But the longer I taught and evaluated these skills, the more I noticed the limitations of our assessments.   

We commonly use multiple-choice questions and Observed Structured Clinical Examinations (OSCEs) and draw inferences from outcomes, such as questions asked, diagnoses identified and justifications of those diagnoses. This type of task-level feedback may help a learner think through a case scenario but rarely informs overall skill development. Students do not receive insight into how they reached a conclusion and, if they erred, where in the process their reasoning broke down. This always requires a 1:1 coaching session to gain the learner’s perspective and identify specific gaps in their reasoning process to build a personalized plan for improvement. Effective but time consuming!   

Recognizing these challenges, NBME launched its first Creative Community (CC) initiative in January 2022,¹ leveraging the expertise of NBME staff, medical school faculty, and learners to develop innovative clinical reasoning assessments aimed at providing specific feedback to support learner growth.  

The CC recently completed this two-year project. Among the many insights from the work, learner performance trends provided several important lessons. Reflecting on this experience has led me to consider our clinical reasoning assessments and how we can make some incremental improvements.  

1. More practice. Although clinical reasoning skills are widely taught, learners lack sufficient practice and consistent feedback to improve these skills. For example, despite widespread understanding of the definition of a problem representation, post-clerkship students struggled to provide a summary statement. Most learners were either overly general or expansive, but rarely succinct and descriptive. 

2. Challenging cases. Learners need to encounter challenging cases to assess the range of clinical reasoning behaviors. In contrast to their typical OSCEs, which they felt were structured to have a clear single diagnosis, the post-clerkship students noted that our cases required them to pivot their thinking to new information and reconsider their hypotheses throughout the patient encounter. In fact, several described it as a “great learning experience” despite not receiving performance feedback. 

3. Process-level feedback. A different approach to assessment design is necessary to provide process feedback on clinical reasoning. As a complex, multi-dimensional construct, a principled approach to assessment design can help ensure that we provide effective feedback to grow clinical reasoning skills. We used Evidence Centered Design² to develop cases and assessment tools with the goal of providing timely, process-level feedback that a learner can immediately implement in their next patient encounter.     

Our work created a novel approach to assessment focusing on the process of clinical reasoning, with an emphasis on the skill of hypothesis-directed information gathering, which is essential regardless of the clinical scenario. Our pilot studies demonstrated initial validity evidence, and we are continuing to build on that work to develop an approach that can provide robust feedback to learners. We look forward to continued collaboration with the health professions education community and to share future updates, which you can sign up to receive below. 

[1] NBME. https://www.nbme.org/research/research-collaborations/assessment-alliance. Accessed August 7, 2024.

[2] Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence‐centered design. ETS Research Report Series, 2003(1), i-29.