Assessing the acceptability of script concordance testing: a nationwide study in otolaryngology

Assessing the acceptability of script concordance testing: a nationwide study in otolaryngology

Can J Surg 2021;64(3):E317-E323 | PDF

Andrée-Anne Leclerc, MD, MSc; Lily H.P. Nguyen, MD CM, MSc; Bernard Charlin. MD, PhD; Stuart Lubarsky, MD, MHPE; Tareck Ayad, MD

Abstract

Background: Script concordance testing (SCT) is an objective method to evaluate clinical reasoning that assesses the ability to interpret medical information under conditions of uncertainty. Many studies have supported its validity as a tool to assess higher levels of learning, but little is known about its acceptability to major stakeholders. The aim of this study was to determine the acceptability of SCT to residents in otolaryngology – head and neck surgery (OTL-HNS) and a reference group of experts.

Methods: In 2013 and 2016, a set of SCT questions, as well a post-test exit survey, were included in the National In-Training Examination (NITE) for OTL-HNS. This examination is administered to all OTL-HNS residents across Canada who are in the second to fifth year of residency. The same SCT questions and survey were then sent to a group of OTL-HNS surgeons from 4 Canadian universities.

Results: For 64.4% of faculty and residents, the study was their first exposure to SCT. Overall, residents found it difficult to adapt to this form of testing, thought that the clinical scenarios were not clear and believed that SCT was not useful for assessing clinical reasoning. In contrast, the vast majority of experts felt that the test questions reflected real-life clinical situations and would recommend SCT as an evaluation method in OTL-HNS.

Conclusion: Views about the acceptability of SCT as an assessment tool for clinical reasoning differed between OTL-HNS residents and experts. Education about SCT and increased exposure to this testing method are necessary to improve residents’ perceptions of SCT.

Résumé

Contexte : Le test de concordance de script (TCS) est une méthode objective d’évaluation du raisonnement clinique qui mesure la capacité d’interpréter les renseignements médicaux en contexte d’incertitude. Beaucoup d’études en appuient la validité en tant qu’outil pour évaluer l’enseignement supérieur, mais on en sait peu sur son acceptabilité auprès des principales parties prenantes. Le but de cette étude était de déterminer l’acceptabilité du TCS chez les résidents en otorhinolaryngologie – chirurgie de la tête et du cou (ORL – chirurgie tête et cou) et un groupe de référence composé d’experts.

Méthodes : En 2013 et 2016, une série de questions de TCS, de même qu’un questionnaire post-test, ont été inclus dans l’examen national en cours de formation NITE (National In-Training Examination) pour l’ORL – chirurgie tête et cou. Cet examen est administré à tous les résidents en ORL – chirurgie tête et cou au Canada qui sont entre leurs deuxième et cinquième années de résidence. Les mêmes questions de TCS ont été envoyées à un groupe de chirurgiens en ORL – chirurgie tête et cou de 4 université canadiennes.

Résultats : Pour 64,4 % des membres facultaires et des résidents, l’étude était leur première exposition au TCS. Dans l’ensemble, les résidents ont trouvé difficile de s’adapter à cette forme de test, même si les scénarios cliniques étaient clairs, et ils ont estimé que le TCS était peu utile pour évaluer le raisonnement clinique. En revanche, la grande majorité des experts ont jugé que les questions du test reflétaient la réalité des cas cliniques et recommanderaient le TCS comme méthode d’évaluation en ORL – chirurgie tête et cou.

Conclusion : Entre les résidents et les experts en ORL – chirurgie tête et cou, les points de vue quant à l’acceptabilité du TCS comme outil d’évaluation du raisonnement clinique ont différé et il faudrait y exposer les résidents davantage pour améliorer leur perception du TCS.


Presented at the Annual Meeting of the Quebec Association of Otolaryngology and Head and Neck Surgery, Oct. 16–18, 2015, Québec, Que.; at the 69th Annual Meeting of the Canadian Society of Otolaryngology – Head and Neck Surgery, June 6–9, 2015, Winnipeg, Man.; and at the 2nd International Conference on Clinical Reasoning, Oct. 28–31, 2014, Montréal, Que.

Accepted July 28, 2020

Affiliations: From the Department of Otolaryngology – Head and Neck Surgery, Université de Montréal, Montreal, Que. (Leclerc, Charlin, Ayad); the Department of Otolaryngology – Head and Neck Surgery, McGill University, Montréal, Que. (Nguyen); the Centre for Medical Education, Université de Montréal, Montreal, Que. (Charlin); the Centre for Medical Education, McGill University, Montreal, Que. (Lubarsky); and the Department of Neurology, McGill University, Montreal, Que. (Lubarsky).

Competing interests: None declared.

Contributors: A.-A. Leclerc, L. Nguyen, B. Charlin, S. Lubarsky and T. Ayad designed the study. A.-A. Leclerc acquired the data, which A.-A. Leclerc and T. Ayad analyzed. A.-A. Leclerc and T. Ayad drafted the manuscript, which A.-A. Leclerc, L. Nguyen, B. Charlin, S. Lubarsky and T. Ayad critically reviewed. All authors provided final approval of the version to be published.

Content licence: This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY-NC-ND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use), and no modifications or adaptations are made. See: https://creativecommons.org/licenses/by-nc-nd/4.0/

Data sharing: The data sets analyzed in this study are not publicly available as they were provided by the National In-Training Examination (NITE) committee, but they are available from the corresponding author on reasonable request.

DOI: 10.1503/cjs.014919

Correspondence to: A.-A. Leclerc, Department of Otolaryngology, Hôpital Maisonneuve-Rosemont, 5415 Assumption Blvd, Montréal QC H1T 2M4, andree-anne.leclerc.1@umontreal.ca