Abstract
Background: Positive correlation between the orthopedic in-training examination (OITE) and success in the American Board of Orthopaedic Surgery examination has been reported. Canadian training programs in internal medicine, anesthesiology and urology have found a positive correlation between in-training examination scores and performance on the Royal College of Physicians and Surgeons of Canada (RCPSC) certification examination. We sought to determine the potential predictive value of the OITE scores of Canadian orthopedic surgery residents on their success on their RCPSC examinations.
Methods: A total of 118 Canadian orthopedic surgery residents had their annual OITE scores during their 5 years of training matched to the RCPSC examination oral and multiple-choice questions and to overall examination pass/fail scores. We calculated Pearson correlations between the in-training examination for each postgraduate year and the certification oral and multiple-choice questions and pass/fail marks.
Results: There was a predictive association between the OITE and success on the RCPSC examination. The association was strongest between the OITE and the written multiple-choice examination and weakest between the OITE and the overall examination pass/fail marks.
Conclusion: Overall, the OITE was able to provide useful feedback to Canadian orthopedic surgery residents and their training programs in preparing them for their RCPSC examinations. However, when these data were collected, truly normative data based on a Canadian sample were not available. Further study is warranted based on a more refined analysis of the OITE, which is now being produced and includes normative percentile data based on Canadian residents.
Canadian orthopedic surgery residency training programs require residents to write the orthopedic in-training examination (OITE) annually. This is a multiple-choice examination overseen by the Committee on Examinations and Evaluation of the American Academy of Orthopaedic Surgeons (AAOS), administered continuously since 1963.1 It covers all aspects of orthopedic surgery with questions designed to test recall, comprehension, application, problem solving, evaluation and synthesis (taxonomy levels 1 through 6, respectively).
Correlation between the OITE and success in the American Board of Orthopaedic Surgery (ABOS) examination has been studied. One study reported a high risk of failure (63%; 5 of 8 failed) on the ABOS Part-I examination when a resident scored below the 29th percentile for postgraduate year (PGY)-3 and below the 20th percentile for PGY5. No failures occurred (50 passed) when either the PGY3 score was above the 32nd percentile or the PGY4 score was above the 27th percentile.2 Another study reported that those who averaged in the 27th percentile or lower on the OITE had a 57% chance of failing the ABOS Part-I examination.3 Crawford and colleagues4 reported that PGY3 OITE percentile scores predicted ABOS Part-I and Part-II passage, with residents who scored in the lower quartile having a 5.2 times greater risk of failure on ABOS Part-I. Herndon and colleagues5 reported that the OITE percentile score in the final year in training was a predictor of success on the ABOS Part-I and Part-II examinations.
Canadian training programs in internal medicine, anesthesiology and urology make use of examinations similar to the OITE and have studied the correlation between resident performance on these examinations and their subsequent success on their Royal College of Physicians and Surgeons of Canada (RCPSC) certification examinations.6–8 All 3 specialties have found a positive correlation between in-training examination scores and performance on the RCPSC certification examination. Specifically, in internal medicine, it has been reported that there is a high correlation between the results of the in-training examination and the written component of the RCPSC certification examination. In-training examination scores above the 50th percentile were predictive of a low failure rate (< 1.5%), and scores below the 10th percentile were associated with a high failure rate (24%) on the written component of the RCPSC examination6. In anesthesiology, in-training scores above the 50th percentile were highly predictive of success on the written component of the RCPSC examination, and scores above the 60th percentile were highly predictive of success on the oral component of the examination. In-training scores below the 20th percentile were predictive of failure on both the written and oral components.7 This provides useful feedback to the residents and their training programs in preparing them for RCPSC certification examinations. To our knowledge, no similar investigation has been done for orthopedic surgery.
The purpose of this study was to determine the potential predictive value of OITE scores of Canadian orthopedic surgery residents for their success on their RCPSC certification examinations. The RCPSC certification in orthopedic surgery is based on a compensatory examination combining oral and written components to yield an overall mark that determines whether the candidates pass or fail.
Methods
Thirteen English-speaking residency training programs elected to participate in this study. The study cohort consisted of 118 Canadian orthopedic surgery residents (38 in 2000–2001, 44 in 2001–2002, 36 in 2002–2003). The annual OITE scores obtained by each resident during their 5 years of training were collected by their program directors and matched to their corresponding residents’ RCPSC ID numbers in a nonidentifying blinded fashion on a data sheet. Similarly, a staff member of the Educational Research Unit of the RCPSC Office of Education entered the residents’ RCPSC certification examination scores on a data sheet matched to their corresponding residents’ RCPSC ID numbers, with no reference to resident names. The data sheets were sent to another individual at the RCPSC to match the certification oral and multiple-choice examinations and overall pass/fail examination marks using the nonidentifying resident RCPSC ID numbers. No resident names were attached to any of the data, and their confidentiality was preserved.
Statistical analysis
The Pearson correlations between the OITE raw and percentile scores for each of the candidates’ final PGY and the certification scores on the oral and written multiple-choice examinations and on the overall pass/fail marks were calculated.
Results
The correlation between the OITE percentile and RCPSC oral and written multiple-choice examination marks was significant at the 0.01 level in each of the 3 final academic years. The correlation between the OITE raw score and the RCPSC pass/fail marks was significant at the 0.01 level in 2 of the 3 years; the calculation in the third year was not possible because all the candidates passed. There was a stronger association between the RCPSC oral and written multiple-choice examinations and overall pass/fail marks and the OITE raw score than with the OITE percentile in 2 of the 3 years. The strongest OITE association was with the RCPSC multiple-choice examination, and the weakest association was with the overall pass/fail mark (Table 1).
Discussion
The RCPSC certification examination in orthopedic surgery was a compensatory examination combining oral and written components. The OITE was a multiple-choice examination representing the spectrum of clinical orthopedics. It is not surprising that the strongest correlation was with the RCPSC multiple-choice examination, which is a similar assessment measure. There was also a significant correlation with the RCPSC oral examination marks, indicating that the OITE was a useful tool for preparing residents for this component as well. There was a significant correlation between the OITE raw scores and the RCPSC overall pass/fail marks in 2 of the 3 years; the calculation in the third year was not possible because all the candidates were successful. These results indicate that the OITE was able to provide useful feedback to the residents and their training programs concerning their acquisition of appropriate knowledge in preparation for RCPSC certification.
There was a stronger association between the RCPSC oral and written multiple-choice examinations and the overall examination pass/fail marks with the OITE raw scores than with the OITE percentiles. The percentile scores provided were not an accurate reflection of the competence of the Canadian residents. The percentile is an individual’s raw score compared with their peers in the same year in training (YIT) with a resident in YIT-1 defined as having completed 6 months of orthopedic training. In Canada, individual training programs vary in terms of the amount of time spent in orthopedics during the first 2 years of core surgery training, thus resulting in the possibility of different YIT assignment for the OITE between programs of residents in the same PGY. We believe that this inconsistent association between PGY and YIT in different programs is one reason that we could not find an OITE threshold percentile for those passing or failing the RCPSC multiple-choice and oral examinations and the overall examination. A second reason that the percentile scores were inaccurate is that they were based on the entire sample of residents taking the OITE, most of whom were American candidates.
Conclusion
This study has demonstrated that the OITE had the potential to predict the success of Canadian residents in the years 2000 through 2003 on the RCPSC certification examination. Since 2009, substantial improvements have been made to the OITE reports. Prior to 2009 there was no standardization of scores on the OITE and no breakdown of norms into different groups. The OITE reports now contain Canadian norms (i.e., percentiles) as well as reports for the different content domains in orthopedics. This presents the possibility that greater prediction may be possible not only based on the overall percentile scores, but also based on a regression analysis representing the content domains. In addition, an objective structured clinical examination format has been added to the oral component of the RCPSC certification examination. Overall, further study into the current association between the OITE and the RCPSC certification examination is warranted.
Acknowledgements
We acknowledge the orthopedic surgery training program directors who offered support and encouragement to this project. This work was supported by the Wigle Award from the Office of Postgraduate Medical Education, Queen’s University.
Footnotes
Competing interests: None declared.
Contributors: D. Yen and G. Athwal designed the study. D. Yen and G. Cole acquired the data, which G. Cole analyzed. D. Yen wrote the article, which G. Athwal and G. Cole reviewed. All authors approved the paper for publication.
- Accepted February 5, 2014.