Abstract
Background: Educational videos have become valuable resources and can address some of the pitfalls of traditional learning. To ensure clerkship students have adequate exposure to curriculum objectives, a series of objective-aligned self-directed learning video podcasts covering core surgical concepts were developed by medical students and surgical residents. The objective of the study was to evaluate the efficacy of the video podcasts in the surgery clerkship rotation.
Methods: Nineteen video podcasts were created, housed at www.surgicaleducationportal.com, and distributed to third-year medical students completing their surgical clerkship. A 10-question multiple-choice quiz was administered before and after students viewed each video, and they were also asked to complete a satisfaction survey.
Results: A total of 302 paired pretests and posttests were completed. There was a mean increase of 2.7 points in posttest scores compared with pretest scores (p < 0.001). On a Likert scale from 1 to 5, with 5 being excellent, students rated the usefulness of the videos as 4.3, the quality of the content as 4.3 and the quality of the video as 4.2. Ninety-eight percent of students would recommend these videos to their classmates.
Conclusion: Video podcasts are an effective modality for engaging medical students and may improve standardization of learning during their surgical clerkship.
Didactic in-class lectures have traditionally been the primary method of knowledge dissemination in undergraduate medical education. Although instructional lectures are an acceptable approach for delivering knowledge, there has been a shift toward other teaching modalities, such as self-directed learning, electronic learning (e-learning) and problem-based learning, which have been well received by students and have demonstrated improved outcomes in learners.1–3 In recent years, computer-based educational videos have become valuable resources in medical education, and they can address some of the pitfalls of traditional lecture-based learning. E-learning has been shown to be an effective tool for medical students during medical school and beyond.4–6
A particular modality of e-learning that has become increasingly popular in the medical education community is podcasting.7–9 Podcasts are a digital format available for downloading or streaming over the Internet.10 The content of a podcast has historically been audio; however, advancements in consumer electronics capabilities have recently enabled podcasters to include pictures and videos.11 The term enhanced podcasts has been used in the literature to refer to podcasts that incorporate the use of videos and pictures alongside audio.11–13
The use of enhanced video podcasts may be an effective and efficient way to standardize the curriculum in the surgical clerkship. During this time, medical students have many responsibilities, including inpatient rounding, seeing patients as consultations in the emergency department and taking overnight in-hospital call. Students may have limited dedicated study time. Faculty educators also have demanding schedules filled with clinical, educational and administrative responsibilities. Although learning on the wards may be a rich experience, such opportunities are often sporadic and informal. As such, these experiences are not amenable to standardization. With the move toward competency-based medical education, the standardization of educational exposure is paramount.
The purpose of this project was to create objective-aligned self-directed video podcasts that would educate third-year surgical clinical clerks on core topics in general surgery, urology and orthopedic surgery. The videos were created by medical students in collaboration with senior surgical residents, focusing on the core competencies and objectives of the surgical clerkship. We aimed to test the efficacy of the videos and user satisfaction using pre- and posttests and satisfaction questionnaires.
Methods
Video podcast design
Nineteen enhanced video podcasts were created: 10 in general surgery, 5 in urology and 4 in orthopedic surgery (Appendix 1, available at canjsurg.ca/019019-a1). High-yield core topics seen in the surgical clerkship were selected for the videos, as determined by the undergraduate education lead from each specialty. Each video was developed around the University of Ottawa curriculum objectives, which are aligned with the Licentiate of the Medical Council of Canada (LMCC) objectives. Content transcripts were created by medical students using standard surgical textbooks and review articles and edited for content accuracy and quality by senior surgical residents in the appropriate specialty. The project was overseen by the director of undergraduate surgical education (B.K.L., followed by R.F.)
Final transcripts for each video were converted into a Prezi (Prezi Inc.) presentation and recorded as a video podcast using Camtasia (TechSmith). Each video ranged from about 10 to 20 minutes in duration. Videos included embedded interactive questions termed “Residents Corner” to immediately test students’ understanding of core concepts and to emphasize salient points. Each video podcast was accompanied by a transcript that could be downloaded and used to supplement the video podcasts. Final recorded video podcasts were exported onto YouTube (Alphabet Inc.) and subsequently placed on a private website (www.surgicaleducationportal.com) that was housed on a third-party server (wix.com). Each video podcast was reviewed by the director of undergraduate surgical education (R.F.) before it was posted online and made available to the students. This study was approved by the Ottawa Health Science Network Research Ethics Board (protocol ID 20180155-01H).
Setting and participants
A cohort of 161 third-year medical students at the University of Ottawa were invited to participate in this study. Within the medical school clinical clerkship curriculum, there were 8 parallel streams with 21 or 22 students in each group during the clerkship year. Each group did a 6-week surgical clerkship consisting of 2 weeks of general surgery, 2 weeks of orthopedic surgery or urology, 1 week of elective surgery of the student’s choice and 1 week of a didactic and hands-on surgery “boot camp” at the beginning of the surgery rotation. Boot camp primarily consisted of classroom teaching wherein students reviewed surgical topics with residents or staff physicians with associated clicker-based quizzes. The majority of the students completed their surgical clerkship at The Ottawa Hospital, a tertiary academic hospital affiliated with the University of Ottawa.
From September 2017, at the beginning of every 6-week rotation, an email was sent from the surgical education administrator regarding the use of these video podcasts. Students were encouraged to use the videos before starting on the wards and before their surgery examinations. A general username and password were provided to all students within the cohort, and a unique 4-digit identifier was assigned to each student. Students were able to access the videos on any personal electronic device. They were also provided with access to the video transcripts for use as a study adjunct if they so desired.
Tests and evaluations
To assess the efficacy of the video podcasts, a 10-question multiple-choice test was created for each video podcast. Test questions were designed to focus on clinically relevant aspects of the video podcast such as physical examination findings, diagnostic tests, prognosis and complications associated with certain conditions (Appendix 1). The same 10-question test was administered to students before they watched the video and immediately after. The pretest did not provide solutions to the questions, but students were provided with their pretest score. The posttest provided solutions with explanations. Students were reminded to do the pretest and posttest at the start and end of the video. They were able to access the tests via hyperlinks adjacent to the video (Figure 1). The hyperlinks led users to a Google Form (Alphabet Inc.) wherein they could complete the tests.
Screenshot of website (www.surgicaleducationportal.com) displaying the layout of a video podcast on a topic in general surgery (Approach to the Acute Abdomen).
Students were informed that their results would be formative and anonymous. The posttest ended with an optional questionnaire in which students were asked whether they found the videos useful and invited to rate the quality of the content and the video. Quality was assessed using a subjective 5-point Likert scale, with 1 indicating poor or not useful at all and 5 indicating excellent or very useful. As part of the questionnaire, students were also asked if they would recommend the videos to their classmates, if they used the video transcript and if they preferred this learning modality to traditional modalities such as textbooks. The participants answered these questions with a “yes,” “no” or “indifferent.”
Statistical analysis
Scores for completed pretests and posttests were recorded and compared for each student, using the paired samples t test. Results for students who completed only the pretest or posttest (not both) were excluded from the final analysis. To test for selection bias, excluded pretest and posttest scores were compared with the included scores for each video, using the unpaired samples t test. Data from this study were analyzed using Excel (Microsoft). Data from the satisfaction survey were analyzed using descriptive statistics.
Results
In total, 925 tests were completed for the video podcasts. We included 302 paired pretests and posttests in the final analysis of the study (183 in general surgery, 66 in urology and 53 in orthopedic surgery); the remaining tests did not have either a corresponding pretest or posttest with the same 4-digit identifier. A total of 54 unique students completed both the pretests and posttests in the final analysis. The mean pretest score was 6.0 out of 10 (standard deviation [SD] 1.80) and the mean posttest score was 8.7 out of 10 (SD 1.35), which corresponded to a mean increase of 2.7 (SD 1.73) points (p < 0.001) (Figure 2). The scores that were excluded because a corresponding pretest or posttest score was missing had an average difference of −0.87 and 0.6 points, respectively, when compared with the included pretest and posttest scores (p > 0.08 for all videos).
Change in mean pretest and posttest scores for video podcasts on topics in general surgery, urology and orthopedic surgery and overall change in mean pretest and posttest scores for all video podcasts combined. *p < 0.001.
In the general surgery category, the acute abdomen video podcast had the most quiz completions, with 37. The acute appendicitis video podcast had the second highest number of quiz completions, with 26. The acute abdomen podcast also had the highest increase between pretest and posttest scores, with a mean increase of 4.0 (SD 1.56) points (p < 0.001). The topic with the lowest increase in score, but with the highest mean pretest score, was urinary tract infections, with an increase of 0.80 (SD 0.94) points (p < 0.01). Overall, the videos were rated highly by the students in terms of quality and usefulness (Figure 3). On the 5-point Likert scale, the students rated the usefulness of the videos as 4.3, the quality of the content as 4.3 and the quality of the video as 4.2. Ninety-eight percent (n = 88) of the students reported they would recommend these videos to their classmates and 81% (n = 73) preferred this learning modality to more traditional didactic lectures or textbook reading (Figure 4).
Responses to qualitative questionnaire administered after posttests using a 5-point Likert scale with 1 being poor or not useful and 5 being extremely useful or excellent.
Responses to qualitative questionnaire asking participants about their use of the transcript and their learning modality preference and asking if they would recommend the videos to their classmates.
Discussion
The objective of this study was to test the efficacy of student- and resident-created enhanced video podcasts in the surgical clerkship rotation. A total of 302 paired pretests and posttests were completed in this study; there was an overall average score increase of 2.7 points, which was statistically significant. Students found the videos useful and preferred the videos to traditional learning modalities. The findings from this study provide evidence that enhanced video podcasts that are aligned with curriculum objectives and created by peers and near-peers are an effective learning tool to complement the surgical clerkship.
Standardization of learning among medical students during the surgical clerkship was an important consideration in the creation of these videos. In addition to clinical learning not being easily amenable to standardization, many surgical clerkship students do rotations at peripheral teaching sites. These sites may not provide the same learning experience for students as there are typically no residents to participate in teaching. These videos allow for standardization and help ensure all students get exposure to core topics because they are objective-aligned, up-to-date and clinically relevant resources that all students can use. Furthermore, enhanced podcasts in medical education for undergraduate students are traditionally in the form of recorded in-person lectures, which tend to be lengthy. Our video podcasts ranged from 10 to 20 minutes in duration and were available to students across multiple electronic devices including smart phones, tablets and laptops. As such, they are portable and each video can be viewed without a substantial time commitment. This is ideal for busy surgical clerks who have time constraints and limited dedicated study time. This was reflected in the overwhelmingly positive response of the students, with 81% preferring this method of learning over more traditional modalities like textbooks or lectures and 98% willing to recommend these videos to their classmates.
One of the reasons students enjoyed this learning modality may be that they were able to control the pace of learning. Students could pause, rewind and speed up videos as needed, according to their personal educational needs and preferences. Narula and colleagues published a study on the use of 5-minute video podcasts by clerkship students completing their internal medicine rotation and found that the majority of students agreed that video podcasts were time-efficient and effective learning tools.14 Similar to our study, Narula and colleagues also found that the majority of students preferred video podcasts over traditional textbooks and online resources.14 Furthermore, in a comprehensive review paper on video podcasts Kay and-colleagues found that the most common benefits reported by participants using video podcasts included improving learning, increased control over learning, and making up for missed classes.11 Video podcasts were also found to improve students’ study habits and motivation to learn.11 With respect to control over learning, students appreciated being able to dictate when and where they did their learning.15–17
Our video podcasts had intentional clinical cases and interactive questions embedded within the video to facilitate immediate reinforcement of concepts as they were introduced. In addition, the use of Prezi as the visual medium for the podcasts provided a uniquely different platform relative to traditional slideshow presentation software. Prezi allows for creative transitions between topics while enabling the viewer to continually be reminded of the “bigger picture.” This may have enabled the students in our study to remain engaged and focused and to have improved comprehension during the video, which is reflected in the significant increase in score from pretest to posttest. Our results demonstrate that the use of video podcasts can increase students’ knowledge of core concepts, which may translate into an increase in their comfort level when approaching patients in a clinical setting. In a similar study assessing the use of 5-minute video podcasts in the internal medicine clerkship, Narula and colleagues found that 74% of students who used 5-minute video podcasts felt more comfortable when approaching a symptom or disease in a clinical setting after using the video podcasts.14
Although our study is limited by the absence of a comparator control group, other studies have reported findings similar to ours with respect to improvements in test scores after a podcast or video podcast intervention.18,19 A 2011 study by Bhatti and colleagues compared knowledge acquisition between medical students split into 2 groups.19 One group received a traditional lecture on hemorrhoids while the other group received a podcast augmented with pictures and text about the same topic. The study found that students given the podcast with adjuncts did better in the multiple-choice posttest than the group that received the lecture.19 That study had 2 groups with 2 separate interventions being compared whereas our study only looked at 1 intervention. As a result, the 2 studies are not comparable. However, both studies demonstrated an improvement in test results using e-learning adjuncts conveying core surgical knowledge.
Limitations
Our study has several limitations that are worth commenting on. First, the absence of randomization may predispose our study to a selection bias with respect to the students who voluntarily chose to participate in our study. However, randomization raised ethical concerns regarding restricting some students’ access to a potentially beneficial learning resource. Second, only 54 of 161 eligible students used the video podcasts, indicating there may have been a selection bias of the participants included in our study. Furthermore, many test scores were not used in the final analysis because some students completed only the pretest or the posttest (not both). This could be a possible confounder; however, no significant difference was found when we compared the paired tests that were included with the unpaired tests that were not. In addition, the pretest and posttest questions were reviewed by senior residents involved in our study but had not been previously validated. Another limitation is that the test questions were not developed to reflect each of the levels in Bloom’s taxonomy. The questions used in this study were primarily knowledge-, comprehension- and application-based questions with a minimal focus on higher level analysis or synthesis. This limitation may have resulted in posttest scores that are more representative of short-term memory retention than true comprehension of the content. In future studies, we could test for longer term retention through correlation with examination scores. In addition, we could ask students to indicate specifically when they are using the videos so as to better predict correlations with clinical performance. At our institution, we have continued to grow our video library to include more surgical specialties, including cardiac surgery, otorhinolaryngology, plastic surgery and neurosurgery. By increasing promotion of the videos during the surgical bootcamp week and involving more surgical subdisciplines we expect to see increased engagement of our students with this useful educational tool.
Conclusion
Objective-aligned, student- and resident-created, self-learning video podcasts are an effective tool in the surgical clerkship. Medical students enjoy video podcasts as a learning tool and prefer it to traditional modalities such as textbooks. The use of video podcasts should be strongly considered as an adjunct to the undergraduate surgery curriculum and may have a role in other clerkship rotations as well.
Footnotes
Presented at Meredith Marks Day for Research and Innovation in Medical Education, Apr. 6, 2018, Ottawa, Ont.; the Ontario Student Medical Education Research Conference, Mar. 31, 2019, Toronto, Ont.; the Canadian Surgery Forum, Sept. 5–7, 2019, Montréal, Que.; and the Canadian Conference for the Advancement of Surgical Education, Sept. 25–26, 2019, Ottawa, Ont.
Funding: This research was partially funded through the Innovation in Undergraduate Medical Education Summer Studentship Program of the University of Ottawa.
Competing interests: None declared.
Contributors: N. Mookerji, J. El-Haddad, T. Vo, E. Grose, C. Seabrook, B. Lam and S. Bennett designed the study. N. Mookerji, J. El-Haddad, E. Grose, C. Seabrook and S. Bennett acquired the data, which N. Mookerji, E. Grose, R. Feibel and S. Bennett analyzed. N. Mookerji, E. Grose and R. Feibel wrote the article, which N. Mookerji, J. El-Haddad, T. Vo, E. Grose, C. Seabrook, B. Lam, R. Feibel and S. Bennett revised for important intellectual content. N. Mookerji, J. El-Haddad, T. Vo, E. Grose, C. Seabrook, B. Lam, R. Feibel and S. Bennett gave final approval of the version to be published.
- Accepted September 9, 2020.
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY-NC-ND 4.0) licence, which permits use, distribution and reproduction in any medium, provided that the original publication is properly cited, the use is noncommercial (i.e., research or educational use), and no modifications or adaptations are made. See: https://creativecommons.org/licenses/by-nc-nd/4.0/