In January 2001, the Royal College of Physicians and Surgeons of Canada (RCPSC) established the first 5-year cycle of the Maintenance of Certification program (MainCert). MainCert certification is now required for admission to the RCPSC and to renew Fellowship privileges. For practising surgeons, involvement in MainCert implies that continuing professional development (CPD) activities must be completed and documented by all surgeons wishing to maintain active Fellowship status.1 As indicated in Table 1, initial surgeon participation has been encouraging.2
A framework for CPD activities has been established (Table 2).1 These activities have been divided into 6 sections with different characteristics and achievable “credits.” Of all these activities, the traditional format of continuing medical education (CME) that is offered through meetings and symposia (section 1) is the most convenient and common CPD activity for practising surgeons. Recent data from RCPSC (Fig. 1) confirm this and illustrate how participation is distributed by section.2
Nevertheless, concern has been raised over the effectiveness of CME interventions and their capacity to improve clinical practice.3–6 A systematic review7 of the evidence evaluating CME interventions has identified 3 factors that are most effective in producing a change in physician behaviour: a physician-based needs assessment (recognition of the need to change physician behaviour); interactive learning among physician colleagues (discussion and review with peers); and sequenced or multifaceted learning activities (use of more than 1 CME activity to achieve change). CME interventions incorporating these activities will be more successful in improving physician performance. The expense and inconvenience of CME interventions may also limit their effectiveness.
Section 5 of the framework comprises “practice review and appraisal” activities. This includes the activity of practice audits or self-audit. Self-audit has been described3,4 as the most educationally sound and possibly the most effective method for CPD. Currently, only general guidelines exist for surgeons wanting to complete a self-audit. This article will review and discuss an approach to self-audit for the surgeon in practice.
Clinical scenario
In conversation, a surgeon colleague comments on his anecdotal impression of a dramatic reduction in wound infection rates after laparoscopic appendectomy (LA) compared with open appendectomy (OA). [The male pronoun is used in this article to refer to all surgeons, irrespective of gender.] You are impressed by this statement and wonder whether there is support for this impression in the literature. You have completed LA selectively, but remain concerned about the overall benefit to your patients. Because evidence of a significant reduction in wound infection rates may cause you to reconsider your selective approach, you decide to review the literature yourself and see if it justifies an audit of your own experience with appendectomy.
You recall seeing an article recently in the “Users’ Guide to the Surgical Literature” series that outlines the steps for completing a literature review, 8 and you decide to use this as a reference to complete your task. Using PubMed, you complete the literature search and identify the Cochrane systematic review of LA versus OA.9 In this review of 39 randomized controlled trials, the overall wound infection rate following LA (3.8%) was significantly lower than after OA (7.6%).
This firms up your decision to audit your experience with appendectomy and establish the wound infection rate for your patients, but you are unsure how to proceed and wonder how much time this task might take.
Defining self-audit
Self-audit is defined by the RCPSC as “An educationally oriented review of a particular aspect of a Fellow’s practice for the purpose of assessing current practices and identifying areas of potential improvement in either delivered care or practice delivery.”1 This process may be initiated not only by the Fellow but also by specialty societies, local medical organizations or provincial licensing bodies.
The process of self-audit
The RCPSC has provided a general description of how to perform a self-audit. In this process, a surgeon is expected to review his practice to identify specific educational needs or ways in which his practice may be modified or improved. The inciting event for this process may relate to a memorable case, an educational activity (local rounds, a journal article) or a conversation with a colleague. This encounter stimulates the surgeon to reflect on his practice and to ask a question about a certain aspect of patient care that he provides, an approach to a surgical problem or a specific surgical procedure. This process may be further defined as an analysis of process-of-care or outcome-of-care variables. Either approach is considered an appropriate method for making judgements on work-based performance.
Measuring outcomes of care (morbidity, mortality) seems logical in assessing the performance of a physician. However, confounding problems can limit the interpretation of outcome variables (i.e., attributing the outcomes only to the actions of the physician versus the health care team). Process-of-care variables (diagnosis, management, prescribing, counselling) are assumed to be more directly related to the actions of the physician but can have a limited impact on changes in patient outcomes, depending on the variable chosen and outcome studied.10
Reflection on one’s practice is an effective method of learning and has been described and acknowledged as an essential component of CPD.11,12 Ultimately, the surgeon is then encouraged to review, evaluate and compare his practice to the current standards of practice or clinical practice guidelines. The emphasis of the process is educational and not punitive, providing an atmosphere that is most conducive to adult learning.5
Documentation
The RCPSC requires that the participating surgeon formally document the self-audit process. The required documentation includes an audit proposal; appropriate data-collection forms; dates of initiation and completion; data interpretation; anticipated outcomes for the surgeon’s practice; and summaries of the findings and actual participation of the surgeon in the self-audit. The college does not require the participating surgeon to keep raw data from the audit.
An approach to self-audit
Self-audit can be seen as a logical process that proceeds according to the “Plan, Do, Study, Act” steps of quality improvement, which follow in turn the principles developed from evidence-based quality assessment.13,14 This process can be further interpreted as a cycle (Fig. 2). In this article, we describe a stepwise approach to self-audit (Box 1) that will provide the documentation required, and facilitate this process for surgeons in practice.
Self-audit template
Identify a topic of interest (from discussion, patient incident, meeting): __________
Develop a question that forms the basis for the audit:
Population: __________
Intervention: __________
Comparison: __________
Outcome: __________
Complete a literature search.
Choose a method for capturing your patients of interest from hospital and office charts:
Diagnostic code: __________
Treatment code: __________
Time span to consider: __________
yy/mm/dd–yy/mm/dd
Decide upon the variables to be collected and design a data collection form (Box 2).
Conduct a pilot chart review and modify the variables, forms or scope of the audit as needed to create an acceptable timeline for completion:
Time to review one chart: __________
No. of patients to be reviewed: __________
Total data collection time: __________
Complete the collection of all data and record it on the forms. (Consider transferring it into a computerized spreadsheet.)
Summarize the data into a table (Box 3).
Interpret data with specific reference to the initial question and standard chosen.
Discuss the audit with your colleagues and consider changes to your patterns of practice.
Consider a follow-up audit in future.
In the initial phase of an audit, the surgeon observes and reflects upon his practice. He will likely choose to review a specific aspect of care that he has become interested in as a result of a patient encounter, a conversation, or a formal or informal CME event. Essentially, the surgeon is asking himself, “How well do I perform in this aspect of patient care in comparison with my colleagues or the ‘standard of care’?”
Not unlike the process of doing an effective literature search, the key to obtaining relevant and accurate information is to develop a focused question. With a specific topic in mind, the surgeon can use the patient–intervention– comparison–outcome (PICO) method to design his question. This method describes the patient or clinical problem, the intervention or exposure, a relevant comparison and the outcome of interest.8,15 This approach is further described in the article8 in this Users’ Guide series that outlines an approach for searching the literature (the reader is advised to review that article when performing a literature search). This method is a concise and simple way to state the question and set up the self-audit. In our scenario, you formulate the resulting question as “In patients with acute appendicitis undergoing open appendectomy, as compared with laparoscopic appendectomy, what is my wound infection rate?” To gather a reasonable sample, you choose to review your last 3 years in practice.
Most surgeons will not have an efficient method for gathering data (prospective or retrospective) for an audit, unless a relatively sophisticated database has been established in their practice. Certain computerized proprietary billing packages will, however, allow for chart retrieval based on diagnostic or procedural codes. Alternatively, a surgeon can take advantage of a hospital’s electronic medical registry (EMR) or medical records department to facilitate the audit. In some cases, hospital discharge codes may allow for retrieval of data, but additional data and review of outpatient records will nevertheless also be required.
It is important to develop a checklist or form to use for data collection (Box 2). This will facilitate data retrieval and is a simple means to ensure accurate and complete data collection. Developing a simple form for your data also ensures that all relevant data are recorded at the time of chart review. A data collection form can be created manually or with a readily available computer software package such as Microsoft Excel, which can allow for automated data summation and use on hand-held or portable computers.
Data collection form
Patient | Age (y) | Sex M/F | Antibiotics Yes /No | Approach* LA/LAC/OA | OR time (min) | Appendix pathology† 0/NPA/PA | Drain Yes /No | LoS (d) | Wound infection Yes /No | Post-op abscess Yes /No | Other complications |
---|---|---|---|---|---|---|---|---|---|---|---|
1 | |||||||||||
2 | |||||||||||
3 | |||||||||||
… |
* LA = laparoscopic appendectomy; LAC = laparoscopic appendectomy, converted to open; OA = open appendectomy
† 0 = none (no pathology); NPA = non-perforated appendicitis; PA = perforated appendicitis
Collecting data for the audit by reviewing charts may be tedious and time-consuming, but carries certain rewards. This should not be a delegated task. Chart review often reveals interesting facts about your practice and patient care, and is an important part of the learning process. The consistency, completeness and legibility of your documentation of patient care will also be revealed during the data-abstraction process.
In determining the key variables to collect, it is important to reflect on the variables that may be most important in influencing the outcome of interest. Recalling the Cochrane systematic review, in our scenario you draft a data collection form with the variables of patient age, sex, preoperative antibiotics, operative approach (open appendectomy [OA] or laparoscopic appendectomy [LA]), conversion of LA to OA, operating time, pathology of the appendix (gross and histopathologic: normal appendix; non-perforated appendicitis; and perforated appendicitis), duration of postoperative antibiotics, length of hospital stay, and the development of wound infections and intra-abdominal abscesses.
It is often helpful to review the first few charts (2 or 3) as an initial assessment of your audit proposal, data forms and chosen outcome variables. This allows you to determine the feasibility of the audit as planned, estimate the time to review each patient chart and then to adjust the numbers of charts to be reviewed, if need be. It is important to consider an appropriate timeline, so that your audit can be completed successfully. In addition, modifications of the data collection form are often necessary if the initial variables are not readily available or if new variables of interest are identified. (On your data collection form, you decide to add columns to collect data on the use of postoperative drains and to document “other” complications not already categorized.) This pilot chart review also provides a timeline for the audit: about 10 minutes are required per patient to review a chart and collect data.
Once all of the data are collected, they should be summarized for review. This can be done by characterizing the patients (basic demographics and clinical variables: total number of patients, mean age, and ratio of the sexes) and summarizing the quantitative data into means and ranges (minimum to maximum), then summarizing the qualitative data by calculating incidences and rates. Data summaries can be reviewed in a simple table (Box 3) to allow a general assessment of the data and understanding of the important trends. Patients can be stratified according to key prognostic features (i.e., perforated v. non-perforated appendices). Often, this is all that will be necessary to generate clinically important conclusions from the data.
Summary table of data
Characteristic | Open appendectomy | Laparoscopic appendectomy | Laparoscopic appendectomy, converted |
---|---|---|---|
No. of patients, % of total | |||
Age, years (mean, min–max) | |||
Sex (frequency of Males/Females) | |||
Antibiotics (freq. Yes/No) | |||
OR time, minutes (mean, min–max) | |||
Frequency of: • Normal appendix | |||
• Non-perforated appendicitis | |||
• Perforated appendicitis | |||
Drains (frequency of Yes/No data) | |||
Length of stay, d (mean, min–max) | |||
Wound infections (freq. Yes/No) | |||
Post-op abscess (freq. Yes/No) | |||
Other complications, frequency |
freq. = data frequency; min–max = range of minimum to maximum
The final challenge for the surgeon is to interpret the data correctly and to consider its implications for his practice. Whether you should alter a practice pattern after comparison to the chosen standards depends on the data and the conclusions made. Understanding the meaning of the data and the reasons for any variation from the standard of practice may require further reflection, additional review of the literature or presentation to a group of colleagues for further discussion and consideration.
Barriers to self-audit
Clear strategies to assist and facilitate a surgeon in completing a self-audit have been unavailable. It is therefore reasonable for surgeons to be concerned that excessive time and effort may be required to complete the process. However, an argument against CPD that is predicated on time constraints may be inappropriate for physicians.16 As Fig. 1 shows, participation in self-audit or practice appraisal by surgeons is minimal (~ 5% of total hours of participation).
As a distinct incentive, participating in a self-audit earns 2 credits per hour with no ascribed maximum, compared with most group learning activities that earn 1 credit per hour and have maximums in certain areas.
Few studies have addressed the cost of audit or feedback interventions. 17 The cost to individual surgeons will vary dramatically according to the extent and depth of the self-audit and the methods for data collection. Computerized patient records may facilitate data management, but this infrastructure demands high capital costs for the appropriate software and hardware. However, traditional CME interventions (i.e., conferences, meetings and courses) also typically incur high costs, including time away from home and clinical practice.
Self-audit is unfamiliar to many surgeons. Perceived complexity and the logistics of this process will undoubtedly influence overall acceptance and participation. Coupled with this, there may well be feelings of suspicion or threat due to the process itself or with those who recommend and mandate this activity.
All provinces in Canada except Ontario have enacted legislation to try to protect the physician and surgeon in the collection and analysis of quality-of-care data, and Ontario was scheduled to pass Bill 31 in the summer of 2004.18 This Bill has 2 parts: Schedule A, the Personal Health Information Protection Act; and Schedule B, the Quality of Care Information Protection Act.
Schedule A discusses the appropriate discretion that must be taken when patient information is collected and analyzed. Patient confidentiality must be maintained in all respects. Data collection should be done in such a way that individual patients remain anonymous; patients can be identified by assigning a code or unique identifier, but the code key must be kept separate and stored in a secure location. Data sharing must maintain respect for the patient; patient confidentiality must remain paramount. The surgeon can collect data only on his patients and must use the data expressly for the purposes of quality-of-care review (i.e., for self-audit by the individual surgeon).
Although Schedule B provides protection for surgeons functioning on designated hospital quality-of-care committees, it is unlikely to provide the same degree of protection to a surgeon performing an independent self-audit.19 Therefore, a surgeon who plans to complete a self-audit and keep the data and data summary should be made aware that this information may be legally discoverable, if a civil action is made against the surgeon and relevant data from a self-audit are requested. The surgeon who discards raw data after a self-audit would not be considered to have acted improperly.19 Preserving the remaining documentation of the self-audit will fulfil RCPSC requirements for documentation.
Conclusion
As a surgeon, you complete a self-audit of your experience with open and laparoscopic appendectomies. You find that wound infection rates are calculated and found to be comparable to the Cochrane data. Knowing the morbidity that accompanies wound infections, you resolve to reconsider your surgical management of appendicitis and to discuss the issue further with your colleagues.
No “magic bullet” exists to alter physician and surgeon behaviour to improve clinical performance. Improvements in clinical performance will lead to better patient care and ultimately have an important effect on and contribute to the control of health care costs.6 Physician behaviour is, however, complex, and further research is necessary to understand what will induce physicians and surgeons to alter their behaviour and practice patterns.20 The RCPSC strategy for maintenance of certification reflects an approach that combines many methods and techniques to influence practice.
In a self-audit, a surgeon asks himself a challenging, introspective question related to his current surgical practice. As the question is being considered, proposed and resolved by the surgeon the educational process of self-audit and self-directed learning is taking place.11 Participation by governing societies and specialty groups may further contribute to learning and could play an important role in stimulating the individual surgeon to conduct a self-audit. Discussion at meetings and conferences may then confirm or validate findings and facilitate the difficult process of changing practice. A database of templates for self-audit could be further developed and maintained by specialty groups to facilitate the process for individual surgeons and provide needed incentive and structure. Ideally, this will lead to an improved overall acceptance of this challenging but rewarding process.
Footnotes
The Evidence-Based Surgery Working Group members include Stuart Archibald, MD;*†‡ Mohit Bhandari, MD;† Charles H. Goldsmith, PhD;‡§ Dennis Hong, MD;† John D. Miller, MD;*†‡ Marko Simunovic, MD, MPH;†‡§¶ Ved Tandan, MD, MSc;*†‡§ Achilleas Thoma, MD;†‡ John Urschel, MD;†‡ and Sylvie Cornacchi, MSc†‡
*Department of Surgery, St. Joseph’s Hospital, †Department of Surgery, ‡Surgical Outcomes Research Centre and §Department of Clinical Epidemiology and Biostatistics, McMaster University, and ¶Hamilton Health Sciences, Hamilton, Ont.
Competing interests: None declared.
- Accepted June 8, 2004.