Users’ guide to the surgical literature: how to evaluate clinical practice guidelines
=======================================================================================

* Christopher J. Coroneos
* Sophocles H. Voineskos
* Sylvie D. Cornacchi
* Charlie H. Goldsmith
* Teegan A. Ignacy
* Achilleas Thoma

## Clinical scenario

You are a young general surgeon in a community practice. A new consultation is scheduled in your clinic: a 60-year-old woman presenting with chronic venous ulcers on both legs. She has no other notable medical history. Over the previous 5 years, she had a number of debridements and skin grafts performed by a surgeon who recently retired. She is very frustrated by her odorous oozing ulcers and is embarrassed to visit her family. You review the records at your hospital but cannot find much useful information. You do not know why her previous surgeries and nonsurgical treatments have failed. You know that venous ulcers have different etiologies, such as persisting edema, superinfections or concomitant arterial insufficiency. You decide to review the literature guidelines that give specific recommendations on the management of venous ulcers to ensure an option on the treatment algorithm has not been overlooked.

## Literature search

As described in a previous article in the “users’ guide to the surgical literature” series,1 you begin with a Medline search. The terms “venous ulcer” and “guideline” are entered separately. Based on medical subject heading (MESH) terms, Medline prompts inclusion of the terms “leg ulcer/” or “varicose ulcer/” and “guideline” or “practice guideline,” respectively. These terms are combined and results limited to the English language, yielding 10 articles.2–11 Three of them do not relate to venous ulcers,2,3,6 and 3 were published before the year 2000.8–10 Two are nonspecific for ulcers,5,7 and 1 focuses on prevention.4 You select the article entitled, “Guidelines for the treatment of venous ulcers,”11 which appears to address your question. You print the guideline and review it before your next visit with the patient.

## Introduction

Clinical practice guidelines (CPGs) are defined in the literature as “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.”12–15 They distill a large body of literature on a topic into a format that is high-yield and easy for physicians to use.

Worldwide, surgeons perform 200 million procedures annually.16 There is constant effort to optimize this complex and expensive health care facet.16 Surgeons are faced with difficult management decisions while balancing evidence-based recommendations.17 When trial evidence exists, it often cannot be perfectly applied to specific patient presentations. It is difficult to independently condense primary research for each patient. Moreover, health care providers and insurers are increasingly concerned with quality improvement and cost effectiveness. Guidelines aim to balance these factors,12,15 and direct consistent and reliable care.17 The number of surgical guidelines available in the literature is increasing. However, CPGs vary in their quality and sometimes deviate from high methodological rigor.16 It is necessary for surgeons to be able to appraise CPGs before deciding to adopt their recommendations.

Since 1990, CPGs have been an increasingly popular tool influencing physician practice.18,19 More than 20 tools to interpret and appraise CPGs have been published;14 the latest is the AGREE-II instrument (appraisal of guidelines for research and evaluation).13 It was originally released in 2003 to address guideline development, reporting and evaluation. Two further studies,20,21 have refined the instrument, now recognized as the methodological standard in guideline evaluation.14

In this article, we discuss a practical approach to the appraisal of a CPG; Box 1 contains the key items readers should consider when using a CPG in surgery. As in previous users’ guide to the surgical literature articles,22 we use a condensed framework to approach a guideline from a surgical perspective. This will provide surgeons with a practical approach to interpreting and applying recommendations in a CPG, using the guideline by Robson and colleagues11 as an example.

### Box 1. Users’ guides for an article on clinical practice guidelines

1.  Are the recommendations valid?
    
    *   1. Is there a clear statement of a clinical problem?
    
    *   2. Who was involved in guideline development (i.e., authors, reviewers, patients, readers)?
    
    *   3. How is the guideline reviewed?
    
    *   4. What literature are recommendations based on?

2.  What recommendations are made?
    
    *   5. Are useful recommendations presented?
    
    *   6. How do authors move from evidence to recommendations?

3.  Will the results help me in caring for my patients?
    
    *   7. Were all outcomes considered (surgical outcomes versus natural course of disease)?
    
    *   8. Will I be able to implement these recommendations?

### Are the recommendations valid?

#### Is there a clear statement of a clinical problem?

Like other publications, CPGs address a defined problem in a specific group of patients. Surgeons must always consider whether the CPG recommendations can be applied to their own patients.15,23 The PIPOH items (patient population, intevention(s), professionals/patients, outcomes to be considered, health care setting) are suggested in the ADAPTE process ([www.adapte.org](http://www.adapte.org)) to frame the content and clinical question in a guideline.24 Readers should use these categories to decide if the recommendations presented are representative of their patient and treatment goals. Surgeons are cautioned in applying CPGs not designed for their patient populations.25 Subtle differences in any category can alter the CPG’s applicability.

Robson and colleagues11 summarize the management of venous ulcers in 8 categories: diagnosis, compression, infection control, wound bed preparation, dressings, surgery, adjuvant agents and long-term maintenance. However, the guideline does not include specific PIPOH criteria. For example, the authors need to be more specific in Recommendation #6.3: “Less extensive surgery on the venous system, such as superficial venous ablation, endovenous laser ablation, or valvuloplasty, especially when combined with compression therapy, can be useful in decreasing the recurrence of venous ulcers (Level I).” The reader must carefully consider patient population and health care setting in this recommendation. Venous ulcers are associated with comorbidity. If our hypothetical patient had diabetes or an inflammatory disorder, interventions would differ. Further, procedures such as endovenous laser ablation may not be available in every health care setting.

#### Who was involved in guideline development? (ie. authors, reviewers, patients, readers)

No guideline is developed in isolation, free from potential bias.26,27 These biases may be subconscious and difficult to detect.28 Surgeons must critically consider how and why the guideline has been created. What inherent biases may the authors have? Organizing committees and professional organizations beyond those listed in the authorship may have reviewed the guideline. While reviewing evidence and providing recommendations, each group will have their own influence.29 Who are the people in these roles? Who are they representing? What is their expertise? Guideline panels and authorship are often sponsored by the pharmaceutical industry in some capacity.28

The authorship should be analyzed for surgical input. Review by a surgical association or publishing in a surgical journal demonstrate evaluation with surgical familiarity. Even when presented with the same research evidence, professional groups can differ in their recommendations.30 Shaneyfelt and colleagues15 identify examples in breast and prostate cancer: a cancer interest group may support adoption of new, costly population screening interventions despite limited effectiveness, whereas public health groups may not view the intervention as a cost-effective strategy in the general population.

Robson and colleagues’ work11 is developed by the Wound Healing Society, with grant support from its educational/charitable arm, the Wound Healing Foundation.31 The guideline is published in *Wound Repair and Regeneration* by the Wound Healing Society. The guideline has been developed and published by the same association, indicating a potential conflict of interest. Further, CPGs authored by research/care societies may be of lower quality than those published by guideline societies.16 The CPG by Robson and colleagues listis its authors and their affiliations and positions. The group is composed of academicians, private practice physicians, podiatrists, nurse clinicians, research nurses, industrial scientists and an epidemiologist. The diverse author group reflects the multidisciplinary approach to chronic wounds and helps to reduce professional bias.32 However, little detail is given to the roles of each professional, and there is no mention of surgeons.

#### How is the guideline reviewed?

Surgeons should scrutinize the review and revision process of CPGs. Like any other publication, CPGs are subject to peer review. Beyond the guideline’s sponsoring association and authors, surgeons should be sure that independent experts are involved.33 This includes experts in medical and research methodology and possibly patient groups. The process should be transparent. Commentary and editing from the review panel should be included or available in a supplement. Prior to dissemination, guidelines may be pilot tested on small patient samples to ensure applicability. Authors should describe the process of reviewing and updating the guideline on an ongoing basis.26 Some groups establish a team monitoring for new evidence, whereas others provide a predetermined schedule of updates to their guidelines.32 Guidelines with extensive readership and consistent new research findings (e.g., ACCP34 and ACLS35 guidelines) often schedule new releases.

In Robson and colleagues’ work,11 specific revision methodology is lacking. There is no indication the Wound Healing Society has reviewed the guideline, despite their sponsorship. Details of expert review, review scales, specialties/disciplines of reviewers and edits suggested during the review process would all be pertinent to the surgeon. Without insight into who approved the CPG, it is difficult to discern the potential biases that would impact surgical decision making.33 No procedure is defined for revision. Given that leaders in the ever changing field of wound management provided this CPG, surgeons should expect a schedule of updates.

Editorial independence and funding should be declared with all forms of research. Surgeons should be critical in assessing the interests of governing bodies or pharmaceutical/equipment sponsors. The translation of primary literature to clinical recommendations requires judgment.15 Surgeons must ensure this judgment is not biased.36 For example, does a company marketing dressings have any stake in these recommendations? Do involved professional organizations have a monetary or public interest? The interests of authors, *Wound Repair and Regeneration* and the Wound Healing Society are not discussed in the CPG by Robson and colleagues. It is difficult to interpret the biases authors may impose on the CPG outside of credentials listed. No information is available for financial or research support of members. Robson and colleagues succeed in not emphasizing the use of brand name products. Instead it indicates the evidence-based properties of a dressing that improve wound care.

#### What is the evidence base?

Authors should use appropriate methodology to support their recommendations. A transparent and structured methodology reflects rigorous development. This is evaluated stepwise, beginning with the search strategy, appraising evidence and grading recommendations. Similar to the rigor of a systematic review, a good CPG will reflect a body of high-quality research with coherent results.17 Quality of referenced studies should be clear. Issues with blinding, allocation concealment and equal expertise among groups are unique challenges in surgical RCTs.37 Unfortunately, CPGs in surgery can rarely depend solely on high-level evidence (systematic reviews, randomized controlled trials [RCTs]).38 In some surgical areas, observational studies and case reports may be the only evidence available,39 and these must be analyzed for confounding and bias. Often the available evidence is not of high quality.38 However, guidelines addressing questions without available high-quality evidence are still important in guiding physician decision making.40 Moreover, these complex situations require a transparent and rigorous methodology.40

Guidelines should include, either in the text or supporting documentation, a statement detailing the development process. The availability of this process is a good predictor of the CPG’s overall rigor.41 Search strategies should incorporate multiple databases and a search of grey literature (unpublished sources, such as conferences and thesis work). Explicit inclusion and exclusion criteria should be defined, and the assessment of the validity of the evidence should be reproducible and consistent among studies.36 A lag time exists between guideline development and publication. The CPG by Robson and colleagues11 was developed in October 2005 and published in the November/December 2006 issue of *Wound Repair and Regeneration*. In some rapidly changing specialties, it is possible that new evidence becomes available within this lag time.

The CPG by Robson and colleagues11 includes a methods section. While databases are listed, no search terms are specified to ensure a reproducible search methodology for the references cited. Without search terms, transparency is difficult to establish. Robson and colleagues specify that their methodology differs from that of previous publications, including laboratory/animal studies and findings extrapolating from treatment of other ulcers. Beyond this, their process for selecting evidence is vague.

Robson and colleagues11 succeed in defining the level of evidence for each recommendation. For example, they cite the following literature for Recommendation #2.1: “Cullum N, Nelson EA, Fletcher AW, Sheldon TA. Compression for venous leg ulcers. The Cochrane Database of Systematic Reviews. (2001 Issue 2) The Cochrane Collaboration. John Wiley & Sons Ltd. [STAT, 23 RCT].” This illustrates that the recommendation is based on a meta-analysis of 23 RCTs and provides readers with a reference to the original data. Recommendations are followed by contributing references, each marked with 1 of 8 levels of evidence: STAT (Statistical analysis, meta-analysis, consensus statement by commissioned panel of experts), RCT, LIT REV (literature review), CLIN S (clinical case series), RETRO S (retrospective series review), EXP (experimental laboratory or animal study), TECH (technique or methodology description) or PATH S (pathological series review). Of the 41 grouped recommendations made, 5 do not reference RCT or higher levels of evidence: 1.3, 1.4, 4.3, 5.4 and 6.4. While RCTs, systematic reviews and meta-analyses represent the top of the level of evidence hierarchy, there is no discussion of the merits of each reference. Preferably, the RCTs should each be evaluated for individual methodological quality, especially given the unique issues in surgical trials.

### What recommendations are made?

#### Are useful recommendations presented?

Surgeons use guidelines for specific and practical evidence-based advice to direct patient care. An RCT measuring physician practice finds specific recommendations leading to more appropriate and fewer inappropriate clinical tests when compared with unspecific recommendations.32 For surgical CPGs and decision making, choices for patient care can often be reduced to a decision tree (e.g., nonoperative v. procedure X v. procedure Y). From a surgical standpoint, attention to this paradigm is critical. Given a patient presentation, readers will turn to CPGs to illustrate both when a procedure should be performed and which procedure should be performed if different options exist.

In Robson and colleagues’ work,11 recommendations are specific in most cases. For example, their Recommendation #1.4 states, “Apparent venous ulcers that have been open continuously without signs of healing for 3 months or that do not demonstrate any response to treatment after 6 weeks should be biopsied for histological diagnosis (Level III),” and Recommendation #1.1 states, “Gross arterial disease should be ruled out by establishing that pedal pulses are present on physical examination and/or that the ankle: brachial index (ABI) is > 0.8. (Any ABI < 1.0 suggests a degree of vascular disease and compression therapy is usually considered to be contraindicated with an ABI < 0.7) […] (Level I).” These 2 examples reflect objective recommendations for venous ulcers.

The clarity of other recommendations could be improved. For example, Recommendation #4.1 states, “Examination of the patient as a whole is important to evaluate and correct causes of tissue damage. This includes factors such as (A) systemic diseases and medications, (B) nutrition, and (C) tissue perfusion and oxygenation (Level II).” What specific diseases and medications are most important for venous ulcers? What components of a nutrition workup are relevant? What typically needs to be supplemented? This information should be provided from the primary literature. The guideline succeeds in presenting surgical, nonsurgical and preventative surgical options for venous ulcers where applicable.

#### How do authors move from evidence to recommendations?

Arriving at a guideline recommendation is complex, combining best evidence, clinical decision making and patient preferences.42 Good CPGs will provide simple, straightforward care recommendations despite the complexities behind them. When authors use a systematic method to arrive at a judgment, recommendations are more clear and accurate in guiding practice.43 Using this methodology, CPG authors should provide a strength or grade for each recommendation.43 This provides surgeons an indication of the confidence authors have in the literature, level of evidence and real-world effectiveness behind each of their recommendations.17 While CPG authors use a variety of methods to grade recommendations,17 use of a consistent and transparent methodology allows CPGs to be compared across different fields and specialties.43 The GRADE39 methodology is used widely, including the Cochrane Collaboration and UpToDate.39 The GRADE methodology uses a simple system to categorize the quality of evidence into 4 levels (high, moderate, low and very low) and strength of recommendations (strong or weak). Authors interpret methodology, heterogeneity, directness, precision and publication bias of each primary paper.44 For example, the Society for Vascular Surgery adopts the GRADE framework and has a transparent methodology in forming their rigorous, patient-important guideline recommendations.45

Robson and colleagues11 do not describe the strengths and limitations in the body of evidence for each recommendation. There is no formal tool used to illustrate the quality of each paper cited. A classification is used to indicate the strength of each recommendation. This helps illustrate the judgment process for each recommendation. However, the authors do not include patient values in their judgment:

*   “Level I: Meta-analysis of multiple RCTs or at least 2 RCTs support the intervention of the guideline. Another route would be multiple laboratory or animal experiments with at least 2 clinical series supporting the laboratory results.”

*   “Level II: Less than Level I, but at least 1 RCT and at least 2 significant clinical series or expert opinion papers with literature reviews support the intervention. Experimental evidence that is quite convincing, but not yet supported by adequate human experience, is included.”

*   “Level III: Suggestive data of proof of principle, but lacking sufficient data, such as meta-analysis, RCT or multiple clinical series.”

The suggestion in the guideline can be positive or negative at the proposed level (e.g., meta-analysis and 2 RCTs stating intervention is not of use in treating venous ulcers).”

A high level of evidence may not lead to a strong recommendation. For example, Recommendation #7b.4 states, “Negative pressure wound therapy may be useful prior to a skin graft/flap by helping promote the development of granulation tissue in the wound base, or postoperatively by preventing shearing and removing exudates. However, its reported experience in venous ulcers is limited (Level II).” Despite high-level evidence there has not been an illustration of clinical effectiveness, and the impact of therapy may outweigh its potential benefits to patients.

A classification of each recommendation’s strength is missing. Grading recommendations based on this system would allow for comparisons among recommendations in this guideline. No consensus methodology (e.g., Delphi method46) is included. Without explicit methodology, it is difficult to ascertain how the CPG committee arrived at their recommendations. Insight on how decisions were made is necessary for surgeons to apply findings in their own decision making. The aforementioned GRADE methodology provides structure to the review process and limits the bias of “expert opinion” where evidence is unclear.

### Will the results help me care for my patients?

#### Are all outcomes considered?

The process used to select the relevant outcomes and importance of these outcomes must be explicit and sensible. The importance of a certain outcome is directly related to what a patient cares about most. Therefore, CPG authors need to describe the methods with which the outcomes were chosen and a description of the process used to decide on the importance of each outcome. Information on who was involved in outcome choice as well as how values were assigned to outcomes should be apparent in the guideline.

Surgical decision making, like other recommendations, can often be reduced to analysis of benefit versus risk and harm.44 Guidelines should identify not only the interventions of interest, but also sensible alternatives. Surgeons must consider whether the benefits of the treatment discussed outweigh not only the side effects and risks of treatment, but also the implications of another treatment or no treatment. For example, under what circumstances does the benefit of diagnostic laparoscopy outweigh the risk? In a CPG for basal cell carcinoma,43 authors weigh surgical excision against curettage and desiccation, cryotherapy, radiation, chemotherapy and carbon dioxide laser. Considerations include the clinical situation, availability of equipment and patient values/risk profiles among other variables.

Robson and colleagues11 provide a thorough approach to workup and treatment of venous ulcers. Surgical CPGs are sometimes guilty of focusing on the surgical aspects of care while ignoring other aspects of patient management. The nonsurgical multidisciplinary approach is well defined for workup, allowing readers a guide to the workup and preoperative preparation of a venous ulcer. Operative interventions should be compared more directly. For example, Recommendation #6.3 states, “Less extensive surgery on the venous system, such as superficial venous ablation, endovenous laser ablation, or valvuloplasty, especially when combined with compression therapy, can be useful in decreasing the recurrence of venous ulcers (Level I).” This recommendation should be scrutinized because authors can expand on the specific indication of each procedure compared with traditional deep ligation of multiple perforating veins and previously mentioned subfascial endoscopic perforator surgery. This approach to surgical decision making would be helpful to readers.

#### Will I be able to implement these recommendations?

Moving from primary evidence to CPGs, authors consider the potential barriers in offering these procedures to patients.32 When using a guideline, surgeons interpret recommendations in their own setting. An academic tertiary care centre and community hospital have different patient populations, resources and support personnel.36 Applicability and assessment of barriers is often overlooked,47 especially in surgery. Guidelines are expected to illustrate how recommendations can be applied in the settings the authors intended.41

Robson and colleagues11 describe the necessary components for proper management of venous ulcers. While the CPG touches on the multidisciplinary care required in the preoperative workup, operative/postoperative management and follow-up, the barriers and difficulties in this process are not specifically discussed. The CPG focuses solely on interventions. Practically, surgeons are most often limited by the resources available to them. Using a CPG, surgeons must consider if their own resources would support recommendations. Are new, expensive dressings more effective in treatment? Questions of cost-effectiveness and economic analysis are increasingly important to answer.18

## Resolution

Although Robson and colleagues’ work11 is not specific to a particular population or any comorbid conditions, you have no reason to believe that it is not applicable to your patient. You consider the recommendations in this guideline in a stepwise manner. A biopsy of the ulcer first rules out malignancy. A quantitative biopsy rules out clincially important bacterial contamination. You proceed to debride the ulcer in your clinic to minimize the bacterial medium. Home care services are used for daily moist dressing in addition to compression to minimize edema. Two weeks later, the ulcer has a clean base, and you perform a split thickness skin graft. With weekly outpatient follow-up, to the patient’s surprise the ulcer proceeds to heal for the first time in 5 years. This is not the end of the story though. You recommend that the patient should continue applying the compression dressings for life to avoid recurrence.

## Footnotes

*   **Competing interests:** None declared.

*   **Contributors:** C. Coroneos, S. Voineskos, T. Ignacy and A. Thoma designed the study. All authors acquired and interpreted the data. C. Coroneos, T. Ignacy and A. Thoma wrote the article, which all authors reviewed and approved for publication.

*   Accepted October 21, 2013.

## References

1.  Birch DW, Eady A, Robertson D, et al.Users’ guide to the surgical literature: how to perform a literature search.Can J Surg 2003;46:136–41.
    
    

2.  Wienert V, Gerlach H, Gallenkemper G, et al.Medical compression stocking (MCS).J Dtsch Dermatol Ges 2008;6:410–5.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1610-0387.2007.06410.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=18093215&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

3.  Hopf HW, Ueno C, Aslam R, et al.Guidelines for the prevention of lower extremity arterial ulcers.Wound Repair Regen 2008;16:175–88.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1524-475X.2008.00358.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=18318803&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000253761000007&link_type=ISI) 

4.  Robson MC, Cooper DM, Aslam R, et al.Guidelines for the prevention of venous ulcers.Wound Repair Regen 2008;16:147–50.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1524-475X.2008.00355.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=18318800&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000253761000004&link_type=ISI) 

5.  Nicolaides AN, Allegra C, Bergan J, et al.Management of chronic venous disorders of the lower limbs: guidelines according to scientific evidence.Int Angiol 2008;27:1–59.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=18277340&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

6.  Hopf HW, Ueno C, Aslam R, et al.Guidelines for the treatment of arterial insufficiency ulcers.Wound Repair Regen 2006;14:693–710.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1524-475X.2006.00177.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=17199834&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000242872100006&link_type=ISI) 

7.  Agus GB, Allegra C, Antignani PL, et al.Guidelines for the diagnosis and therapy of the vein and lymphatic disorders.Int Angiol 2005;24:107–68.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=15997218&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

8.  McInnes E, Cullum N, Nelson A, et al.RCN guideline on the management of leg ulcers.Nurs Stand 1998;13:61–3.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9923348&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

9.  Update: venous leg ulcer guidelineUniversity of PennsylvaniaOstomy Wound Manage 1997;43:80–2.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9369747&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

10. Douglas WS, Simpson NB.Guidelines for the management of chronic venous leg ulceration. Report of a multidisciplinary workshop. British Association of Dermatologists and the Research Unit of the Royal College of Physicians.Br J Dermatol 1995;132:446–52.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1365-2133.1995.tb08681.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=7718464&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1995QM41200018&link_type=ISI) 

11. Robson MC, Cooper DM, Aslam R, et al.Guidelines for the treatment of venous ulcers.Wound Repair Regen 2006;14:649–62.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1524-475X.2006.00174.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=17199831&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000242872100003&link_type=ISI) 

12. 1.  Field M, 
    2.  Lohr K
    
    , eds (1990) Committee to Advise the Public Health Service on Clinical Practice Guidelines: clinical practice guidelines directions for a new program (National Academy Press, Washington (DC)).
    
    

13. Brouwers MC, Kho ME, Browman GP, et al.AGREE II: advancing guideline development, reporting and evaluation in health care.CMAJ 2010;182:E839–42.
    
    [FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTgyLzE4L0U4MzkiO3M6NDoiYXRvbSI7czoxODoiL2Nqcy81Ny80LzI4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

14. Vlayen J, Aertgeerts B, Hannes K, et al.A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit.Int J Qual Health Care 2005;17:235–42.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1093/intqhc/mzi027&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=15743883&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000229441200009&link_type=ISI) 

15. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J.Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature.JAMA 1999;281:1900–5.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1001/jama.281.20.1900&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=10349893&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000080427300029&link_type=ISI) 

16. Barajas-Nava L, Solà I, Delgado-Noguera M, et al.Quality assessment of clinical practice guidelines in perioperative care: a systematic appraisal.Qual Saf Health Care 2010;19:e50
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjE5LzYvZTUwIjtzOjQ6ImF0b20iO3M6MTg6Ii9janMvNTcvNC8yODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

17. Dahm P, Yeung LL, Gallucci M, et al.How to use a clinical practice guideline.J Urol 2009;181:472–9.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=19084846&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

18. Ray-Coquard I, Philip T, De Laroche G, et al.A controlled “before-after” study: impact of a clinical guidelines programme and regional cancer network organization on medical practice.Br J Cancer 2002;86:313–21.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1038/sj.bjc.6600057&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=11875690&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000174449300001&link_type=ISI) 

19. Woolf SH, Grol R, Hutchinson A, et al.Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines.BMJ 1999;318:527–30.
    
    [FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTgvNzE4Mi81MjciO3M6NDoiYXRvbSI7czoxODoiL2Nqcy81Ny80LzI4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

20. Brouwers MC, Kho ME, Browman GP, et al.Development of the AGREE II, part 1: performance, usefulness and areas for improvement.CMAJ 2010;182:1045–52.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTgyLzEwLzEwNDUiO3M6NDoiYXRvbSI7czoxODoiL2Nqcy81Ny80LzI4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

21. Brouwers MC, Kho ME, Browman GP, et al.Development of the AGREE II, part 2: assessment of validity of items and tools to support application.CMAJ 2010;182:E472–8.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTgyLzEwL0U0NzIiO3M6NDoiYXRvbSI7czoxODoiL2Nqcy81Ny80LzI4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

22. Thoma A, Farrokhyar F, Bhandari M, et al.Users’ guide to the surgical literature. How to assess a randomized controlled trial in surgery.Can J Surg 2004;47:200–8.
    
    

23. Grol R, Dalhuijsen J, Thomas S, et al.Attributes of clinical guidelines that influence use of guidelines in general practice: observational study.BMJ 1998;317:858–61.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTcvNzE2Mi84NTgiO3M6NDoiYXRvbSI7czoxODoiL2Nqcy81Ny80LzI4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

24. Graham ID, Harrison MB, Brouwers M, et al.Facilitating the use of evidence in practice: evaluating and adapting clinical practice guidelines for local use by health care organizations.J Obstet Gynecol Neonatal Nurs 2002;31:599–611.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1552-6909.2002.tb00086.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=12353740&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000178724600016&link_type=ISI) 

25. Hayward RS, Wilson MC, Tunis SR, et al.Users’ guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? The Evidence-Based Medicine Working Group.JAMA 1995;274:570–4.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1001/jama.1995.03530070068032&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=7629987&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1995RN46600029&link_type=ISI) 

26. Thomson R, Lavender M, Madhok R.How to ensure that guidelines are effective.BMJ 1995;311:237–42.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTEvNjk5OS8yMzciO3M6NDoiYXRvbSI7czoxODoiL2Nqcy81Ny80LzI4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

27. McNicol M, Layton A, Morgan G.Team working: the key to implementing guidelines?.Qual Health Care 1993;2:215–6.
    
    [FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czozOiJxaGMiO3M6NToicmVzaWQiO3M6NzoiMi80LzIxNSI7czo0OiJhdG9tIjtzOjE4OiIvY2pzLzU3LzQvMjgwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

28. Shaneyfelt TM, Centor RM.Reassessment of clinical practice guidelines: go gently into that good night.JAMA 2009;301:868–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1001/jama.2009.225&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=19244197&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000263609900031&link_type=ISI) 

29. Thomas S.Standard setting in the Netherlands: impact of the human factor on guideline development.Br J Gen Pract 1994;44:242–3.
    
    [FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo0OiJiamdwIjtzOjU6InJlc2lkIjtzOjEwOiI0NC8zODMvMjQyIjtzOjQ6ImF0b20iO3M6MTg6Ii9janMvNTcvNC8yODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

30. Coulter I, Adams A, Shekelle P.Impact of varying panel membership on ratings of appropriateness in consensus panels: a comparison of a multi- and single disciplinary panel.Health Serv Res 1995;30:577–91.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=7591782&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1995RZ24500004&link_type=ISI) 

31. Wound Healing Foundation About the Wound Healing Foundation [website], Available: [www.woundhealfoundation.net/about/index.htm](http://www.woundhealfoundation.net/about/index.htm). accessed 2012 Apr. 15.
    
    

32. Shekelle PG, Woolf SH, Eccles M, et al.Clinical guidelines: developing guidelines.BMJ 1999;318:593–6.
    
    [FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTgvNzE4My81OTMiO3M6NDoiYXRvbSI7czoxODoiL2Nqcy81Ny80LzI4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

33. Murphy MK, Black NA, Lamping DL, et al.Consensus development methods, and their use in clinical guideline development.Health Technol Assess 1998;2:i–iv1–88.
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9561893&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

34. Holbrook A, Schulman S, Witt DM, et al.Evidence-based management of anticoagulant therapy: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines.Chest 2012;141Supple152S–84S.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1378/chest.11-2295&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=22315259&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000208839900006&link_type=ISI) 

35. Neumar RW, Otto CW, Link MS, et al.Part 8: adult advanced cardiovascular life support: 2010 American Heart Association guidelines for cardiopulmonary resuscitation and emergency cardiovascular care.Circulation 2010;122Suppl 3S729–67.
    
    [FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MTQ6ImNpcmN1bGF0aW9uYWhhIjtzOjU6InJlc2lkIjtzOjE5OiIxMjIvMThfc3VwcGxfMy9TNzI5IjtzOjQ6ImF0b20iO3M6MTg6Ii9janMvNTcvNC8yODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

36. Burgers JS, Grol RPTM, Zaat JOM, et al.Mokkink HG a. Characteristics of effective clinical guidelines for general practice.Br J Gen Pract 2003;53:15–9.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czo5OiI1My80ODYvMTUiO3M6NDoiYXRvbSI7czoxODoiL2Nqcy81Ny80LzI4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

37. Poolman RW, Struijs PAA, Krips R, et al.Reporting of outcomes in orthopaedic randomized trials: Does blinding of outcome assessors matter?.J Bone Joint Surg Am 2007;89:550–8.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiamJqc2FtIjtzOjU6InJlc2lkIjtzOjg6Ijg5LzMvNTUwIjtzOjQ6ImF0b20iO3M6MTg6Ii9janMvNTcvNC8yODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

38. Cook DJ, Greengold NL, Ellrodt AG, et al.The relation between systematic reviews and practice guidelines.Ann Intern Med 1997;127:210–6.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.7326/0003-4819-127-3-199708010-00006&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=9245227&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=A1997XN12900006&link_type=ISI) 

39. 1.  Shanahan JF, 
    2.  Pancotti R
    
    Guyatt GH, Prasad K, Schünemann HJ, et al.How to use a patient management recommendationShanahan JF, Pancotti RUsers’ guides to the medical literature — a manual for evidence based clinical practice2nd edNew York (NY)McGraw-Hill Professional2008597–618.
    
    

40. Loblaw DA, Prestrud AA, Somerfield MR, et al.American Society of Clinical Oncology clinical practice guidelines: formal systematic review-based consensus methodology.J Clin Oncol 2012;30:3136–40.
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjEwOiIzMC8yNS8zMTM2IjtzOjQ6ImF0b20iO3M6MTg6Ii9janMvNTcvNC8yODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

41. Fervers B, Burgers JS, Haugh MC, et al.Predictors of high quality clinical practice guidelines: examples in oncology.Int J Qual Health Care 2005;17:123–32.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1093/intqhc/mzi011&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=15665068&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000228269900005&link_type=ISI) 

42. Guyatt G, Vist G, Falck-Ytter Y, et al.An emerging consensus on grading recommendations?.ACP J Club 2006;144:A8–9.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1136/ebm.11.1.2-a&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=16388549&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

43. Atkins D, Best D, Briss PA, et al.Grading quality of evidence and strength of recommendations.BMJ 2004;328:1490
    
    [Abstract/FREE Full Text](http://canjsurg.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjgvNzQ1NC8xNDkwIjtzOjQ6ImF0b20iO3M6MTg6Ii9janMvNTcvNC8yODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

44. 1.  Shanahan JF, 
    2.  Pancotti R
    
    Schünemann HJ, Vist GE, Jaeschke R, et al.Grading recommendationsShanahan JF, Pancotti RUsers’ guides to the medical literature — a manual for evidence based clinical practice2nd edNew York (NY)McGraw-Hill Professional2008679–701.
    
    

45. Murad MH, Montori VM, Sidawy AN, et al.Guideline methodology of the Society for Vascular Surgery including the experience with the GRADE framework.J Vasc Surg 2011;53:1375–80.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1016/j.jvs.2011.01.036&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=21575755&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

46. Vonk Noordegraaf A, Huirne JAF, Brölmann HAM, et al.Multidisciplinary convalescence recommendations after gynaecological surgery: a modified Delphi method among experts.BJOG 2011;118:1557–67.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1111/j.1471-0528.2011.03091.x&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=21895950&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 

47. Browman GP.Improving clinical practice guidelines for the 21st century. Attitudinal barriers and not technology are the main challenges.Int J Technol Assess Health Care 2000;16:959–68.
    
    [CrossRef](http://canjsurg.ca/lookup/external-ref?access_num=10.1017/S0266462300103034&link_type=DOI) 
    
    [PubMed](http://canjsurg.ca/lookup/external-ref?access_num=11155844&link_type=MED&atom=%2Fcjs%2F57%2F4%2F280.atom) 
    
    [Web of Science](http://canjsurg.ca/lookup/external-ref?access_num=000165817300003&link_type=ISI)