Methodology for clinical practice guidelines for the management of arteriovenous access

Article Outline

• Abstract
• Using research evidence to formulate recommendations
• The role of evidence synthesis in formulating high quality practice guidelines
• From evidence to practice: formulating clinical recommendations
• The future
• References
• Copyright

The Society for Vascular Surgery considers the placement and maintenance of arteriovenous hemodialysis access to be an important component of any vascular surgery practice. Therefore, the Society has long been involved in setting the standards for the management of arteriovenous access. Formulating clinical recommendations in this area is the latest effort by the Society to improve the management of arteriovenous access on a national level. To provide an unbiased study of the evidence and to help in formulating the recommendations, the Society used the Knowledge and Encounter Research (KER) Unit of the Mayo Clinic College of Medicine, Rochester, Minn, to review the available evidence and advise a multidisciplinary group of access surgeons and nephrologists in formulating the clinical recommendations. To review the evidence, randomized and observational study designs were both considered. Whenever possible, systematic reviews and meta-analyses of the literature were used because, compared with individual studies, they generate more precise estimates of treatment effects and their results are applicable to a wider range of patients. On behalf of the Society, the group issued its recommendations following the Grading of Recommendations Assessment, Development and Evaluation (GRADE) format; this format disentangles the strength of recommendations from the quality of the evidence and encourages statements about the underlying values and preferences relevant to the particular recommendation. The recommendations are classified as strong (denoted by the phrase “we recommend”) or weak (denoted by the phrase “we suggest”); and the quality of evidence is classified as high, moderate, low, or very low. These recommendations are not meant to supersede clinical judgment; rather, they should be used as a guide for the practicing surgeon and nephrologist as the decision is being made for the placement and subsequent procedures and management of arteriovenous hemodialysis access are being considered.

The Society for Vascular Surgery undertook the task of developing evidence-based guidelines to assist patients and clinicians in the process of decision making. To provide better understanding of these guidelines and facilitate their integration into daily practice, we present the methodologic framework that guideline developers have used to formulate their recommendations. In this supplement, practice guidelines for arteriovenous hemodialysis access and the study of the evidence available in this field are included. In this article, we discuss the evaluation and synthesis of research evidence and the formulation of clear and helpful clinical practice recommendations using the vascular access reviews as examples.

Back to Article Outline

Using research evidence to formulate recommendations 

Randomized controlled trials (RCTs) are the only study design known to balance known and unknown prognostic risk factors between study arms. Hence, RCTs are considered the gold standard study design to evaluate the effectiveness of therapeutic and diagnostic interventions. However, not all randomized trials offer similarly strong inferences because trialists may or may not use the various measures intended to reduce the risk of bias, such as allocation concealment, blinding, and the intention-to-treat principle. For instance, when the Society for Vascular Surgery evaluated the efficacy of vascular access surveillance, only three of the 10 relevant RCTs protected randomization by assuring that the person assigning patients to a study arm was unaware of the randomization sequence, a procedure called allocation concealment. This procedure usually requires the use of central randomization (eg, the investigator called a central location with patient characteristics and received the arm allocation) or the use of sealed, numbered, and opaque envelopes.¹ Patients and surgeons were appropriately not blinded in these trials; however, data collectors and researchers ascertaining the outcomes could have been blinded to reduce bias; the former was done in five of 10 RCTs and the latter was done in only one RCT.

In the comparison between autogenous and prosthetic accesses, it was unclear whether trials did not report or did not adhere to the intention-to-treat principle, by which researchers set up procedures and resources to keep randomized participants in the arm to which they were randomized, with no crossover to the other arm. Hence, the inferences from these trials need to be interpreted in the context of the described methodologic limitations.

Moreover, the evaluation of surgical interventions presents challenges to the use of the randomized trials. Patients are reluctant to be randomized to procedures that may be associated with permanent consequences or scars, blinding surgeons and sham surgeries are often unfeasible, and the duration of required follow-up may be too long and impractical for RCTs.² As a consequence, evidence-based surgeons have relied extensively on observational studies to this point: only 3.4% of all publications in the leading surgical journals are RCTs, and more than half of these RCTs compared medical therapies in surgical patients rather than alternative surgical procedures or surgical vs medical or radiologic interventions.³

Recent innovations in the design of surgical trials may overcome some of the limitations of traditional RCTs in surgery, provide the bias protection that randomization offers, and gain the ability to make strong inferences about the efficacy of surgical interventions. An example of these innovations is the expertise-based RCT. Whereas participants in conventional RCTs are randomized to one of two interventions (A or B) and individual clinicians give intervention A to some participants and B to others, participants in expertise-based RCTs are randomized to clinicians with expertise in intervention A or clinicians with expertise in intervention B and the clinicians perform only the procedure they are expert in.⁴ Trials of carotid endarterectomy performed by vascular surgeons vs stenting performed by interventional radiologists are examples of expertise-based RCTs.

Observational studies are unlikely to provide the same degree of protection against bias in ascertainment of the efficacy of interventions compared with high-quality randomized trials. Their quality can be improved, however, when investigators include measures aimed at reducing bias. Some of these measures (parallel to those that enhance the quality of randomized trials) include selecting adequate populations, making them appropriately comparable by means of matching or statistical adjustment by key predictors of outcome, ensuring adequate ascertainment of the exposures and outcomes at baseline, planning long enough follow-up to allow time for critical outcomes to develop, and blinding the assessment of outcomes in both groups.⁵, ⁶

Hence, in developing clinical practice guidelines for the Society for Vascular Surgery, we considered evidence generated by both study designs: RCTs and observational studies that included concurrent comparison cohorts. In addition, it is important to recognize the inherent limitations of both designs when applied to surgical interventions in that surgical outcomes are often influenced by other factors such as the surgeons' learning curve and experience, surgical volume, institution characteristics, and the quality of perioperative care.⁴ These factors are difficult to assess and are often not reported in the published literature.

Back to Article Outline

The role of evidence synthesis in formulating high quality practice guidelines 

The ideal evidence-based recommendation should consider all pertinent evidence. However, with >1800 new citations indexed in MEDLINE each day and >16 million citations accrued to date,⁷ it is unrealistic to expect that one or a few experts would be aware of the totality of evidence about a particular subject. Furthermore, if experts choose the evidence they bring to bear in formulating recommendations without explicit and reproducible criteria, they run the risk of reviewing an incomplete and biased sample of the available research. Thus, in pursuit of finding the best evidence for guideline development, a rigorous systematic review of the literature is paramount. In contrast to traditional literature reviews, which are nonexhaustive, unsystematic, and can be biased, systematic reviews address a focused clinical question using methods designed to reduce the likelihood of bias in the identification, selection, critical appraisal, description, and summary of the totality of the relevant literature. Compared with the primary studies they seek to summarize, systematic reviews offer greater precision in estimating treatment effects, particularly when these reviews include well-conducted quantitative summary of the results (ie, meta-analysis) and provide results that are applicable to a wider range of patients.⁸

The process of conducting systematic reviews starts by defining the clinical question in terms of the population, intervention, comparison, and outcomes. Librarians with expertise in systematic reviews conduct a sensitive—but not very specific—search of relevant bibliographic databases. Reviewers, working independently and in pairs, appraise studies with predetermined criteria for inclusion and exclusion, assess the quality of studies, and abstract relevant data. Results are summarized qualitatively or quantitatively. To ensure high-quality reviews, we monitor and report the agreement among reviewers by using agreement statistics (eg, κ statistic), contact the authors of primary studies to ensure correct representation of their results and to ask for missing data, explore heterogeneity in the results by conducting subgroup and sensitivity analyses, and assess the potential impact of publication bias on review results (Table I).

Table I. Conducting a systematic review

Define the question

•Specify inclusion and exclusion criteria Population Intervention or exposure Outcome Methodology

•Establish a priori hypotheses to explain heterogeneity

•Conduct literature search

•Decide on information sources: databases, experts, funding agencies, pharmaceutical, companies, hand-searching, personal files, registries, citation lists of retrieved articles •Determine restrictions: time frame, unpublished data, language •Identify titles and abstracts

Apply inclusion and exclusion criteria

•Apply inclusion and exclusion criteria to titles and abstracts •Obtain full articles for eligible titles and abstracts •Apply inclusion and exclusion criteria to full articles •Select final eligible articles •Assess agreement on study selection

Extract data

•Data abstraction: participants, interventions, comparison interventions, study design •Results •Methodologic quality •Assess agreement on validity assessment

Conduct analysis

•Determine method for pooling of results •Pool results (if appropriate) •Decide on handling missing data •Explore heterogeneity, sensitivity, and subgroup analysis •Explore possibility of publications bias

Back to Article Outline

From evidence to practice: formulating clinical recommendations 

In using evidence to formulate recommendations, it is helpful for the user of such recommendations to understand how valid, precise, and applicable is the available body of evidence supporting a recommendation (ie, the quality of the evidence). High-quality evidence (ie, estimates of effect for safety and efficacy that are unlikely to change substantially as new research accumulates) provides greater confidence in formulating recommendations. Similarly, it is helpful for the user of practice guidelines to note whether a recommendation is strong (ie, “must do,” “must not do”) or weak (ie, “may do,” “may not do”).

Multiple systems have been used to rate the quality of evidence and grade the strength of clinical recommendations. Thankfully for clinicians and other decision makers, there is an emerging consensus among professional organizations toward using one system, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system, for this purpose.⁹, ¹⁰ The GRADE system classifies recommendations as strong (grade 1) or weak (grade 2) and the quality of the evidence (ie, risk of bias) into one of four categories (high, moderate, low, or very low). To further enhance the interpretation and clarity of the recommendations, guideline developers use the terms “we recommend” to denote strong recommendations, and the less definitive wording “we suggest” to denote weak recommendations.

When guideline developers are confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects, they will make a strong recommendation within the context of a described intervention. Typically, this requires high- or moderate-quality evidence on patient-important outcomes, but on occasion, a strong recommendation can be based on lower-quality evidence. This may occur when the values and preferences that guideline developers bring to bear are such that when considering even low-quality evidence, they are confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects.

A weak recommendation is one for which a guideline panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but the panel is not confident. Thus, if guideline developers believe that benefits and downsides are finely balanced, or appreciable uncertainty exists about this balance, they offer a weak recommendation. Overall, low- or very low-quality evidence usually leads to weak recommendations because of uncertainty about the balance between risks and benefits. Guideline panels may offer weak recommendations even when high-quality evidence is available because that evidence clearly demonstrates that the benefits and risks are closely balanced.

As described above, study design and conduct are important determinants of the quality of evidence (Table II).¹¹ RCTs allow decision makers to draw causal inferences linking interventions and outcomes with protection against bias. Therefore, RCTs begin with a “high” quality rating, whereas observational studies start with a “low” quality rating. This rating can be downgraded when:

Table II. Grading of Recommendations Assessment, Development and Evaluation guidelines

Rating of evidence quality	Clarity of risk/benefit	Description of supporting evidence	Implications
Strong recommendations
High-quality evidence	Benefits clearly outweigh harms and burdens, or vice versa	Consistent evidence from well-performed randomized controlled trials or exceptionally strong evidence from unbiased observational studiesa	Recommendation can apply to most patients in most circumstances. Further research is very unlikely to change our confidence in the estimate of effect.
Moderate-quality evidence	Benefits clearly outweigh harms and burdens, or vice versa	Evidence from randomized controlled trials with important limitations (inconsistent results, methodologic flaws, indirect or imprecise evidence), or unusually strong evidence from unbiased observational studies	Recommendation can apply to most patients in most circumstances. Further research (if performed) is likely to have an impact on our confidence in the estimate of effect and may change the estimate.
Low-quality evidence	Benefits clearly outweigh harms and burdens, or vice versa	Evidence for at least one critical outcome from observational studies, from randomized controlled trials with serious flaws, or indirect evidence	Recommendation may change when higher quality evidence becomes available. Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.
Very low-quality evidence (very rarely applicable)	Benefits clearly outweigh harms and burdens, or vice versa	Evidence for at least one of the critical outcomes from unsystematic clinical observations or very indirect evidence	Recommendation may change when higher quality evidence becomes available; any estimate of effect for at least one critical outcome, is very uncertain.
Weak recommendations
High-quality evidence	Benefits closely balanced with harms and burdens	Consistent evidence from well-performed randomized controlled trials or exceptionally strong evidence from unbiased observational studies.	The best action may differ depending on circumstances or patient or societal values. Further research is very unlikely to change our confidence in the estimate of effect.
Moderate-quality evidence	Benefits closely balanced with harms and burdens	Evidence from randomized controlled trials with important limitations (inconsistent results, methodologic flaws, indirect or imprecise evidence), or unusually strong evidence from unbiased observational studies	Alternative approaches likely to be better for some patients under some circumstances. Further research (if performed) is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.
Low-quality evidence	Uncertainty in the estimates of benefits, harms, and burdens; benefits may be closely balanced with harms and burdens	Evidence for at least one critical outcome from observational studies, from randomized controlled trials with serious flaws, or indirect evidence	Other alternatives may be equally reasonable. Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.
Very low-quality evidence	Major uncertainty in the estimates of benefits, harms, and burdens; benefits may or may not be balanced with harms and burdens	Evidence for at least one critical outcome from unsystematic clinical observations or very indirect evidence.	Other alternatives may be equally reasonable. Any estimate of effect, for at least one critical outcome, is very uncertain.

Modified from Schunemann HJ et al.¹¹

For example, when evaluating the effectiveness of access surveillance on the survival of the vascular access of hemodialysis we note that (1) there was important inconsistency between study results, with 60% of the observed variability being attributed to true differences in the results across trials rather than to chance, (2) some RCTs reported only on the effect of the interventions on access thrombosis, which is a surrogate of the more important outcome, access loss or abandonment, and (3) only six of 12 studies evaluated access failure with a total of only 94 events (ie, accesses lost) in the surveillance group vs 88 events in the no surveillance group, which produced a wide confidence interval. Therefore, the overall clinical trial evidence was downgraded due to inconsistency, indirectness and imprecision, respectively; that is, from high-quality evidence to very low-quality evidence.¹

On the other hand, the quality of evidence of observational studies can be upgraded if (1) a very large treatment effect was observed, (2) all plausible confounders would reduce the magnitude of the treatment effect, yet the effect remains sizable, and (3) a dose-response gradient was noted. An example of evidence that was upgraded comes from the comparison of autogenous access with prosthetic access for hemodialysis. We noted that placing an autogenous access produced a large effect in decreasing the risk of access infection (relative risk, 0.22; or five times reduction of risk), which led to upgrading the quality of the evidence.¹²

Finally, the GRADE system offers insights into the role of values and preferences when it disentangles the strength of recommendations from the quality of the evidence and when it encourages statements about the underlying values and preferences relevant to the recommendations. For example, the vascular access committee issued a strong recommendation for using distal arm autogenous access as a first-line access despite very low-quality evidence. The evidence for this recommendation was considered to have very low quality because most of the studies were nonrandomized (80 of 83), the study cohorts were prognostically imbalanced at baseline, outcome assessors were mostly unblinded, apparent reporting bias was present, and results were inconsistent among studies to the extent that the proportion of heterogeneity that is not attributable to chance in most outcomes >50% and often >90%.

These limitations in the quality of evidence notwithstanding, the committee issued a strong recommendation because they placed higher value on optimizing patient-important outcomes such as preventing death, access infection, local complications, and achieving a longer period of time with successful dialysis; all were features thought to be more associated with the autogenous access. In recommending distal upper extremity access sites despite the very low quality evidence, they also placed high value on the preservation of proximal veins or future access placement. The committee placed lower value on some of the characteristics of the prosthetic access such as higher reimbursement for placement, ready off-the-shelf availability, and shorter time to first use for dialysis. Hence, the guideline authors made a strong recommendation when they assumed that the pertinent values and preferences across a broad range of informed patients are consistent with the recommended action.

Back to Article Outline

The future 

The Society for Vascular Surgery belongs to a select but expanding group of professional organizations that has endorsed an approach that is the state-of-the-art in guideline formulation and grading. This position is not without challenges. The GRADE Working Group (www.gradeworkinggroup.org) is identifying the best approaches to the grading of diagnostic recommendations. There is considerable uncertainty on how to incorporate considerations about resource utilization and societal priorities in health care. Also, mechanisms to incorporate patient preferences into guideline formulation are in their infancy.

Other challenges for the Society involve the use of the guidelines to inform policies in support of quality improvement efforts. For example, it is plausible that strongly recommended procedures could lead to process-of-care parameters that can be part of quality assessment and incentive schemes. Weak recommendations, on the other hand, may indicate areas not ready for quality improvement. How the Society, and other organizations, can move from guideline formulation to practice implementation represents yet another frontier in this field.

In future articles, we will keep the vascular surgery community abreast of developments in the methodology of guideline development while we continue to improve on the procedures we have began to use in response to users' feedback.

Back to Article Outline

References 

1. Casey E, Murad MH, Rizvi A, Sidawy AN, Elamin MB, Flynn D, et al. Surveillance of arteriovenous hemodialysis access: a systematic review and meta-analysis. J Vasc Surg. 2008;48(Suppl):48S–54S
   • View In Article
2. Poolman RW, Petrisor BA, Marti RK, Kerkhoffs GM, Zlowodzki M, Bhandari M. Misconceptions about practicing evidence-based orthopedic surgery. Acta Orthopaedica. 2007;78:2–11
   • View In Article
   • MEDLINE
   • CrossRef
3. Wente MN, Seiler CM, Uhl W, Buchler MW. Perspectives of evidence-based surgery. Dig Surg. 2003;20:263–269
   • View In Article
   • MEDLINE
   • CrossRef
4. Devereaux PJ, Bhandari M, Clarke M, Montori VM, Cook DJ, Yusuf S, et al. Need for expertise based randomised controlled trials. [see comment] BMJ. 2005;330:88
   • View In Article
   • CrossRef
5. Levine M, Haslam D, Walter S, Cumming R, Lee H, Haines T. In:  Guyatt G,  Rennie D editor. Users' guides to the medical literature: a manual for evidence-based clinical practice. Chicago, Ill: AMA Press; 2002;p. 81–100
   • View In Article
6. Wells G, Shea B, O'Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. http://www.ohri.ca/programs/clinical_epidemiology/oxford.htmAccessed: Feb 15, 2007
   • View In Article
7. National Library of Medicine. Detailed indexing statistics: 1965-2006. http://www.nlm.nih.gov/bsd/index_stats_comp.htmlAccessed: Feb 15, 2007
   • View In Article
8. Oxman A, Guyatt G, Cook D, Montori V. Summarizing the evidence. In:  Guyatt G,  Rennie D editor. Users' guides to the medical literature: a manual for evidence-based clinical practice. Chicago, Ill: AMA Press; 2002;p. 155–173
   • View In Article
9. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. BMJ. 2004;328:1490
   • View In Article
   • CrossRef
10. GRADE Working Group. http://www.gradeworkinggroup.org/society/index.htmAccessed: Nov 1, 2007
    • View In Article
11. Schunemann H, Jaeschke R, Cook D, Bria W, El-Solh A, Ernst A, et al. ATS Documents Development and Implementation Committee An official ATS statement: grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. Am J Respir Crit Care Med. 2006;174:605–614
    • View In Article
    • MEDLINE
    • CrossRef
12. Murad MH, Elamin MB, Sidawy AN, Malaga G, Rizvi A, Flynn D, et al. Autogenous vs prosthetic vascular access for hemodialysis: a systematic review and meta-analysis. J Vasc Surg. 2008;48(suppl):34S–47S
    • View In Article