Objective scoring systems of medical risk: A clinical tool for selecting patients for open or endovascular abdominal aortic aneurysm repair

Methods 

The database at our university-affiliated medical center consists of prospectively collected information on all major vascular surgeries performed by one of four surgeons. It contains patient demographic data, intervention modality, outcome, and specific grading of a number of medical risk factors as defined by the Ad Hoc Committee on Reporting Standards of the Society for Vascular Surgery (SVS) (Appendix 1, online only).6 This database was retrospectively reviewed for elective open and endovascular aneurysm repairs performed from January 1999 to December of 2004. The primary outcome variable was mortality ≤30 days of an elective infrarenal aneurysm repair. The analysis was performed as intention to treat.

Scoring system 

Several scoring systems were applied to our patient database (Table I).

Table I. Scoring system formulas

	Scoring system	Formula
	Glasgow Aneurysm Score	Age + 7 if CAD + 10 if CVD + 14 if RF
	Leiden Score (raw)	(Age-70)/2.5 + 4 if female + 3 if prior MI + 8 if Hx of CHF + 8 if ECG evidence of ischemia + 12 if RF + 7 if COPD
	Leiden Score (% mortality calculation)	= 10/{1 + EXP [−({Leiden raw score)/10} − 6.14)]}
	Comorbidity Severity Score	4 × cardiac status + 2 × pulmonary status + 2 × renal status + hypertension status + age status (0 if <55, 1 if 55 to 69, 2 if 70 to 79, 3 if >80)

CAD, Coronary artery disease; CVD, cerebrovascular disease; RF, renal failure; MI, Myocardial infarction; Hx, history; CHF, congestive heart failure; ECG, electrocardiogram; COPD, chronic obstructive pulmonary disease; EXP, exponential.

Results 

During this 6-year period, 862 patients underwent repair: 558 had open repair and 304 had EVAR. Demographic information is listed in Table II. EVAR patients tended to be older (average 75 vs 71) and had a higher incidence of CHF or angina (52% vs 17%), any coronary disease (70% vs 56%), and COPD (45% vs 12%).

Table II. Patient demographics

		Open repair N (%)	EVAR No (%)
	Patients	558	304
	Female	105(18.8)	42(13.8)
	Asymptomatic CAD⁎	220(39.4)	55(18.1)
	Compensated CHF/stable angina⁎	95(17)	159(52.3)
	COPD⁎	65(11.6)	136(44.7)
	CRF	11(2)	7(2.3)
	CVD	12(2.2)	9(3)
	Hypertension	480(86)	261(85.9)
	Average age (years)⁎	71	75
	Deaths	26(4.7)	6(2)

EVAR, Endovascular aneurysm repair; CAD, coronary artery disease; CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; CRF, chronic renal failure; CVD, cerebrovascular disease.

⁎ Significant.

Univariate analysis of association of individual risk factors with mortality risk in the open repair data set identified age, grade I CAD (asymptomatic heart disease), grade II CAD (compensated CHF or stable angina), and male gender as statistically significant risks (Table III). Four patients crossed over from the EVAR group to open repair, all within the first 2 years. Success of EVAR was 300 (98.7%) in 304 patients. None died within the perioperative period for either intervention. No patients were converted from open repair to EVAR.

Table III. Univariate analysis of comorbidities

	Risk factor	RR (95% CI)	P
	Male sex	5.8(1.03-33.8)	<.05
	Asymptomatic CAD	3.2(1.29-8.24)	<.025
	CHF/angina	3.04(1.44-6.38)	<.01
	COPD	1.40(0.01-3.41)	NS
	CRF	2.1(0.57-7.00)	NS
	CVD	0.81(0.14-4.25)	NS
	Hypertension	0.71(0.29-1.80)	NS
	Age >70	5.5(1.75-17.5)	<.01

RR, Relative risk; CI, confidence interval; CAD, coronary artery disease; CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; NS, not significant; CRF, chronic renal failure; CVD, cerebrovascular disease.

Application of the scoring systems to our open repair population showed that all three scoring systems were valid. Quartiles of increased score were associated with significantly increased risk of perioperative mortality (Fig, Table IV). Logistic regression analysis also revealed a highly significant association between death and increased score for each of the scoring systems (P < .001 for GAS, P < .001 for M-LS, and P < .0001 for M-CSS). Hosmer-Lemeshow goodness-of-fit testing supported this association for each of the three scoring systems.

View Large Image Download to PowerPoint

Fig. Open aneurysm repair mortality with quartiles of risk by scoring system. GAS, Glasgow Aneurysm Score; M-LS, Modified Leiden Score; M-SVS, Modified Society of Vascular Surgery Comorbidity Severity Score.

Table IV. Comparison between open repair and endovascular repair outcomes using open repair defined quartiles

	Score quartiles	Open mortality (%)	EVAR mortality (%)	RR (95% CI)	P
	Glasgow Aneurysm Score
	<70	0.70	0	N/A	NS
	70-77	2.80	3.30	0.85(0.19-3.92)	NS
	77-83	6.90	2.40	2.95(0.75-11.90)	NS
	>83	8.50	1.60	5.25(1.34-20.98)	.02
	Modified Leiden Score
	<1.8	0.70	0	N/A	NS
	1.8-5.0	2.10	0	N/A	NS
	5.2-8.4	6.70	0	N/A	NS
	>8.4	9.20	3.00	3.03(1.22-7.59)	.015
	Modified Comorbidity Severity Score
	0-3	0	0	N/A	NS
	4-6	2.80	0	N/A	NS
	7-8	4.30	0	N/A	NS
	>8	10.20	3.40	2.98(1.23-7.33)	.01

EVAR, Endovascular aneurysm repair; RR, Relative risk; CI, confidence interval; NA, not applicable.

Application of ROCs to the data revealed an area under the curve (C statistic) of 0.72 for GAS, 0.71 for M-LS, and 0.74 for the M-CSS (Table V). Cut points were also identified that allowed dichotomous definition of groups at a statistically different risk of perioperative mortality for each of the scoring systems (Table VI).

Table V. Cut-point analysis of open aneurysm repair and endovascular repair data sets

	Repair	Scoring system	C stat	Cut point	Sensitivity	Specificity
	Open	GAS	0.72	76.5	88.5	49.1
		M-LS	0.71	5.2	84.6	53.4
		M-CSS	0.74	8	57.7	75.4
	EVAR	GAS	0.47	70.9	100	13.1
		M-LS	0.7	11.8	83.3	57.9
		M-CSS	0.69	9	100	42.9

GAS, Glasgow Aneurysm Score; M-LS, Modified Leiden Score; M-CSS, Modified Comorbidity Severity Score.

Table VI. Comparison between open repair and endovascular repair outcomes using open repair defined cut points

	Score	Open mortality	EVAR mortality	RR (CI)	P
	Glasgow Aneurysm Score
	<76.5	1.1	2.2	0.52(0.10-2.57)	NS
	>76.5	7.8	1.9	4.17(1.54-11.45)	<.01⁎
	Modified Leiden Score
	<5.2	1.4	0	n/a	NS
	>5.2	8.1	2.5	3.29(1.40-7.81)	<.01⁎
	Modified Comorbidity Severity Score
	<76.5	2.7	0	n/a	NS
	>76.5	10.3	3.4	3.06(1.26-7.54)	.02⁎

EVAR, Endovascular aneurysm repair; RR, Relative risk; CI, confidence interval; NS, not significant.

⁎ Significance set at P < .05.

Application of the same scoring systems to our EVAR population revealed that EVAR patients were significantly higher-risk patients as defined by each of the scoring systems (average GAS, 81.0 vs 76.4; average M-LS, 10.7 vs 5.3; average M-CSS, 8.9 vs 6.9). When the quartiles of risk that had been defined for the open repair population were applied to the EVAR population, observed mortality was clearly different (Table IV). This difference, however, was only statistically significant for the highest risk quartiles (GAS >83, M-LS >8.6, M-CSS >9). Application of the cut points identified through open repair ROCs showed highly significant reductions in observed mortality for high-risk patients treated with EVAR (Table VI).

ROC curves were also created directly from the EVAR data set. Predictive ability (C statistic) for the scoring systems for mortality was lower for the EVAR population (Table V); however, the M-LS and the M-CSS appeared to generate some predictive ability. Cut points were 11.8 for the M-LS and 9 for the M-CSS in the EVAR population. Mortalities above and below these cut points were 0.6% and 3.9% (P < 0.05) for the M-LS, and 0% and 3.73% for the M-CSS (P < 0.025).

Discussion 

The vascular community has accepted endovascular treatment of AAAs as an option that trades uncertain long-term durability and increased overall cost for decreased perioperative surgical risk. However, currently no clear guidelines are available to objectively define when this trade-off is medically justified. Published randomized trials of EVAR vs open repair loosely defined “good risk” as a patient who could be considered “medically well enough for elective” surgery, but did not consider already available stratification systems for objectively defining risk with open repair.1 As a result, perioperative mortality in the open repair group closely resembles that seen with pooled data for open repair in all comers on a national or nonselected level.15 This situation leaves room for stricter definition.

The first step was to choose a stratification system. From a practical standpoint, scoring systems were chosen that used elements already recorded in our aneurysm database. The GAS was chosen because of its simplicity and its track record of validity.8, 9, 10 The Leiden scoring system, although fairing no better than the GAS in a comparative validation of multiple scoring systems,10 is compelling because of its increased weighting of major CAD in the form of CHF or ECG changes suggesting angina. Given that in clinical practice cardiac risk is the predominant cause of postoperative mortality and major morbidity and weighs considerably into our clinical decision-making against open repair, it was included in this analysis.

As outlined, our database did not allow for distinction between patients who had ECG changes of ischemia and those with compensated CHF. It is recognized that this modification will score patients with both CHF and ECG evidence of ischemia too low. This scoring system was initially developed with a complex algorithm to convert the score to an absolute predicted risk of death in percent open repair. However, we chose to report our data with raw Leiden scores because this allows for rapid calculation in a clinical setting and avoids mathematical dilution of results. Of interest was that applying the conversion factor to our data revealed remarkable congruity between expected and observed mortalities; for example, patients with an expected mortality of 2% to 5% by Leiden score actually had a mortality of 3.8% in our open repair population (Appendix 4, online only).

The SVS’s CSS was also applied to our data set. It was assessed because it is the only stratification system that was specifically intended for EVAR risk stratification. To our knowledge, this is the first attempt at validation of this scoring system.

Our quartile data for open repair show that all three scoring systems are highly valid in our patient population. The receiver-operator curves echo this finding. The area under the curve (C statistic) for each of these scoring systems compares well with that observed in other published validations.9 It is interesting that the M-CSS performed at least as well as the validated scoring systems for stratification of our open repair population (C statistic: 0.74 vs 0.72 for GAS and 0.71 for M-LS).

When each of the scoring systems is applied to our EVAR population, several points are worthy of note. For the highest quartile of risk with open repair, statistically significant reductions in mortality were clearly observed for EVAR with all scoring systems. For any given degree of medical risk, regardless of stratification system used, no patient was at statistically significantly higher risk with EVAR than with open repair.

These validated scoring systems offer an objective means to assess risk of death from open repair and EVAR for a given patient. As such, they may be used as tools for choosing between surgical options. This decision is facilitated by use of cut-point values that can be derived from the ROCs (Table IV). Patients with scores below the cut points for the open repair data have a very low risk of mortality with open repair. This is perhaps less true with use of the M-CSS (2.7% mortality vs 1.1% as defined by GAS and 1.4% with M-LS). Patients with scores above these cut points have a much higher mortality with open repair but a low mortality with EVAR overall: GAS >76.5 has open repair mortality of 7.8% vs 1.9% with EVAR, M-LS >5.2 has open repair mortality of 8.1% vs 2.5% with EVAR, and M-CSS has open repair mortality of 10.2% vs 3.4% with EVAR. All three of the systems are simple to calculate, but the GAS is certainly the most user friendly. Given that it cedes little to the other scoring systems in comparing open repair risk with EVAR risk, it would likely be the objective scoring system of choice.

Despite a small number of deaths in our EVAR group, both the M-LS and the M-CSS appear to have predictive ability with EVAR that is similar to that seen in some reports with open repair (C statistic is 0.70 for M-LS and 0.69 for M-CSS).9, 12 Mortalities with these stratification systems were only seen in patients in the highest stratum. In our EVAR patient population, the GAS showed poor stratification of risk of death (C statistic, 0.47). This is contrary to the observation of the European Collaborators on Stent-Graft Techniques for AAA and Thoracic Aortic Aneurysm and Dissection Repair (EUROSTAR) investigators, who found the GAS to be predictive of mortality when applied retrospectively to their far larger database (C statistic, 0.70).16 It is probable that our database is simply not large enough to corroborate the validity of the GAS in EVAR.

Consideration of patients who might be high risk for EVAR brings about a more complex algorithm. A recent analysis of the United States Investigational Device Exemption (US IDE) trials showed that patients with major comorbidities could undergo EVAR with a relatively low—but higher than baseline—mortality of 2.9%.17 The EVAR-2 trial has shown us that there is no overall mortality benefit of EVAR over observation in a fragile cohort, whose perioperative mortality with EVAR was 9%.5 The EUROSTAR database identified tertiles of risk with EVAR using the GAS.16 This literature suggests that AAA patients might be stratified by medical comorbidity into groups that are low, moderate, and prohibitive risk for EVAR.

Of the tested scoring systems, the M-LS and the M-CSS both support risk discrimination with EVAR in our database. Using the cut point of 11.8, low-risk patients have a predicted mortality of 0.6%, whereas high-risk patients have a mortality of 3.8%. The M-CSS defines similar groups: mortality rates at a cut point of 9 are 0% for low risk and 3.7% for high risk.

Patients defined at high risk for EVAR in our population still had a perioperative mortality of <4%. This fits more with the US IDE high-risk cohort than with the EVAR-2 population. As a result, although statistically significant, our definition of a high-risk group for EVAR may not be clinically meaningful: we cannot use these scoring systems to define who might have a predictably poor outcome from EVAR.

This study’s findings are limited by several factors. As a retrospective study, objective definition of medical risk was not assigned before operation. It is possible that poor anatomy for EVAR is an important confounder in poor outcome for open repair. A clear scenario is the juxtarenal aneurysm, which has been shown to have poorer outcome.18 Although our database did not document clamping above the renal arteries, the presence of a juxtarenal aneurysm was noted with which clamping tended to be suprarenal. Together these comprised 34 patients, of which two died in the perioperative period (5.8% mortality).

We do not believe that these potential anatomic differences invalidate the relationship between medical comorbidity and outcome with method of repair. If we consider the lowest medical risk subgroup, it is hard to conceive how poor anatomy for EVAR confounds toward improved outcome with open repair. Anatomic exclusion from EVAR is more of a concern in the high-risk open repair outcome data. However even here, comorbidity—not anatomy—likely played an overriding role. An important step toward supporting this impression would be to retrospectively apply one of the scoring systems to the patient data from the large randomized trials because all patients enrolled were anatomic EVAR candidates.

Another limitation is the relatively small number of deaths in the EVAR group as a whole. As a result, regression analysis in this data set could not confirm the association between score and mortality risk. The small number of deaths may predispose to a type 2 error in data interpretation. As a consequence, although it appears that the GAS has no predictive ability in the EVAR data set, this is not conclusively proven by this study. Indeed, given the results of the EUROSTAR analysis,16 our lack of findings most likely reflect inadequate power. Conversely, there may be a role for cautious interpretation of the positive results of the ROC analysis with the EVAR data for the M-LS and the M-CSS. Overall, our data set suggests a utility of these scores for defining operative risk with EVAR, but application to a larger data set may allow for clarification of groups that are at low, moderate, and prohibitive risk for aneurysm repair.

Another possible concern is the large proportion of aortouniiliac grafts in this series. This is not a reflection of anatomy that is not amenable to bifurcated graft placement in most cases; rather, it has largely come about as a result of physician preference. In our experience, aortouniiliac endografts are simpler to plan and have fewer possibilities for limb disconnect, and the femoral crossover confers no increase in thrombosis risk.

Conclusion 

Taking all patients together, a cursory assessment of the EVAR-1 trial seemed to suggest that both open repair and EVAR are equivalent options viewed from the standpoint of 4-year follow-up. This report is a first attempt at dispelling this simplification. It is initial support for our hypotheses that criteria can be defined by which a subgroup of patients who are high risk for open aneurysm repair—but low risk for EVAR—can be objectively defined, and conversely, that another subgroup for whom EVAR confers no major mortality advantage over open repair, even in the short term, can also be identified. Given the retrospective nature of this analysis, our study cannot conclusively prove these hypotheses; however, it aims to set some objective parameters that would be required of a prospective trial that may. In an era where medical cost containment is evermore important, defining populations that might truly benefit from more advanced, expensive, and less proven technology is a necessity.

Author contributions