Challenges in analysis and interpretation of cost data in vascular surgery

Article Outline

• Abstract
• Methods
  • Review
  • Data analysis
• Results
  • Review of studies comparing cost of OR and EVAR for AAA
  • Effect of statistical methodology on interpretation of cost data: Analysis of data from a population-based study
• Discussion
• Conclusions
• Author contributions
• References
• Copyright

Objective

Health economic arguments have become increasingly important in clinical decision making, especially when new treatment modalities are introduced. This study reviews the methods used in health economic reports of abdominal aortic aneurysm (AAA) repair and uses original cost data to study how different methods affect interpretation of results in terms of cost differences and economic efficiency.

Design

Publications referenced in PubMed from 2003 to 2008 studying cost of AAA repair were reviewed. Original population-based cost data of AAA repair were analyzed, comparing open (OR) and endovascular repair (EVAR). Means, medians, and cost distributions were calculated, and differences were analyzed with four different statistical methods.

Results

The review showed a mixture of statistical methods used in AAA treatment cost-comparison studies. Presentation of cost data and inclusion criteria varied between studies. The analysis of original data showed skewed distribution of cost data, with large differences between mean and median cost. Although mean values indicated a lower total, perioperative, and postoperative cost for EVAR, the median values indicated OR was the least costly method. Exclusion of extreme values lowered mean perioperative cost of OR by 10%, while cost of EVAR was unaffected. Inferential testing of cost differences by means of four statistical methods showed that P values were highly dependent on test methodology.

Conclusions

Conclusions of health economic reports can be highly dependent on how the data are presented and the statistical methods that are used. We recommend that cost data be presented as mean values with distributions. Exclusion of outliers and focus on P values should be avoided.

Health economics has become increasingly important in health care decision making during the past decades. In some countries, health economic evaluation of new medications and treatment strategies is mandatory before these are reimbursed within the health care system.¹, ² In vascular surgery, discussions on the health economic benefits or burdens of new endovascular techniques are common, and the number of publications in PubMed on cost for vascular surgical procedures has quadrupled since 1990.

Health economic data pose specific challenges to the researcher. Cost data are almost always skewed, making it difficult to use the usual statistical methods to analyze cost differences between alternative treatment strategies.³ The median cost disregards the skewedness of data and thus underestimates the effect of seldom but regularly occurring cost-intensive cases and their effect on the total resources that are needed over time (Fig 1). Thus, in contrast to other medical research areas where skewed data are best presented with median values, mean cost is a more relevant value to the economic decision maker in a budgeting situation.

• 
  • View Large Image
  • Download to PowerPoint

• Fig 1. 

  Schematic histograms illustrate the relationship of mean and median values to the total in (A) a normal and (B) a skewed distribution.

Several inferential statistical methods exist for comparing the costs of different treatment strategies, all having their pros and cons.³, ⁴, ⁵, ⁶, ⁷ Although several publications recommend a presentation of overall mean cost in health economic evaluations, statistically comparing data with parametric t test or bootstrap technique,³, ⁴ there is no consistency in how cost data are analyzed and reported in the literature. It is common practice in health economic studies to test the sensitivity of cost-effectiveness calculations for variables such as cost and morbidity; however, we have not encountered any study that tests how presentation of data and statistical methods affect interpretation of results.

The aim of this study was to review the methods used in studies evaluating the cost of abdominal aortic aneurysm (AAA) treatment with open (OR) and endovascular repair (EVAR) and to compare how different methods for presentation and analysis of cost data affect the interpretation of results in terms of cost differences and economic efficiency. To visualize these differences, original data from a previously published study⁸ that compared the cost of OR with the cost of EVAR to repair AAA were analyzed with different statistical techniques encountered in published reports.

Back to Article Outline

Methods 

Review 

The PubMed database was searched for articles published between January 1, 2003, and December 31, 2008. The Medical Subject Heading (MeSH) criteria were used to identify articles with MeSH code “aortic aneurysm, abdominal/economics.” To focus on publications concerning the cost of AAA repair, articles regarding cost of screening (with MeSH code “mass screening”) were excluded. The 37 articles that remained were reviewed by the primary author (K. M.). All articles that included a comparison of the cost of AAA repair between OR and EVAR were scrutinized to identify how cost data had been analyzed statistically and what data were presented.

Data analysis 

To analyze the effect of statistical methodology on the interpretation of the results, data from a population-based study of the cost of AAA repair with OR and EVAR⁸ were used. The study, which aimed to analyze cost differences between elective OR and EVAR in a population-based setting, included 109 patients who underwent elective AAA repair during a 5-year period in Uppsala County in Sweden, comprising 58 with OR and 51 with EVAR. Cost data on preoperative, perioperative, and postoperative care, including a mean 2.5 years of follow-up, were gathered for all patients.

Distribution of the cost data was displayed in box-and-whisker plots as well as histograms of the original data, after logarithmic transformation of data and after bootstrap simulation of the mean value.⁹ Medians, means, and confidence intervals were calculated for patients who had undergone OR and EVAR. The two groups were compared statistically using four different methods (Table I; Fig 2), as suggested by the literature review (Table II).

Table I. Description of statistical methods

• 
  • View Large Image
  • Download to PowerPoint

• Fig 2. 

  Schematic presentation shows the bootstrap simulation technique.

Table II. Publications comparing cost of abdominal aortic aneurysm repair with open and endovascular technique between 2003 and 2008, and methods for presentation and statistical analysis of cost data

First author, year	Presentation of data		Statistical method					Significance
	Averages	Distribution	t Test	MW U test	Bootstrap	LT	None
Tarride,11 2008	Mean	___			X			At5%level
Lesperance,12 2008	Median	IQR		X				P
Mani,8 2008	Mean	Range, SD, histogram			X			P
Hynes,13 2007	Mean	—					X	___
Prinssen,14 2007	Mean	95% CI			Xa			___
Visser,15 2006	Mean	Range, 95% CI, histogram		Xb	Xb	Xb		P
EVAR trial,16 2005	Mean	Standard error			Xc			___
Hayter,17 2005	Mean excluding extreme casesd	___	X					P
Rosenberg,18 2005	Mean excluding extreme casesd	SEM	X					P
Watson,19 2004	Mean	___	X					___
Dryjski,20 2003	Mean	___					X	___

CI, Confidence interval; IQR, interquartile range; LT, log transformation; MW, Mann-Whitney; SD, standard deviation; SEM, standard error of the mean.

Statistical evaluation of the data was achieved with computer software packages Stata 9 (StataCorp LP, College Station, TX) for bootstrap analysis and SPSS 14.0 (SPSS, Chicago, IL) for all other statistical analysis. All costs are presented in Euro 2006 values (€1 = US$ 1.3).

Back to Article Outline

Results 

Review of studies comparing cost of OR and EVAR for AAA 

We identified 11 articles⁸, ¹¹, ¹², ¹³, ¹⁴, ¹⁵, ¹⁶, ¹⁷, ¹⁸, ¹⁹, ²⁰ that assessed the cost of EVAR for AAA treatment compared with OR (Table II). One presented median cost data,¹² and 10 presented mean cost data.⁸, ¹¹, ¹³, ¹⁴, ¹⁵, ¹⁶, ¹⁷, ¹⁸, ¹⁹, ²⁰ Skewedness of cost data and its effect on statistical analysis was discussed in three reports,⁸, ¹⁴, ¹⁵ and two articles excluded cases with extreme cost.¹⁷, ¹⁸ Three studies discussed the cost of specific subgroups, such as patients with complications vs patients without complications, and their effect on total cost,¹¹, ¹⁷, ²⁰ but only one study specifically discussed the effect of outlier values on the results.¹⁸ Distribution of cost data was visualized with histograms in two reports⁸, ¹⁵ and with cost-effect plots in two reports.¹¹, ¹⁴ A variety of methods were used for inferential statistics (Table II). Only one study analyzed differences in cost of treatment between OR and EVAR with several separate statistical methodologies,¹⁵ but there was no discussion on the effect of the statistical methodology on results of cost analysis.

Effect of statistical methodology on interpretation of cost data: Analysis of data from a population-based study 

All original cost data for AAA repair were highly skewed, with numerous extreme values, as visualized in box-and-whisker plots and histograms (Fig 3; Table III). Histograms of cost data after logarithmic transformation and bootstrap simulation are presented in Fig 4. Median cost was consistently lower than mean cost for preoperative, perioperative, and postoperative cost for OR and EVAR (Table III). Mean and median values pointed in different directions in terms of what treatment strategy was least costly in three of four parameters: median cost was lower for OR compared with EVAR in parameters of total cost, perioperative cost, and postoperative cost; whereas, the mean value was lower for EVAR in same parameters.

• 
  • View Large Image
  • Download to PowerPoint

• Fig 3. 

  Box-and-whisker plots and histograms of the cost of abdominal aortic aneurysm repair with open repair (OR) and endovascular repair (EVAR) using a population-based study⁸. (A) total, (B) preoperative, (C) perioperative, and (D) postoperative cost. In the box-and-whisker plots (left in each figure), the box indicates the interquartile range (IQR; left line is the lower quartile [Q1], middle line is the median, right line is the upper quartile [Q3]); the whiskers indicate the smallest and largest non-outlier observations within 1.5 IQR below Q1 or above Q3. The black circles (•) indicate mild outlier (1.5-3 IQR below Q1 or above Q3). The asterisk (*) shows an extreme outlier (>3 IQR below Q1 or above Q3). In the histograms (right in each figure) staples indicate the number of observations (N) in a specific cost interval. The full line indicates the mean; dashed lines indicate the 95% confidence interval. N, Number of observations in each interval.

Table III. Cost of open and endovascular treatment for elective AAA repair in Euros (€1.00 = US $ 1.30)

Cost	OR, €	EVAR, €	t Test		Bootstrap	MW U
			Mean	Log mean
	(n = 58)	(n = 51)	P	P	P	P
Total cost
Mean	29,786	26,382	.347	.877	.336	.180
Median	19,876	22,183
95% CI
Arithmetic	23,353-36,218	23,140-29,624
Bootstrap	24,648-37,139	23,676-30,017
Pre-op cost
Mean	661	1494	.006a	.003a	.002a	.001a
Median	400	1114
95% CI
Arithmetic	363-959	984-2003
Bootstrap	449-1041	1078-2087
Peri-op cost
Mean	24,512	20,484	.172	.442	.135	.524
Median	17,411	18,366
95% CI
Arithmetic	19,041-29,983	18,418-22,550
Bootstrap	20,434-31,127	18,716-22,634
Post-op cost
Mean	4613	4403	.901	<.001a	.209	<.001a
Median	308	2588
95% CI
Arithmetic	1826-7399	2546-6262
Bootstrap	2340-7795	2940-6556

CI, Confidence interval; EVAR, endovascular aneurysm repair; MW, Mann-Whitney; OR, open repair.

• 
  • View Large Image
  • Download to PowerPoint

• Fig 4. 

  Histograms show the total cost of abdominal aortic aneurysm repair based on a population-based study.⁸ Distribution of original cost data, logarithm of cost data, and bootstrap simulation of mean cost are presented. N, Number of observations in each interval.

To test for the effect of extreme cost values in the data set, mean values were calculated when excluding the only patient with a mean total cost of treatment >3 standard deviations above the overall average. The patient was a 73-year-old man with a 50-mm infrarenal aneurysm who underwent OR. The extremely high cost resulted from perioperative problems, postoperative hemorrhage, and very long intensive care period. Exclusion of this patient lowered mean total cost of OR by 8% to €27,525 and the mean perioperative cost for OR by 10% to €22,161.

Results of inferential statistics with four different methods for these parameters are presented in Table III. The values for P were homogenous irrespective of test method for the parameter of preoperative cost. Large variations in P values occurred, depending on test methodology, for the other three parameters, and in the parameter postoperative cost, two test methods indicated a highly significant cost difference between OR and EVAR, with P < .001; whereas, two other methods did not indicate any significance at all, with P = .901 and P = .209.

Back to Article Outline

Discussion 

The present report visualizes the challenges in the interpretation of health economic cost data. Conclusions drawn from cost analysis are highly dependent on how the data are presented and the statistical methodology used to test for inferences. This is evident when the various statistical methods encountered in the current literature are applied on one data set.

Overall, cost data for AAA repair were skewed, and important differences in mean and median values were observed in most cases, with median values consistently lower than mean values. Medians are often accepted in medical research when a data set is small, not normally distributed, and is distorted by a few samples with extremely high values. For the economic decision maker, however, median cost would not be useful. For example, a budget for a clinic performing 100 elective open AAA repairs annually based on median cost would result in an underestimation of the budget by 30% in the current study example. Thus, to predict the overall cost, the most relevant value is the arithmetic mean. Cost comparison studies should present the distribution of the data as thoroughly as possible, for example, with means and confidence intervals (Table III). Furthermore, to allow the reader to appraise the whole data set— including outliers— data can preferably be visualized in histograms or box-and-whisker plots (Fig 3).

The management of outliers in cost studies is complex. The review indicated two studies where extreme cases were excluded by cost value alone.¹⁷, ¹⁸ In the present report, exclusion of one case with extreme cost due to complications resulted in an 8% to 10% lower cost for the OR group. Unexpected clinical complications happen with certain regularity; therefore, such cases should probably be included in cost calculations. Most of the cases that were excluded because of extreme cost in the reviewed studies¹⁷, ¹⁸ were from the OR group. It could be assumed that OR is more prone to complications requiring reoperations and costly intensive care, thus producing a higher rate of cases with extreme cost. Exclusion of the complicated cases from cost calculations for AAA treatment would in that case erroneously favor OR. It should, however, be noted that the effect of extreme cases may be over estimated in studies with small samples. In addition, outliers in nonrandomized settings may be clustered in one treatment group due to patient selection, for example, selection of patients with complex aneurysm anatomy to OR rather than EVAR.⁸ Direct comparison of cost is hazardous in such situations.

No optimal technique exists for inferential testing of cost data,³ and the cost comparison in this article underlines the risks of excessive use of P values in health economics. As a matter of fact, P values tend to be more dependent on the statistical method used than on the underlying cost estimates they are testing (Table III). We tested the differences in cost between OR and EVAR with four established statistical methods (Table I). All of these techniques have previously been used in the cost comparison studies on AAA repair identified in the review (Table II).

One study reported mean values for cost but tested the differences in mean cost with the nonparametric Mann-Whitney U test.¹⁵ As shown in the analysis of postoperative cost in the present study, this can be very misleading (Table III). The Mann-Whitney U test indicated a significant difference in cost of postoperative care between OR and EVAR, although the mean values were practically identical and the median values were highly different in the opposite direction. Being free of assumptions about the distribution of the data, the Mann-Whitney U test is more representative when presented with medians rather than means and thus is less applicable to cost comparison studies.

Although transforming cost data to a logarithmic scale made the distribution more normalized (Fig 4) and thereby allowed use of the t test, interpretation of the results based on transformed data is complex. Furthermore, differences in logarithmic cost data are of little relevance to the health economist. Differences in mean cost may be analyzed with t test and bootstrap. The t test is generally not regarded as appropriate for skewed data unless the sample size is large. For cost data specifically, Thompson et al³ argued that the t test may be a relevant estimation of P value and that this should be verified with bootstrap. P values calculated with these two methods are, however, not always overlapping (Table III). The two randomized trials included in the review both used bootstrap to compare costs.¹⁴, ¹⁶

From this discussion, it is obvious that focus on a specific P-value limit (eg, P < .05) can give a false sense of security in terms of assuming significant differences between two groups. P values depend on methodology, especially in skewed cost data. Their role is solely to estimate the risk of chance explaining the difference between two groups. P values do not reflect effect size, and even a highly significant P value can be irrelevant in practice if the quantitative difference it refers to is small. In this case, the €200 difference in postoperative cost between OR and EVAR can be regarded as irrelevant in relation to the overall treatment cost of €26,000 to €29,000, regardless of significance.

The clinical importance of a difference in cost is better presented with mean values and confidence intervals or with an indication of the difference in cost as a percentage of the total cost. Another often-encountered problem in cost studies is that they are based on patient data from clinical trials. Sample sizes in these studies are often based on clinical outcomes, while cost is a secondary outcome. A power calculation can be helpful to determine the cost-difference possible to detect within a clinical trial.

To abstain from inferential statistics is an alternative that is often disregarded in current medical research where there is still a strong focus on P values and statistical tests despite a well-underpinned notion that they cannot form the base of scientific inference and medical decision making.²¹ Presenting the original data and their distribution may give the reader a more accurate understanding of the similarities and differences in cost than focusing on P values alone. A problem, however, is that data presented without inferential statistics are often seen with skepticism in medical research and may result in difficulties in publishing the results. Even if P values are presented, a sounder interpretation may follow from playing down the importance of chance and focusing more on evaluating the validity of the study (ie, evaluating bias and confounding) and discuss the quantitative estimates and the confidence intervals.²¹

When choice of statistical methodology is not obvious, such as in this form of cost analysis, an alternative can be to sensitivity test the statistical significance of differences in cost with several inferential methods. This would be a complement to the current practice of sensitivity analysis in economic evaluations.

When data from the population-based study were interpreted according to this discussion, we could conclude that the total cost of AAA repair was similar for OR and EVAR when mean values and their distributions were compared (Table III; Fig 3).⁸ This finding was also robust when tested with several inferential statistical methods. Analysis of postoperative cost with different methods resulted in diverging results (Table III). However, the most relevant values (means and confidence intervals) and the most powerful test methods (bootstrap and t test) did not indicate any difference in postoperative cost between OR and EVAR.

Back to Article Outline

Conclusions 

Analysis and interpretation of cost data can be challenging. An understanding of the specific qualities of cost data and the economic perspective in which these data are used is important. Conclusions of health economic reports on the cost of treatment modalities can be highly dependent on how data are presented and the statistical methodology used. A high degree of openness when presenting cost data in medical reports is desirable to avoid misleading conclusions. Five principles for analysis of cost data are suggested in Table IV.

Table IV. Suggested principles for analysis of cost data

Back to Article Outline

Author contributions 

Back to Article Outline

References 

1. Jonsson B. Changing health environment: the challenge to demonstrate cost-effectiveness of new compounds. Pharmacoeconomics. 2004;22(suppl 4):5–10
   • View In Article
   • CrossRef
2. EUROMET 2004: The influence of economic evaluation studies on health care decision-making—a European survey. Amsterdam: IOS Press; 2005;
   • View In Article
3. Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed?. BMJ. 2000;320:1197–1200
   • View In Article
   • MEDLINE
   • CrossRef
4. Barber JA, Thompson SG. Analysis of cost data in randomized trials: an application of the non-parametric bootstrap. Stat Med. 2000;19:3219–3236
   • View In Article
   • MEDLINE
   • CrossRef
5. Thompson SG, Nixon RM. How sensitive are cost-effectiveness analyses to choice of parametric distributions?. Med Decis Making. 2005;25:416–423
   • View In Article
   • MEDLINE
   • CrossRef
6. Rascati KL, Smith MJ, Neilands T. Dealing with skewed data: an example using asthma-related costs of medicaid clients. Clin Ther. 2001;23:481–498
   • View In Article
   • Abstract
   • Full-Text PDF (990 KB)
   • CrossRef
7. Nixon RM, Thompson SG. Parametric modelling of cost data in medical studies. Stat Med. 2004;23:1311–1331
   • View In Article
   • MEDLINE
   • CrossRef
8. Mani K, Bjorck M, Lundkvist J, Wanhainen A. Similar cost for elective open and endovascular AAA repair in a population-based setting. J Endovasc Ther. 2008;15:1–11
   • View In Article
   • CrossRef
9. Efron B, Tibshirani R. An introduction to the bootstrap. Boca Raton: CRC Press; 1998;
   • View In Article
10. Altman DG. Practical statistics for medical research. 1st edition. London: Chapman & Hall/CRC; 1991;
    • View In Article
11. Tarride JE, Blackhouse G, De Rose G, Novick T, Bowen JM, Hopkins R, et al. Cost-effectiveness analysis of elective endovascular repair compared with open surgical repair of abdominal aortic aneurysms for patients at a high surgical risk: a 1-year patient-level analysis conducted in Ontario, Canada. J Vasc Surg. 2008;48:779–787
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (650 KB)
    • CrossRef
12. Lesperance K, Andersen C, Singh N, Starnes B, Martin MJ. Expanding use of emergency endovascular repair for ruptured abdominal aortic aneurysms: disparities in outcomes from a nationwide perspective. J Vasc Surg. 2008;47:1165–1167
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (223 KB)
    • CrossRef
13. Hynes N, Sultan S. A prospective clinical, economic, and quality-of-life analysis comparing endovascular aneurysm repair (EVAR), open repair, and best medical treatment in high-risk patients with abdominal aortic aneurysms suitable for EVAR: the Irish patient trial. J Endovasc Ther. 2007;14:763–776
    • View In Article
    • CrossRef
14. Prinssen M, Buskens E, de Jong SE, Buth J, Mackaay AJ, van Sambeek MR, et al. Cost-effectiveness of conventional and endovascular repair of abdominal aortic aneurysms: results of a randomized trial. J Vasc Surg. 2007;46:883–890
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (483 KB)
    • CrossRef
15. Visser JJ, van Sambeek MR, Hunink MG, Redekop WK, van Dijk LC, Hendriks JM, et al. Acute abdominal aortic aneurysms: cost analysis of endovascular repair and open surgery in hemodynamically stable patients with 1-year follow-up. Radiology. 2006;240:681–689
    • View In Article
    • MEDLINE
    • CrossRef
16. EVAR trial participants. Endovascular aneurysm repair versus open repair in patients with abdominal aortic aneurysm (EVAR trial 1): randomised controlled trial. Lancet. 2005;365:2179–2186
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (109 KB)
    • CrossRef
17. Hayter CL, Bradshaw SR, Allen RJ, Guduguntla M, Hardman DT. Follow-up costs increase the cost disparity between endovascular and open abdominal aortic aneurysm repair. J Vasc Surg. 2005;42:912–918
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (95 KB)
    • CrossRef
18. Rosenberg BL, Comstock MC, Butz DA, Taheri PA, Williams DM, Upchurch GR. Endovascular abdominal aortic aneurysm repair is more profitable than open repair based on contribution margin per day. Surgery. 2005;137:285–292
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (101 KB)
    • CrossRef
19. Watson DR, Tan J, Wiseman L, Ansel GM, Botti C, George B, et al. The clinical and fiscal impact of endovascular repair of abdominal aortic aneurysms. Heart Surg Forum. 2004;7:E503–E507
    • View In Article
    • CrossRef
20. Dryjski M, O'Brien-Irr MS, Hassett J. Hospital costs for endovascular and open repair of abdominal aortic aneurysm. J Am Coll Surg. 2003;197:64–70
    • View In Article
    • Full-Text PDF (109 KB)
    • CrossRef
21. Rothman K, Greenland S. Modern epidemiology. 2nd edition. Philadelphia: Lippincott-Raven; 1998;
    • View In Article