Potential problems with industry-supported clinical research☆☆☆★★★

Comparative groups: inclusion and exclusion criteria 

The outcome of a trial and its generalizability are greatly influenced by the inclusion and exclusion criteria. In the early studies of endografts designed for abdominal aortic aneurysm (AAA) repair, inclusion of only cases with favorable anatomy is a cautious approach and furthers one of the company's goals of demonstrating safety and successful deployment, but the results cannot be generalized to most patient with an AAA. Liberal inclusion criteria can have the same effect; examples include trials of laser and atherectomy devices, and stent studies that included patients with lesions that would have yielded good results with PTA alone, and trials of endografts for occlusive lesions for which good results have been established for PTA and stenting. After initial studies that demonstrate safety, each new advance in technology (and cost) should be tried on the marginal lesions that the simpler technology cannot satisfactorily address.

Blinding and randomization 

Blinding and randomization in prospective trials are obviously desirable but not always possible. Blinding is not feasible when the compared treatments are perceptibly different; unfortunately, this opens the door not only to investigator bias, but also to patient bias. In some of the AAA endograft trials, patients randomized to surgery backed out and went to another participating institution for another roll of the dice. Randomization may also present a problem if the individual investigator concludes that one treatment is clearly preferable. This raises the question, should that individual participate in the trial or continue entering patients? Freedman9 addressed this ethical challenge in clinical research by suggesting that clinical equipoise (ie, uncertainty) exists if the informed professional group (not an individual) has not reached a consensus and that it is ethical for the individual to enter patients into a trial that will give a definite answer to the question at hand. Recent debates over clinical trials on carotid angioplasty have repeatedly raised the issue of clinical equipoise.10, 11

It is important that the mechanism for randomization be predefined and independent of investigator and sponsor input. Further, validation of the suitability of the patient for inclusion in the trial must be efficient, and not too complex or time-consuming, so that the process does not interfere with recruiting a broad spectrum of cases. In trials of thrombolytic agents for acute arterial occlusion, those with acute limb-threatening ischemia were grossly underrepresented, in part because of the complexity of the study design and the time consumed by the randomization process.12 Another contributor to this was a natural selection bias in the group with acute ischemia because the treating physician could exclude a patient from randomization if the physician thought that patient safety was at risk.

End points 

The choice of end points of a trial is critical. What end point constitutes evidence of efficacy of treatment? Under what conditions will the trial be stopped? Some end points are desirable but unrealistic. In pharmacotherapy trials for critical limb ischemia, the Food and Drug Administration (FDA) has insisted that the primary end points include salvaging limbs and saving lives rather than using the useful and potentially achievable clinical goals of relieving rest pain and healing ischemic ulcers. This may have resulted in falsely negative studies. Similarly, in the case of the thrombolytic trials, restoring graft or artery patency for a reasonable period would be a more realistic end point, considering that the treatment of the underlying lesions, not the method of clot removal, determines long-term outcome. This too has contributed to negative results of trials.

Combining end points may add strength to the statistical analysis, decrease the number of cases required, and save costs, but this approach may reduce the generalizability. In the STILE trial,12 four clinical end points were combined into one clinical outcome, resulting in the premature stopping of the trial when two less important end points, major morbidity and ongoing ischemia, created a statistically significant difference in combined clinical outcome in favor of surgery, at a time when the more important end points of mortality and amputation were not significantly different. Again, the result was a negative trial for the tested therapy.

The appropriate selection of objective end points is essential but does not solve the problem of a trial that does not have a proper comparative control group. This is particularly true of treatments for severe limb ischemia, because patients may improve spontaneously with time rather than as the result of treatment (eg, sympathectomy, use of hyperbaric oxygen, spinal cord stimulation, and, up until now, VEGF).

Data analysis 

Unfortunately, it has become increasingly apparent that some presenters and primary authors of manuscripts at our national and regional meetings have reviewed only the summarized analysis of the data that was supplied by the company. They have not reviewed the raw data or performed or checked the statistical analyses and cannot supply the additional information requested by discussants or reviewers.

For proprietary reasons, device manufacturers and pharmaceutical companies rarely release the results of their own “in-house” research; there is also a perception that they may be selective about the data from clinical studies that they choose to release to their clinical investigators to make public. Although the company and its consultants are not likely to try to hide unfavorable data, by not releasing all the data to the investigators for analysis, there is the risk that significant relationships in the data set are not analyzed. Thus, the use of summary data provided by company officials can lead to problems.

There are many examples of the control of trial data by industry. The often-quoted European randomized prospective trial in which the benefit of primary stenting, with Palmaz stent, was compared with iliac percutaneous transluminal balloon angioplasty (PTA) alone, completed many years ago, is one example. The results, claiming statistically significant advantages for primary stenting in both “clinical success” and secondary patency at 5 years, were presented at several meetings and in abstract form in 1993,14 but the study has not yet been published as a peer-reviewed article. It is regrettable that the principal authors and the company have not persisted in achieving publication by providing the necessary data to satisfy peer review. Approval by the FDA has been obtained for this indication, and it is presumed that this study was among the data presented. With regulatory approval and marketing success achieved, the scientific aim of the study seems to have taken a backseat.

Recently, summary data were the sole basis for two abstracts submitted to major vascular societies. This is a risky but not uncommon practice. One paper had to be withdrawn from the 1998 American Venous Forum meeting because the company that sponsored the study did not reply to repeated requests to supply the necessary background information (the denominators) to put the study's findings in proper perspective in sufficient time to allow presentation.15 When the actual data were ultimately obtained, they did not match the summary data originally submitted to the investigators for preparation of the abstract. In presenting and publishing a recent analysis of the cost-effectiveness of an aortic endograft device,16 the authors were accused of not having data to support their direct cost analysis. Although an inquiry subsequently demonstrated that a company consultant had raw data from most of the individual centers and had supplied the authors with summary data on which to base their projections, nonetheless, they were vulnerable to the criticism of using summarized data rather than analyzing the raw data themselves.

Finally when the results of a trial depend significantly on the interpretation of diagnostic studies, arrangements should be made for these studies to be independently interpreted by blinded individuals (eg, those in a core laboratory who have no vested interest in the outcome and are independent of company or investigator interference).

Interim data release 

With nonrandomized studies of new devices, preliminary interval reports of the progress of the trial are possible and may even be desirable. Unfortunately, many of the early reports contain soft data and are often presented with enthusiastic bias. At the beginning of such a study, during the active patient-recruitment phase, results can often be obtained from investigators' meetings, and the participants may be allowed (if not encouraged) to present the data at meetings and symposia. In the case of the evaluation of the Endovascular Technologies (EVT) device, restrictions were subsequently placed on the presentation of data released at investigators' meetings. It was announced that, because of the need for compliance with “inside trader rules” when the company “went public,” official data updates would only be released by the company. Such data disclosures took the form of periodic press releases, seemingly intended more for investors than for clinical investigators. A more appropriate reason for a company not releasing data during the course of a trial is the desire to avoid having to retract impressions from preliminary analyses, and rather wait for full analysis on completion of one of the phases of the trial. This has been the explanation given for the Gore Excluder Trial. Other companies have been willing to release their device data from the Eurostar registry, but increasingly companies are wary of releasing interim data from ongoing FDA trials and will generally designate one of the principal investigators to present updates at well-intended vascular symposia. Such updates of data are rarely critical and often present summarized data supplied by the company. This deprives the vascular surgical community, who mostly attend a limited number of meetings, of objective updates of trials that could help in patient management decisions (eg, open AAA repair vs referral to an endograft trial and, if so, which one). This dilemma could be avoided because in a properly constructed prospective device evaluation, the data are collected in a way that will be suitable for submission to the FDA and for publication; consequently, there is no reason why interim data releases, such as presentations at symposia, should not contain objective data and be reported by using standard reporting practices. The details of the device and its use are proprietary; objective data from a clinical trial into which patients are still being actually recruited should not be. One of the main reasons for regular objective and complete disclosure of the outcomes of new devices is the need to assess the failure rate and complications, particularly significant adverse events. Even periodic investigators' meetings may not fulfill this need when the data are prepared and presented by company officials or the investigators are instructed not to disseminate the information.

Adverse events 

Significant delays may occur between the initial discovery of device-specific problems and their eventual dissemination at meetings or in published reports. Reports of serious complications should be circulated immediately by the company to all investigators, who then have a responsibility to notify their patients, institution, and the IRB. In a recent editorial, one of the authors of this editorial17 emphasized the importance of promptly disseminating information if a problem is noted that potentially reflects on the design or deployment of the device. When hook fractures occurred with the EVT device, the problem was quickly disclosed.18 However, when fabrication flaws were detected in another commonly used device, the Min Tec Stentor graft, it was some time before there was public acknowledgment at meetings or in publications,19 although one heard of “disappearing (breaking) sutures” from discussions at investigators' meetings. In an analysis of the Eurostar data20 it was reported that “breakage of polypropylene sutures connecting the rings of the metal stent frame has recently been observed in one of the commercially available types of endografts,” but the device was not identified by name. Several important issues are raised by these adverse events: Is it the company's or the principal investigator's responsibility to disseminate information about adverse events? Who should receive such information—only potential device implanters or the vascular community? What is the appropriate middle ground between premature allegations of device failures and delayed disclosure? And is it appropriate to accede to company requests not to identify complications by device name in registry presentations and publications? These issues have been the subject of an exchange in letters to the editors of this journal regarding the disclosures of adverse events associated with the Stentor and the Vanguard devices.21, 22

Min Tec's Stentor device has been taken over by Boston Scientific Corporation and superceded by the redesigned and differently fabricated Vanguard device. A case report of a Vanguard graft developing fabric erosions was published in this journal in June 1999.23 The authors quoted a November 1998 letter that Boston Scientific Corporation sent to “all its customers,” noting that “late endoleaks due to holes in the fabric covering now have been reported to the manufacturer in six cases.” This letter also noted that “the reported events occurred between five and twelve months post-implant” and “only one of the events described in this letter occurred within this Eurostar data set.” Five of these six cases had been discussed at a breakfast session at the Montefiore Symposium in November 1998. A year later, at this same symposium, the nine cases were reported to have fabric holes, but overall data on this unique complication have yet to be published a year and a half after its first appearance. This may yet be found to be simply a new and relatively low frequency form of endoleak with this device, but this potentially important information has not been promptly and widely transmitted. How else can practicing surgeons be sure that it is safe to refer their patients with AAAs for implantation of a particular endograft or, if they have appropriate endovascular skills, begin using a device that is now being marketed? Thus, the early disclosure of possible device failures remains a nagging issue. The Vanguard device was recently withdrawn, but according to a company spokesman this was not because of concern over the increasing number of fabric holes, but because of microscopic particles separating from the sheath, although the latter have not had any known clinical consequence.

Full disclosure of adverse events has subsequently been seen with the problem of late rupture of aortic aneurysms “repaired” with the AneuRx device. Three cases were reported by Politz et al24 in the March issue of this journal, and an analysis by Zarins, Fogarty, and White of all seven cases known to date appears in this issue.25 Although these reports differ somewhat in their interpretation of the potential seriousness of this particular adverse event, at least the vascular community has been alerted and has the opportunity to review the details. Journals should promptly report and fast-track articles addressing potentially important adverse events.

Negative trials 

There is a tendency not to submit negative results for publication, and it is difficult to get negative trials published. Just like surgeons with poor results, companies are not likely to want to report negative trials. However, failure to publish negative results may result in the continuation of harmful patient treatment, unnecessary duplication of studies, and failure to release important scientific data; it also subverts the ethical principle of honesty in publishing research data. In some instances, the contract between the investigator and the company may appear to give the company the right to block publication. Recently, an investigator was obliged to step to the podium at a major national scientific meeting and announce that he was being prevented from presenting the results of the study by the pharmaceutical company sponsor.26 Permission was withdrawn the evening before presentation. Because of contractual agreements with the investigators, the industry-sponsor maintained a right to block public disclosure. In an editorial discussing this incident,27 it was pointed out that the signed American Heart Association abstract submission document was also binding regarding presentation and that informed consent documents generally include such statements as “the patients and their families have agreed to participate in an activity with risk and have the right to be assured that the results of the study will be transmitted to the scientific community in a timely fashion.” The authors of the editorial maintained “that these other documents (and the informed consent in particular) may also have legal standing as contracts, and that inappropriate delays or limitations in the scope of presentations may represent breaches of these contracts.” Prevention of harm to patients has ethical standing that would likely outweigh and supercede the obligations to follow the terms of a contract requiring nondisclosure.

A company may not have included a clause to prevent disclosure, but if it controls the pooled data, it can effectively prevent publication. This was observed a few years ago, when investigators from one center, who had recruited by far the most cases in a negative trial, finally decided to exercise their right to present and publish their own results28 after it became apparent that the company was not going to have the overall trial data published. After the presentation, company officials tried unsuccessfully to undermine that publication through telephone contact with one of the former editors of this journal. When told that it was undergoing peer review and if accepted would be published, they then asked to publish a rebuttal article but declined when they found it too would have to be submitted for peer review. After the article was published, they published a letter to the editor29; in addition, there were two seemingly spontaneous letters published by others,30, 31 one of whom had a contract with the study sponsor, Curative Technologies, Inc. The primary author's reply32 effectively dismissed their objections, but the company officials' response demonstrates the extent to which a study sponsor may go to prevent or oppose publication of negative trial results. To not allow publication of results interferes with the investigator's right to report scientific data from a trial in a timely fashion regardless of outcome and with the patient's right to know the outcome since this information may modify their current or subsequent management.

Hailey33 quotes two examples where companies have used the court system to delay publication. Bristol-Myers-Squib Canada, Inc, tried unsuccessfully to have the courts block the release of a summary document on statins prepared by the Canadian Coordinating Office for Health Technology Assessment. Release of the report was delayed for almost a year. Recently, the Ontario Ministry of Health requested that a group of experts formulate guidelines on the use of drugs aimed at treating ulcers and related diseases. When the panel concluded that cheaper alternatives to Losec might be equally effective, a lawyer for Astra Zeneca wrote the chair of the committee an intimidating letter requesting that she not finalize and distribute the guidelines and stating if she persisted, legal action would be instigated. The company subsequently stated that it has never had the intention of restricting a researcher from publishing the study results.

Delays in publication 

Although the company may encourage early publication of favorable results, conflict between the company and the investigator may result if the results are not favorable to the company. Last summer, it was communicated to participating investigators that enrollment had been stopped in an almost 5-year randomized trial of carotid stenting using the Schneider Wallstent (now owned by Boston Scientific Corp) versus carotid endarterectomy for carotid stenosis. More than 200 patients had been entered, and the procedural mortality and neurologic morbidity were said to be much higher for the stent group than for those undergoing carotid endarterectomy. Inquiring about possibly fast-tracking a report of this study, the senior editor of this journal was informed by a clinical research investigator coordinating the trial for the company that “enrollment was stopped in June [of 1999, but] we are continuing to follow the patients and plan to do so for an indefinite period of time. The data have not yet been presented or submitted for publication. The investigators agreed that a publication committee should be identified and that they would write the manuscript.” The reply continued “the committee is just in the process of being selected” and “there is no principal investigator for the trial.” The trial sponsors appeared to be in no rush to present or publish the initial outcome (ie, mortality and neurologic morbidity) of the compared procedures, as is common practice for treatment of carotid stenosis, yet it would seem that if the outcomes had been reversed and in favor of carotid stenting, their response regarding prompt presentation and publication might have been quite different.

In addition to a responsibility to inform the vascular community, patients consenting to such a randomized trial are normally assured of obtaining prompt disclosure of the results of a trial if there is a clear difference between groups. Investigators should be certain that a publication committee is established before the start of the study and that provision for prompt publication of the results is assured in the research contract they sign. The research contract should specify that the company can only delay publication for a reasonable time to allow them to review the manuscript and complete application for patent protection or submission to a regulatory body. Ordinarily, 90 days is sufficient for these purposes, assuming the latter contingency has been arranged for in advance.

Multiple publications 

One of the problems with multiple publications of the same longitudinal data is that subsequent accounts may have different starting points, ostensibly because the device itself has undergone redesign. However, this practice hides “learning curve” data that may be important information for others who are going to start to use the technique. Although it is reasonable to show the impact that improvements in the device or the skills of those implanting it have on the results, it is more appropriate to present the entire experience and then carry out subgroup analysis to provide the reader with the full perspective. Detailed publications covering results, complications, and special aspects of a device are clearly justified, but others, in which essentially the same data are revisited with a somewhat different discussion, are clearly examples of “salami slicing,” which prevents the reader from getting a broad perspective on the reported results. To avoid this problem, a publications committee should decide ahead of time how to fairly present the data to avoid multiple publications and should resist company pressures to broadly publish similar data.

As pointed out by Rennie,34 multiple publications from the same data set, along with the tendency to publish only positive results, bias the opinion in the literature in favor of a device or a drug. A proposed solution is to register trials before they are carried out and to ensure publication of all results.35