Journal of Vascular Surgery
Volume 46, Issue 5 , Pages 1055-1064, November 2007

Endovascular skills training and assessment

  • Simon K. Neequaye, BSc, MSc, MRCS

      Affiliations

    • Regional Vascular Unit, St Mary’s Hospital, London, United Kingdom
    • Department of Biosurgery and Surgical Technology, Imperial College, London, United Kingdom.
    • Corresponding Author InformationReprint requests: Simon K. Neequaye, Department of Biosurgery and Surgical Technology, Imperial College, Praed Street, Paddington London W2 1NY UK.
    • ,
  • Rajesh Aggarwal, MA, MRCS

      Affiliations

    • Department of Biosurgery and Surgical Technology, Imperial College, London, United Kingdom.
    • ,
  • Isabelle Van Herzeele, MD

      Affiliations

    • Regional Vascular Unit, St Mary’s Hospital, London, United Kingdom
    • Department of Biosurgery and Surgical Technology, Imperial College, London, United Kingdom.
    • ,
  • Ara Darzi, KBE, MD, FRCS

      Affiliations

    • Department of Biosurgery and Surgical Technology, Imperial College, London, United Kingdom.
    • ,
  • Nicholas J. Cheshire, MD, FRCS

      Affiliations

    • Regional Vascular Unit, St Mary’s Hospital, London, United Kingdom
    • Department of Biosurgery and Surgical Technology, Imperial College, London, United Kingdom.

Received 26 April 2007; accepted 20 May 2007.

Kenneth Ouriel, MD, Review Section Editor

Article Outline

Objective

Evolving endovascular therapies have transformed the management of vascular disease. At the same time, the increasing use of non-invasive vascular imaging techniques has reduced the opportunities to gain the required basic wire and catheter handling skills by performing diagnostic catheterizations. This article reviews the evidence for alternative tools currently available for endovascular skills training and assessment.

Methods

A Literature search was performed on pubmed using combinations of the following keywords; endovascular, skills, training, simulation, assessment and learning curve. Additional articles were retrieved from the reference lists of identified papers as well as discussion with experts in the arena of medical education.

Results

Available alternatives to training on patients inclue synthetic models, anesthetized animals, human cadavers and virtual reality (VR) simulation. VR simulation is a useful tool enabling objective demonstration of improved skills performance both in simulated performance and in subsequent in-vivo performance. Assessment modalities reviewed include time action analysis, error analysis, global rating scales, procedure specific checklists and VR simulators. Assessment in training has been widely validated using VR simulation. Rating scales and checklists are presently the only assessment modalities that have demonstrated utility outside the training lab.

Conclusion

The tools required for a structured proficiency based endovascular training curriculum are already available. Organization of training programs needs to evolve to make full use of modern simulation capability for technical and non-technical skills training.

 

Endovascular treatment options are being increasingly applied to all territories of vascular disease.1, 2, 3 As a result of the expansion of therapeutic endovascular treatment options, there is a need to address the specific issue of endovascular skills training. At present, training is mainly carried out on patients which raises a number of concerns in terms of patient safety and expense.4 Endovascular skills training shares some of the problems associated with the early experience of therapeutic laparoscopy.5 There is a different set of skills required when compared with open surgery, in combination with additional cognitive factors in decision making, judgment, and communication.

In addition, there are relatively few experts worldwide for newer techniques such as carotid artery stenting (CAS), which leads to difficulties in developing structured training programmes.6 CAS is emerging as an alternative to carotid endarterectomy.7, 8 A range of clinical specialists have expressed an interest in performing CAS ranging from vascular surgeons to interventional radiologists, cardiologists, neurologists, and neurosurgeons.9 There are, however, potentially catastrophic results of technical error in CAS as evidenced by the recently published EVA 3S trial which was terminated prematurely as a result of an excess stroke and death risk in the endovascular arm (30-day incidence of any stroke or death 3.9% after endarterectomy vs 9.6% after CAS).10 A major criticism of this study was that CAS was performed by relatively inexperienced interventionalists. In order to ensure that only appropriately trained individuals carry out such high stakes procedures, it is necessary to develop structured training programs underpinned by objective measures of proficiency.

Simulator based training is likely to play an increasingly important role in skills based training.11 Virtual reality simulator training in particular appears to be well suited to endovascular skills training, enabling novice subjects to learn basic wire and catheter handling skills, and expert practitioners the opportunity to rehearse new procedures in the skills laboratory prior to intervention on patients. Integrated simulator based training may help to overcome some of the inherent difficulties of learning to operate using a two-dimensional view and offers a unique opportunity to objectively demonstrate technical competence as part of the credentialing process. This article reviews the evidence with regards to endovascular skills training and assessment. The relative merits of the different training tools currently available are discussed.

Back to Article Outline

Endovascular training 

Basic wire and catheter handling skills have previously been acquired by performing diagnostic catheterization studies, the number of which has been significantly reduced due to increased availability of noninvasive techniques such as duplex ultrasonography, computed tomography, and magnetic resonance angiography.12 There are, however, alternatives to acquiring new skills on patients. These include synthetic and animal based models, though more recently virtual reality (VR) trainers have been developed (Table I).

Table I. Available simulation models for endovascular skills training
Simulation modelAdvantagesDisadvantagesPublished results of endovascular training studiesRelative cost
Synthetic
•Standardized task

•Cheap

•Easily transported

•X-ray screening not required


•Lacks validated assessment tool

•Advanced task simulation poor ie, carotid/intracranial

•Single use endovascular devices

•Dynamic behavior of arterial system not reproduced

•None
•Unit cost $3000

•Additional costs (Endovascular tools, skills laboratory)

Animal
•High degree of realism using live anaesthetized animal

•Full procedure simulation (including arteriotomy and closure)


•Lacks validated assessment tool

•Single session use of animal

•Endovascular tools not re-usable

•Anesthesia required

•Specialist facilities

•Legal and ethical problems

•Size and anatomical differences to human

•Cost high


•Significant improvement in skills performance of endovascular novices using porcine training model35


•Unit cost $1000†

•Additional costs (Specialist facility, disposal of carcasses, veterinarian, specialist caretaker)

Human cadaver
•Full procedure simulation (including arteriotomy and closure)

•Realism high


•Lacks validated assessment tool

•Single session use endovascular tools not re-usable

•Cost high

•None
•$1000-$3000 per cadaver⁎

•Additional costs (Specialist facility, preparation of body, transport by undertaker, cremation)

Virtual reality
•Standardized Task

•High degree of realism

•Validated assessment metrics

•Multiple modules in various anatomical territories

•Patient specific simulation possible

•Endovascular tools re-usable


•Significant setup, transport and maintenance cost

•Frequent breakdown


•Significant improvement in simulator performance of novice interventionalist.31, 32, 33, 34, 35, 36, 55

•Significant improvement in in vivo performance35, 41


•Unit cost approximately $100,000 to $200 000 (dependant on software and modules)‡

•Annual service contract $10,000 to $16,000

•Additional costs (Skills lab transportation and insurance)

⁎Reusable if refrigerated but once stent deployed this part of circulation no longer available for device deployment.

†Not reusable therefore cumulative cost is high.

‡Simsuite (MSC) lease agreement over 3 to 6 years $200,000 to $ 500,000 per annum, dependant on curriculum requirements and includes provision of a clinical educator, maintenance, and technology upgrades.

Synthetic models 

Synthetic models range from low fidelity solid plastic models to high fidelity systems with pulsatile flow and fluoroscopy13, 14, 15 (Fig 1). These models are relatively inexpensive costing from $3000 per unit and benefit from being portable and simple to set up. They cannot, however, fully replicate the dynamic behavior of the human arterial circulation in terms of the elasticity of arterial walls or blood flow. Advanced models such as carotid territory simulation are also limited by the effect of friction during passage of devices through curves.16

Low-fidelity simulation is an effective method of minimally invasive skills training.17 Catheter and wire behavior, stent deployment, and balloon inflation can be taught effectively using real devices with the benefit of force feedback. The draw back of using real tools, however, is that deployment of single use devices is wasteful and adds to the cost of training. These tabletop demonstrations are an essential early step in training, though for advanced skill training higher fidelity options such as animal models and VR simulation may be needed.

Animal models 

Animal models offer a high degree of realism with the possibility of artificially inducing stenotic and aneurysmal disease by endothelial injury18, 19 and sutured patches,20 respectively. Use of animal models is limited by expense, requirement for specialist facilities, legal and ethical issues, as well as anatomical and size differences between animals and humans. Furthermore, the animals can only be used for one session. Despite these limitations, large animal models offer a highly realistic training option for advanced interventions.

Human cadavers 

A human cadaveric model has also been described that offers realistic conditions for training and testing endovascular devices. The process establishes pulsatile flow in the arterial tree of a human cadaver following a thrombolytic process.21 Fresh frozen human cadavers were used. Antegrade arterial flow was established by pumping fluid into an inflow cannula placed in the descending aorta via the axillary artery and an outflow cannula in the superficial femoral artery, thus, providing antegrade pulsatile flow. The cadaveric model allows full procedures to be performed including arterial puncture and closure though preserved cadaveric tissue differs in feel and deformation from living tissue. Limited availability and high costs related to preservation and appropriate storage limit the potential use of human cadavers for endovascular training.22

Virtual reality systems 

Virtual reality (VR) simulation uses a computer-generated three-dimensional model of the vascular tree allowing the user to interact with the simulation through an interface device.23 Reusable, modified instruments are used with the active tip produced by the simulation. Recent developments in computing power and volume rendering techniques enable a high degree of realism in simulated fluoroscopic images (Fig 2). Patient specific simulations are also possible which may allow rehearsal of a procedure prior to performing the real case. High fidelity VR simulation is already available. These simulators can be reused ad infinitum, with no ethical issues related to their use c.f. animal models and the ability to objectively and instantly assess training performance.

There are disadvantages to VR simulation though. These devices represent a significant capital cost with each unit retailing for approximately $ 200,000, with additional maintenance costs. The devices are still prone to technical failure and require regular calibration and maintenance. Despite the cost, computer based simulation may be an attractive option for endovascular training. Although they will not replace training on patients completely, a realistic training experience is possible, with the added benefit of objective and immediate assessment of performance.24 Trainee’s can repeatedly perform a procedure or indeed a maneuver until proficiency has been demonstrated. The use of a standardized task may allow the development of a proficiency based curriculum with subjects demonstrating a predetermined benchmark level of expertise prior to interventions on patients.25, 26

Commercially available VR endovascular simulators can be described as part-task simulators as arterial puncture and closure are not performed. Currently available endovascular simulators include Procedicus VIST™ (Mentice, Gothenburg, Sweden), Angiomentor (Simbionix, Cleveland, Ohio), Simsuite (Medical Simulation Corporation, Denver, Colo), Endovascular Accutouch (Immersion Medical, Gaithesburg) (Table II). The Procedicus VIST simulator comprises a mechanical unit housed within a plastic mannequin cover, a high-performance desktop computer, and two display screens. Modified instruments are inserted through the access port using a haptic interface device. The term haptic relates to tactile feedback which is created by a series of motorized carts which lock onto the inserted instrument allowing the subject to manipulate the simulated instrument in real-time with force-feedback ie, mechanical stimulation of the sense of touch. Commercially available simulation modules include occlusive arterial disease in the coronary, carotid, renal and ileo-femoral regions, and over the wire lead placement for biventricular pacing. The subject is able to select appropriate instruments and perform interventional procedures using the simulated fluoroscopic screen. Performance is measured using metric parameters such as volume of contrast fluid used, fluoroscopy time, and markers of stent placement accuracy.

Table II. Comparison of VR endovascular trainers
DeviceDescriptionModulesAssessment parametersValidation studies
Angiomentor Symbionix
(1)Part procedure simulator

(2)Haptic feedback

(3)Neurological and pharmacological responses

(4)Metric assessment

Carotid, renal, iliac/SFA, coronaryQuantitative metrics, qualitative metrics, clinical errors, hemodynamic features, handling of complications(Ongoing study by SIR, CRF)
Accutouch Immersion Medical
(1)Part procedure simulator.

(2)Haptic feedback

(3)Physiological responses

(4)Metric assessment

Carotid, renal, iliac, coronary arteryQuantitative metrics, qualitative metrics, procedure complications, hemodynamic features• Wang et al, construct validity for cardiac lead placement66
Procedicus VIST™
(1)Part procedure simulator.

(2)Haptic feedback

(3)Metric assessment

Neurointerventions, carotid, renal, iliac/SFA, coronaryQuantitative metrics, qualitative metrics, clinical parameters, technical errors
•Face validity (4 studies)31, 32, 35, 55

•Construct validity (4 studies)31, 32, 33, 60

•Transfer of training (2 studies)35, 41

Simsuite Medical Simulation Corp.
(1)Part procedure trainer

(2)Haptic feedback

(3)Neurology and pharmacology responses

Neurointerventions, carotid, coronary, renal, Iliac and closure of patent foramen ovaleQuantitative metrics, qualitative metrics, clinical parameters, technical errors, hemodynamic features, handling of complications
Dawson et al, improved technical skill of residents following simulator based training.34

(Pilot studies on behalf of ABIM and SCAI to determine construct validity and benchmark performance underway.)

SIR, Society of Interventional Radiology; CRF, Cardiovascular Research Foundation; SCAI, Society for Cardiac Angiography; ABIM (American Board of Internal Medicine).

The Angiomentor Ultimate endovascular trainer has a similar range of arterial procedures to the VIST and also boasts advanced haptic technology. It differs from the VIST in that there is greater emphasis on patient monitoring, drug administration, and response to physiological disturbance (Fig 3). For example, administration of atropine to correct bradycardia induced by carotid sinus stimulation and appropriate therapeutic responses to chest pain or breathlessness. Two cheaper and more portable editions have become available recently, the Angiomentor Express and the Angiomentor Mini. These have a similar simulation package but less peripheral attachments such that the Mini can fit into a hand held case.

The Simsuite is a larger simulator system with up to six interactive screens to facilitate multidisciplinary team training. Similarly to the Angiomentor system, response to patient physiology features substantially in the simulation. Additionally though, appropriate case selection and management are also taught. The Endovascular Accutouch simulator also boasts peripheral arterial, carotid, and coronary simulation modules with metric based assessment.

The reliability of simulator devices remains problematic, and there is a noteworthy requirement for regular maintenance and calibration. The main problem with the use of computer based simulation is not setting up VR labs; it is keeping the simulators up and running. Technical difficulties do occur frequently in particular following rough handling by inexperienced subjects. In the authors’ institution, the most problematic component is the haptic interface mechanism. This part of the device is composed of motorized carts that grip inserted instruments producing tactile feedback. Regular calibration is required to ensure optimum levels of force-feedback. This needs to be checked daily and is dependent on environmental temperature which needs to be stable. Inserted tools are detected by the haptic interface using an optic mechanism which distinguishes both the presence and the diameter of the instrument allowing the selected tool to be simulated. This part of the unit requires calibration less frequently (weekly), though is subject to interference from dust particles and debris (typically from broken catheters). The majority of these calibration and maintenance tasks are carried out by research fellows in our institution (following manufacturer training) but skilled technical support is required for heavy usage periods especially involving more challenging cases such as carotid artery stent procedures. A service contract with the various companies is possible and varies in price depending on the number of simulators and institution. For example, the annual service contract for angiomentor (Simbionix) is approximately 10% of the simulator price, $10,000 to $16,000. Simsuite (MSC) on the other hand retains responsibility for servicing their machines as part of their lease agreement. With regard to endovascular tools, real life tools can be used but the floppy tips of guide wires, stents, or embolic protection device need to be removed.

The evidence for VR in endovascular skills training 

It is incorrect to assume that a realistic simulation equates to an effective training or assessment model.27 Compared with skills training in open and laparoscopic surgery there is a relative paucity of research in the domain of endovascular skills training. Recently, there have been a handful of papers reporting on VR simulation in endovascular skills training (Table III). In addition to developments in the realism of VR simulation, demonstration of reliability, feasibility, and validity is necessary (Table IV). These studies have so far sought to establish the value of these devices as training and assessment devices.

Table III. VR endovascular training studies
StudySimulator deviceModuleFace validityConstruct validityTraining potentialTransfer of training to in vivo
Wang et al (2001)66AccutouchCardiac lead placement Yes
Dayal et al (2004)31VISTCarotidYesYesYes
Hsu et al (2004)32VISTCarotidYesYesYes
Aggarwal et al (2006)33VISTRenal YesYes
Hislop et al (2006)60VISTCarotid Yes
Berry et al (2006)77VISTRenal No
Patel et al (2006)55VISTCarotidYes Yes
Chaer et al (2006)41VISTIliac/SFA Yes
Dawson et al (2007)34SimsuiteIliac Yes
Berry et al (2007)35VISTIliacYes YesYes
Neequaye et al (2007)36VISTIliac and renal Yes
Table IV. Qualities of the ideal surgical assessment tool (Aggarwal et al; Ann Surg 2007)
Feasibilityis a measure of whether something is capable of being done or carried out
Validity
Face validityis the extent to which the examination resembles real life situations
Content validityis the extent to which the domain that is being measured is measured by the assessment tool—for example, while trying to assess technical skills we may actually be testing knowledge
Construct validityis the extent to which a test measures the trait that it purports to measure. One inference of construct validity is the extent to which a test discriminates between various levels of expertise
Concurrent validityis the extent to which the results of the assessment tool correlate with the gold standard for that domain
Predictive validityis the ability of the examination to predict future performance
Reliability
Test-retestis a measure of a test to generate similar results when applied at two different points
Inter-rateris a measure of the extent of agreement between two or more observers when rating the performance of an individual

Training potential of VR endovascular simulation 

The term “learning curve” used in the context of skills training refers to the time taken and/or the number of procedures an average practitioner needs to be able to perform a procedure independently with an acceptable outcome.28 Two types of variables are generally used; measures of patient outcome such as complications and survival or measures of surgical process such as blood loss and operative time.29 Endovascular practitioners have a procedure related learning curve. Lin et al analyzed the outcomes of sequential groups of patients undergoing carotid artery stenting and demonstrated decreased procedure-related complications, fluoroscopic time, and contrast volume used with increased physician experience.30 Simulation based training may allow the early part of this learning curve to take place without exposing patients to unnecessary risk.

Studies examining the potential for using VR systems in endovascular skills training have analyzed the learning curves of both novice and expert subjects. Dayal et al demonstrated improved simulated performance of a CAS procedure by novice subjects in terms of improved procedure time, fluoroscopy time, and supervisor assessment of catheter handling following a minimum of 2 hours of supervised training on the VIST simulator. Expert subjects (>300 endovascular procedures) did not show any statistically significant improvement following training.31 Hsu et al performed a randomized study in which both novice and expert subjects (>50 endovascular procedures) were randomized to receive supervised simulator based CAS training or no training. Significant improvement in procedure completion time was reported in the group randomized to simulator training in both novice and expert subjects.32 Aggarwal et al analyzed the learning curves of experienced open vascular surgeons and demonstrated improved performance (procedure time and contrast fluid used) following VR simulator training using a renal artery stenting model.33 Similar improvements in simulator performance following training have been reported for Iliac and renal angioplasty.34, 35, 36

These training studies tend to suggest that inexperienced subjects in particular derive significant benefit in terms of improved performance on the simulator with repetitive practice compared with expert subjects who also have a short learning curve as they become familiar with the simulator. The suggested benefit of simulation based practice is that subjects gain basic psychomotor skills that become automated by the time they perform procedures in real patients.37 Before widespread adoption of simulators into the endovascular curriculum, it is necessary to demonstrate transfer of endovascular skill to real procedures.

Transfer of skills from endovascular training lab to patient 

Skills transfer ie, significant improvement in operative performance following a period of dedicated skills training, has been demonstrated following VR training in laparoscopy,38 lower gastrointestinal endoscopy,39 and bronchoscopy.40 Recent evidence of skills transfer using VR simulation for endovascular skills training is encouraging with improved performance in the catheterization lab demonstrated in vivo. Berry et al performed a randomized trial comparing a live porcine model with VR simulation training to perform an iliac artery angioplasty task. Total score (combined global rating scale and task specific checklist) improved significantly with repetitive practice in both the porcine and VR groups. Notably, this improvement was shown to transfer from the VR simulator to the porcine model.35 The first randomized trial examining transfer of VR endovascular training to the human model was carried out by Chaer et al who assessed performance of two supervised iliofemoral angioplasty procedures by twenty general surgery residents in vivo. Following didactic teaching, half were randomized to receive a maximum of 2 hours of VR simulation training; the remainder received no further training. The simulator trained group improved significantly compared with the control group using a procedure specific checklist and a global rating scale to assess performance.41

Performance benchmarks 

Simulator derived performance reporting also allows the learning curve of an individual trainee to be tracked, leading to the development of a proficiency based training curriculum with progress determined by demonstration of skills performance to a predetermined benchmark level. Personalized training such as this may be a more effective way of training than undertaking a set number of repetitions or time.42 Further work is required to define appropriate benchmark levels of skill both within the VR simulation environment and in vivo.

Back to Article Outline

Assessment in endovascular training 

Current guidelines with regards to who is competent to perform endovascular procedures are largely based on the number of procedures performed and time in training.43 However, these recommendations rely on crude data that are recognized to be unreliable and indirect measures of technical skill.44, 45 There is increasing recognition that the number of procedures performed and time in training does not equate to expertise. As a result, the trend in medical skills training is a move towards using objective assessment tools to demonstrate technical competence.

Objective measures of skills performance are not well reported in endovascular interventions. A number of assessment tools are available including time-action analysis, error analysis, global rating scales and procedure specific checklists, motion analysis, and, perhaps most promising, VR simulators.

Time-action analysis 

Time-action analysis has been used as a method of objective assessment of performance in open and minimally invasive surgery.46, 47, 48 The method can be applied to real life or simulator performance and involves breaking down the procedure into a series of steps with performance analyzed by how long an individual takes to complete each step.49, 50 This procedure is, however, man-power intensive in terms of setup and video analysis time. In addition, the amount of time taken to complete an individual procedural step does not offer any measure of quality of performance. Therefore time action-analysis may be more useful as a research tool offering an insight into instrument design and procedural efficiency.

Error analysis 

Human reliability and error analysis is an evolving field in healthcare. Error scores have been proposed as discriminators of technical skill though inherent difficulties exist in defining surgical or medical error as there is no standardized taxonomy.51 It is, however, possible to differentiate technical skill by examining both the frequency and type of error committed during laparoscopic cholecystectomy52, 53 and pyloromyotomy.54 To date, error analysis in endovascular training and assessment is at an early stage with no reported studies examining this question in vivo. Modern simulator technology allows reporting of catheter and device handling errors. Patel et al reported a reduction in the composite catheter handling error scores of interventional cardiologists performing a VR carotid angiogram following simulator training.55

Simulation and in vivo error analysis may be a feasible option though further work is required in terms of the definition and relative weighting of different types of error. This is a potentially exciting area and could include clinical errors such as case selection and drug administration as well as technical errors related to device handling; though they require further refinement these errors are already reported by the current generation of VR simulation devices.

Rating scales and procedure specific checklists 

A global rating scale is a quantitative assessment tool based on appraisal of seven aspects of quality in operative performance. Each component is marked using a five-point Likert scale.56 This method has been demonstrated to differentiate between experience levels in both open and minimally invasive surgery.57, 58, 59 A modified global rating scale has been shown to differentiate endovascular experience and training using a virtual reality simulator.60 The first two studies examining VR transfer of training to the catheterization lab both used modified rating scales.35, 41 Procedure specific checklists used in conjunction with global rating scales have been shown to be effective and reliable assessment tools of surgical dexterity using synthetic and cadaveric models as well as in live operating.61, 62 Post hoc video analysis though not mandatory does reduce the potential for bias. The main disadvantage of this mode of assessment is that a large amount of time is required from expert assessors. Full length video viewing is required as edited video assessment appears to reduce the reliability of assessment.63 The feasibility of using global rating scales and checklists for endovascular skills assessment is not proven though this system has been extensively validated in other fields of minimally invasive surgery and is a promising prospect for in vivo assessment of endovascular skill.

Motion analysis 

Motion analysis may offer a less time consuming option. Efficient and purposeful hand movements are a discriminator of technical skill in surgery.64 The technology is already available and indeed surgical dexterity is currently assessed using this modality for the open surgery portion of the European Board of Surgery Qualification in Vascular Surgery (EBSQ-VASC) examination. The Imperial College Surgical Assessment device (ICSAD) is used to track hand movement in three-dimensions using electromagnetic sensors with a composite score based on economy of movement and qualitative analysis.65 This is a potentially exciting area for future research with no published studies to date examining hand motion analysis in the endovascular arena.

VR simulators 

The major advantages of VR simulation is the ability to automatically and instantly provide an objective performance report based on quantitative and qualitative assessment parameters. Error scores and rating scales can also be used in conjunction.55, 60. Used in a standardized setting, it is possible to distinguish between subjects of different levels of experience.31, 32, 33, 66 Assessment of nontechnical skills such as appropriate drug administration and physiological monitoring is also possible with most of the current generation of simulators. For example SimSuite (Medical Simulation Corp) requires appropriate case selection and Angiomentor (Symbionix, Cleveland, OH) has advanced patient physiology reporting with the ability to administer a range of drugs including heparin, atropine, and glycerine trinitrate.

The validity of this method of assessment is under evaluation. Currently, performance reporting remains unsatisfactory. Quantitative measures of performance related to procedure time and use of the C-arm are well reported but further work is necessary in developing more subtle indicators of performance and judgment such as clinical outcome and technical error. Though further work is required, simulation based assessment is potentially a mechanism for selecting candidates for surgical or interventional training programs and may be a requirement for revalidation or gaining credentials to perform procedures.67

Back to Article Outline

Discussion 

The current trend in medical skills training is a move away from the traditional apprentice model of graded responsibility to a more structured approach towards the attainment of technical competence.68 The concept of a “pretrained novice” used by educational psychologists refers to one who has automated basic psychomotor skills and spatial judgements.69 Moving a trainee forward to this position is attractive because he or she is able to come into the operating/interventional suite armed with basic technical skills that allow them to learn procedures more effectively with less risk to the patient.70 Mentorship within the curriculum will be the key to learning via simulation, using simulation as a tool.67

As part of a well planned training curriculum, simulator training and in particular VR training offers significant benefits in the context of a competence or proficiency based training program because simulators can be used early on in the learning experience to acquire and reinforce basic wire and catheter handling skills prior to learning on patients.41 The available evidence tends to suggest that inexperienced subjects are able to improve their endovascular skills performance with repetitive practice using VR simulation.32 Animal and synthetic models are likely to offer similar benefits though cost and ethical implications make animal models impractical. Both modalities, animal and synthetic suffer because of the absence of a reliable and objective performance assessment tool.

Further work to validate VR simulators as assessment tools is required especially to examine the question of skills transfer from simulation to real life if they are to fulfill their potential to provide a high fidelity training experience and objective assessment of endovascular skill.67 In particular, the transfer effectiveness of VR endovascular training is not established. This concept, which is derived from aviation simulation, can be defined as the ratio of time spent in simulator training to time spent in the aircraft to achieve a given level of competence.71 Assessment tools for in vivo performance are a fundamental requirement to demonstrate the efficacy of simulator based training and assessment. A modified global rating scale may be an option for in vivo assessment, but motion and error analysis systems already described in other fields of minimally invasive surgery hold much promise in the identification of expert levels of performance.

Simulation based training is unlikely to replace real life experience but is an adjunct for training to allow novices to learn basic skills away from the patient leading to a shorter and flatter learning curve. The effectiveness of simulation in this regard ie, effectively moving a trainee forward in their individual learning curve is not proven. Despite rapid progress in the development of simulator technology, in particular computer based simulation, there is still a considerable gap in knowledge about how best to make use of this technology. Further work is required to develop and validate training curricula such that they are able to deliver the required end product, a safe and proficient practitioner. Ideally, in addition to pure technical skill the curriculum should also teach cognitive knowledge and decision making skills to ensure appropriate case selection as well as the ability to detect and correct complications. What device is required at various stages of training is yet to be established. Task analysis is required to determine what core skills are required as well as how they are best delivered by the available modalities of training. Advanced simulation may not be necessary early in the learning curve. Basic skills tasks such as wire and catheter may be effectively taught by relatively cheap synthetic models prior to performing complete procedures using VR simulators or anesthetized animals in “wet labs.” Additionally, the use of simulation for nontechnical skills training such as leadership and communication is required. There are a variety of good tools already available for endovascular skills training. Organization of these into a stepwise proficiency based training curriculum requires further work in terms of the process of training rather than attempts to develop increasingly sophisticated simulation models.

Patient specific simulation allows so called mission rehearsal. Cates et al report using a patient’s magnetic resonance angiogram to generate a VR simulation with which they were able to prepare for a live case by practicing a CAS procedure using a reproduction of the patients own vascular anatomy. Preprocedure review of the arch angiogram usually allows identification of difficult arch anatomy, however, using VR simulation for rehearsal affords the opportunity to experiment with different catheter shapes to selectively cannulate the target artery and size the stent and embolic protection device. This may improve procedure efficiency and also reduce risk to the patient from catheter manipulation in the arch.72

VR simulators may also be useful in the context of nontechnical skills training and assessment. Communication and team working skills are increasingly recognized as important determinants of patient outcome. Rehearsing crisis scenarios in particular may allow the team to deal with real life crises in a safe and ordered way.73 The endovascular interventional suite is a complex environment requiring interaction between a large multidisciplinary team including doctors, nurses, radiographers, porters, and clerical staff. The importance of this team interaction cannot be underestimated. Effective and safe practice requires that each member is suitably trained, competent, and familiar with their surroundings. Simulator based training may allow the extended team to practice their individual roles. For example, the scrub nurse and assistant can be taught about device characteristics and specific setup requirements at a safe and convenient time. The virtual operating room concept seems an ideal way of improving team dynamics and communication with the ultimate aim of improving patient safety and reducing the risk of adverse events.74 Patient specific procedure training and planning is also a real possibility with reconstructions reviewed in a multidisciplinary setting prior to the procedure.

The ideal skills laboratory has not been established. There are significant infrastructure requirements both physical and organizational. The physical infrastructure includes well located facilities of an appropriate size and layout to enable didactic teaching as well as skills based training. Utilizing existing facilities may reduce the associated cost as would collaboration with other specialties with similar training requirements. For example, vascular surgery trainees require a similar grounding in cardiovascular physiology and pathology to cardiology and neurology trainees. They require similar open surgical skills to general surgeons and perhaps most importantly they must develop a similar skills-set to interventional radiology trainees for both endovascular intervention and imaging techniques such as duplex ultrasound scanning. The organization of a successful skills lab may require combining some or all of these groups at various stages in their training. This would be a more efficient use of resources and increase the faculty pool available. Above all, in addition to being well funded and organized, the ideal skills laboratory should have clear aims as to who is to be trained and how to deliver this training.75

The organization of training program must also be addressed to take full advantage of modern simulation capability in individual and team training. Centralized programs with didactic teaching followed by a period of VR training with objective assessment of proficiency prior to graded training on real patients is a possibility and would reduce the costs associated with setting up and running VR skills labs. The politics of who pays for training and where this training will be provided is beyond the scope of this review, but it appears that the tools are already available, and in the UK, there is already a move towards regional radiology academies which may be ideally placed to provide the training infrastructure.76

Simulation based training offers the opportunity to shorten the trainee’s learning curve, improve patient safety, and reduce expense in terms of operating room time. Simulation does not remove the requirement for specialist supervision. The particular device used is less important than the curriculum within which it is used, that includes error identification in addition to skills acquisition.77 In order to benefit fully from this new technology, training structures need to evolve to make use of the available technology.

Back to Article Outline

References 

  1. Greenhalgh RM, Brown LC, Epstein D, Kwong GPS, Powell JT, Sculpher MJ, et al. Endovascular aneurysm repair versus open repair in patients with abdominal aortic aneurysm (EVAR trial 1): randomized controlled trial. Lancet. 2005;365:2179–2186
  2. Bradbury AW, Ruckley CV, Fowkes FGR, Forbes JF, Gillespie I, Adam DJ, et al. Bypass versus angioplasty in severe ischemia of the leg (BASIL): multicenter, randomized controlled trial. Lancet. 2005;366:1925–1934
  3. Molyneux AJ, Kerr RSC, Yu LM, Clarke M, Sneade M, Yarnold JA, et al. International subarachnoid aneurysm trial (ISAT) of neurosurgical clipping versus endovascular coiling in 2143 patients with ruptured intracranial aneurysms: a randomized comparison of effects on survival, dependency, seizures, rebleeding, subgroups, and aneurysm occlusion. Lancet. 2005;366:809–817
  4. Babineau TJ, Becker J, Gibbons G, Sentovich S, Hess D, Robertson S, et al. The “cost” of operative training for surgical residents. Arch Surg. 2004;139:366–369
  5. Meyers WC, Sue S, Bennett CL, Chari R, Iannacchione MA, Moore MJ, et al. The Learning-Curve for Laparoscopic Cholecystectomy. Am J Surg. 1995;170:55–59
  6. Rosenfield KM. Clinical competence statement on carotid stenting: training, and credentialing for carotid stenting - Multispecialty consensus recommendations. J Vasc Surg. 2005;41:160–168
  7. Coward LJ, Featherstone RL, Brown MM. Safety and efficacy of endovascular treatment of carotid artery stenosis compared with carotid endarterectomy - a Cochrane systematic review of the randomized evidence. Stroke. 2005;36:905–911
  8. Yadav JS, Wholey MH, Kuntz RE, Fayad P, Katzen BT, Mishkel GJ, et al. Protected carotid-artery stenting versus endarterectomy in high-risk patients. N Engl J Med. 2004;351:1493–1501
  9. Sacks D, Connors JJ. Carotid stenting, stroke prevention, and training. J Vasc Interv Radiol. 2004;15:1381–1384
  10. Mas J, Chatellier G, Beyssen B, Branchereau A, Moulin T, Becquemin JP, et al. Endarterectomy versus stenting in patients with symptomatic severe carotid stenosis. N Engl J Med. 2006;355:1660–1671
  11. Reznick RK. Surgical simulation - a vital part of our future. Ann Surg. 2005;242:640–641
  12. Nederkoorn PJ, Mali WPTM, Eikelboom BC, Elgersma OEH, Buskens E, Hunink M, et al. Preoperative diagnosis of carotid artery stenosis - accuracy of noninvasive testing. Stroke. 2002;33:2003–2008
  13. Chong CK, How TV, Black RA, Shortland AP, Harris PL. Development of a simulator for endovascular repair of abdominal aortic aneurysms. Ann Biomed Eng. 1998;26:798–802
  14. Lermusiaux P, Leroux C, Tasse JC, Castellani L, Martinez R. Aortic aneurysm: construction of a life-size model by rapid prototyping. Ann Vasc Surg. 2001;15:131–135
  15. Berry E, Marsden A, Dalgarno KW, Kessel D, Scott DJA. Flexible tubular replicas of abdominal aortic aneurysms (Proceedings of the Institution of Mechanical Engineers Part H). J Eng Med. 2002;216:211–214
  16. Suzuki Y, Fujitsuka M, Chaloupka JC. Simulation of endovascular neurointervention using silicone models: imaging and manipulation. Neurol Med Chir. 2005;45:567–572
  17. Rosser JC, Rosser LE, Savalgi RS. Skill acquisition and assessment for laparoscopic surgery. Arch Surg. 1997;132:200–204
  18. Lin PH, Chen CY, Surowiec SM, Conklin B, Bush RL, Chaikof EL, et al. A porcine model of carotid artery thrombosis for thrombolytic therapy and angioplasty: application of PTFE graft-induced stenosis. J Endovasc Ther. 2000;7:227–235
  19. Ishii A, Vinuela F, Murayama Y, Yuki I, Nien YL, Yeh DT, et al. Swine model of carotid artery atherosclerosis: experimental induction by surgical partial ligation and dietary hypercholesterolemia. Am J Neuroradiol. 2006;27:1893–1899
  20. Maynar M, Qian Z, Hernandez J, Sun F, DeMiguel C, Crisostomo V, et al. An animal model of abdominal aortic aneurysm created with peritoneal patch: technique and initial results. Cardiovasc Intervent Radiol. 2003;26:168–176
  21. Garrett HE. A human cadaveric circulation model. J Vasc Surg. 2001;33:1128–1130
  22. McLachlan JC, Bligh J, Bradley P, Searle J. Teaching anatomy without cadavers. Med Educ. 2004;38:418–424
  23. Satava RM. Virtual reality surgical simulator - the first steps. Surg Endosc Ultrasound Intervent Tech. 1993;7:203–205
  24. Gallagher AG, Cates CU. Approval of virtual reality training for carotid stenting - What this means for procedural-based medicine. JAMA. 2004;292:3024–3026
  25. Aggarwal R, Grantcharov T, Moorthy K, Hance J, Darzi A. A competency-based virtual reality training curriculum for the acquisition of laparoscopic psychomotor skill. Am J Surg. 2006;191:128–133
  26. Aggarwal R, Grantcharov TP, Eriksen JR, Blirup D, Kristiansen VB, Funch-Jensen P, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg. 2006;244:310–314
  27. Sutherland LM, Middleton PF, Anthony A, Hamdorf J, Cregan P, Scott D, et al. Surgical simulation - A systematic review. Annals of Surgery. 2006;243:291–300
  28. Muir G. The “learning curve” in surgery: what is it, how do we measure it and can we influence it?. BJU Int. 2004;93:1173–1174
  29. Cook JA, Ramsay CR, Fayers P. Statistical evaluation of learning curve effects in surgical trials. Clin Trials. 2004;1:421–427
  30. Lin PH, Bush RL, Peden EK, Zhou W, Guerrero M, Henao EA, et al. Carotid artery stenting with neuroprotection: assessing the learning curve and treatment outcome. Am J Surg. 2005;190:850–857
  31. Dayal R, Faries PL, Lin SC, Bernheim J, Hollenbeck S, DeRubertis B, et al. Computer simulation as a component of catheter-based training. J Vasc Surg. 2004;40:1112–1117
  32. Hsu JH, Younan D, Pandalai S, Gillespie BT, Jain RA, Schippert DW, et al. Use of computer simulation for determining endovascular skill levels in a carotid stenting model. J Vasc Surg. 2004;40:1118–1124
  33. Aggarwal R, Black SA, Hance JR, Darzi A, Cheshire NJW. Virtual reality simulation training can improve inexperienced surgeons’ endovascular skills. Eur J Vasc Endovasc Surg. 2006;31:588–593
  34. Dawson DL, Meyer J, Lee ES, Pevec WC. Training with simulation improves residents endovascular procedure skills. J Vasc Surg. 2007;45:149–154
  35. Berry M, Lystig T, Beard J, Klingestierna H, Reznick R, Lonn L. Porcine Transfer Study: virtual Reality Simulator Training Compared with porcine training in endovascular novices. Cardiovasc Intervent Radiol. 2007;30:455–461
  36. Neequaye SK, Aggarwal R, Brightwell R, Van Herzeele I, Darzi A, Cheshire NJ. Identification of skills common to renal and iliac endovascular procedures performed on a virtual reality simulator. Eur J Vasc Endovasc Surg. 2007;33:525–532
  37. Gallagher AG, Cates CU. Virtual reality training for the operating room and cardiac catheterization laboratory. Lancet. 2004;364:1538–1540
  38. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg. 2004;91:146–150
  39. Ahlberg G, Hultcrantz R, Jaramillo E, Lindblom A, Arvidsson D. Virtual reality colonoscopy simulation: a compulsory practice for the future colonoscopist?. Endoscopy. 2005;37:1198–1204
  40. Blum MG, Powers TW, Sundaresan S. Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy. Ann Thorac Surg. 2004;78:287–291
  41. Chaer RA, DeRubertis BG, Lin SC, Bush HL, Karwowski JK, Birk D, et al. Simulation improves resident performance in catheter-based intervention - results of a randomized, controlled study. Ann Surg. 2006;244:343–352
  42. Brunner WC, Korndorffer JR, Sierra R, Massarweh NN, Dunne JB, Yau CL, et al. Laparoscopic virtual reality training: are 30 repetitions enough?. J Surg Res. 2004;122:150–156
  43. Gaines P, Nicholson T. A suggested training program for carotid artery stenting (CAS). Eur J Radiol. 2006;60:37–39
  44. Moorthy K, Munz Y, Sarker SK, Darzi A. Objective assessment of technical skills in surgery. BMJ. 2003;327:1032–1037
  45. Warf BC, Donnelly MB, Schwartz RW, Sloan DA. Interpreting the judgment of surgical faculty regarding resident competence. J Surg Res. 1999;86:29–35
  46. Ruurda JP, Broeders IAMJ, Pulles B, Kappelhof FM, van der Werken C. Manual robot assisted endoscopic suturing - Time-action analysis in an experimental model. Surgical endoscopy and other interventional techniques. 2004;18:1249–1252
  47. Minekus JPJ, Razing PM, Valstar ER, Dankelman J. Evaluation of humeral head replacements using time-action analysis. J Shoulder and Elbow Surg. 2003;12:152–157
  48. den Boer KT, de Wit LT, Dankelman J, Gouma DJ. Peroperative time-motion analysis of diagnostic laparoscopy with laparoscopic ultrasonography. Br J Surg. 1999;86:951–955
  49. den Boer KT, Dankelman J, Gouma DJ, Stassen HG. Time-action analysis of laparoscopic procedures - input for clinically driven instrument design. Minim Invasive Ther Allied Technol. 2001;10:139–144
  50. Bakker NH, Tanase D, Reekers JA, Grimbergen CA. Evaluation of vascular and interventional procedures with time-action analysis: A pilot study. J Vasc Interv Radiol. 2002;13:483–488
  51. Cuschieri A. Medical errors, incidents, accidents and violations. Minim Invasive Ther Allied Technol. 2003;12:111–120
  52. Sarker SK, Chang A, Vincent C, Darzi AW. Technical skills errors in laparoscopic cholecystectomy by expert surgeons. Surg Endosc Other Intervent Tech. 2005;19:832–835
  53. Tang B, Hanna GB, Joice P, Cuschieri A. Identification and categorization of technical errors by observational clinical human reliability assessment (OCHRA) during laparoscopic cholecystectomy. Arch Surg. 2004;139:1215–1220
  54. Tang B, Hanna GB, Bax NMA, Cuschieri A. Analysis of technical surgical errors during initial experience of laparoscopic pyloromyotomy by a group of Dutch pediatric surgeons. Surg Endosc Other Intervent Tech. 2004;18:1716–1720
  55. Patel AD, Gallagher AG, Nicholson WJ, Cates CU. Learning curves and reliability measures for virtual reality simulation in the performance assessment of carotid angiography. J Am Coll Cardiol. 2006;47:1796–1802
  56. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative “bench station” examination. Am J Surg. 1997;173:226–230
  57. Nielsen PE, Foglia LM, Mandel LS, Chow GE. Objective structured assessment of technical skills for episiotomy repair. Am J Obstet Gynecol. 2003;189:1257–1260
  58. Datta V, Bann S, Beard J, Mandalia M, Darzi A. Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg. 2004;199:603–606
  59. Eubanks TR, Clements RH, Pohl D, Williams N, Schaad DC, Horgan S, et al. An objective scoring system for laparoscopic cholecystectomy. J Am Coll Surg. 1999;189:566–574
  60. Hislop SJ, Hsu JH, Narins CR, Gillespie BT, Jain RA, Schippert DW, et al. Simulator assessment of innate endovascular aptitude versus empirically correct performance. J Vasc Surg. 2006;43:47–55
  61. Anastakis DJ, Regehr G, Reznick RK, Cusimano M, Murnaghan J, Brown M, et al. Assessment of technical skills transfer from the bench training model to the human model. Am J Surg. 1999;177:167–170
  62. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondre K, Stanbridge D, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005;190:107–113
  63. Scott DJ, Rege RV, Bergen PC, Guo WDA, Laycock R, Tesfay ST, et al. Measuring operative performance after laparoscopic skills training: edited videotape versus direct observation. J Laparoendosc Adv Surg Tech A. 2000;10:183–190
  64. Bann SD, Khan MS, Darzi AW. Measurement of surgical dexterity using motion analysis of simple bench tasks. World Journal of Surgery. 2003;27:390–394
  65. Datta V, Mackay S, Mandalia M, Darzi A. The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg. 2001;193:479–485
  66. Wang T, Darzi A, Foale R, Schilling RJ. Virtual Reality Permanent pacing: validation of a novel computerized permanent pacemaker implantation simulator. J Am Coll Cardiol. 2001;37(suppl):493A–494A
  67. Gould DA, Reekers JA, Kessel DO, Chalmers NC, Sapoval M, Patel AA, et al. Simulation devices in interventional radiology: validation pending. J Vasc Interv Radiol. 2006;17:215–216
  68. Hamdorf JM, Hall JC. Acquiring surgical skills. Br J Surg. 2000;87:28–37
  69. Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, et al. Virtual reality simulation for the operating room - Proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 2005;241:364–372
  70. Gallagher AG, Cates CU. Virtual reality training for the operating room and cardiac catheterization laboratory. Lancet. 2004;364:1538–1540
  71. Rantanen EM, Talleur DA. Incremental transfer and cost effectiveness of ground-based flight trainers in university aviation programs (Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting). In: 2005;p. 764
  72. Cates CU, Patel AD, Nicholson WJ. Use of virtual reality simulation for mission rehearsal for carotid stenting. JAMA. 2007;297:265–266
  73. Aggarwal R, Undre S, Moorthy K, Vincent C, Darzi A. The simulated operating theatre: comprehensive training for surgical teams. Quality & Safety in Health Care. 2004;13:I27–I32
  74. Moorthy K, Munz Y, Adams S, Pandey V, Darzi A. A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Ann Surg. 2005;242:631–639
  75. Aggarwal R, Darzi A. Organizing a surgical skills center. Minim Invasive Ther Allied Technol. 2005;14:275–279
  76. http://site.riti.org.uk/public/default.aspx?PageID=5. Accessed 2006.
  77. Satava RM. Identification and reduction of surgical error using simulation. Minim Invasive Ther Allied Technol. 2005;14:257–261

 Competition of interest: none.

PII: S0741-5214(07)00970-6

doi:10.1016/j.jvs.2007.05.041

Journal of Vascular Surgery
Volume 46, Issue 5 , Pages 1055-1064, November 2007

Article Tools

* Image Usage & Resolution