Discussion

Article Outline

• Copyright

Dr David L. Dawson (Sacramento, Calif). Skeptics would say that simulation is a credentialing tool of the future—and it always will be. I suspect, however, that we will see procedural simulation become a more substantial part of the certification process in the future. To date, procedural training has been the primary application of simulation technology. Dr Tedesco and her Stanford surgical colleagues have addressed another important role for simulation—the potential use of the simulator as an assessment tool. Further, their observation that the machine may not work well as a stand-alone assessment tool is an important one.

Test developers recognize five characteristics of a good assessment tool: reliability, validity, usability, comprehensiveness, and discrimination, the last being the focus of this report. In this small study, there were no measures of test reliability. The validity of the testing methodology is yet to be established, but I think it certainly could be with more studies. Simulation, I believe, is a usable test method, but one associated with substantial costs. As reported, this particular test was not comprehensive, as this pilot project was of limited scope.

Thus, the authors focus primarily on the ability of simulation-based endovascular skills assessment to discriminate between low and high endovascular case experience in residents, assuming that the volume of prior experience is a reasonable surrogate for either clinical competence or proficiency. Their observations suggest that an expert observer can evaluate performance with a structured assessment tool and using this evaluation can distinguish between low and high performance.

They found that self-assessments were not particularly useful, with most subjects grading themselves as midlevel in performance. These self-assessment scores do not appear to effectively discriminate between those who actually did perform well or not, an observation that provides additional support for the argument that other, better, and perhaps more independent assessment tools are needed.

I have three questions. First, this evaluation model is an example of a criterion-based test. How were the criteria established for expected or desired performance? That is, how were the definitions of “poor performance” and “flawless performance” defined and how were the gradations between delineated?

Second, in contrast to the reports of others working with this type of technology, you were not able to use the objective performance metrics of the stimulator to distinguish between the levels of performance of the test subjects. Do you think that this might be attributable to the limited time spent on the single case simulation that each subject performed?

And finally, it is generally accepted that expert observers can meaningfully evaluate professional competence. I think this is a concept that the American Board of Surgery has bought into, as “expert observers” in small hotel rooms often do assess professional competence of people seeking certification. Time and resource constraints, however, limit the practicality of always depending on evaluations by senior subject matter experts. Do you think it would be valid to have subjective assessments made by trained educational testing specialists who are not physicians or endovascular specialists?

I appreciate the opportunity to discuss this interesting and timely paper and I expect that we will be hearing more about this topic as the technologies mature and experience grows.

Dr Tedesco. Thank you, Dr Dawson, for your discussion and questions and especially thank for asking three, not 17, questions. With respect to the first question, our senior author and second-year endovascular fellow developed the questionnaire based on reports by Reznick and others. Specifically addressing how we scaled the Likert scale, we graded the applicants from 1 to 5, and where 1 on the scale indicated a poor performance and a 5, flawless performance. Criteria for a fail grade or poor performance were frequently stopping the procedure, clearly being unsure of the next move, awkward or inappropriate movements that would result in potential injury to the vessels, sizing the target lesion that might result in rupture. Those would be indications for a fail or a poor performance. We defined flawless or a superior performance with a score of 5 as consistently handling the wires and catheters with minimal damage to the vessels, clear economy of motion and efficiency, a well-thought-out plan of procedure with effortless flow or a demonstration of sound knowledge of the appropriate wires and catheters for the renal angioplasty and stenting procedure. That is how we measured the scale that was developed by our senior author.

To address your second question: yes, you are correct. Objective criteria have been used to differentiate novice and expert subjects. However, the expert subjects that were used in prior studies had performed over 300, sometimes over 1000 endovascular cases. In our study, the objective criteria was not able to discern the small difference between the experienced groups which were between—the low experience group, with less than 20 endovascular cases, and the moderate experience group with between 20 and 100. Perhaps the limitation of this study and this simulator is that it is unable to detect small differences, which also highlights the importance and need for the expert observer. In addition, there was limited time for each subject, which created a definite limitation for this study. Perhaps if the low experience group had more time to practice they would have performed better and that is certainly a limitation of our study.

With respect to your last question, the self-assessment was not an accurate assessment of skill level in this particular study and perhaps testing specialists could perform the assessment. However, I think it is potentially dangerous to remove physicians from the testing scenario, as physicians are the people performing the actual live endovascular skill on a day-to-day basis. There are nuances and style points that perhaps only physicians can understand and would be able to score better than trained specialists. This of course has implications for the application of simulation-based skill assessment with respect to physician time and cost.