|LETTER TO EDITOR
|Year : 2015 | Volume
| Issue : 2 | Page : 270-271
Simulation for "Evaluation" and teaching "Standard operating procedures"
Pankaj Kundra, Anusha Cherian
Department of Anaesthesiology and Critical Care, JIPMER, Puducherry, India
|Date of Web Publication||16-Apr-2015|
Department of Anaesthesiology and Critical Care, JIPMER, Puducherry - 605 006
Source of Support: None, Conflict of Interest: None
|How to cite this article:|
Kundra P, Cherian A. Simulation for "Evaluation" and teaching "Standard operating procedures". J Anaesthesiol Clin Pharmacol 2015;31:270-1
|How to cite this URL:|
Kundra P, Cherian A. Simulation for "Evaluation" and teaching "Standard operating procedures". J Anaesthesiol Clin Pharmacol [serial online] 2015 [cited 2020 Jun 4];31:270-1. Available from: http://www.joacp.org/text.asp?2015/31/2/270/155208
To the Editor,
Current evaluation systems are based on the assumption that if a student spends adequate time on a clinical situation, he will master it. The existing model of evaluation therefore relies on a logbook of clinical exposure coupled with a written and oral examination, to serve as an assessment of the student's mastery of the subject. It is obvious that this system of examination is a poor surrogate of direct observation of the students overall skills and at best test the lower levels of cognitive and psychomotor domains. The affective domain and higher cognitive and psychomotor areas are largely not tested.
In clinical medicine and allied areas the best test of a student's readiness to practice is his/her ability to manage a real life situation. However to test a student in a real life situation is impractical due to several reasons. Principal among them are concerns regarding safety of the student, patient and environment as well as the inability to provide a standardized scenario. The solution to this conundrum is to a large extent provided by using simulation. Using a simulated environment allows the educator to test the cognitive, psychomotor as well as the affective learning outcomes in a safe environment.
Competency in performing complex tasks in a simulated environment assesses the higher level of cognitive domain such as application, synthesis and evaluation as opposed to traditional methods, which assesses only the ability to recall concepts and facts. Evaluation of psychomotor domain by simulation focuses on more comprehensive set of skill tests on a broad range. Affective domain is assessed to find out if the student has internalized the knowledge of values, attitudes and beliefs to influence his professional behavior. Simulation can be used to integrate all 3 domains from low to high level. The ability to receive information and respond by integration of knowledge, judgment, communication and teamwork can be assessed in a simulated setting. This is particularly useful in evaluating anesthesia residents as there is currently no other way to assess the nontechnical skills which are important in crew resource management.  Nontechnical skills are a combination of cognitive and affective skills such as communication, situation awareness that cannot be assessed by traditional methods. Moreover, evaluation of all students takes place using the same scenario under the same conditions.
There are a few prerequisites before designing simulation for evaluation of students. The scenarios must be realistically programmed and evaluated. Their validity and reliability must be established. Most of the evaluation tools use scores for each of the items. Hence there should be internal consistency, which is the measure of quality of the items in the score. It should be comparable with other methods of evaluation (criterion validity), should be realistic and a legitimate indicator of concept being tested, the extent to which a test reflects the concept to be learnt (construct validity) and produce consistent results. 
There are several evaluation tools currently available but none of them are ideal. A review of assessment during simulation done by Byrne and Greaves concluded that the methodologies currently in use were largely indeterminate.  Most of the studies assessed performers on their compliance to standard algorithms or procedures, ability to solve complex cases or the time taken to make decision based on observations and judgments. The assessment tools were mainly in the form of scores and replay of videos tapped during their performance. A study by Devitt et al. assessed if a simulator-based performance evaluation could demonstrate construct validity and realism. They found that certain scenarios could discriminate skill levels between anesthesia residents and staff without loss of realism.  They also found considerable inter observer agreement among different educators while assessing the same simulated anesthesia event on video.  Gaba et al. also found inter-rater agreement during simulator-based assessments.  However, most studies do not test the whole range of skills to be mastered in the given curriculum. Hence there is lack of content. Currently few countries have incorporated simulation-based scenarios to test students in their curriculum to meet specific objectives. However, this is at its infancy and needs more validation.
Simulation is an extremely important resource in teaching standard operating protocols (SOPs) and procedures. SOPs are components of clinical pathways, which are based on current recommendations from experts and professional organizations to improve diagnostic and therapeutic management with in medicine. Introduction of these protocols have led to a fall in morbidity and mortality and have shown economic benefits in the medical organization. Currently, there is much stress laid on protocol based training practice of appropriate management of a clinical situation. SOPs are also implemented when there is low frequency of exposure to a particular clinical situation or procedure. Simulation helps to develop the right attitude and skills necessary to handle these situations. An offshoot to this is the crisis resource management (CRM). Following the report by the institute of medicine in America in 2000, it was found that nearly 1 lakh deaths occurred due to avoidable errors in the health profession. It recommended adaptation of the aviation model of crew resource management, which was called CRM in medicine. By definition, it is the promotion of safety by addressing behavioral and cognitive skills needed to effectively manage all health resources, especially in a crisis situation through team training. The key components of CRM for building team effectiveness are:
This principle is especially effective when applied to high acuity situations like in the intensive care units, operating rooms and emergency departments. However, there is presently no evidence to state that CRM results in effective team building. But, it is still believed to help in developing medical institutes as high reliability organization. Pioneering work in anesthesia was done by Gaba et al. in the development of anaesthesia CRM.  This is a simulation based training which focuses on team effectiveness in managing critical high-risk situation, video taping them and debriefing the performers to achieve optimum output. It has helped in the recognition of abnormal situations; effectively perform what needs to be done as a team by training systematically on simulators to effectively perform leadership, team work, communication and resource management. Several programs have followed the success of ACRM and currently obstetrics, pediatrics, surgery and emergency medicine have their own high-risk simulation program based on this model.
- Team leadership,
- Mutual performance monitoring,
- Backup behavior,
- Adaptability, and
- Team orientation.
To conclude, simulation can create realistic situations and exposes several shortcomings in human behavior and skills, which cannot be assessed by traditional methods. Hence the future direction in assessment would be to integrate classroom training with multidisciplinary skill and behavior assessment on a simulation based platform. Nontechnical skills and CRM is key in the development of high reliability medical organization.
| References|| |
Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998;89:8-18.
Byrne AJ, Greaves JD. Assessment instruments used during anaesthetic simulation: Review of published studies. Br J Anaesth 2001;86:445-50.
Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Noel AG, et al.
Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg 1998;86:1160-4.
Devitt JH, Kurrek MM, Cohen MM, Cleave-Hogg D. The validity of performance assessments using simulation. Anesthesiology 2001;95:36-42.
Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Murphy PM, et al.
Testing the raters: Inter-rater reliability of standardized anaesthesia simulator performance. Can J Anaesth 1997;44:924-8.
Gaba DM, DeAnda A. A comprehensive anesthesia simulation environment: Re-creating the operating room for research and training. Anesthesiology 1988;69:387-94.