Featured Article| Volume 45, P50-59, August 2020

The Impact of Emerging Simulation-Based Technologies on the Management of Deteriorating Patients: Aiming for a Gold Standard Educational Evaluation


      • Simulation evaluation frameworks are discussed in relation to the management of deteriorating patients.
      • The impact of face to face and virtual simulation modalities are described.
      • The impact of simulation programs on clinical practice and patient safety should be considered.



      Measuring the impact of patient safety interventions is challenging. This article aims to illustrate a “gold standard” model of program evaluation incorporating examples from two patient deterioration programs.


      Australian nurses were trained in primary responses to emergencies in four hospitals using either face-to-face (F2F) or screen-based simulation versions of a simulation program. Evaluation outcomes were measured using Kirkpatrick's evaluation hierarchy covering participant ‘reaction’, ‘learning’, ‘behaviour’ change and ‘results’—based on 1,564 pre–post intervention vital signs chart reviews.


      Seventy-four nurses participated. Reaction—participant confidence/competence ratings and Learning improved significantly in both modalities (p < .001). Behaviour—oxygen delivery systems were used more appropriately after training in the F2F group (p = .037). Applicable recording of oxygen saturation (SpO2) improved significantly (p ≤ .008) in both modalities. Results—at least a two-fold increase in the overall initiation of a clinical review after training, in both modalities (p < .001; effect: d = 0.41 F2F and d = 0.35 screen-based simulation).


      Kirkpatrick's evaluation model enables a suitable template for gold standard education evaluations.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Clinical Simulation In Nursing
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Australian Commission on Safety and Quality in Healthcare (ACSQH)
        NSQHS Standards.
        2nd ed. 2017 (Retrieved from)
        • Bates R.
        A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence.
        Evaluation and Program Plannin. 2004; 27: 341-347
        • Bogossian F.
        • Cooper S.
        • Cant R.
        • Beauchamp A.
        • Kain V.
        • Porter J.
        • FIRST2ACT Research Team
        • et al.
        Undergraduate nursing students’ performance in recognising and responding to sudden patient deterioration in high fidelity simulated environments: Quantitative results from an Australian multi-centre study.
        Nurse Education Today. 2013; 34: 691-696
        • Bogossian F.
        • Cooper S.
        • Cant R.
        • Porter J.
        • Forbes H.
        • FIRST2ACT™ Research Team
        A trial of e-simulation of sudden patient deterioration (FIRST2ACTWEB™) on student learning.
        Nurse Education Today. 2015; 35: 36-42
        • Cant R.
        • Cooper S.J.
        The benefits of debriefing as formative feedback in nurse education.
        Australian Journal of Advanced Nursing. 2011; 29: 37-47
        • Buykx P.
        • Kinsman L.
        • Cooper S.
        • McConnell-Henry T.
        • Cant R.
        • Endacott R.
        • Scholes J.
        FIRST2 ACT: Educating nurses to identify patient deterioration – a theory-based model for best practice simulation education.
        Nurse Education Today. 2011; 31: 687-693
        • Cant R.
        • Cooper S.
        Simulation-based learning in nurse education: Systematic review.
        Journal of Advanced Nursing. 2009; 66: 3-15
        • Cantrell M.A.
        • Franklin A.
        • Leighton K.
        • Carlson A.
        The evidence in simulation-based learning experiences in nursing education and practice: An umbrella review.
        Clinical Simulation In Nursing. 2017; 13: 634-667
        • Care Quality Commission
        The fundamental standards.
        2017 (Retrieved from)
        • Chung C.
        • Cooper S.J.
        • Cant R.P.
        • Connell C.
        • McKay A.
        • Kinsman L.
        • Breakspear T.
        • et al.
        The educational impact of web-based and face-to-face patient deterioration simulation programs: An interventional trial.
        Nurse Education Today. 2018; 64: 94-98
        • Connell C.J.
        • Endacott R.
        • Jackman J.A.
        • Kiprillis N.R.
        • Sparkes L.M.
        • Cooper S.J.
        The effectiveness of education in the recognition and management of deteriorating patients: A systematic review.
        Nurse Education Today. 2016; 44: 133-145
        • Cook D.
        • Hatala R.
        • Brydges R.
        • Zendejas B.
        • Szostek J.
        • Wang A.
        Technology-enhanced simulation for health professions education a systematic review and meta-analysis.
        JAMA. 2011; 306: 978-988
        • Cooper S.
        • Cant R.
        • Bogossian F.
        • Bucknall T.
        • Hopmans R.
        Doing the right thing at the right time: Assessing responses to patient deterioration in electronic simulation scenarios using Course-of-Action analysis.
        Computers, Informatics, Nursing. 2015; 33: 199-207
        • Cooper S.
        • Cant R.
        • Bogossian F.
        • Kinsman L.
        • Bucknall T.
        Patient deterioration education: Evaluation of face-to-face simulation and e-simulation approaches.
        Clinical Simulation in Nursing. 2015; 11: 97-105
        • Cooper S.
        Simulation versus lecture? Measuring educational impact: Considerations for best practice.
        Evidence-Based Nursing. 2016; 19: 55
        • Cooper S.J.
        • Hopmans R.
        • Cant R.P.
        • Bogossian F.
        • Giannis A.
        • King R.
        Deteriorating patients: Global reach and impact of an E-simulation program.
        Clinical Simulation in Nursing. 2017; 3: 562-572
        • Cooper S.
        • Kinsman L.
        • Chung C.
        • Cant R.
        • Boyle J.
        • Bull L.
        • Rotter T.
        • et al.
        The impact of web-based and face-to-face simulation on patient deterioration and patient safety: Protocol for a multi-site multi-method design.
        BMC Health Services Research. 2016; 16: 475
        • Cooper S.
        • Kinsman L.
        • Chung C.
        • Cant R.
        • Boyle J.
        • Cameron A.
        • Rotter T.
        • et al.
        The impact of face-to-face and web-based simulation on patient deterioration and patient safety. [final report].
        (Retrieved from)
        • Cooper S.
        • Hopmans R.
        • Hopmans K.
        2019 (Retrieved from)
        • Crofts J.
        • Bartlett C.
        • Ellis D.
        • Hunt L.P.
        • Fox R.
        • Draycott T.J.
        Training for shoulder dystocia: A trial of simulation using low-fidelity and high-fidelity mannequins.
        Obstetrics & Gynecology. 2006; 108: 1477-1485
        • Draycott T.
        • Crofts J.F.
        • Ash J.P.
        • Wilson L.V.
        • Yard E.
        • Sibanda T.
        • Whitelaw A.
        Improving neonatal outcome through practical shoulder dystocia training.
        Obstetrics & Gynecology. 2008; 112: 14-20
        • Gavin N.R.
        • Satin A.J.
        Simulation training in obstetrics.
        Clinical Obstetrics and Gynecology. 2017; 60: 802-810
        • Grohmann A.K.S.
        Evaluating training programs: Development and correlates of the questionnaire for professional training evaluation.
        International Journal of Training and Development. 2013; 17: 2
        • Heale R.
        • Twycross A.
        What is a case study?.
        Evidence-Based Nursing. 2018; 21 (Retrieved from): 7-8
        • Hegland P.A.
        • Aarlie H.
        • Stromme H.
        • Jamtvedt G.
        Simulation-based training for nurses: Systematic review and meta-analysis.
        Nurse Education Today. 2017; 54: 6-20
        • IBM Corp
        IBM SPSS Statistics for Windows, Version 24.0.
        IBM Corp, Armonk, NY2016
        • International Liaison Committee on Resuscitation (ILCOR)
        (Retrieved from)
        • Jolly B.
        Control and validity in medical education research.
        Medical Education. 2001; 35: 920-921
        • Kardong-Edgren S.
        Striving for higher levels of evaluation in simulation.
        Clinical Simulation in Nursing. 2010; 6: e203-e204
        • Kirkpatrick D.
        Seven keys to unlock the four levels of evaluation.
        Performance Improvement. 2006; 45: 7
        • Kirkpatrick D.L.
        Evaluating Training Programs.
        Tata McGraw-Hill Education, Chicago1975
        • Kolb A.Y.
        • Kolb D.A.
        Learning styles and learning spaces: Enhancing experiential learning in higher education.
        Academy of Management Learning & Education. 2005; 4: 193-212
        • Laerdal
        Courses and Learning: vSim for Nursing.
        2019 (Retrieved from)
        • Liaw S.
        • Chng Y.K.
        • Wong L.F.
        • Ho J.T.Y.
        • Mordiffi S.Z.
        • Cooper S.
        • Ang E.N.K.
        • et al.
        The impact of a web-based educational program on the recognition and management of deteriorating patients.
        Journal of Clinical Nursing. 2017; 26: 4848-4856
        • Liaw S.Y.
        • Wong L.F.
        • Lim E.Y.
        • Ang S.B.
        • Mujumdar S.
        • Ho J.T.
        • Ang E.N.K.
        • et al.
        Effectiveness of a web-based simulation in improving nurses’ workplace practice with deteriorating ward patients: A pre-and post-intervention study.
        Journal of Medical Internet Research,. 2016; 18: e37
        • McGaghie W.C.
        • Issenberg S.B.
        • Barsuk J.H.
        • Wayne D.B.
        A critical review of simulation-based mastery learning with translational outcomes.
        Medical Education. 2014; 48: 375-385
        • Murphy M.
        • Curtis K.
        • McCloughen A.
        What is the impact of multidisciplinary team simulation training on team performance and efficiency of patient care? An integrative review.
        Australasian Emergency Nursing Journal. 2016; 19: 44-53
        • Nagy S.
        • Mills J.
        • Waters D.
        • Birks M.
        Using Research in Healthcare Practice.
        1st ed. Lippincott Williams & Wilkins Pty Ltd, New South Wales, Australia2010
        • O'Leary J.
        • Nash R.
        • Lewis P.
        Standard instruction versus simulation: Educating registered nurses in the early recognition of patient deterioration in paediatric critical care.
        Nurse Education Today. 2015; 36: 287-292
        • Robertson J.M.
        • Dias R.D.
        • Yule S.
        • Smink D.S.
        Operating room team training with simulation: A systematic review.
        Journal of Laparoendoscopic & Advanced Surgical Techniques. 2017; 27: 475-480
        • Sullivan N.
        An integrative review: Instructional strategies to improve nurses' retention of cardiopulmonary resuscitation priorities.
        International Journal of Nursing Education Scholarship. 2015; 12: 37-43
        • Tjoflåt I.
        • Brandeggen T.
        • Strandberg E.
        • Dyrstad D.
        • Husebø S.
        Norwegian nursing students’ evaluation of vSim for Nursing.
        Advances in Simulation. 2018; 3: 10
        • Walton M.
        • Shaw T.
        • Barnet S.
        • Ross J.
        Developing a national patient safety education framework for Australia.
        BMJ Quality & Safety. 2006; 15: 437-442
        • Wang G.G.
        • Wilcox D.
        Training evaluation: Knowing more than is practiced.
        Advances in Developing Human Resources. 2006; 8: 528-539
        • Yardley S.
        • Dornan T.
        Kirkpatrick’s levels and education ‘evidence’.
        Medical Education. 2012; 46: 97-106