Testing the validity of three acute care assessment tools for assessing residents’ performance during in situ simulation: the ACAT-SimSit study

^a Emergency Department, Pitié-Salpêtrière Hospital, Sorbonne Université, GRC 14, BIOFAST, AP-HP, Paris

^b Laboratoire Interuniversitaire des Sciences de l’Education (LISEC) – Learning Sciences Department, Strasbourg University, Strasbourg

^c Emergency Care Training Center (CESU-76A), Department of Anaesthesiology, Intensive Care and Emergency Medical Services, Rouen University Hospital

^d Centre d'Enseignement des Soins d'urgence, Medical Training Center (MTC), Rouen University Hospital, Rouen

^e Emergency Department, Lariboisière Hospital, Assistance Publique – Hôpitaux de Paris (AP-HP)

^f Faculté de Médecine, Université de Paris-Cité, Paris

^g Unité de simulation Européenne en santé (UNISIMES), Faculté de Médecine, maïeutique et science de la santé, Université́ de Strasbourg, Strasbourg

^h Emergency Department, Hôpital Bichat, Assistance Publique – Hôpitaux de Paris (AP-HP)

ⁱ Emergency Department, Georges Pompidou European Hospital, Assistance Publique – Hôpitaux de Paris (AP-HP), Paris

^j Emergency Department, Hôpitaux Universitaires de Strasbourg, Strasbourg

^k Emergency Department, Cochin Hospital, Assistance Publique – Hôpitaux de Paris (AP-HP), Paris, France

The assessment of technical and nontechnical skills in emergency medicine requires reliable and usable tools. Three Acute Care Assessment Tools (ACATs) have been developed to assess medical learners in their management of cardiac arrest (ACAT-CA), coma (ACAT-coma) and acute respiratory failure (ACAT-ARF).

This study aims to analyze the reliability and usability of the three ACATs when used for in situ (bedside) simulation.

This prospective multicenter validation study tested ACATs using interprofessional in situ simulations in seven emergency departments and invited training residents to participate in them. Each session was rated by two independent raters using ACAT. Intraclass correlation coefficients (ICC) were used to assess interrater reliability, and Cronbach’s alpha coefficient was used to assess internal consistency for each ACAT. The correlation between ACATs’ scores and the learners’ level of performance was also assessed. Finally, a questionnaire and two focus groups were used to assess the usability of the ACATs.

A total of 104 in situ simulation sessions, including 85 residents, were evaluated by 37 raters. The ICC for ACAT-CA, ACAT-coma and ACAT-ARF were 0.95 [95% confidence interval (CI), 0.93–0.98], 0.89 (95% CI, 0.77–0.95) and 0.92 (95%CI 0.83–0.96), respectively. The Cronbach’s alphas were 0.79, 0.80 and 0.73, respectively. The ACAT-CA and ARF showed good construct validity, as third-year residents obtained significantly higher scores than first-year residents (P < 0.001; P < 0.019). The raters supported the usability of the tools, even though they expressed concerns regarding the use of simulations in a summative way.

This study reported that the three ACATs showed good external validity and usability.