Enhancing residents’ neonatal resuscitation competency through unannounced simulation-based training

Background Almost half of pediatric third-year residents surveyed in 2000 had never led a resuscitation event. With increasing restrictions on residency work hours and a decline in patient volume in some hospitals, there is potential for fewer opportunities. Purpose Our primary purpose was to test the hypothesis that an unannounced mock resuscitation in a high-fidelity in-situ simulation training program would improve both residents’ self-confidence and observed performance of adopted best practices in neonatal resuscitation. Methods Each pediatric and medicine–pediatric resident in one pediatric residency program responded to an unannounced scenario that required resuscitation of the high fidelity infant simulator. Structured debriefing followed in the same setting, and a second cycle of scenario response and debriefing occurred before ending the 1-hour training experience. Measures included pre- and post-program confidence questionnaires and trained observer assessments of live and videotaped performances. Results Statistically significant pre–post gains for self-confidence were observed for 8 of the 14 NRP critical behaviors (p=0.00–0.03) reflecting knowledge, technical, and non-technical (teamwork) skills. The pre–post gain in overall confidence score was statistically significant (p=0.00). With a maximum possible assessment score of 41, the average pre–post gain was 8.28 and statistically significant (p<0.001). Results of the video-based assessments revealed statistically significant performance gains (p<0.0001). Correlation between live and video-based assessments were strong for pre–post training scenario performances (pre: r=0.64, p<0.0001; post: r=0.75, p<0.0001). Conclusions Results revealed high receptivity to in-situ, simulation-based training and significant positive gains in confidence and observed competency-related abilities. Results support the potential for other applications in residency and continuing education.


Introduction
Four million babies are born in the United States per year. Approximately 10% of all neonates require some level of resuscitation by health care providers at the time of birth, and less than 1% will require advanced resuscitative measures (1,2). The first course to teach a standardized neonatal resuscitation program (NRP) was developed in 1985 jointly via the American Academy of Pediatrics (AAP) and the American Heart Association (AHA). More than 1.5 million health care professionals have been trained in NRP with regular updates to the program (2). Despite formal NRP education and training, real-life situations often prove difficult to manage. Studies have shown that retention of skills learned in these courses is only 6Á12 months (3).
Becoming proficient in the NRP usually requires routine participation in real-life events. The Accreditation Council for Graduate Medical Education (ACGME) and the Residency Review Committee (RRC) allow a maximum of 6 months of intensive care in a pediatric training program (4). Based on ACGME and RRC guidelines, residents can complete pediatric training with only three, and no more than four, rotations in a neonatal intensive care unit, with exposure to patients of variable acuity and inconsistent delivery room experiences (5). Given that new restrictions (e.g., duty hours) in training guidelines have shifted through recent years, residents are finishing training with fewer hands-on experiences in neonatal resuscitation and cite resuscitation management as a deficiency (6,7). Further, with the increased presence of neonatal nurse practitioners and other health professionals and a decline in patient volume in some hospitals, residents are faced with limited opportunities to gain resuscitation experience (8Á10). A survey of pediatric residents in 2000 found that 44% of third-year residents had never led a resuscitation event (6).
In the last decade, simulation-based training has been increasingly adopted for clinical training across the medical education continuum. Such training often immerses trainees in realistic scenarios. These scenarios replicate real-life situations with sufficient fidelity to achieve 'suspended disbelief' (11). Such training was first adopted for medical training in the anesthesia community with the use of mannequins for basic life Support training (12Á14). Successful use of simulation-based training has been reported more recently in other specialties (e.g., emergency medicine, surgery, obstetrics and gynecology, pediatrics) (15Á26). The potential for medical simulation in pediatric residency training is clear, particularly in light of increased national concerns for providing safe, highquality, affordable patient care, and the declining opportunities for real-life neonatal resuscitation experiences (24,27,28). Simulated neonatal resuscitation offers hands-on, guided experience with realistic scenarios that use after-action debriefing that includes both focused and deliberate feedback and practice (10,26,29). Consequently, simulation-based training may offer greater opportunity than real-life situations for explicit teaching and learning and the purposeful refinement of relevant knowledge and skills. Further, such training has the potential to improve clinical decision making, sharpen skills, reduce medical error, increase confidence as team leaders, and improve overall teamwork. Conducting such training in the real-life clinical setting also offers the opportunity to identify setting-specific latent conditions (e.g., location of specific resources), practice with real-life team members, and enhance feasibility of embedding such training into the already-packed curriculum and work expectations of busy residents and faculty. The following research questions guided the study: Residents were informed about the project prior to initiation, that each would participate in only one session, but they would not know when they would be called to a training session; thus, creating the unannounced mock code feature.

Training intervention methods
An area of the neonatal intensive care unit (NICU) was set up to support the training environment. Upon arrival to an unannounced mock code, the resident encountered and responded to HFIS resuscitation scenario that required him/her to perform both bag and mask positive pressure ventilation (PPV) and chest compressions in an infant delivered by emergent cesarean section for nonreassuring fetal heart tones. A staff nurse participated as an actor in the scenario to reflect a real-life situation.
Resuscitation proceeded up to 10 min followed immediately by a structured debriefing session in the same setting between the resident and faculty trainer, lasting 10Á15 min (30). The scenarioÁdebriefing cycle was repeated with a second scenario that required the resident to perform neonatal resuscitation involving PPV and intubation due to respiratory distress associated with meconium-stained amniotic fluid. Following the debriefing of the second simulation scenario, the training session concluded with an emphasis of key targets for enhancing future performance. The total length of a training session was one hour.
In the focused debriefings, the faculty trainer/facilitator prompted the resident to reflect on what went well and what could be improved. The trainer also provided specific feedback and coaching, targeting specific improvement strategies to focus practice on enhancing targeted knowledge and behaviors/skills. Closure was achieved by summarizing the scenario-based experiences, reinforcing strengths, and targeting specific strategies for further improvement. Because debriefing occurred in the simulation setting, video recordings were not necessary to achieve accurate recall. Residents were asked to honor the confidentiality of the session and scenarios by not discussing these with anyone outside of the session.

Training evaluation methods
Kirkpatrick's four levels of training effectiveness provided a model for program evaluation (31). Residents completed pre-and post-training questionnaires (Level 1: reaction, motivation; Level 2a: change in confidence/ perceived competence). Direct observations of residents' performance of critical NRP behaviors was conducted by trained assessors during training, and later, by assessment of video-recorded performances conducted by a single content expert (i.e., 'Gold Standard') who was a content expert for instrument development (Level 2b: change in knowledge; Level 3: behavior change). For logistical reasons, we were not able to assess Level 4 outcomes (real-life impact, outcomes).

Instrumentation
The pre-and post-training questionnaire contained several demographic items (e.g., prior experience in NRP, PGY level) and 14 specific behaviors reflecting adopted NRP best practices derived from literature review and input from content experts experienced with resident education (e.g., stimulating and drying, administration of supplemental oxygen, positive pressure ventilation, intubation, and chest compressions). Residents responded to the 14 NRP items using a four-point Likert-type scale (00not confident at all to 40completely confident) that yielded a maximum possible total score of 56. Residents responded to the same 14 NRP items on the post-training questionnaire and also to 10 items pertaining to session effectiveness (5-point, Likert-type scale, 10Definitely no to 50Definitely yes). Three open-ended questions provided additional opportunities for residents to comment on their desire for future training, input on increasing realism of scenarios, and other feedback for improving training sessions. The direct observation assessment instrument used for both live and video-recorded resident performances included the specific behaviors reflected in 14 NRP critical behaviors and additional performance quality and time measurements defined for specific NRP behaviors, yielding 23 discrete assessment judgments. The overall structure of the instrument was an adaptation of an assessment protocol developed and used previously for evaluating simulation-based training in pediatric resuscitation using unannounced mock codes in which the instrument demonstrated good psychometric qualities. Scoring of the observation-based assessments yielded a maximum possible total score of 41.

Data collection
Residents completed the pre-program questionnaire at the start of the project period before any training was initiated. Residents completed the post-training session questionnaire immediately upon completion of their respective training event. For each questionnaire, residents followed directions to generate a record identification code for matching individual responses across measures, while maintaining anonymity. At each training session, one trained observer only recorded the directly observed behaviors and did not assign any numerical scores. A senior member of the team representing an expert clinician ('gold standard') completed a separate and independent observation-based assessment of each video-recorded resident performance using the same instrument that was used in the live session. All observation-based assessments were subsequently scored by two members of the research team using a standard 'grading' rubric reflecting the critical NRP behavior standards. Three trained assessors were neonatology Fellows, of which one was a senior neonatology faculty member who served as the 'Gold standard' content expert.

Data analysis
After data compilation and initial screening, descriptive statistics were calculated. Data for the pre-and postself-confidence questionnaire responses were examined using the Wilcoxon Signed Rank Test. Content analysis was used to examine narrative responses to open-ended questions for common perspectives. Comparison of residents' performances in scenarios 1 and 2 was accomplished using the paired t-test for observation-based assessment data. To examine relationships between residents' self-confidence and actual observed performance and between observation data for live and video-based assessments, correlational analyses were conducted.

Results
Results are reported by the research questions set forth at the beginning of this article.

Question 1: Feasibility of training
Each of the 32 residents rotating through CHNOLA during the study period completed one training session and participated voluntarily in the study (100%, 11 PGY-1, 11 PGY-2, 10 PGY-3). One faculty member was able to conduct one or more in situ unannounced mock code training sessions per training day. No narrative feedback from residents or feedback from faculty and staff participants indicated any significant interruption or inconvenience with other typical responsibilities.
Question 2: Change in residents' self-confidence Pre-training data were not available for five residents due to a logistical glitch, so the analysis included matched pre/ post data for 27 residents. Table 1 shows positive gains for 13 of the 14 self-confidence items, with statistical significance for 8 (Wilcoxon signed rank test, p 00.00Á0.03) and for the total scale score (p00.00): Analysis of performance by post-graduate year in training revealed positive preÁpost gains for all three groups, but results were not statistically significant (Fig. 1).

Question 3: Change in residents NRP performance, based on direct observation assessment
With a maximum possible total score of 41, the average preÁpost gain was 8.28 and statistically significant (paired t-test, n 032, p B0.0001). Results of the observationbased assessments using a video recording also revealed statistically significant performance gains (pB0.0001). Results showed an improvement in response time and quality of actions from the first simulation scenario to the second. The greatest improvement was found in first-year trainees (PGY-1). The average performance scores for each of the two scenarios, represented per year of training, are shown in Fig. 2.

Question 3: Correlation between self-reported self-confidence and observed performance
The correlation between self-confidence and actual performance was not statistically significant (pre: r 00.12, p B0.55; post: r 00.06, p B0.76).

Question 4: Correlation between live and video-based observation assessments
Correlation between the live and video-based observations was strong for both pre-and post-training scenario performances (pre: r 00.64, p B0.0001; post: r 00.75, p B0.0001). The differences in total scale scores between live and video-based observations were greater for pre than post-training performances (pre: 3.82, p 00.0011; post: 0.43, p 00.43). Pre-to post-training gains were noted for both live and video-based observations, and  item ratings were generally higher for video-based than live assessments.

Discussion
Several limitations of this study are worthy of mention. First, the study occurred in one setting at one institution with pediatric residents. Second, the study did not include a control group, but used residents as their own control through preÁpost training measures. Given the existing structure of residents' educational program and limited hands-on NRP experience, and faculty agreement that NRP knowledge and skill was limited, priority was given to providing training to all currently enrolled residents. Third, training evaluation included measures addressing Kirkpatrick's Levels 1Á3 (i.e., reaction, self-confidence, performance in simulated setting), but Level 4 measurement of actual practice outcomes in the real-life setting was not possible because of logistical constraints. The extent to which the results generalize to other pediatric residency programs will depend on the similarity of contexts and residents' prior experiences in neonatal resuscitation. Also, replication of the curriculum and training with an intervention and control group and the inclusion of Level 4 outcomes measurement would contribute to strengthening the program evaluation design and resulting evidence from conclusions could be generalized to other pediatric residency settings.
Regarding Question 1, results demonstrated that the in situ NRP simulation-based training program was logistically feasible. Training fit well within the context of the existing curriculum and clinical care requirements. Using an unannounced/unscheduled format provided the flexibility necessary to complete activities within the context of the residents' schedule and the requirements of a typical work day within a reasonable study time frame. Most residents responded positively to the unannounced mock NRP simulation training.
Results pertaining to Question 2 (self-reported selfconfidence) provided strong evidence of the efficacy of simulation-based training for NRP. While self-confidence is not entirely a proxy for actual performance in real-life settings, its contribution should not be under-estimated. According to Bandura's construct of self-efficacy and the work on developing expertise by Ericsson et al., selfconfidence is a significant influence on individuals' motivation and self-regulation of behavior (29, 32Á35). Further, there is increasing evidence that self-confidence is an important mediating factor that contributes to the extent to which one approaches learning and persists toward achievement of goals and expertise (29, 31Á38). High fidelity simulation is an expensive technology for instruction and assessment, but a key contribution is its ability to support an authentic environment, particularly in situ, for learning and assessing clinical skills when reallife experiences are either insufficient or inappropriate for teaching, learning, and assessment. In the context of developing future physicians, self-confidence is an important measure at certain points within a curriculum and for certain learning experiences (e.g., when and how to incorporate simulation-based training). Consequently, from an educational and practical perspective, the statistically significant gains in self-confidence for 8 of 14 NRP critical performance behaviors are considered an important contribution of this study.
The results pertaining to Question 3 (correlation between self-reported self-confidence and actual observed performance) reiterate the caution previously communicated about using self-confidence as an overall proxy measure for actual performance or behavior change. However, as mentioned in the previous paragraph, there are other valid reasons for attending to and measuring self-confidence in the teaching and learning process. Upon closer examination of the data, we observed that third-year residents' self-confidence ratings were higher than those of less experienced residents, perhaps simply reflecting a higher level of motivation and persistence to strive toward achievement and expertise, given their closer proximity to graduation and entry into professional practice. Similarly, we observed that three of the first-year residents reported less self-confidence on posttraining than pre-training, possibly an indication of their inflated espoused abilities prior to having hands-on experiences in real-life and simulation scenarios. Their post-training responses may be indicative of achieving a more accurate self-appraisal that resulted from the simulation-based training. This may be an important consideration for the use of simulation-based training as a strategy for enhancing both individual motivation and self-regulation in professional learning and for improving safety in patient care for the future.  Question 4 was important to consider the feasibility of using video-based observation as an accurate alternative to conducting assessment during the live simulation-based training sessions. We recognize that seeing and hearing performances is easier and more accurate in the live observations, as the placement and fidelity of video cameras cannot substitute for the incredible abilities of the human senses. The positive and statistically significant correlation between live and video-based assessments was encouraging. The higher correlation between live and video assessments for observations of post-training performances may have been due the residents' more explicit or enhanced demonstration of specific behaviors in posttraining performances or it may have been due to the influence of practice on assessors' abilities in observing and recording judgments on the assessment instrument. With increased demands and time constraints, the use of a video-based approach for performance-based assessment appears to be a practical option, at least for low-stakes, informal assessments used for coaching and providing formative feedback.

Conclusions
In summary, we are encouraged by the study results regarding the development of a sustainable, standardized curriculum to enhance NRP competency in pediatric residents. While self-confidence and demonstration of improved abilities in a realistic, simulated scenario is important, there is still the need to include examination of the transfer of learning and abilities to real-life practice and to relevant outcomes (e.g., patient, team, organizational). Furthermore, the NRP simulation-based training model may be a viable option for maintaining adequate levels of competency for specific clinical knowledge, technical, and non-technical skills reflected in low frequency, high stakes situations or when learning and training in procedural skills is not feasible or appropriate in the context of real patient care. In addition to residency training, such use of in-situ and center-based simulation could be highly relevant for physicians in practice, particularly in certain types of communities where frequency and consequence may be even more critical (e.g., rural, critical access hospitals, medically underserved communities) (24Á26, 39,40). Members of the research team have begun to explore this area of simulation-based training. Another area of simulationbased training that still needs further attention is the extent to which gains achieved are maintained and how much time must lapse before knowledge and skills degrade to a point when re-training or re-calibration is necessary to reinforce or sustain competency? What type and to what extent must follow-up training reflect the original format of training? These and other questions are opportunities on which to build upon this work and the work of others in simulation-based training for clinical knowledge and skills development across the medical education continuum.