Abstract
Background : While most would agree that utilizing the literature to enhance individual educational practice and/or institutional success is the ideal method for improving medical education, methods to focus attention on the most relevant and valuable information have been heretofore lacking in the pediatric medical education literature.
Methods : We performed a review of the medical education literature for the year 2010. Utilizing a similar strategy employed by others in Internal Medicine, we selected 12 high-yield education journals and manually reviewed the table of contents to select titles that would have grassroots applicability for medical educators. A broad search through PubMed was then completed using search terms adopted from prior studies, and titles from this search were similarly selected. The abstracts of selected titles (n=147) were each reviewed by two of the authors, then all authors reached consensus on articles for full review (n=34). The articles were then discussed and scored to achieve consensus for the 11 articles for inclusion in this paper.
Results : Several themes emerged from reviewing these publications. We did not select topics or sections of interest a priori. The themes, grouped into four areas: supervision and leadership, hand-off communication, core competencies: teaching and assessment, and educational potpourri, reflect our community's current concerns, challenges, and engagement in addressing these topics. Each article is summarized below and begins with a brief statement of what the study adds to the practice of pediatric medical education.
Discussion : This review highlights multiple ‘articles of value’ for all medical educators. We believe the value of these articles and the information they contain for improving the methods used to educate medical students, residents, and fellows are significant. The organically derived thematic areas of the representative articles offer a view of the landscape of medical education research in pediatrics in 2010. Readers can use these individual articles as both tools to improve their practice, as well as inspiration for future areas of research.
For many academic educators, the breadth and volume of educational literature can be intimidating. The sheer number of articles, as well as the range of relevance to any one educator, makes evaluating and utilizing the literature a challenging task. This can lead to missed opportunities to incorporate or build upon medical education innovations resulting in a paralysis of scholarship. While most would agree that utilizing the literature to enhance individual educational practice and/or institutional success is the ideal method for improving medical education, methods to focus attention on the most relevant and valuable information have been heretofore lacking in the pediatric medical education literature.
As a result of this gap, we modeled the successful efforts of Chheda et al. 1 to develop a similar analysis of the educational literature that we believe is the most compelling and relevant for individuals engaged in pediatric medical education. In this review we therefore present 11 articles of high value for pediatric educators from the medical education literature in 2010.
Methods
We sought articles in the medical education literature that we felt would be of interest to pediatric medical educators and be potentially capable of changing practice. Though the rigor of the study methods was considered, the potential grassroots applicability (or the ability of a medical educator to incorporate the results into their daily practice) was weighted more heavily in our process of article selection. On the basis of our experiences as medical educators and the strategies detailed by Chheda et al. 1, we selected 12 journals that were of high yield for practical pediatric medical education research [Appendix A]. We reviewed the table of contents for each of these journals (one author per journal) during the period of January 1–December 31, 2010 for titles that could meet the above defined criteria. We also ran a broad search through PubMed using search terms modified from prior studies 1 [Appendix B] and each author reviewed one quarter of this list to select additional titles that could meet the criteria.
Two authors reviewed and scored the abstracts of each of the selected titles (n=147). All authors discussed the titles to achieve consensus, and 34 articles were selected for full review. Each article was read by two of the authors and scored based on the above criteria. The selections were again discussed and consensus was achieved on the 11 articles to include in this report. All discussions were influenced by the scores of the reviewers, experience of the authors with medical education topics, and relevance and suitability to criteria. During this inductive process of article selection, the authors were able to determine emerging themes within the works. The final articles were then grouped under these themes to allow more robust discussion of particular topic areas as well as individual articles. The discussion presented in this report was further influenced by comments and insights obtained from workshop participants as this review was presented as a workshop at the Association of Pediatric Program Directors Annual Meeting in Miami, FL, on April 3, 2011.
Results
Several themes emerged from reviewing these publications. We did not select topics or sections of interest a priori. The themes were grouped into four areas: supervision and leadership, hand-off communication, core competencies: teaching and assessment, and educational potpourri. These themes reflect our community's current concerns, challenges, and engagement in addressing these topics. Table 1 identifies the specifics of each of these articles, including what we believe is the most important contribution of each as well as specific content matter from each article.
Table 1. Update from key pediatric medical education articles from 2010
This review is based on 11 articles 2–12, which focused on four separate areas: (1) supervision and leadership, (2) hand-off communication, (3) core competencies: teaching and assessment, and (4) educational potpourri. What follows is a synthesis of the impact of the articles, by topic theme, for medical educators.
Section 1: supervision and leadership
There were two articles that addressed improved methods for development of important resident skills: supervision of trainees and leadership competencies. Farnan and colleagues 2 presented a locally developed method for organizing on-call supervision of internal medicine residents [the SUPERB/SAFETY model], and most importantly, presented qualitative research evidence that this model addresses the significant concerns of both resident and attending physicians regarding the balance between supervision and autonomy.
Their method provides a bidirectional framework for how inpatient faculty can balance supervision while preserving resident autonomy, and how residents can appropriately utilize faculty while on-call. The lead author interviewed attending physicians and residents on the general medicine inpatient service at the University of Chicago in 2006. The interview consisted of two parts: 1 an appraisal of critical incidents to identify individual roles in decision-making while on overnight call and 2 an appreciative inquiry to elicit perspectives on improving the supervision system. Major themes were identified and used to create a model for attending supervision and a separate model for when residents should seek attending input. The themes were divided into the bidirectional SUPERB and SAFETY models:
SUPERB model
Suggestions for attending physicians providing supervision
S et expectations for when to be notified
U ncertainty is a time to contact
P lanned communication
E asily available
R eassure resident not to be afraid to call
B alance supervision and autonomy for resident
SAFETY model
Suggestions for when to seek attending input
S eek attending input early
A ctive clinical decisions
F eel uncertain about clinical decisions
E nd-of-life care family/legal discussions
T ransitions of care
H elp with the system/hierarchy
The value of adequate and appropriate supervision is a lesser-discussed aspect of the new Accreditation Council for Graduate Medical Education (ACGME) duty hour regulations 13. While an issue of great importance to patient safety, resident training and attending physician effort, there is surprisingly little published regarding what constitutes adequate supervision in the medical education literature. This study provides a simple model that uses easy to remember mnemonics for faculty and residents to contextualize how they should regard and enact clinical supervision. Although this study is limited by single site and specialty implementation, the concepts seem well derived and fit into an effective model that can be used to provide consistent actions and decrease variability in this important aspect of patient care and medical education. We believe that this logical construct could easily be translated to inpatient pediatric learning experiences. This paper also provides a simple but logical training program to help inform faculty who attend on the inpatient service, as well as guide residents as they take inpatient call.
The second paper by Kuo and colleagues 3 describes the development and implementation of an effective curriculum designed to develop leaders in providing care to children in poor and underserved areas. The authors describe a comprehensive curriculum at the University of California, San Francisco (UCSF) to develop leaders in pediatric underserved care called the Pediatric Leadership for the Underserved (PLUS) Program. UCSF implemented this new residency training program in 2004 with the ‘express purpose of incorporating leadership development into standard clinical training’. Participants engaged in active learning in three domains: leadership training, critical-thinking skills, and community engagement. An important component is the organized mentoring of residents by faculty advisors and peers within the program as each resident is helped to develop and implement a 3-year longitudinal child health project.
The lessons from this model, which reviews early outcome data 2004–2009, are presented by the authors in this paper: 1 the need for curricula to be skill-based and the need to encourage reflection and feedback, 2 the need to link curricular elements to everyday leadership roles, 3 the importance of highlighting physician role models, and 4 the requirement of balance when incorporating specialized tracks within a larger training program. Important outcomes for participants in this program included general satisfaction with the program, improved individual resident self-perception of leadership and clinical skills, a positive impact on future career goals, increased resident scholarship (awards, grants, publications), and notable postgraduate leadership and policy positions. Beyond the successful education of the residents in the issues that confront underserved children, this curriculum also provides valuable insights into how to develop leadership abilities in trainees. As medicine continues to involve more emphasis on teamwork and team provision of care, the value of training residents for leadership roles is becoming increasingly evident. This paper provides a novel training curriculum in pediatric leadership development. While the authors have created a robust curriculum that may not always be easily implemented in total, the components of the program may be quite useful to faculty and program directors interested in developing leadership skills in their trainees.
Section 2: hand-off communication
The second section presents three articles that all deal with a very prominent issue in patient safety and medical education – the most effective methods to provide ‘Hand-off’ training. Multiple lines of evidence support the need for more effective hand-offs as current resident and fellow hand-offs are often incomplete and of poor quality. Poor hand-offs can contribute to adverse events 14. The ACGME recently focused more emphasis on the development of effective hand-over training 13, so these papers are very timely.
Farnan and colleagues 4 present a model hand-over education and evaluation process that relies on an observed simulated hand-off experience (OSHE). Given the increased attention to the impact of effective hand-offs in insuring best quality patient care outcomes and promoting patient safety 15 16, recent ACGME directives now define requirements for organized methods for implementing, educating, and evaluating hand-off methods in graduate medical education programs. Recently there has been additional attention to developing hand-off education for medical students in preparation for residency work, so this paper is particularly timely and, we believe, very useful to faculty and education leaders.
The focus of this study was to quantitatively study fourth-year medical students in acquisition and demonstration of this most important patient care skill. The organized curriculum began with a 90-min interactive, case-based workshop that focused on key principles of effective hand-offs. Students received examples of high-quality sign-outs (available electronically) and a pocket card with helpful tips. One week post-workshop, students participated in an observed simulated hand-off experience (OSHE) to evaluate their skills. During the OSHE, students received a mock patient history and physical exam transcript to review. They then watched a video of a dynamic patient encounter containing important clinical updates intended to trigger anticipatory guidance and to-do items. Each student was evaluated on her/his written sign-out using a standardized scoring rubric. Following this, each student was given 10 min to verbally hand-off the patient to a standardized receiving resident. The trained resident evaluators completed a modified mini-CEX using a Likert-type scale (scored on a scale of 1–9) to evaluate each student's performance.
Improved student performance and student self-perceived preparedness for performing effective hand-off were established by this curriculum. This article provides a feasible specific method to improve hand-off education and procedures. With the advent of new duty hour restrictions there is a compelling need for this type of curriculum as well as tools to evaluate hand-offs among trainees. This article offers a detailed description and actual tools for implementation of an effective hand-off training experience.
Gakhar 5 presented a series of efforts designed to improve internal medicine resident skills in hand-off practices. They provided a contextualized framework for teaching sign-out and a rubric for evaluating these skills. This study evaluated the spoken and written sign-out practices of interns at a single institution. The intervention consisted of a 1 hour learning experience. Residents were presented with the principles of spoken and written sign-out practices using the SIGN-OUT mnemonic (S – status; I – identification data; G – general hospital course; N – new events of the day; O – overall health; U – upcoming possibilities; T – tasks to do) and provided opportunities to practice these skills in small group practice sessions. Eight weeks after intervention, trained resident evaluators assessed intern practices in verbal and written hand-offs. Improved intern performance in both verbal and written hand-offs was demonstrated through this process, again suggesting that organized education and evaluation delivers improved performance in this important skill. A major benefit of this approach of Gakhar is the excellent framework that can be used to successfully build a curriculum based on local needs and customs while defining a feasible method to improve sign-out processes and education.
Discharge communication is another essential process designed to promote patient safety and deliver better patient care. The article of Key-Solle and colleagues 6, which details a feasible model for improving discharge communication skills, is another important contribution to the literature about education in this essential skill. The intervention consisted of a brief educational intervention and individual reinforcement session. The pediatric and medicine-pediatric interns who participated in the intervention were compared with historical controls in terms of very defined outcomes derived from review of their medical records. The discharge summaries created by the interns were assessed with a defined rubric and significant improvement in completeness was demonstrated over time. While this outcome is not surprising, a remarkable aspect of this educational program was the incorporation of individual reinforcement to assist in the development of this important skill. The value of this additional educational component cannot be over-emphasized. All too many times, organized educational efforts are expected to deliver effective skills, neglecting the beneficial impact of informed feedback and practice 17 on skill development. This article should serve as a reminder of the benefit of incorporating deliberate feedback into educational efforts designed to develop important skills and result in real behavior change.
Section 3: core competencies: teaching and assessment
Methods to better teach core competencies in medicine have now extended beyond the ACGME 18 (for graduate medical education trainees) to the LCME (for medical students) and ACCME (for practicing physicians). A number of useful publications in 2010 were devoted to defining better methods in this important focus for medical education. These papers relate to Evidence Based Medicine (EBM), Professionalism and Advocacy competency development – all areas of importance to the development of effective medical students and graduate medical education trainees.
Chernick 7 described a novel instrument for assessing pediatric resident ability in EBM. This validated tool defines the ability of learners to assess their ability and demonstrate knowledge and effectiveness in applying EBM principles and methods to pediatric care. This tool relies on demonstrating EBM knowledge and solving pediatric patient care scenarios that require abilities to develop searchable questions and select evidence to address diagnostic and treatment issues. An important contribution of this paper is the investigation of the reliability and validity of the EBM instrument. Important results from this educational program included improvements in the learners’ EBM comfort level and self-reported practice, as well as the ability to demonstrate EBM knowledge and solve pediatric patient care scenarios. Interestingly, scores in these trainees increased significantly with training level, expertise level and prior EBP experience. While this tool does not assess all EBM skills (i.e. harm and prognosis), Chernick and his co-authors have developed a reliable and valid instrument to assess proficiency of EBP skills. This is the first validated instrument specifically designed for pediatric residents and can be beneficial in programs devoted to evaluating trainees over time.
The paper by Rademacher 8 explores using critical incidents as a teaching technique to develop professionalism skills. This Critical Incident Technique (CIT) is a versatile teaching strategy that is applicable in many settings with many different learners. The authors analyzed ‘real-life’ occurrences of unprofessional behavior with learners using the CIT established by Flanagan 19. This process resulted in identification of common terminology and definitions of professionalism that could be explored with learners within a structured discussion framework. A valuable aspect of this study is the successful use of CIT in four settings with varied audiences in: (1) a third-year medical student half-day intersession, (2) faculty as educators workshops, (3) Medical Grand Rounds (focused on faculty professionalism in education), and (4) faculty development program for primary care physicians focused on recording critical teaching incidents that occurred during their daily teaching interactions (critical teaching incident casebook).
Of note this CIT approach was well received by over 250 students and faculty at the Medical College of Wisconsin as well as diverse attendees at two national meetings. CIT was shown to be useful in multiple settings and disciplines as an analysis framework for teaching, assessment, and quality improvement. Since CIT relies on real-life experiences, it has proved to be a powerful strategy for reframing the daily experiences for learners. While this study of Rademacher did not address ‘hard’ outcomes such as behavior change, the wide acceptance by the learners is noteworthy. Since engaging effective methods to develop professionalism continue to be a challenge in pediatric as well as other areas of medical education, the versatility and authenticity of CIT to promote better understanding and learning in professionalism should be further explored.
Developing advocacy skills is another challenge in Core Competency education. Klein and co-authors 9 describe how medical students and residents are increasingly likely to encounter patients and families in poverty in their work and in their careers. In an attempt to more effectively develop residents’ knowledge and attitudes about individuals in poverty, this group used Kolb's Experiential Learning Cycle to develop a novel approach for educating pediatric residents about the impact of social determinants on children's lives. Their locally developed social–legal curriculum was modeled after the framework of experiential learning first described by David Kolb 9 to combine didactic and experiential learning experiences. In this curriculum, defined immersion experiences and reflection activities complemented the learning. Although the educational intervention was brief, the impact on the residents was defined as ‘powerful’. This assessment was derived qualitatively by the authors based on their analysis of the reflective pieces and through demonstrated improvements in the specific knowledge of the residents about advocacy issues and community partnerships for solutions.
While this educational experience of Klein and colleagues was brief (10% of a 2-week curriculum) and long-term outcomes were not assessed, the impact on these pediatric interns appeared meaningful and was achieved in a manner that could be generalized to other important curricular areas. Further studies could assess whether these types of experiences impact learners in a long-term manner and if they impact patient care decisions and practice. The Kolb experiential learning process offers real promise to expand learners’ meaning in experiences and thereby go beyond the all too frequent ‘observational’ type experiences common in many clerkships and graduate medical education programs.
Section 4: educational potpourri
By the end of the second review there were 34 articles remaining and these were reduced through an iterative group process down to the top 11 articles (Appendix C for the articles that were fully reviewed but not included). As described in the methods, several of these articles naturally fell into ‘themes’. These articles were often hot topics in medical education for 2010 and dealt with the most pressing issues at the time. However, there were three articles that stood out based on their outcomes and methodological rigor that could not be categorized into a central overarching theme. These were subsequently labeled ‘educational potpourri’. This section contains articles which dealt with subjects of importance that are always in need of better answers but not necessarily the hot topics in medical education.
The first article by Raman et al. 10 sought to determine whether long-term retention was enhanced by teaching in small portions dispersed over time or by delivering the material en masse. Determining the most effective and efficient methods to teach for learner retention is a significant and often overlooked issue in medical education. One of the main goals of education is to promote long-term knowledge retention and memory after educational sessions 20. Learning is basically the transfer of information from working memory to long-term memory. Trainees, however, are required to process a significant amount of information daily, and this may result in excessive cognitive load. Cognitive load refers to the total amount of mental activity imposed on the working memory at an instance in time 10. It has been postulated that learning may be ineffective if the capacity of the working memory is exceeded. A common curricular structure in education is the academic half-day, where learners are relieved of other duties to spend the time in a classroom setting listening to a constant flow of information. The work by Raman and colleagues demonstrated improved long-term retention after spaced learning experiences in comparison with learning en masse. This is consistent with the work of Kerfoot 20 in spaced education. The Raman article is significant in that it debunks the traditional curricular structure of long lectures or continuous teaching sessions that encompass a half or whole day utilized in many medical institutions. They recommend abandonment of the academic half-day and replacement with a daily academic hour. The major limitation of this study besides the small sample size and generalizability is that it does not answer the question whether or not distributed learning improves higher-level cognitive tasks, such as diagnostic reasoning. Raman and colleagues state that consideration should be given to teaching more difficult concepts and less commonly encountered clinical scenarios in a dispersed as opposed to massed fashion. They also recommend the avoidance of cognitive fatigue although state that it is unclear how best to achieve this task.
The second study by Andolsek 11 was a comparison of two different methods to orient trainees to their new work environment. Upon review, we felt this study to be very important to medical educators as orientation sessions are ubiquitous to all programs and the amount of time spent in orientation seems to be increasing yearly. Formal orientation programs for new employees are proven tools in the business world 11. Optimally, orientation should be an effective, efficient introduction to one's new work environment. However, there are many obstacles to the ideal orientation; the largest of these barriers is time. Residents transitioning to fellowship may leave one job on June 30th to start their new job on July 1st. Additionally, all trainees do not start on the same day. It is particularly difficult for those residents transitioning to programs in different cities or states from their previous training or who are not starting at the same time as the majority of other trainees. Also, as we learned from the previous study, excessive cognitive load is likely to occur from sitting in a classroom 8 hours a day. Inevitably this will result in loss of some information that exceeds the working memory and learning will be less effective.
Andolesek and colleagues demonstrated that those trainees who participated in the webcast orientation significantly outperformed those who participated in the live lecture orientation. The increased effectiveness initially comes at a price though. The webcast orientation cost $181/person, whereas the live lecture was only $119/person. Over time Andolesek et al. was able to amortize the cost so that 2 years later the webcast orientation was only $122/person. Trainees commented positively on the flexibility in when to study the material, options on how to view the material (podcasts were made available later), and opportunities to review it as needed in order to master it. Another advantage is that having a web-based orientation freed up time in the orientation curriculum to add more opportunities for hands on learning and social activities to facilitate interaction with peers, faculty and staff.
The last study deals with an age-old issue in medical education, the ‘problem resident’. Educational leaders are always looking for ways to identify the learner in difficulty before he or she has problems. Brenner et al. 12 sought to determine whether data available at the time of residency application could be used to predict future problems of performance, both during and after residency. In our review of the 2010 literature, we felt this was an article of great significance. Each year, residency programs expend significant resources in the recruitment process. Committees charged with identifying the ‘best’ candidates for their programs are often at a loss of knowing which applicants will become successful residents and which will experience significant difficulties. Brenner et al. conducted a review of the literature assessing the ability to predict future performance of residents and found mixed results. Therefore, they took data from their own program for the past 20 years and identified 40 problem residents and then identified 40 matched controls. Brenner demonstrated that any negative comment in the Dean's letter was a potential correlate of future problems. Upon review of some of their definitions of negative comments such as, ‘very nervous, initially timid’, ‘displayed little curiosity’, and ‘needs to read more on her own’, this paper demonstrated the importance of careful review of the Dean's letter to assess for any negative comments as a potential correlate of future problems. The authors also point out an even more relevant applicability to the study. The identification of negative comments can alert the program director to potential problems that may be anticipated and therefore mitigated before they emerge as actual problems.
These articles provide evidence to questions that clinician-educators, curriculum developers and program leadership face on a daily basis. The authors spur the reader to always challenge the status quo and focus on the outcomes of their educational products – be it curricula or the trainees who participate in their programs. Additionally, the relevancy and practicality of the suggestions provided by these articles can be used in a wide variety of learning environments and across multiple specialties.
Discussion
The number of articles published in 2010 that were directly relevant to pediatric medical educators was vast. In focusing on 12 high-value journals and doing a broad PubMed search, we identified 147 articles that we felt could be directly applicable to pediatric medical educators and easily identified 34 that we thoroughly reviewed. In this manner we were able to narrow down the reviewed articles to a ‘top eleven’. However, this limitation to 11 was self-imposed and a number of the remaining 23 articles were found to be relevant to medical education as a whole and were methodologically sound (Appendix C). We believe the value of these articles and the information they contain for improving the methods used to educate medical students, residents, and fellows are significant. This review highlights multiple ‘articles of value’ for all medical educators.
Several themes emerged from reviewing these publications. As stated previously, we did not select topics or sections of interest a priori.
In the area of supervision and leadership the SUPERB/SAFETY model of Farnan 2 offers much promise as an effective tool that can be learned and assessed over time. There will undoubtedly be many different ways to organize supervision goals and objectives in residency training. We believe this is a strong candidate for future exploration, validation, and use. Leadership programs for pediatric residents and fellows are an area of increasing interest to residency/fellowship programs and institutions. This model from UCSF 3 has much to teach us and components that many institutions can adopt.
Patient care hand-offs and communication are arguably among the highest profile issues in graduate medical education today and have been highlighted because of concerns about the impact of increased duty hour limitations on patient safety 15 16. It has also been recognized that the traditional method of informal hand-offs is no longer defensible in terms of the needs of our learners or our patients. The methods of Farnan 4 and Gakhar 5 (particularly the SIGN OUT mnemonic) provide thoughtful constructs to use in developing educational methods for this important skill. Similarly, the importance of hand-offs from the inpatient setting to the primary care physician is a critical skill in fostering collaborative relationships and assuring patient care quality. Key-Solle's 6 study offers an easily implemented intervention for improving the quality of this skill.
Improved methods to perform competency assessments are consistently of interest to medical educators and the work of several investigators merited attention in this area. Chernick 7 presents an excellent validated tool to measure competency in Evidence-Based Practice, a skill where assessment tools offer opportunities to significantly improve learner assessment and mastery. Rademacher's use of the Critical Incident Technique 8 details a potent method to teach and assess understanding of the hidden curriculum that comprises much of professionalism education for our students and trainees. Klein's development of a curricular model for teaching social determinants of health 9 provides an example of combining didactic and experiential learning components that could be replicated for other competency areas.
Lastly, three articles stood out as poignant reflections of enduring truths in medical education: the more powerful impact on learning of teaching in small doses over time 10, the fact that more ‘efficient’ methods of learning demand validation as they are increasingly adopted into the curricula 11, and that predictions of future problem behaviors in residents are available to us at the point of recruitment, if only we look in the right places 12. These are significant, if not novel lessons for us all.
We hope this review will help interested medical educators assess new knowledge in the field of medical education and make important decisions about new techniques, program improvements, and opportunities for future research and scholarship. It is important for medical educators to step back periodically to assess what has been published as part of a commitment to continuous improvement of our methods and our craft. Glassick's definition of scholarship, in fact, requires that a ‘scholar show an understanding of existing scholarship in the field’ 21. This article might be used by faculty for medical education journal club selections, self-study, new program implementation, and/or further opportunities for medical education research and scholarship. We hope to see such benefit from this review.
Conflict of interest and funding
None of the authors has any relevant financial disclosures or conflicts of interest.
| Study | Significance | Study design | Results | Limitations |
|---|---|---|---|---|
| Farnan et al. 2 | Provides a bidirectional framework for how inpatient faculty supervises while preserving autonomy and how residents work while utilizing faculty appropriately. | Qualitative study of supervision of on-call teams in an internal medicine setting. Fifty-four attending physicians and 46 residents on general medicine inpatient service were interviewed via (1) Critical Incident Technique to identify individual roles in decision-making on-call and (2) appreciative inquiry to elicit perspectives on improving the supervision system. Major themes used to create a model for attending supervision and directions for when residents should seek attending physician input. | The themes were divided into bidirectional models:SUPERB modelSuggestions: attending physicians providing supervision 1. S et expectations for when to be notified 2. U ncertainty is a time to contact 3. P lanned communication 4. E asily available 5. R eassure resident not to be afraid to call 6. B alance supervision and autonomy for resident SAFETY modelSuggestions: residents to seek attending input when 1. S eek attending input early 2. A ctive clinical decisions 3. F eel uncertain about clinical decisions 4. E nd-of-life care family/legal discussions 5. T ransitions of care 6. H elp with the system/hierarchy. | Limited to single site and single specialty. No data on effectiveness of this method for physicians, residents or patients. |
| Kuo et al. 3 | Presents effective comprehensive curriculum to develop leaders in pediatric underserved care; though a complex program, smaller parts could be used by others. | Non-randomized select cohort study of participants in a Pediatric Leadership for the Underserved (PLUS) Program. This residency track has the ‘express purpose of incorporating leadership development into standard clinical training’. Participants engage in active learning in three domains: leadership, critical-thinking, and community engagement via PLUS-focused clinic, advisor meetings, conferences and curriculum meetings. | Lessons learned include the need for curricula to be skill-based, encourage reflection and feedback, linked to everyday leadership roles, highlight role models, and achieve balance. Outcomes for participants included improved resident self-perception (impact on future career, improved leadership/clinical skills), increased resident scholarship (awards, grants, publications), and notable postgraduate leadership & policy positions. | Single site and single specialty. Long term outcomes not available. Resident involvement selective; no control non-intervention group. |
| Farnan et al. 4 | Defines a specific detailed method for teaching sign-out skills to medical students. | Non-randomized cohort study of results of 32 fourth year medical students’ acquisition of hand-over skills based on (1) 90-min interactive case-based workshop, (2) examples of high quality sign-outs, and (3) pocket card with helpful tips. One week post-workshop, observed simulated hand-off experience (OSHE) used to evaluate skills. Each student evaluated on written hand-off and verbal hand-out by trained resident evaluators who completed modified mini-CEX to evaluate each student's performance. | Resident receivers rated overall student performance with a mean score of 6.75 (range 4–9). Improvement was noted in student self-perceived preparedness for performing an effective hand-off (67% post-reporting vs. 27% pre-reporting ‘well prepared,’ P<0.009). Sixty-six percent of students felt that the experience should be required for all fourth year students and 100% agreed that the experience would impact their practice as interns. | Small sample; limited to fourth year medical students; single site; no patient care outcomes. |
| Gakhar and Spencer 5 | Presents a specific detailed method for teaching hand-over skills to interns including assessment tool with scoring rubric. | This study evaluated the spoken and written sign-out skills acquisition of 25 internal medicine interns. Intervention consisted of 1 hour learning experience including (1) 30 min didactic presentation to teach the principles of spoken and written sign-out practices using the SIGN-OUT mnemonic (S-status, I-identification data; G-general hospital course, N-new events of the day; O-overall health; U-upcoming possibilities; T-tasks to do) and (2) 30 min interactive small group practice session. Trained resident evaluators assessed two components: intern satisfaction and pre/post objective assessments of both spoken and written sign-outs. | Eight weeks after educational intervention, evaluators observed 4–6 hand-overs for 12 randomly chosen interns on ward rotations for spoken hand-overs; 28 randomly selected sign-out sheets were audited. When compared with pre-training assessment, interns more frequently reported each of 7 items in SIGN-OUT mnemonic. Complete written sign-out sheets rose from 16 to 77% (P<0.001). Overall score for written sign-out sheet completeness rose from 5.8 to 7.6 (P<0.001). Interns showed significant improvement from pre-training to post-training scores in 3/4 items evaluated: identification data (64% pre, 89% post), code status (82% pre, 100% post), and medication list (4% pre, 79% post) (P<0.001 for each item). | Small sample size; single site. |
| Key-Solle et al. 6 | Provides a feasible model for improving discharge communication skills of residents. | Interventional trial utilizing historical controls (36 previous interns) of effect of brief educational intervention and individual reinforcement session on quality of discharge summary (DS) communication in 28 pediatric and medicine-pediatric interns. Two part intervention: (1) 1 hour interactive large-group session to teach six key elements of high-quality and timely DS: diagnosis; timely completion; pending laboratory work/studies; medications; length ≤ 3 pages; and discharge weight and (2) individual 5–10 min 1 on 1 reinforcement sessions during second week of 1 month inpatient general pediatrics rotation. Assessment involved audits of all intern DS's to define how many of the key elements contained (max score = 6). | Examination of the first week audits revealed similar DS scores for both groups. However, statistically significant difference noted in comparison between third week post-reinforcement scores for two groups. Experimental group had higher overall total DS scores (4.4 vs 3.38, P<0.001) with DS that were more timely (P<0.000); less likely to have medication list in the DS (P<0.004); more likely to include discharge weight (P<0.001); more likely to include pending lab work (P<0.004); and more likely to contain greater than five of the six key components of discharge (P<0.0001). | Single learner group; single site; historical controls. |
| Chernick et al. 7 | Presents a well validated tool to assess learners’ and practitioners’ Evidence Based Practice (EBP) self-efficacy, knowledge, and ability. | A validated EBP (Evidence Based Medicine – EBM) assessment instrument composed of demographics, comfort level (self-efficacy), self-reported practice, and EBM knowledge. Assessment of ability to identify best evidence and appropriate study design using scenarios to develop searchable questions, and select evidence to address diagnostic and treatment issues. Four groups studied (students, interns, residents, and local EBP faculty experts). Reliability and validity of the instrument defined. | Fifty-six subjects participated (13 MS-2s; 13 PGY-1s; 22 PGY2-3s; 8 experts). Internal reliability good (Cronbach's alpha = 0.80). Inter-rater reliability high (kappa = 0.94). Items were free of floor or ceiling effects. Comfort level and self-reported practice of EBP increased significantly with expertise level and prior EBP experience. Scores on the knowledge section rose with training level and correlated with prior EBP education. | Single site; EBM competency assessment does not address all EBM activities. |
| Rademacher et al. 8 | Uses the Critical Incident Technique (CIT) to develop a versatile teaching strategy for exploring and teaching professionalism with varied audiences in a wide array of settings. | Descriptive review of ‘real-life’ occurrences of unprofessional behavior with learners via CIT to define issues for structured small group discussions. CIT used in four varied settings and audiences: (1) third year medical student half-day intersession, (2) faculty as educators workshops, (3) Medical Grand Rounds (focused on faculty professionalism in education) and, (4) faculty development program for primary care physicians based on critical teaching incidents that occurred during their daily teaching interactions (critical teaching incident casebook). | CIT well received by over 250 medical students and faculty at one medical school as well as diverse attendees at two national meetings. CIT useful in multiple settings and disciplines as framework for teaching, assessment, and quality improvement. CIT emphasis on participants’ real-life experiences proved to be powerful strategy for reframing daily experiences for learners. Authors speculate that addressing these issues in various learning environments may promote change in professional behavior and enhance understanding of the challenges that learners and professionals encounter. | Did not explore patient care outcome measures. |
| Klein and Vaughn 9 | Application of Kolb's Experiential Learning Cycle to develop a novel approach for educating pediatric residents about the impact of social determinants on children's lives. | Descriptive presentation of new social-legal curriculum added to 2 weeks advocacy block rotation for pediatric residents. Using Kolb's experiential learning framework, 37 interns participated in organized immersion experiences by visiting local public benefit organizations (i.e. family services and food banks) and writing reflective pieces (‘Memo to Myself’ (2) to reflect on 2–3 items). These reflective pieces were analyzed qualitatively for salient themes to define categories, themes, and linkages. | Based on analysis of ‘Memos to Myself’ generated by interns, impact on residents was ‘powerful’. On the basis of the written output, the authors assessed the impact of the social-legal curriculum on the learners by identifying three primary areas of enhanced understanding and interest: (1) realization regarding family circumstances; (2) reflections regarding self and personal practice; and (3) specific knowledge about advocacy issues and community partnerships for solutions. | Single learner group; single site; brief intervention; knowledge and attitude only outcomes measured – no skill assessment. |
| Raman et al. 10 | Long term retention of material is enhanced with a dispersed curriculum. | Prospective interventional cohort study involving 20 fellows. One group received 1 hour weekly didactic sessions on nutrition education over 4 weeks and another group received the same content from the same lecturer in one ‘massed’ 4-hour educational session. Both groups completed knowledge examinations at baseline, 1 week and 3 months post-intervention. | Both cohorts scored significantly higher on the 1-week post-intervention examination. The absolute pretest difference was significantly higher among the participants in the dispersed curriculum (p=0.048). Test scores declined from the one week post-intervention level in both groups but the absolute pretest to three month post-test difference was significantly higher in the dispersed curriculum group (p=0.021). | Small sample size (10 in each group) and only one learner type (gastroenterology fellows). |
| Andolsek et al. 11 | Webcasts are an efficient, flexible, and effective method to introduce trainees to their new work environment. | A non-randomized cohort study with all new residents and fellows in the 2005 and 2006 academic years. In 2005, 249 participants received a live lecture orientation with 12.5 hours of didactic content. In 2006, 241 participants received 11.9 hours of didactic content via webcast. Both groups were tested on content and were allowed three attempts to select correct answers to the questions. | The webcast cohort had a significantly greater proportion of trainees who passed the exam after three attempts (p<0.001). It took slightly less time for the residents to complete orientation in the webcast cohort, however, cost per person was higher. By 2008, the amortized cost of the webcasts was essentially the same as the costs for the live lecture in 2005. | One institution and no randomization. |
| Brenner et al. 12 | The presence of any negative comments (even mild concerns) in the dean's letter of residency candidates correlates with future behavior problems. | Retrospective review of all residents files with problematic behavior over a 20 year time period. Problems cases were identified as directly affecting performance, leaves of absence due to stressors or development of significant post-residency problems. These problem cases were then divided into ‘minor’ and ‘major’ problems (resolution with disciplinary action). A total of 40 problematic residents were identified and a set of matched controls was created. | Any negative comments in the Dean's letter were strongly correlated with problem outcomes. Problem cases had significantly more negative comments than controls. Residents with ‘major’ problems had significantly more negative comments than did those of residents with ‘minor’ problems. Other factors such as USMLE scores, failed courses, letters of recommendation, and interviewer evaluations did not predict future problems. | Retrospective study, one training program (adult psychiatry) from one institution, small sample size; lack of standardization of the interview evaluation process. |
| Academic Medicine | Medical Education |
| Academic Pediatrics | Medical Education Online |
| BMC Medical Education | Medical Teacher |
| British Medical Journal | New England Journal of Medicine |
| Journal of the American Medical Association | Pediatrics |
| Journal of Graduate Medical Education | Teaching & Learning in Medicine |
Acknowledgements
The authors are grateful to Reena Karani for encouraging this endeavor and sharing her experience along the way. Presented in part at the APPD Annual Spring Meeting in Miami, Florida on April 3, 2011.
References
- Chheda SG, Karani R, Dunn K, Babbott S, Bates CK. Update in medical education. JGIM. 2008; 23: 195–201. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Farnan JM, Johnson JK, Meltzer DO, Harris I, Humphrey HJ, Schwartz A. et al. Strategies for effective on-call supervision for internal medicine residents: the SUPERB/SAFETY model. JGME. 2010; 2: 46–52. [Crossref], [Google Scholar]
- Kuo AK, Thyne SM, Chen HC, West DC, Kamei RK. An innovative residency program designed to develop leaders to improve the health of children. Acad Med. 2010; 85: 1603–8. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Farnan JM, Paro JAM, Rodriguez RM, Reddy ST, Horwitz LI, Johnson JK. et al. Hand-off education and evaluation: piloting the observed simulated hand-off experience (OSHE). JGIM. 2010; 2: 129–34. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Gakhar B, Spencer AL. Using direct observation, formal evaluation, and an interactive curriculum to improve the sign-out practices of internal medicine interns. Acad Med. 2010; 85: 1182–8. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Key-Solle M, Paulk E, Bradford K, Skinner AC, Lewis MC, Shomaker K. Improving the quality of discharge communication with an educational intervention. Pediatrics. 2010; 126: 734–9. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Chernick L, Pusic M, Liu H, Vazquez H, Kwok M. A pediatrics-based instrument for assessing resident education in evidence-based practice. Acad Pediatr. 2010; 10: 260–5. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Rademacher R, Simpson D, Marcdante K. Critical incidents as a technique for teaching professionalism. Med Teach. 2010; 32: 244–9. 10.3402/meo.v17i0.14433. [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
- Klein M, Vaughn LM. Teaching social determinants of child health in a pediatric advocacy rotation: small intervention, big impact. Med Teach. 2010; 32: 754–9. 10.3402/meo.v17i0.14433. [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
- Raman M, McLaughlin K, Violato C, Rostom A, Allard JP, Coderre S. Teaching in small portions dispersed over time enhances long-term knowledge retention. Med Teach. 2010; 32: 250–5. 10.3402/meo.v17i0.14433. [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
- Andolsek KM, Murphy C, Pinheiro S, Petrusa E, Tuck T, Weinerth J. Efficacy and efficiency of webcast orientations versus live resident orientations: results of a 2-year survey. JGME. 2010; 1: 136–40. [Crossref], [Google Scholar]
- Brenner AM, Mathai S, Jain S, Mohl PC. Can we predict “problem residents”?. Acad Med. 2010; 85: 1147–51. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Nasca TJ, Day SH, Amis ES Jr, ACGME Duty Hour Task Force. The new recommendations on duty hours from the ACGME Task Force. N Engl J Med. 2010; 363: e3.10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. Consequences of inadequate sign-out for patient care. Arch Intern Med. 2008; 168: 1755–60. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Clarke CM, Persaud DD. Leading clinical handover improvement: a change strategy to implement best practices in the acute care setting. J Patient Saf. 2011; 7: 11–8. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Johnson JK, Barach P. Patient care handovers: what will it take to ensure quality and safety during times of transition?. Med J Aust. 2009; 190: S110–2. [Web of Science ®], [Google Scholar]
- McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Medical education featuring mastery learning with deliberate practice can lead to better health for individuals and populations. Acad Med. 2011; 86: e8–9. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Williams RG, Dunnington GL. Accreditation Council for Graduate Medical Education core competencies initiative: the road to implementation in the surgical specialties. Surg Clin North Am. 2004; 84: 1621–46. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Flanagan JC. The critical incident technique. Psychol Bull. 1954; 51: 327–58. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
- Kerfoot BP, Conlin PR, Travison T, McMahon GT. Web-based education in systems-based practice: a randomized trial. Arch Intern Med. 2007; 167: 361–6. 10.3402/meo.v17i0.14433. [Google Scholar]
- Glassick CE. Boyer's expanded definition of scholarship, the standards of assessing scholarship, and the elusiveness of the scholarship of teaching. Acad Med. 2000; 75: 877–80. 10.3402/meo.v17i0.14433. [Web of Science ®], [Google Scholar]
Appendix A: Appendix A: Pediatric and medical education journals for manual review
Table
Appendix B: Appendix B: PubMed search terms used
((‘medical education’ OR teaching OR teach OR training) AND (outcomes OR evaluation OR ‘randomized controlled trial’ OR ‘systematic review’ OR ‘meta-analysis’ OR assessment) AND (student OR resident OR pgy OR intern)) NOT (‘Acad Med’ [Journal] OR ‘Med Educ’ [Journal] OR ‘Teach Learn Med’ [Journal] OR ‘Med Teach’ [Journal] OR ‘Pediatrics’ [Journal] OR ‘New England J Med’ [Journal]) OR ‘Academic Pediatrics’ [Journal] OR ‘JAMA’ [Journal] OR ‘BMJ’ [Journal] OR ‘BMC Med Ed’ [Journal] OR ‘Medical Education Online’ [Journal]))
Appendix C: Appendix C: Articles that were fully reviewed but not included
Adamshick EA, Fanning R, Piro N. Leadership lessons from military education for postgraduate medical curricular improvement. Clin Teach 2010; 7: 26–31.
Bonnetain E, Boucheix J, Hamet M, Freysz M. Benefits of computer screen-based simulation in learning cardiac arrest procedures. Med Educ 2010; 44: 716–22.
Brustman LE, et al. The effect of blinded versus nonblinded interviews in the resident selection process. JGME 2010; 2: 349–53.
Campbell J, Pochazka AV, Yamashita T, Gopal R. Predictors of persistent burnout in internal medicine residents: a prospective cohort study. Acad Med 2010; 85: 1630–4.
Chun MBJ, et al. Using the cross-cultural care survey to assess cultural competency in graduate medical education. JGME 2010; 2: 96–101.
Cook DA, et al. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med 2010; 85: 909–22.
Dattner L, Lopreiato JO. Introduction of a direct observation program into a pediatric resident continuity clinic: feasibility, acceptability, and effect on resident feedback. Teach Learn Med 2010; 22: 280–86.
Durning SJ, et al. Making use of contrasting participant views of the same encounter. Med Educ 2010; 44: 953–61.
Gerner B, et al. Using simulated patients to develop doctors’ skills in facilitating behaviour change: addressing childhood obesity. Med Educ 2010; 44: 706–15.
Gigante J, Swan R. A simplified observation tool for residents in outpatient clinic. JGME 2010; 1: 108–10.
Kluger AN, Van Dijk D. Feedback, the various tasks of the doctor, and the feedforward alternative. Med Educ 2010; 44: 1166–74.
Jeffries A, Skidmore M. Evaluation of a collaborative mentorship program in a multi-site postgraduate training program. Med Teach 2010; 32: 695–7.
Joyce BL, Steenbergh T, Scher E. Use of the Kalamazoo essential elements communication checklist (adapted) in an institutional interpersonal and communication skills curriculum. JGME 2010; 2: 165–9.
Li ST, Paterniti DA, Co JP, West DC. Successful self-directed lifelong learning in medicine: a conceptual model derived from qualitative analysis of a national survey of pediatric residents. Acad Med 2010; 85: 1229–36.
Li ST, Tancredi DJ, Co JP, West DC. Factors associated with successful self-directed learning using individualized learning plans during pediatric residency. Acad Pediatr 2010; 10: 124–30.
Link EA, Kreiter CD, D'Alessandro DM. An evaluation of pediatric residency education in conveying immunization knowledge. Teach Learn Med 2010; 22: 176–9.
McMahon GT, et al. Evaluation of a redesign initiative in an internal medicine residency. N Engl J Med 2010; 362: 1304–11.
Murad MH, et al. The effectiveness of self-directed learning in health professions education: a systematic review. Med Educ 2010; 44: 1057–68.
Roels P, Van Roosmalen G, Van Soom C. Adaptive feedback and student behaviour in computer-assisted instruction. Med Educ 2010; 44: 1185–93.
Schichtel M. Core-competence skills in e-mentoring for medical student: a conceptual exploration. Med Teach 2010; 32: e248–62.
Schwartz A, Weiner SJ, Harris IB, Binns-Calvey A. An educational intervention for contextualizing patient care and medical students’ abilities to probe for contextual issues in simulated patients. JAMA 2010; 15: 1191–7.
Thomas EJ, et al. Team training in the neonatal resuscitation program for interns: teamwork and quality of resuscitations. Pediatrics 2010; 125: 539–46.
Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ 2010; 10: 12.