Narrative information obtained during student selection predicts problematic study behavior.

Abstract Introduction: Up to now, student selection for medical schools is merely used to decide which applicants will be admitted. We investigated whether narrative information obtained during multiple mini-interviews (MMIs) can also be used to predict problematic study behavior. Methods: A retrospective exploratory study was performed on students who were selected into a four-year research master’s program Physician-Clinical Investigator in 2007 and 2008 (n = 60). First, counselors were asked for the most prevalent non-cognitive problems among their students. Second, MMI notes were analyzed to identify potential indicators for these problems. Third, a case-control study was performed to investigate the association between students exhibiting the non-cognitive problems and the presence of indicators for these problems in their MMI notes. Results: The most prevalent non-cognitive problems concerned planning and self-reflection. Potential indicators for these problems were identified in randomly chosen MMI notes. The case-control analysis demonstrated a significant association between indicators in the notes and actual planning problems (odds ratio: 9.33, p = 0.003). No such evidence was found for self-reflection-related problems (odds ratio: 1.39, p = 0.68). Conclusions: Narrative information obtained during MMIs contains predictive indicators for planning-related problems during study. This information would be useful for early identification of students-at-risk, which would enable focused counseling and interventions to improve their academic achievement.


Introduction
A variety of selection procedures is used at medical schools to pick those candidates that are best able to perform successfully during their study and beyond (Siu & Reiter 2009;Prideaux et al. 2011;Monroe et al. 2013). Until recently, the focus has been primarily on predictors of cognitive academic performance (Salvatori 2001;Siu & Reiter 2009). Nowadays, however, it is clear that, besides cognitive skills, non-cognitive qualities like professionalism, time management and communication are important competencies of future medical students and doctors. Recent studies have shown that problematic professional behavior during medical school is predictive of future professional problems (Papadakis et al. 2005;Tamblyn et al. 2007). Other difficulties students often experience are in organizing and integrating large amounts of new information (Paul et al. 2009), time management, effectiveness of study strategies, and in self-regulatory and metacognitive skills (Winston et al. 2010). As a result, these students may be at risk with respect to their study progress.
Till date, it has proven to be difficult to assess noncognitive, personal qualities in selection procedures validly. Judgment biases may influence selection outcomes in an unwanted manner. In addition, reliability is at stake due to poor sampling when applicants are subjected to only one interview (Prideaux et al. 2011). Recently, the so-called multiple mini-interview (MMI) procedure was introduced to assess non-cognitive skills more reliably (Eva et al. 2004). In subsequent studies, the MMI method was shown to be positively correlated with early medical school performance, and also with clerkship performance ratings and OSCE performance (Reiter et al. 2007;Eva et al. 2009). These studies show that multiple individual human judgments of

Practice points
Student selection procedures for medical schools can serve a wider purpose than merely to detect which students to accept and which not. Narrative information obtained during selection interviews contains predictive indicators for future noncognitive problems, which can be used to make student mentors more attentive for early warning signs. Such additional use of selection methods would mean an important step forward in terms of social accountability of medical schools. non-cognitive skills when combined predict future performance in a sufficiently reliable way.
In 2007, the MMI method was introduced as part of the selection procedure for the four-year medical research master Physician-Clinical Investigator (P-CI) at Maastricht University (Guyaux et al. 2010). The selection procedure results in a ranking list, representing differences in predicted suitability to perform successfully in this research master. This list is used to offer the available training positions to the applicants with the highest ranks. Each year, 100-200 applicants participate in the selection procedure for the P-CI master, in which 30 training positions are available.
Although most selected students are successful in both cognitive and non-cognitive aspects of the study, some encounter professional lapses or problematic study behavior. Clearly, these problems were not predicted by the MMI scores or any other part of the selection procedure. Theoretically, the narrative information that is written down by the interviewers during the MMIs and stored in the student files could be a better predictor of such problems and could constitute a useful resource for the student mentors (called counselors in the P-CI master), but till now this information has been unused. If this assumption about the notes being better predictors is correct, the selection procedure may not only serve to provide a ranking list purely for selection purposes, but also be the starting point for custom-made teaching and an assessmentfor-learning program right from the start (Shepard 2000). Such additional use of selection methods would mean an important step forward in terms of social accountability of educational institutes that are required to use their resources as effectively as possible (Boelen & Woollard 2009). Therefore, we investigated whether narrative information obtained during MMIs in a selection procedure predicts problematic non-cognitive and study behavior during medical school, and, if so, which behavior is predicted.

Context
The four-year P-CI research master is a graduate-entry program that enables students to become medical doctor as well as clinical investigator. This combination makes it a challenging program for the students. Each year, a selection procedure decides which 30 students are allowed to enter this master. They must have finished a biomedical bachelor with good results; GPAs as well as a cognitive test are taken into account in the first part of the selection procedure. The second part consists of MMIs on different topics, such as motivation, past performance, empathy and communication skills. The applicants' performances on each individual interview are graded independently by the interviewers as being ''sufficient'', ''doubtful'' or ''insufficient'', and the combination of all individual scores adds up to a ranking list. In each station, interviewers also make notes that are not used in the procedure itself; both notes and grading are completed in the time interval between individual interviews. The notes are stored for possible use in appeals, to underpin the interviewers' judgments.

Students and counselors
In this study, we focused on students who enrolled into the P-CI master in 2007 (cohort 2007; n ¼ 30) and 2008 (cohort 2008; n ¼ 30). In this master, each student is assigned to a counselor at the start of the first year, who mentors the student on an individual basis throughout his/her study. Each counselor typically takes care of 3-8 students per cohort. Every year, student and counselor meet at least four times. Before each meeting, the student had to update his/her portfolio and reflect on his/her performance based on test results, feedback and experiences. This way, the counselors get to know 'their' students very well, including their non-cognitive performance and study behavior.
Seven counselors mentored the 60 students in cohorts 2007 and 2008 (five in cohort 2007 and six in cohort 2008; four of them were active in both cohorts). In cohort 2007, two students discontinued their study within the first year, while in cohort 2008 three students dropped out within the first month. In the end, 54 out of 60 students have finished their study within four to five years, while one student is currently finishing the last part.
All students included in this study completed a consent form during the selection procedure, in which they consented to the inclusion of their data in a de-identified way in future scientific research. We collected all data during selection and in the course of regular educational activities, and collection did not influence selection or study results in any way. Protection of participants and their privacy was guaranteed because all information was de-identified so that the data could not be traced to individual students. Given these precautions, the study did not need formal ethical approval in our context.

Study design
This retrospective exploratory study was subdivided into three parts. First, the seven counselors were asked to name the three most prevalent non-cognitive problems they encountered in 'their' students, and grade them (3-2-1) to indicate the frequency of occurrence (3 ¼ most frequent). From their reactions the two most highly-graded problems were selected for further analysis.
Second, two independent and blinded investigators (MoE and LS) analyzed the de-identified notes written down during the MMIs of 15 randomly chosen students out of the total of 55, and identified what they thought to be possible indicators for these two most frequent non-cognitive problems.
Third, a case-control study design was used. The counselors were asked to identify the students who exhibited either one or both of these non-cognitive problems during their study (cases). The notes of their MMIs were de-identified and screened by the same two independent and blinded investigators (MoE and LS) to investigate whether the proposed indicators of these problems were indeed present. As a control, the MMI notes of a similar number of control students from the same cohorts (without the identified non-cognitive problems) were screened for the presence of these indicators as well.

Selection notes predict study behavior
Statistical significance of the association between the group of students with the non-cognitive problems (cases) and the presence of indicators for these problems in the MMI notes was tested using odds ratios and confidence intervals. We used www.medcalc.org for calculation of odds ratios, confidence intervals and p-values (Bland & Altman 2000;Sheskin 2004).

Results
Part 1: The two most prevalent non-cognitive problems All seven counselors independently identified three noncognitive problems that they encountered most frequently in 'their' students and graded them (3-2-1; 3 ¼ most frequent). Table 1 shows the resulting list of non-cognitive problems; planning difficulties and self-reflection-related problems were mentioned most frequently and graded highest. Planning difficulties related to problems with time management, underestimation of study load, and problematic prioritizing of tasks. Self-reflection-related problems were addressed as insufficient awareness of (the consequences of) own functioning, indications of defensive behavior, and insufficient or non-effective actions to improve this.

Part 2: Indicators in MMI notes
The narrative information that was written down during MMIs with 15 randomly chosen students was analyzed to investigate whether indications for the two most prevalent non-cognitive problems were already present during the selection procedure preceding the master.
In the MMI notes of five students both investigators found no indicators at all for the two non-cognitive problems. In the MMI notes of the other 10 students one or more potential indicators were found. In four of them potential indicators for both planning-related and self-reflection-related problems were present. The two investigators identified the same indicators in all but one case; this case was discussed and agreement was reached.
Illustrations of quotes that both investigators felt to indicate planning difficulties during secondary school and/or the preceding bachelor study were: ''Without stress she has difficulty to start studying; external pressure is needed for serious planning'' ''When he finds the content of a unit less interesting, he tends to study less'' ''Her perfectionist nature forces her to read everything, which can cause planning problems'' ''An interesting, extracurricular subject distracted her attention, causing bachelor grades to decline''.
Potential indicators of problems related to self-reflection were: ''Student indicates that she studies easily, while her bachelor grades are only just sufficient and her progression is limited'' ''She characterises herself as a good planner, but at the same time (while discussing another item in the interview) indicates that she has trouble planning''.
The latter quote illustrates a lack of consistency in the candidate's opinion about her own functioning, suggesting limited self-reflection.
In addition, the following quotes were found in the notes of the interview involving a discussion with a simulated patient, in which students were asked to reflect on their own functioning: ''Self-reflection is primarily aimed at patient problem, not at own communication'' ''Limited self-reflection'' ''Self-reflection on empathy is insufficient''.
In one case, the notes pointed out that a student presented himself as an easy talker, who was very self-confident. After discussion, the investigators considered this to be a potential indicator of limited insight in own functioning as well.
As a result of this analysis, a limited number of potential indicators for planning-related and self-reflection-related problems were identified (Table 2).

Part 3: Case-control study
Based on the above-mentioned findings, a case-control study was performed to investigate how predictive these indicators were for planning-related and/or reflection-related problems during the research master P-CI.
The seven counselors identified 23 students who exhibited planning-related and/or reflection-related problems during their study (cases). Thirteen students had planning-related problems, while six had reflection-related problems; another four students showed problems in both domains.
reflection-related problems) were de-identified and screened for the indicators mentioned in Table 2. Indicators for planning-related problems were found in MMI notes of 23 of these 44 students, while the notes of 11 students contained indicators for reflection-related problems. Illustrations of quotes that were considered indicators for planning-related problems are: ''Study progress retarded; put a lot of time in nonstudy related business'' ''Not well prepared for interviews; no priority'' ''Has problems starting to study without (external) stress'' ''When a subject is less interesting, study is postponed to the latest moment'' ''Perfectionist, wants to read everything; can be a pitfall''.
Quotes considered as potential indicators for reflectionrelated problems: ''Poor self-reflection'' ''Caveat: often reacts with 'yes, but.,', suggesting a defensive, non-self-reflective nature'' ''Beware, a flood of words, knows how to tell everything in a convincing way''.
Altogether, the data indicate a statistically-significant association between the presence of indicators for planning-related problems in MMI notes and the actual occurrence of such problems during the subsequent study (Table 3A: odds ratio 9.33; 95% confidence interval 2.12-41.07; p ¼ 0.003). No such evidence was found for self-reflection-related problems (Table 3B: odds ratio 1.39; 95% confidence interval 0.29-6.68).

Discussion
The selection of applicants for a medical study is generally aimed at selecting those candidates that are most likely able to perform successfully during their study and also in their subsequent professional life. As a result, the selection procedure is merely used to decide on who is admitted and who is not. In the current study, we propose to use narrative information obtained during selection interviews to predict future problems in non-cognitive behavior with the intent to make counselors more attentive for early warning signs. This may enable early and dedicated counseling and remediation to improve the selected students' study success. This way, selection will not only serve as an assessment-of-learning measure but also as a first assessment-for-learning step (Shepard 2000;Schuwirth & Van der Vleuten 2011). This Table 3. Odds ratios for the association between the presence of narrative indicators for planningrelated (A) or self-reflection-related problems (B) in MMI notes taken during selection on the one hand and the actual occurrence of such problems during study of the same students on the other.  Table 2. Potential indicators for planning-related and self-reflection-related problems in MMI notes of 15 randomly chosen students.

Potential indicators in MMI notes
Planning-related problems -Delayed study progress in bachelor phase -Extrinsic instead of intrinsic motivation to study -Perfectionist character -Priority conflicts Self-reflection-related problems -Insufficient reflection on own functioning -Mismatch between explanation for own functioning in bachelor phase and results/facts -Performance-instead of progression-oriented -External instead of internal locus of control -Easy talker, high level of self-confidence approach may decrease student drop-out and prevent study delay.
Our study shows that narrative information written down during MMIs, as part of the selection procedure, contains indicators of planning-related problems that appear to be significantly predictive for the actual occurrence of such problems during medical school. Indicators of extrinsic instead of intrinsic motivation to study, priority conflicts, and/or a perfectionist character are predictive of problematic selfmanagement that can lead to delayed and/or decreased study progress or even drop-out.
All seven counselors encountered planning-related problems among 'their' students; six of them indicated that this was the most frequently occurring non-cognitive problem. Up to now, counselors advise and stimulate students to improve the organization and management of their study from the moment planning-related problems expose themselves. With the current knowledge, however, counseling can be more focused right from the beginning of a study career, enabling specific educational and even therapeutic interventions. Literature shows that unorganized students suffer most from workload and pressure of progressing in their studies according to a predetermined timetable (Ruohoniemi et al. 2010). More than aptitude, time management and prioritizing are important for academic achievement (West & Sadoski 2011). Organized studying appears to be related to both study progress and success (Rytkonen et al. 2012). Therefore, early and dedicated counseling will help to prevent or diminish planning-related study problems and, as a consequence, improve study success. Retrospective analysis of study progress and success of the students under study suggests that the delay faced by some of them could have been prevented by the currently proposed approach.
Another non-cognitive problem frequently encountered by the counselors was limited self-reflection. Although potential indicators for this problem were identified in the MMI notes, the occurrence of these indicators was not significantly predictive of the actual occurrence of self-reflection-related problems during medical school. Adequate self-reflection is nowadays considered an essential attribute of competent healthcare professionals. This is why it is one of the important goals of our portfolio and counseling system to increase students' awareness of the importance of self-reflection and to stimulate development of their reflective skills (Driessen et al. 2005). Since reflective practice is regarded as an important skill for medical students, and early and focused guidance is expected to stimulate development of this skill, further study is needed to better identify indicators for self-reflection problems.
Drop-out from or delay during medical school, in spite of selection efforts, is a cause for concern (Yates 2011;Stratton & Elam 2014). This is the case for the students involved who suffer from personal distress, for the university that is faced with a disproportionate amount of time and energy spent on struggling students, and for society that has to bear the financial burden for drop-out and delayed students in countries where they receive public funding. Early identification of struggling students, with focused guidance and remediation, will help to avoid some of this inefficiency. Indeed, the additional use of selection data is attractive from a financial perspective. In countries like the Netherlands, where education is publicly funded, the gains of avoiding delay or drop-out will compensate largely for the costs of a selection procedure and counseling system. Education of students of the four-year P-CI master is financed by the Dutch government per registered year, for a maximum of four years, and after successful completion of the study. Costs of the curriculum, including the selection procedure, primarily depend on the number of hours invested by staff members. The total costs of selection through MMIs and intensive counseling for four years approximate 4500 euros per selected student, while drop-out of a student in the first year results in a loss of income that is more than three times higher. In addition, study delay by one year results in extra costs of about 1000 euros per student for additional counseling only, without extra income. Hence, in a context like ours prevention of drop-out or study delay is clearly worthwhile from a financial point of view. This might of course be entirely different in a country where students pay their own fees without public support.

Limitations of the study
There are some methodological issues to consider. The current study is performed using data from two student cohorts of the P-CI master. As a consequence, the population under study was relatively small (n ¼ 55). On the other hand, both cohorts were taken into account entirely, which suggests that the current findings also hold for subsequent cohorts in this particular master. In addition, the P-CI master applies problembased learning (PBL), in which non-cognitive skills like selfmanagement and prioritizing are key elements for students to be successful (Dolmans & Schmidt 1996). It is questionable whether our findings are generalizable to non-PBL programs. Planning-related problems may be less prominent in more traditional, teacher-centered curricula.
Another issue to consider is the fact that students who successfully enter the P-CI program have already completed a prior bachelor with flying colors. Probably, this previous bachelor did not challenge all of them to employ good time management skills. By contrast, the P-CI program turns out to be very challenging for all enrolled students and demands good time management skills. The strength of the predictability of the narratives we found may not generalize to other contexts with other entry criteria. Yet, we think we demonstrated the idea that using information obtained in selection interviews as fully as possible in the form of a first step in an assessment-for-learning program is useful, but will have to be adapted to best serve the local context.

Conclusion
The present study shows that selection procedures can serve a wider purpose than merely to detect which students to accept and which not. Notes made during selection interviews contain predictive indicators for future study problems, which can be used to alert student mentors or learning coaches to be vigilant for early signs of non-cognitive problems during the study. This would provide an opportunity for early prevention of study delay and attrition and produce a larger return on the investment associated with selection procedures, and as such also contribute to the social accountability mandate of many medical schools.