Assessment of cognitive biases and biostatistics knowledge of medical residents: a multicenter, cross-sectional questionnaire study

Purpose The aim of this study is to determine the perceived familiarity of medical residents with statistical concepts, assess their ability to integrate these concepts in clinical scenarios, and investigate their susceptibility to the gambler’s fallacy and the conjunction fallacy. Methods A multi-institutional, cross-sectional survey of Greek medical residents was performed. Participants were asked to indicate their familiarity with basic statistical concepts and answer clinically oriented questions designed to assess their biostatistics knowledge and cognitive biases. Univariate, bivariate, and multivariate statistical models were used for the evaluation of data. Results Out of 153 respondents (76.5% response rate), only two participants (1.3%) were able to answer all seven biostatistics knowledge questions correctly while 29 residents (19%) gave incorrect answers to all questions. The proportion of correct answers to each biostatistics knowledge question ranged from 15 to 51.6%. Residents with greater self-reported familiarity were more likely to perform better on the respective knowledge question (all p<0.01). Multivariate analysis of the effect of individual resident characteristics on questionnaire performance showed that previous education outside Greece, primarily during medical school, was associated with lower biostatistics knowledge scores (p<0.001). A little more than half of the respondents (54.2%) answered the gambler’s fallacy quiz correctly. Residents with higher performance on the biostatistics knowledge questions were less prone to the gambler’s fallacy (odds ratio 1.38, 95% confidence intervals 1.12–1.70, p=0.003). Only 48 residents (31.4%) did not violate the conjunction rule. Conclusions A large number of medical residents are unable to correctly interpret crucial statistical concepts that are commonly found in the medical literature. They are also especially prone to the gambler’s fallacy bias, which may undermine clinical judgment and medical decision making. Formalized systematic teaching of biostatistics during residency will be required to de-bias residents and ensure that they are proficient in understanding and communicating statistical information.


Introduction
Biostatistical literacy is imperative for physicians in training who must keep abreast of current medical knowledge and be able to communicate statistical and epidemiological information to patients and colleagues. Furthermore, medical residents may be particularly vulnerable to the suggestions of the published literature and other more questionable sources such as promotional brochures and websites (1,2). The vast majority of published medical research contains at least some elementary statistical analysis, and it has consistently been shown that errors in such data are frequently found, even in well-respected

Medical Education Online ae
Medical Education Online 2014. # 2014 Pavlos Msaouel et al. This is an Open Access article distributed under the terms of the Creative Commons CC-BY 4.0 License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. textbooks (3Á7). It is therefore crucial for residents to possess the necessary basic biostatistical knowledge to critically evaluate and apply original research data.
Previous surveys conducted in the 1980s and early 1990s demonstrated that physicians have poor understanding of simple biostatistical concepts (8Á11). Physicians today may have even more pronounced difficulties in comprehending and integrating data derived from the increasingly sophisticated statistical methodologies used in contemporary medical literature (7, 12Á14). More recent studies have shown that internal medicine, emergency medicine, and obstetricsÁgynecology residents have considerable difficulties in interpreting statistical results found in published clinical research (15Á17). Therefore, despite the fact that many medical schools include statistical courses in their curricula, it should not be taken for granted that residents can effectively assess biostatistical information and use it to the best advantage of their patients.
A number of predictable biases in human information processing may considerably hinder sound clinical decision making (18Á21). Therefore, documenting and elucidating these processes may improve patient outcomes, prevent the inappropriate utilization of medical resources, and reduce health care expenditures. A previous instrument designed to evaluate such cognitive biases among physicians (22) was recently reported to have very limited measurement properties (23). The representativeness heuristic is a known factor that may lead to bias in probability estimation by physicians (24,25). The gambler's fallacy is a classic cognitive bias produced by the representativeness heuristic and arises when a person assumes that a deviation from what occurs on average or in the long term will be corrected in the short term (26). A common example is when tails are repeatedly obtained while tossing a fair coin leading the gambler to incorrectly expect that heads are 'due' for the next toss. In reality, the previous tosses have no bearing on the outcome of future tosses. The gambler's fallacy has been shown to be by far the most frequent bias arising in this probability judgment situation, particularly among highly educated individuals (27,28). The conjunction fallacy is another cognitive illusion of probability judgment which was first documented using the well-known 'Linda' example whereby the stereotypical description of Linda as a social activist compels respondents to rank the probability that she is a feminist bank teller as greater than the probability that she is a bank teller. However, this response is considered erroneous as feminist bank tellers cannot be more frequent or more probable than the more inclusive class of bank tellers (29).
The aim of the present multicenter, cross-sectional study was to determine the perceived familiarity of medical residents with statistical concepts and assess their ability to integrate these concepts in actual clinical scenarios. Furthermore, the incidence of the gambler's fallacy and the conjunction fallacy among medical residents were investigated using clinical vignettes that were designed to tap these two domains with specific reference to medicine. The effects of individual resident characteristics such as age, gender, residency year, and past biostatistics training on the cognitive biases and biostatistics knowledge of medical residents were also evaluated.

Participants and data collection
The reporting of this cross-sectional study conforms to the STROBE statement (30). The data presented here are part of a large multi-institutional project evaluating the biostatistical competency and cognitive biases of medical residents. Respondents were originally recruited to participate in a cross-sectional randomized experimental study of medical resident's Bayesian reasoning performance and the sample size was determined according to that study's requirements. The survey was conducted from January to March 2010, and the participants were chosen from eight major Greek hospitals in the city of Athens (Table 1). The Greek system does not allow official residency training in private hospitals and therefore all institutions surveyed in this study were public. Every resident was given a number according to the official enrolment lists that had been provided by each of the hospitals. Out of 1,272 eligible residents in all hospitals, 200 (25 from each hospital) were randomly selected by a computerized method to participate in the sample, a number that represents approximately 2% of the total number of residents training in Greece (31). Participation was elective and all participants were approached during breaks from work or training, were informed that responses would be anonymous and were blinded to the scope and purpose of the study. The residents were asked to return the completed questionnaires to a sealed box provided in each hospital. The study received permission from the authorized personnel at each institution (i.e., the Department or Scientific Meeting Chair). The study complied with Greek requirements for survey studies, and ethical approval was not required as responses were fully anonymous, participation was elective, all participants were approached during breaks from work or training, and the study did not contain questions on sensitive topics. Participants' informed consent was indicated by each individual's willingness to complete and return the questionnaire.

Survey measures
The full questionnaire was developed in the Greek language and is available from the corresponding author. The first seven questions queried the socio-demographic profile, specialty choices, previous education outside Greece, current training level, residency year, and past training in biostatistics of residents. We combined residencies according to their conceptual and occupational relations and formed three different medical 'fields': 1) internal medicine (n071; dermatology, pediatrics, neurology, and general practice were also included in this group); 2) surgical specialties (n067); and 3) diagnostic and laboratory specialties (n 015). The participants were then asked to indicate their understanding of the statistical concepts of standard deviation (SD), standard error of the mean (SEM), p-values, confidence intervals (CI), correlation coefficients, relative risk (RR), sensitivity, and positive predictive value (PPV) by rating their familiarity with each concept on a 7-point Likert scale ranging from 'none' (score of 1) to 'excellent' (score of 7). The next set of questions was developed by one of the authors and assessed the biostatistics knowledge ( Table 2) and cognitive biases (Table 3) of medical residents.
A mathematician from the Section of Statistics and Operations Research of the Department of Mathematics of the University of Athens reviewed the questions for clarity and mathematical appropriateness. In order to assess the intelligibility, interpretation, content validity, adequacy of response options, and clinical relevance of the vignettes, the questionnaire was pretested in a separate group of seven medical residents, including two residents with advanced training in epidemiology and biostatistics. The residents completed the questionnaire and provided oral feedback which resulted in the removal of one question to avoid duplication of similar concepts. Both residents with advanced biostatistics backgrounds answered all questions correctly. Of the remaining five residents, four answered four of seven biostatistics knowledge questions correctly while one resident correctly answered two out of seven questions. Four of these five residents selected the 'approximately 10%' choice in the gambler's fallacy scenario and answered that the probability of suffering from coronary artery disease (CAD) and Huntington's disease is greater than the probability of suffering from Huntington's disease. Despite extensive discussion of the conjunction fallacy vignette's probabilistic nature, these residents were very reluctant to accept the strictly mathematical reading of the problem.
A 'total biostatistics knowledge' score, which equally weighted each of the seven questions listed in Table 2, was  CI vignette You read in a medical paper that the RR for a disease in individuals who are exposed to a risk factor is 1.64 (95% CI 01.10Á2.45) compared to otherwise similar individuals who are not exposed to this risk factor.
Which of the following statements is most correct?
95% of those exposed to the risk factor had RR between 1.10 and 2.45 There is a 95% likelihood that the true RR falls between 1.10 and 2.45 Approximately 50% of individuals exposed to the risk factor had a RR of    Table 2). Median differences between more than two groups were evaluated using the KruskalÁWallis one-way analysis of variance by ranks. The Spearman rank correlation coefficient was used to determine the correlation between two continuous variables. Median differences between two groups were analyzed using the MannÁWhitney U test for two non-paired data. Categorical variables were compared using Pearson's Chi-square or Fisher's exact tests where appropriate. The internal consistency of the 'total biostatistics familiarity' and the 'total biostatistics knowledge' scores was assessed by examining the Cronbach's a coefficient. Cronbach's a 0.6 were considered to be acceptable (33). Bivariate analyses were performed to identify individual resident characteristics that may be associated with total biostatistics knowledge score, total biostatistics familiarity score, or performance on the cognitive biases questions. Candidate factors included age, gender, hospital, residency field, residency year, residency program type, possession of other advanced degrees, education abroad, past biostatistics training, and total biostatistics familiarity score. Variables with p B0.05 on bivariate comparisons were further included in multivariate regression analyses. Multivariate analysis was performed with linear regression analyses as well as binary logistic regression tests using the dichotomous coding of responses (correct/ incorrect) as the dependent variable. To adjust for multiple pairwise comparisons, pB0.01 was considered statistically significant.

Results
The survey response rate was 76.5% (153/200). Table 1 details participant demographic profile, education, and specialty choice. The majority of residents (81.7%) reported prior attendance of biostatistics courses, which mainly occurred during medical school. Responses to each of the individual biostatistics knowledge and cognitive bias questions are presented in Tables 2  and 3, respectively. The proportion of correct answers to each biostatistic knowledge question ranged from 15% in the scenario concerning correlation coefficients to 51.6% in the diagnostic sensitivity question. The most popular incorrect answer (selected by 63.4% of respondents) in the correlation coefficient question stated that the correlation between the two variables was strong. However, the correlation coefficient provided in the vignette (r 00.29) was actually low. Only two participants (1.3%) answered all biostatistics knowledge questions correctly while 29 residents (19%) gave incorrect answers to all seven questions. Reliability analysis of the 'total biostatistics familiarity' and the 'total biostatistics knowledge' scores showed acceptable a values of 0.904 and 0.648, respectively. The mean9SD of the 'total biostatistics knowledge' score was 2.191.8 (range 0Á7). No significant association was found between residency field and total biostatistics knowledge score (p 00.066). On the other hand, bivariate analyses indicated a significant association of total biostatistics knowledge score with residents' age (r 0(0.166, p00.041), training year (r 0(0.205, p00.011), total biostatistics familiarity score (r 00.333, pB0.001), and education abroad (p 00.004). Multivariate analysis was further performed in order to estimate the independent effects of these factors on biostatistics knowledge (Table 4). Residents' perceived familiarity with biostatistics was associated with higher total knowledge scores (pB0.001) while previous education outside Greece was associated with lower total knowledge scores (p00.005).
Perceived familiarity ratings of biostatistical concepts by medical residents are listed in Table 5. Thirty-one residents (20.3%) gave scores 54 on all familiarity ratings. Notably, B25% of residents reported having above average knowledge (familiarity rating 4) of correlation coefficients. The mean familiarity score for this statistical concept was 2.991.9 (mean9SD) which was significantly lower (pB0.01) compared to all other familiarity ratings. Bivariate analyses showed that residents with higher selfreported familiarity with SD (p B0.001), SEM (p00.001), p-values (p00.009), CI (P B0.001), and correlation coefficients (pB0.001) performed better on the respective knowledge question. Also, there was no significant association between self-reported understanding of the concepts of RR (p00.134) and sensitivity (p 00.571) with the respective knowledge question. The mean9SD of the 'total biostatistics familiarity' score was 31.6912.4 (range 8Á55). In bivariate analyses, total biostatistics familiarity was significantly associated with past training in biostatistics (p00.006) and possession of other advanced degrees (pB0.001). Conversely, residency field was not significantly associated with total biostatistics familiarity (p 00.236). A linear regression model was fitted with 'total biostatistics familiarity' score as the dependent variable and past biostatistics training and possession of other advanced degrees as the independent variables (Table 6). Residents who have previously attended biostatistics courses tended to report higher familiarity scores although this effect did not reach statistical significance (p 0.01). Residents with other advanced degrees were significantly more likely to report higher familiarity with statistical concepts (pB0.001). Table 3, approximately half of the respondents (54.2%) answered the gambler's fallacy quiz correctly. Furthermore, less than one third of residents surveyed (31.4%) judged the probability that John has Huntington's disease to be greater than the probability that John has both CAD and Huntington's disease. Preliminary bivariate analysis of factors that may affect resident performance on the gambler's fallacy problem was followed by multivariate logistic regression which showed that residents with higher total biostatistics scores were more likely to answer that quiz correctly (odds ratio 01.38, 95% CI 1.12Á1.70; p 00.003). Also, none of the investigated individual resident factors were found to be significantly associated with performance on the conjunction fallacy quiz.

Discussion
The present multicenter survey of biostatistics performance and biases demonstrated that a considerable proportion of medical residents are prone to errors and biases that undermine judgement and decision making. The fact that, on average, only two out of seven biostatics knowledge questions were answered correctly raises serious concerns. This significant lack of biostatistics aptitude was observed despite the fact that the majority of participants have attended undergraduate biostatistical training during medical school. It should be noted, however, that most residents did not reinforce these courses during residency training. Indeed, the biostatistics knowledge score tended to decline with progression through the years of residency training although this observation did not reach statistical significance (Table 4). Furthermore, we did not observe a significant performance difference between residents who have previously received biostatistics education with those that have never received such training at any point in their career. Taken together, these data suggest that biostatistics skills may Percentages in parentheses do not add up to 100% due to rounding. SD, standard deviation; SEM, standard error of the mean; CI, confidence intervals; RR, relative risk; PPV, positive predictive value. atrophy with progression through medical training and further effort should be expended to maintain knowledge in that sphere during residency.
The majority of residents were unable to properly combine significance level information with correlation coefficient data (Table 2). This mistake was observed even in respondents who have selected the correct definition of the p-value in the relevant question. Educators may therefore need to reevaluate how these concepts are taught and represented in a way that medical residents can understand and use in rational, evidence-based decisions.
Further analysis of the effects of individual resident variables (Table 4) revealed that respondents (all Greek citizens) who have received medical education outside Greece, predominantly taking place during undergraduate medical education in international medical schools, had lower biostatistics knowledge scores compared to residents who have never trained abroad. This performance difference may be partly due to variability in medical school experiences obtained by Greek citizens abroad but it may also reflect differences in ability and effort. Further research will be required to elucidate the potential performance differences between graduates of Greek and international medical schools particularly with regards to educational quality, clinical outcomes and quality of care. It should also be noted that other resident characteristics, including age, gender, advanced Graduate training, specialty choice, hospital and residency program type did not affect the biostatistics knowledge score of residents indicating that the insufficient biostatistical literacy reported in the present study is similar across a broad range of medical residents in Greece. Furthermore, biostatistical performance was homogenous among residents training in different specialty programs. To our knowledge, the present study is the first to investigate and directly compare the biostatistics knowledge between different resident specialty fields. Recent studies have assessed the biostatistical competence of internal medicine, emergency medicine, and obstetricsÁgynecology residents (15Á17). Contrary to the present study, Windish et al. reported that prior biostatistics training, gender and advanced degrees were predictive of internal medicine residents' statistics knowledge (15). Also, a Danish study mainly focusing on junior hospital doctors did not find significantly higher statistics performance in physicians with previous biostatistics education or interest in research (10). These reports used different sets of questions compared to this study, thus hindering direct comparisons.
Approximately one out of five medical residents acknowledged below average familiarity with eight biostatistical concepts that are routinely encountered in the medical literature (12,14). This lack of comfort was substantiated by the similar percentage of residents who answered all biostatistics questions erroneously. Residents considered themselves the least familiar with the concept of correlation coefficient. Indeed, their actual performance in the correlation coefficient quiz was clearly lower compared to all other tests. The independent association between self-reported biostatistics familiarity and knowledge score was further verified in a multivariate model (Table 4). It is of note that residents with advanced education through a master's or PhD degree were significantly more likely to report higher familiarity with biostatistical concepts although this effect did not translate into substantial performance improvement on the knowledge questions. Previous education in biostatistics also tended to independently increase familiarity rankings although this association did not reach statistical significance. These observations indicate that although residents with advanced training and prior statistical education are more likely to believe that they are comfortable with statistics compared to other medical residents, they experience similar difficulties in interpreting research statistics. Consequently, formalized systematic teaching of biostatistics during residency will be required to ensure that residents are proficient in understanding and communicating statistical information.
To the best of our knowledge, the present study is the first to investigate the susceptibility of physicians to the gambler's fallacy and the conjunction fallacy in medical settings. A recent survey in Germany reported that approximately 60% of the general population selected the correct answer in a coin toss gambler's fallacy test (27). The pervasiveness of this bias among our study population of medical residents is alarming, especially given the fact that the bias was investigated using a problem within their domain of expertise. Residents who felt that the patient has increased chances of having tuberculosis are likely to overlook other important causes of hemoptysis. Notably, residents with higher biostatistics knowledge scores were less prone to the gambler's fallacy. Therefore, continuing medical education programs focusing in statistical concepts and reasoning may mitigate the probability judgment biases of residents (34,35). Furthermore, specific instruction aimed at increasing awareness of the gambler's fallacy may reduce residents' susceptibility to this bias (35Á37).
Approximately 80Á90% of respondents generally violate the conjunction rule according to previous reports using various presentations of the 'Linda' problem (29,38,39). In the present study, we found a slightly lower percentage (68.6%) of violations of class-inclusion in medical residents using a clinically based problem. This percentage was found to be independent of residents' individual characteristics including age, gender, residency year, past biostatistics training, performance on the biostatistics tests and possession of other advanced degrees. A number of authors have argued that linguistic and pragmatic features in conjunction fallacy scenarios may lead respondents to alternative evaluations (38,40). For example, if residents interpret the statement 'John has CAD' to additionally imply that 'John does not have Huntington's disease' then ranking this statement as more probable than 'John has CAD and Huntington's disease' would not be considered a conjunction fallacy. In addition, similarly to the English and German languages, the word 'probable' is polysemous in Greek and residents faced with multiple possible interpretations may infer a non-mathematical meaning of probability.
A further interpretation of the high number of conjunction violations found in the present study may be that residents apply different inferential rules and heuristic procedures resulting in answers that appear mathematically incorrect (41). Physicians use a repertoire of clinically relevant assumptions to arrive at a differential diagnosis based on the patient's history, symptoms and physical examination. These diagnostic norms tend to focus on the most relevant information and assign that information its appropriate weight taking into account various clinical goals and expectations. This strategy may indeed disregard syntactical systems such as the conjunction rule. However, medical decision making is not dependent solely on the probability of an event (e.g., a disease). More often, clinicians focus on calculating the probability times payoff or cost when constructing a differential diagnosis list. Although this approach can lead physicians to rank the constituent as more probable than the conjunction it may also reflect an effective and intelligent way to tackle clinical uncertainty. During pretesting of our conjunction fallacy vignette, residents were specifically made aware of the mathematical nature of the test and the class-inclusion violation of ranking the probability of CAD and Huntington's disease as more probably than Huntington's disease. However, they still insisted that paying attention to the gist that the patient had multiple risk factors for CAD was far more reasonable than precise mathematical rankings.

Limitations
The cross-sectional design of the present study prevented determination of causality. Furthermore, in order to fully protect residents' anonymity, we were unable to collect any further data on non-respondents. Although the response rate was significantly higher than what is typical in surveys of physicians (42), we cannot exclude the possibility that residents who felt less comfortable with statistical concepts may have been less willing to complete and return the questionnaire. The present survey was purposely kept brief in order to achieve maximum participation. We thus limited the assessment of residents' biostatistics knowledge to seven biostatistical concepts ( Table 2) which however represent some of the most commonly used statistical methods found in the medical literature today.

Conclusions
The measured biostatistics performance in our survey indicates that an alarmingly large number of medical residents are unable to correctly interpret crucial statistical concepts that are commonly found in the medical literature. These difficulties were pervasive, extending even to residents with prior biostatistics education and advanced graduate training. In addition, residents were found to be especially prone to the gambler's fallacy bias, which may undermine clinical judgment and medical decision making. Frequent violations of the conjunction rule in a clinical scenario were also observed and residents were uncomfortable with the mathematical explanation of the problem. The low performance of medical residents in the biostatistics knowledge tests and the gambler's fallacy question was independent of any prior biostatistics education, which mainly occurred during medical school. Therefore, in order to adequately develop and maintain the biostatistical reasoning of medical residents, educators may need to re-evaluate how such information is taught as well as emphasize and systematize the teaching of statistical concepts during residency.