Better but still biased: Analytic cognitive style and belief bias

Belief bias is the tendency for prior beliefs to influence people's deductive reasoning in two ways: through the application of a simple belief-heuristic (response bias) and through the application of more effortful reasoning for unbelievable conclusions (accuracy effect or motivated reasoning). Previous research indicates that cognitive ability is the primary determinant of the effect of beliefs on accuracy. In the current study, we show that the mere tendency to engage analytic reasoning (analytic cognitive style) is responsible for the effect of cognitive ability on motivated reasoning. The implications of this finding for our understanding of the impact of individual differences on belief bias are discussed.

People often think in a more effortful manner when faced with statements with which they disagree.Climate change deniers are an extreme example of this phenomenon: When faced with an unbelievable statement (the climate is changing), they recruit more effortful thinking to discredit the strong evidence in favour of climate change (Lewandowsky, Oberauer, & Gignac, we will refer to as motivated reasoning1 (Evans, Barston, & Pollard, 1983).Beliefs also impact the judgement process in a more shallow way: Believable arguments are more commonly accepted than unbelievable arguments, something we refer to as response bias. 2 We investigate these effects within a syllogistic reasoning paradigm (Evans et al., 1983).
Whereas response bias is a commonly accepted component of belief bias, the role of motivated reasoning is controversial.The controversial nature of motivated reasoning stems from the fact that much evidence for it stems from the problematic analysis of acceptance rates (Dube, Rotello, & Heit, 2010;Heit & Rotello, 2014;Klauer, Musch, & Naumer, 2000).Klauer et al. (2000) demonstrated how a formal modelling approach based on multinomial processing tree (MPT) analysis could overcome these deficits.Following up on this work, Dube et al. (2010) used signal detection theory (SDT) to disentangle the contribution of beliefs to changes in response bias and accuracy.Their data was consistent with the idea that belief bias may just be a response bias.Trippas, Handley, and Verde (2013), using SDT, only partially replicated their finding.In contrast to the lower cognitive ability (CA) participants who operated according to the response bias account, higher CA participants showed more signs of motivated reasoning These findings were replicated and extended in an additional study using the forced-choice reasoning paradigm (Trippas, Verde, & Handley, 2014b).In the current paper, we follow up on the well-established result that the propensity to engage in motivated reasoning differs as a function of individual difference variables linked to analytic processing.We test the alternative hypothesis that the mere willingness to engage deliberative reasoning processes (i.e., analytic cognitive style [ACS]), rather than the ability to do so (CA), is the driving force behind motivated reasoning.

Individual differences in reasoning
While CA plays an important role in reasoning and decision making, theory and research suggest that ACS may be even more important (Stanovich & West, 2000).The precedence of ACS over CA has been established in both low-level (e.g., perceptual judgement: Klein, 1951) and high-level (e.g., education: Sternberg & Zhang, 2001) domains.Recent research indicates that ACS independently predicts the degree of religious and paranormal belief (Pennycook, Cheyne, Seli, Koehler, & Fugelsang, 2012) and moral judgement (Pennycook, Cheyne, Barr, Koehler, & Fugelsang, 2014a), suggesting that the motivation to engage analytic reasoning processes may in fact have a stronger influence on certain reasoning outputs in everyday contexts than does the ability to reason analytically.
The increased predictive power of ACS over CA in a variety of fields suggests that ACS could play an important role in motivated reasoning.This is corroborated by the finding that the cognitive reflection test (CRT; Frederick, 2005), a measure of ACS, explains additional variance in the tendency show various thinking and reasoning biases beyond CA alone (Toplak, West, & Stanovich, 2011, 2014).Given that ACS and CA are highly related, yet separate concepts (r % .50: Frederick, 2005;Toplak et al., 2011Toplak et al., , 2014)), a viable alternative explanation for the CAÀmotivated reasoning link is that perhaps ACS is the critical factor.This leaves two competing hypotheses.If CA is the stronger predictor, then it would suggest the difference in the propensity to show motivated reasoning is a quality effect (Evans, 2007;Thompson & Johnson, 2014): Better and poorer reasoners are equally likely to apply analytic processing when facing unbelievable conclusions, but better reasoners succeed because their analytic processing is superior.Conversely, if ACS is more important, then this would suggest a quantity effect: Better and poorer reasoners differ in the amount of analytical processing they are willing to engage in when faced with unbelievable conclusions, and better reasoners achieve higher reasoning accuracy under these circumstances because they engage in more analytic processing.
We pitted these hypotheses opposite one another in a large-sample, highpowered individual differences study using a formal modelling approach based on SDT.As discussed earlier, the use of a formal framework such as SDT is necessary to overcome the serious shortcomings of analysing raw endorsement rates (Dube et al., 2010;Heit & Rotello, 2014;Klauer et al., 2000;Trippas et al., 2014c;although see Singmann &Kellen, 2014, andTrippas, Verde, &Handley, 2015, for a reply).The role of individual differences in ACS and CA for belief bias has not been investigated within a formal modelling framework, questioning the reliability of previous work.Our main aim was to determine whether the relation between motivated reasoning and CA may be driven by ACS.We also investigated the impact of CA and ACS on reasoning accuracy and response bias.
Typically CA is measured using standard measures of fluid and crystallised intelligence such as IQ tests, whereas ACS is measured using self-report scales such as the Actively Open-Minded Thinking scale (AOT) or the Need for Cognition (NFC) questionnaire.One advantage of these measures is that they are well validated from a psychometric perspective.On the other hand, the fact that the measurement format differs substantially between these two Downloaded by [University of Plymouth] at 07:01 15 September 2017 constructs can be an issue of some concern, particularly when investigating deductive reasoning, itself a performance-based behavioural task.Attempting to equate the measurement format across all three constructs, in the current experiment we opted to use a battery of tasks that use items with a similar format to measure CA (numeracy, WordSum, and neutral base-rate neglect problems) and ACS behaviourally (CRT, ratio bias, and incongruent baserate neglect problems; see the Method section for additional details).
Naturally, using performance-based measures to assess ACS means that all ACS and CA measures necessarily require some degree of ability and cognitive style (for a similar discussion in this journal, see Pennycook et al., 2014a).Theoretically, it is impossible to develop a performance-based measure that only reflects ability or style, but not both.If one does not have the willingness to think analytically when given a problem, the ability to compute the solution to the problem will not help.Likewise, if one does not have a requisite level of CA, the willingness to think analytically will not be at all beneficial.Since this is true of both ACS and CA measures, the measures should be viewed as more reflective of one construct over the other, but not as purely one or the other.The key factor that distinguishes ACS and CA measures used here is the presence of an incorrect intuitive lure that necessitates an additional level of analytic reasoning.For example, whereas incongruent base-rate neglect problems contain a conflict between a salient stereotype and base-rate information (see Table S1 in the supplementary materials for an example), neutral base-rate neglect problems do not contain any stereotypical information.Thus, the incongruent base-rate problem is considered an ACS measure because it cues an intuitive response based on the stereotypical information that requires an extra level of analytic reasoning to override, whereas the neutral base-rate problem is considered a CA measure because it assesses one's ability to use base-rate information in judgement (see De Neys & Glumicic, 2008;Pennycook, Trippas, Handley, & Thompson, 2014b).Based on our previous work (Trippas et al., 2013), we predict systematic individual differences in the tendency to show motivated reasoning as a function of analytic factors.If the quality hypothesis holds, CA should be a better predictor of motivated reasoning than ACS.Conversely, according to the quantity hypothesis, ACS should be a better predictor of motivated reasoning than CA.

Participants
A total of 191 University of Waterloo undergraduates volunteered to take part in the study (62 male, 129 female, age range D 17À50, M D 20, SD D 3).Individual differences data was unavailable for nine participants.
Downloaded by [University of Plymouth] at 07:01 15 September 2017 Design Logical validity (valid vs. invalid) was crossed with conclusion believability (believable vs. unbelievable) in a within-subjects design.

Materials and measures
Belief bias.Eight valid and eight invalid syllogisms were repeated four times with randomly assigned item contents for each participant (cf.Trippas et al., 2013).Belief was manipulated using true or false item contents (e.g., believable: Some animals are not cats, unbelievable: Some cats are not animals).Half the arguments had unbelievable conclusions.Premise believability was controlled for using pseudo-word middle terms (see Table 1).
Participants made validity (valid or invalid) and confidence judgements from 1 (not confident) to 3 (very confident) on each trial.
Cognitive style and ability.Participants were given six different measures that have been successfully used in past research to differentially measure ACS or CA (Pennycook et al., 2012;Pennycook, Cheyne, Barr, Koehler, & Fugelsang, 2013;Pennycook et al., 2014a).The key factor that distinguishes these measures is the presence of a misleading intuitive response cue.Consider the following item from the CRT (Frederick, 2005): around 40% of a standard university sample correctly solve it (Frederick, 2005).This and other CRT items are "difficult" because one must engage effortful reasoning to question, inhibit, and override a salient intuitive response.Differences in ACS are more important for problems such as this because one must be motivated to reflect on an answer that "feels" correct (Thompson, Prowse Turner, & Pennycook, 2011).
Although ACS measures are not particularly difficult in terms of the mental operations required to correctly solve them, CA is still a prerequisite for optimal performance (Pennycook et al., 2012(Pennycook et al., , 2013(Pennycook et al., , 2014a)).Our CA measures were roughly equivalent in terms of difficulty, but did not cue misleading intuitive responses.In some cases, CA measures were selected as matched controls.For example, participants completed both the CRT (an ACS measure) and a numeracy task (a CA measure).As we have argued in the introduction, this is preferable to the traditional approach (Toplak et al., 2011(Toplak et al., , 2014) ) of assessing cognitive style using self-report questionnaires (e.g., the Need for Cognition scale or the Actively Open-Minded Thinking scale) because the syllogistic reasoning task is a performance measure.Given that CA tests are also performance measures, measuring ACS via self-report questionnaires creates a fundamental asymmetry that biases the necessary regression analysis, potentially making CA seem a stronger predictor in the full model than warranted.This explains the relatively poor performance of ACS questionnaires relative to performance-based cognitive style measures, such as the CRT and other heuristics and biases tasks (Toplak et al., 2011(Toplak et al., , 2014)).We want to stress that the CRT explains variance in questionnaire measures of thinking dispositions above and beyond that explained by CA, demonstrating that it is an adequate measure of ACS when CA is controlled for (Toplak et al., 2011(Toplak et al., , 2014)).

Data treatment
Using the confidence rating based receiving operator characteristic (ROC) method, which is part of the SDT framework, we calculated the ROC-logic, ROC-belief, and ROC-interaction indices to quantify the effects of reasoning accuracy, response bias, and motivated reasoning, respectively (see Trippas, Handley, & Verde, 2014a, for a detailed explanation and tutorial).These measures can be thought of as SDT-equivalents of the traditional logic, belief, and interaction index (see Table 2 for an overview on how to derive these indices from raw endorsement rates).Unlike the traditional indices, these measures do not suffer from inflated Type 1 error rates (Heit & Rotello, 2014).The model fit the data well in more than 85% of the cases (i.e., p > .05based on a x 2 test of absolute goodness of fit).Removing participants for who model fit was violated from the analysis did not change the conclusions.The logic index, a measure of reasoning aptitude or accuracy, can be calculated using the formula: VB C VU À IB À IU D .53.Higher values indicate higher reasoning accuracy.The belief index, a measure of the tendency to accept believable problems over unbelievable problems, can be calculated using the formula VB C IB À VU À IU D .49.Higher values indicate a greater tendency to accept believable over unbelievable problems, which we have defined as response bias.
The interaction index, a measure of the tendency to reason better for unbelievable compared to believable problems, can be calculated using the formula VU C IB À VB À IU D .07.Higher values indicate a greater tendency to engage in what we have defined as motivated reasoning.

ANALYTIC STYLE PREDICTS BELIEF BIAS 437
Downloaded by [University of Plymouth] at 07:01 15 September 2017

Preliminary analysis
Based on a boxplot of CA, one outlier was removed from all further analyses.There were no outliers for ACS.We investigated whether the aggregate sample (ignoring individual differences) reasoned above chance, showed a response bias, and showed a motivated reasoning effect, using one-sample t-tests on the ROC-logic, ROC-belief, and ROC-interaction indices.Participants significantly reasoned above chance

Main analysis
Given the continuous nature of our variables, we used multiple linear regressions to investigate the role of ACS and CA on reasoning accuracy, response bias, and motivated reasoning. 3We also investigated whether including one predictor over the other explained additional variance for each of our three criteria of interest.Regression results are reported in Table 3.A priori power analysis for multiple linear regression assuming a small to medium effect size of f 2 D .10, an alpha level of .05,and two predictors indicated that a sample of 99 participants was sufficient to obtain a power level of .80.Our sample size well exceeds this criterion, demonstrating this is a high-powered experiment.
Reasoning ability.CA and ACS were both positive significant predictors of the ROC-logic index, demonstrating that those of higher cognitive capacity and higher cognitive style alike reasoned better.Importantly, cognitive capacity and cognitive style predicted reasoning ability independently, suggesting that the two explain independent sources of variance in reasoning ability.
Response bias.CA and ACS were both negative significant predictors of the ROC-belief index, demonstrating that higher CA as well as higher cognitive style reasoners were less likely to show a belief-based response bias.This result must be qualified by the fact that when both predictors are included, only ACS remains significant.This is consistent with the idea that the effect of CA on belief bias is mediated by ACS.
Motivated reasoning.CA and ACS were both positive predictors of the ROC-interaction index (CA was just shy of significance at p D .058),suggesting that higher ability and style lead to an increased propensity to reason better for unbelievable than for believable problems.Crucially, when both predictors were simultaneously included in the model, only ACS remained significant.This suggests, as with the response bias component, that ACS is the main driving force of the motivated reasoning component of belief bias.We discuss the implications of these findings in more detail in the discussion.
To aid interpretation of these effects, endorsement rates are presented in Table 4.

DISCUSSION
Using an SDT approach, we replicated Trippas et al.'s (2013) finding that CA was positively related to the motivated reasoning component of belief bias.An additional, novel finding was that ACS also positively predicted motivated reasoning.In contrast to the current interpretation of the effect of individual differences on motivated reasoning, cognitive style emerged as the more potent predictor.Interestingly, high CA and ACS also led to a decreased tendency to accept believable over unbelievable problems (i.e., the response bias component of belief bias), once again with the latter being the critical component.Reasoning accuracy was also positively related to ACS and CA.In contrast to the motivated reasoning and response bias findings, ability and style had a similar effect on accuracy.This suggests that a crucial difference between ability and style is that, while both generally support improved reasoning, cognitive style specifically drives both components of belief bias.We want to stress that these findings are based on a formal modelling approach as recommended by Klauer et al. (2000) and Heit and Rotello (2014).
Turning to the quality/quantity distinction introduced by Evans (2007) and further investigated by Thompson and Johnson (2014), we must note that our findings support both interpretations, depending on the dependent variable of interest.According to the quality interpretation, individual differences in thinking are driven by an increased ability to apply analytic thinking.In other words, poor and good thinkers apply similar amounts of effort, but the good ones are better because their analytic thinking is somehow better (e.g., because they are more efficient).According to the quantity interpretation, individual differences in thinking arise from the degree to which people engage analytic thinking, with more capable reasoners simply thinking more than less capable ones.The quality/quantity distinction can be mapped neatly onto the CA/ACS distinction: Higher ability people reason better because they have more resources, allowing for qualitatively better reasoning, whereas higher style people reason more because they choose to do so.Our findings clearly suggest that when it comes to overall reasoning accuracy, ability and style are independent predictors, consistent with the idea that better reasoning happens due to both increase in quality and increase in quantity.In contrast, the tendency to resist applying a simple response bias and the propensity to reason more when faced with Means (standard deviations).Endorsement rates equal the proportions of "valid" responses.
Participants were classified as part of the higher or lower group using median splits.
unbelievable conclusions seem to be a quantity effect.People resist the response bias and show motivated reasoning because they use more analytic processing, not because their effortful thought is better.These findings suggest that all of the currently plausible accounts of belief bias such as mental models theory (MMT; Oakhill, Garnham, & Johnson-Laird, 1989), selective processing theory (SPT; Evans, Handley, & Harper, 2001;Klauer et al., 2000), and the strict version of the response bias account (Dube et al., 2010) require some revision to account for these data.The strict version of the response bias account does not make any explicit predictions about the role of individual differences in belief bias, although it could be extended to do so.Crucially, however, the account predicts that motivated reasoning is not a fundamental component of belief bias.This strong assumption has been refuted in the current experiment (see also Trippas et al., 2013, Trippas et al., 2014a, 2014b).According to MMT, belief bias operates on two levels.Reasoners show a general response bias in terms of a conclusion filtering mechanism, as well as a tendency to generate more mental models if the conclusion under consideration is unbelievable.MMT does not, however, predict that increased levels of ACS are linked to a decrease in response bias.MMT does predict that those of higher working memory capacity are more likely to retrieve additional models, but our findings suggest that the willingness to reason (style), rather than the capacity, is the crucial predictor.A similar argument can be made for SPT, which is highly similar to MMT with the exception that typically only one mental model is thought to be constructed.Unbelievable conclusions cue a difference reasoning style by increasing the propensity to falsify, rather than confirm, the conclusion.However, according to SPT this tendency to falsify is mainly a function of problem features, not individual differences.This assumption has been revised in a recent extension of SPT (Stupple, Ball, Evans, & Kamal-Smith, 2011).According to this individual difference version of SPT, better reasoners (as measured using reasoning accuracy) are more likely to show response latency characteristics typically associated with motivated reasoning (although see Thompson, Morley, & Newstead, 2011, for a counterargument).However, the extended SPT model explicitly assumes that analytic inhibition is a crucial factor in driving this pattern, implying that CA is the critical component.Our findings suggest that not the ability to inhibit, but the willingness to think analytically is linked to motivated reasoning.Identifying ACS, rather than CA, as a key predictor of motivated reasoning provides a focus for the investigation of factors that drive other forms of reasoning strategies.
This pattern of findings paints a novel picture of the role of individual differences in belief bias.One speculative interpretation of these findings could be in terms of an overarching individual difference framework in which ability and style both predict actual reasoning performance, with only style being linked to both components of belief bias.Such a framework may provide a way to integrate multiple algorithmic theories of belief bias (e.g., SPT and the response bias account).Our data suggests that individuals with higher CA but relatively lower levels of ACS will operate as predicted by the response bias account: showing adequate reasoning performance with a response bias, but lacking motivated reasoning.In contrast, individuals with higher levels of both cognitive ability and style might reason more in line with predictions drawn from SPT: i.e., showing above average reasoning performance in combination with motivated reasoning, but with lower levels of response bias.Interpreted in terms of dual process theory (DPT), these findings suggest that the response bias component of belief bias is a marker of T1 processing.In contrast, reasoning accuracy and motivated reasoning appear to be determined by T2 processing.
A potential caveat of the present findings is that we measured CA and ACS using a nonstandardised battery of tasks, including ratio bias, baserate neglect, numeracy, the CRT, and the WordSum, instead of more standardised tests such as Raven's progressive matrices (Raven, 2000) and the Actively Open Minded Thinking scale (AOT: Stanovich & West, 2000).We reiterate in this respect that our measures have been used successfully in the past to compare the relative influence of ACS with CA on multiple outcome variables, including religious beliefs, reasoning aptitude, and moral judgements (Pennycook et al., 2012(Pennycook et al., , 2013(Pennycook et al., , 2014a)).One might even argue that our use of a behavioural rather than a self-report measure of ACS is a strength of the current design.As a parallel, consider the increased predictability of behaviourally measured implicit attitudes (using the implicit association test) over self-reported explicit attitudes (measured using a questionnaire) when it comes to racial discrimination (Greenwald, McGhee, & Schwartz, 1998).
The present findings are important because they indicate that motivated reasoning is linked to something other than the ability to engage in the reasoning process.ACS supposedly reflects the willingness to apply cognitive effort, particularly when a salient response is available.This is a key characteristic of the CRT, where an intuitively appealing response must be resisted in favour of a more reasoned one.An intriguing, and somewhat challenging, aspect of these findings is that ACS predicts not only a tendency to resist the influence of beliefs on responding, but also the extent to which unbelievable conclusions facilitate reasoning accuracy.Perhaps motivated reasoning reflects the tendency for some participants to engage a "sceptical mindset" when faced with an uncertain conclusion, a mindset that could also lead people to question the salient response on the CRT because it seems "too good to be true".ACS possibly reflects an individual's motivation to engage in reasoning in circumstances where such uncertainty is detected.High ACS 436TRIPPAS ET AL.Downloaded by [University of Plymouth] at 07:01 15 September 2017 Procedure Participants were tested individually in laboratory sessions lasting around 45 minutes.Measures were administered in the following order: base rate, ratio bias, CRT, numeracy, WordSum, syllogisms.

TABLE 2
Deriving the traditional logic, belief, and interaction indices from endorsement rate data When multiple predictors are included, b and p values are reported in the order CA; ACS.F refers to the fit of the full model and has denominator degrees of freedom 179 or 178, depending on whether one or two predictors were included, respectively.