The search query filter bubble: effect of user ideology on political leaning of search results through query selection

ABSTRACT It is commonly assumed that personalization technologies used by Google for the purpose of tailoring search results for individual users create filter bubbles, which reinforce users’ political views. Surprisingly, empirical evidence for a personalization-induced filter bubble has not been forthcoming. Here, we investigate whether filter bubbles may result instead from a searcher’s choice of search queries. In the first experiment, participants rated the left-right leaning of 48 queries (search strings), 6 for each of 8 topics (abortion, benefits, climate change, sex equality, immigration, nuclear family, Islam, and taxation). An independent sample of participants were then asked to select one of these queries for each of the 8 topics. With the exception of the topic of Islam, participants were significantly more likely to select a query corresponding to their own political leaning, compared to other queries, explaining between 12% and 39% of the variance. A second experiment investigated the effect of the political leaning of the same queries on the overall political leaning of Search Engine Result Pages (SERPs) in Google Search. The top six results of each SERP were rated collectively by a third group of participants, explaining 36.3% of the variance across all 48 search terms (p < .00001). That is, (1) participants in our experiments tended to select own-side search queries, and (2) using those queries tended to yield own-side search results when using the Google search engine. Our results are consistent with the notion of a self-imposed filter bubble in which query selection plays a salient role.


Introduction
Online search has become an indispensable part of everyday life with search engines such as Google and Yahoo processing billions of queries on a daily basis (Google Search Statistics. 2021).Many queries are quests for knowledge about uncontested facts, such as the capital of France, stock prices, currency exchange rates, et cetera.Others concern deep ideological or political issues about which there may be reasonable disagreement.Whereas the idea of search engines producing rankings based on relevance for queries of the former kind is fairly unproblematic, it is much less clear what 'relevance' would mean in cases of complex political issues, raising the question of the role of search engines in shaping ideological and political beliefs.
Furthermore, if relevance is what the user judges to be relevant, as inferred from the user's search history, the more specific worry arises that the user might end up in a socalled filter bubble, where users are presented with search results that are in line with and reinforce her extant ideological and political views (Pariser, 2011).Filter bubbles are therefore thought to increase undesirable affective polarization and ideological segregation, possibly fragmenting political discourse (Garrett & Resnick, 2011).These and related worries have inspired calls for regulation targeting Google's (and similar search engines') hypothesized filter bubble producing tendencies (Simpson, 2012).More recently, the existence of filter bubbles has even been postulated within the realm of scientific research itself where it has been argued that it might impact its reproducibility and rigor (Ćurković, 2019;Ćurković & Košec, 2018).Within the COVID-19 pandemic context, it has been claimed that retrieved search results and their curation may impact the functioning of society through the uptake of public health measures (Ćurković et al., 2021).
In spite of the public debate and popular opinion regarding this issue, empirical evidence for algorithmically produced filter bubbles has not been forthcoming.Hannak et al. (2013, may) found only small differences in search results between users and no measurable effect of search history on links or their position (cf.Haim et al., 2017).Further, whereas Courtois et al. (2018) did find considerable deviations between users when searching on social and political topics, they also observed that most of the variation could be explained by the time of search as opposed to search history.Finally, Flaxman et al. (2016) found that use of search engines increased users' chances of being exposed to opposing or disconfirming views (cf.Cardenal et al., 2019), seemingly running counter to predictions of the 'filter bubble' hypothesis (Pariser, 2011).Thus, it appears that, contrary to common expectation, algorithmic curation based on search history in Google does not, in fact, have a significant filter bubble producing effect (cf.Zuiderveen Borgesius et al., 2016).
However, the apparent failure to identify algorithmically induced filter bubbles in search results does not exclude that there may yet exist filter bubbles in the context of online search.For example, Yom-Tov et al. (2014) found evidence, using search location data from the Bing search engine and data on zip code voting behavior, indicating that people are more likely to read opinions consistent with their own so that filter bubbles could result from selective exposure (Sears & Freedman, 1967), in which case they would be self-imposed.To set the stage for an elaboration of this point, it is often observed that most items in a Search Engine Results Page (SERP) resulting from a query are ignored in selection, in favor of the top-presented SERP (Joachims et al., 2007), and even within a SERP, a great majority of clicks tend to go to the first five links presented (Petrescu, 2014), due to what is known as the top-link heuristic (Salmerón et al., 2013;Granka et al., 2004).Specifically, as observed in Hotchkiss et al. (2005), gaze is concentrated in the rough form of a right triangle with the right angle facing upwards to the left of the screen dropping significantly outside the hypothenuse, a phenomenon for which they introduced the suggestive term golden triangle.Drawing on these findings, a recent eye-tracking study (Ekström et al., 2022) found that when presented with a politically diverse Google-style SERP, participants tended to attend to and subsequently click on own-side links.Further, there was no observable top-link heuristic when top links in SERPs failed to align with searcher ideology.In other words, participants engaged in self-segregating behavior at early stages of cognitive processing (cf.Kawakami et al., 2014), offering evidence for the hypothesis that filter bubbles may be self-imposed.
Empirical investigations into filter bubbles in search engines typically involve feeding a set of curated queries (i.e., anything you put into the search box, also called 'search terms' or 'search strings') as input to a search engine using different (real or artificial) user accounts (e.g., Courtois et al., 2018Haim et al., 2017;Hannak et al., 2013, may;).The present study departs from this methodology in involving participants selecting search queries from a curated list in a controlled experiment.This makes it possible to investigate the users' choice of (controlled) search query as an act of self-curation that may potentially be, partly or wholly, responsible for filter bubbles.We hypothesize that the choice of query on political topics is partly determined by user ideology in such a way that (1) users will tend to select own-side queries, and that (2) political queries, when used as an as input to a search engines such as Google, will tend to produce own-side search results.The validity of the second hypothesis is sometimes taken for granted, but we are not aware of any experimental or other evidence for either of these two claims.For example, Diakopoulos et al. (2018) remark that '[t]he choice of query terms can itself lead to politically biased results' (p.322) without citing supporting evidence.The present article addresses these gaps.After providing the necessary background on selective exposure and cognitive bias in Section 2, Section 3 describes the first experiment, designed to test whether subjects choose search queries in line with their own political leaning.Section 4 reports the second experiment, designed to test whether the political leaning of queries carries over to the top links on the first SERP when using Google Search.Both experiments were carried out in Sweden, in Swedish, with Swedish-speaking participants.We discuss our findings and, in particular, the extent to which they indicate a self-imposed search query filter bubble in Section 5, together with limitations affecting our investigations, and suggestions for future research.

Selective exposure and cognitive bias
Starting with early work in the 1940s, many researchers have observed that people's ideological views bias their attention to news that confirm beliefs already held (e.g., Garrett & Resnick, 2011;Lazarsfeld et al., 1948;Yom-Tov et al., 2014; see also Gentzkow & Shapiro, 2011).In the social sciences, the term selective exposure denotes this general tendency to seek out information that conforms to extant convictions at the expense of information that does not (see Frey, 1986;Klapper, 1960;Sears & Freedman, 1967, see also Mutz & Martin, 2001).Festinger (1957), famously made this aspect of human cognition a central part of his theory of cognitive dissonance, according to which people strive to reduce the mental discomfort ensuing from simultaneously entertaining two or more incoherent beliefs.Similarly, the term confirmation bias denotes individuals' tendency to seek out confirming evidence (Nickerson, 1998).
Unsurprisingly, given the increasing importance of the internet in information-seeking, much recent work on selective exposure has centered on how users engage with information on various online platforms, such as Facebook (e.g., Bakshy et al., 2015;Cinelli et al., 2020, Dutceac Segesten et al., 2022;Sülflow et al., 2019), Instagram (Parmelee & Roman, 2020) and Twitter (Colleoni et al., 2014;Himelboim et al., 2013; see also Garrett, 2013).For example, Knobloch-Westerwick and Meng (2011) found that participants preferred attitude-consistent over counter-attitudinal messages, the effect being a reinforcement of the existing attitude (as measured by increased attitude accessibility).As mentioned, Ekström et al. (2022) found that participants generally attend to, and subsequently select, own-side search results.Before search results can be attended and selected, however, the user needs to formulate and submit a search query.In the present article, we argue that users' political leanings also affect query formation through the tendency to select own-side queries.For instance, a right-wing or socially conservative user interested in learning more about immigration would be more likely, due to confirmation bias, to search on immigration repatriation than on immigration open society, whereas the opposite would be expected of a left-wing user.

Experiment 1: effect of participants' political leaning on political leaning of their selected search queries
The purpose of Experiment 1 was to measure the extent to which people's choice of search query within a specific topic is affected by their political leaning.

Method
Prior to the experiment proper, search terms were created, rated and selected in order to cover a left-right political dimension with sufficient range for the experiment proper.First, eight topics were derived from Everett's (2013) Social and Economic Conservatism Scale (SECS).The eight topics were (1) abortion, (2) benefits, (3) climate change, (4) sex equality, (5) immigration, (6) nuclear family, (7) Islam, and (8) taxation.To avoid confusion resulting from the fact that the variables consist of ratings of these topics both in terms of a left-right dimension and attitudes towards them in terms of a positive-negative dimension, we refer to the former (left-right) with all capital letters and the latter (positive-negative) with an initial capital letter.Ten pseudo queries were generated for each topic based on phrases or terminology deemed to be commonly occurring in societal debates and selected through an online rating experiment.For example, the climate change queries included (Swedish equivalents of) global warming overrated, raise fuel tax and invest in fossil-free fuel.The sex equality queries included feminism gone too far, gender differences socially acquired, and women discriminated against more than men.
Second, a subset of these search queries were selected so as to represent the full range of the left-right dimension.To this end, 16 raters (9 women) were recruited through online advertisements and directed to the Pavlovia online platform (pavlovia.org),where they performed the rating task, created in PsychoPy (psychopy.org).The eighty search terms (ten for each topic) were presented one by one in all lower-case letters, and participants rated each term according to their perceived political leaning on a 7-point Likert scale, where higher ratings signified more right-wing positions and lower ratings signified more left-wing positions.Terms were presented in white against a gray backdrop.Order of presentation was randomized for all participants.The procedure lasted on average ∼5 min.Participants were not compensated, and except biological sex, no personal information (e.g., age) was sampled.Interrater reliability was high (Cronbach's α = .88),indicating considerable agreement among raters regarding what terms signaled right-wing versus left-wing positions.See Appendix A for a list of search terms and their mean political leaning ratings.
Based on these results, six of the ten queries were selected in such a way as to create an even spread of political leanings within each topic.For example, for IMMIGRATION, the selected search terms and their mean ratings of political leaning were, from left (1) to right (7) immigration a gain (M = 2.5), immigration open society (M = 2.9), immigration sweden needed (M = 2.9), crime immigration (M = 5.6), immigration sweden unsustainable (M = 6.0), and immigration repatriation (M = 6.1).
For the experiment proper, 54 participants (26 women) aged 18-53 (M = 23.2,SD = 5.17) were recruited through university bulletin boards and online advertisements.None had participated in the rating process described above.They were informed that no data would be stored that could associate them with their answers.In each trial, six search queries were presented in configurations randomized according to a 6 × 6 Latin square, such that each query appeared in the same position the same number of times across all participants.This was done to avoid possible artefacts of query presentation position.In total, participants were presented with eight sets of six search queries.For each set, they were asked to select the query they found most interesting, i.e., the query they would normally use.To simulate real-life search, stimuli were presented as all lower-case letters.The procedure lasted on average ∼25 min.
As a verification of the predicted association between the chosen topics and the participants' political leaning, they were asked to rate their attitude toward each of the eight topics post experiment.Each topic appeared on the computer screen, together with a visual analog scale ranging from 1 on the left side to 7 on the right side, with no anchors.Participants were asked to indicate a point on the scale corresponding to their attitude about the topic, with low values indicating negative feelings and positive values indicating positive feelings.After this, they were also asked to select one of two options stating whether they identified as politically more left-or right-wing, with the result that 29 participants identified as the former and 25 as the latter.This portion of the procedure lasted on average ∼10 min.In total, the experimental procedure lasted ∼35 min.Participants were rewarded with a cinema ticket.The experiment was carried out in accordance with the guidelines issues by the Swedish Ethical Authority (Etikprövningsmyndigheten) for research on human subjects.

Results
Three search queries were never selected by any participant, namely feminism gone too far from the SEX EQUALITY topic, arguments against abortion from the ABORTION topic, and no to sex-neutral marriage from the NUCLEAR FAMILY topic.The binary self-selection of being more left -or right-leaning (L-R Binary) was coded 1 for left and 2 for right and sex was coded 1 for male and 2 for female.Note that attitudes are referred to using initial capital letters and search query topics using all capital letters.
Figure 1 summarizes the main results.First, the open markers and dashed lines represent the political leaning of search queries chosen, according to the ratings made in the second step of the search query selection above.Open circles represent participants who identified as left and open squares those who identified as right.Given that higher values on the left ordinate reflect more right preferences, as described in the search term selection procedure, those who identified as more political left should have selected queries that were rated lower.This is also the case for all topics, with marginal differences, such as smaller left-right differences for Abortion and Islam.We now turn to the attitudes, rated post experiment, which are represented by filled markers and solid lines that refer to the right ordinate.Given that higher attitude values reflect more positive feelings, those who identified as more politically left should exhibit higher values for all topics except NUCLEAR FAMILY, which was indeed the case.The left-right differences are substantial for six topics, but again, the differences between left and right are smaller except for Abortion and Islam, with more positive feelings towards Abortion and negative feelings towards Islam for both political leaning groups.
Correlations between age and attitude ratings were small (.01 -.20) except for Abortion (-.34, p < .05),meaning that younger participants were more positive to abortion.Some correlations between sex and attitude ratings were small, but several were of medium strength and statistically significant.Their listing in the third column of Table 1 shows that females were more positive to Abortion, Climate change, Sex equality, and Immigration.These results indicate independent contributions of age and sex, and that these variables should therefore be controlled for in analyses of the association between attitudes and other variables.
The search query political leaning ratings obtained in the search term selection procedure had higher numbers representing more right preferences and are therefore expected to be positively correlated with L-R Binary and negatively correlated with the attitude ratings, again with the exception of NUCLEAR FAMILY.Table 1 shows that this was also the case, except for a few correlations involving Islam.
Factor analysis using principal components extraction was performed for the purpose of creating a continuous variable representing the overall left-right leaning indicated by the attitude self-ratings.As suggested by the small correlations between the attitude to Islam and all other variables, it essentially did not share any variance with the other attitudes, presenting a factor loading of 0.084.Being a poor indicator of left-right preferences, it was dropped from the factor analysis.Factor loadings for the remaining variables in the final model (explaining 45.6% of the variance, EV = 3.54) are listed in column 2 of Table 1.The factor scores for each participant were then copied into the data matrix and named the Left-Right factor.The Left-Right factor was positively correlated with all search query political leaning ratings, as well as with the Left-Right Binary in the second row, supporting its construct validity.
Next, the associations between the participants' political leaning, in terms of the selfreported Left-Right factor, and the political leaning of their selected search queries as assessed by raters, were quantified through eight multiple regression analyses, controlling for age and sex.Table 2 shows that all these associations, except for ISLAM, were statistically significant and substantial, explaining between 12% and 39% of the variance.
The regression models -with the political leaning of the selected search terms as dependent variable -do not exhibit a similar pattern of sex and age influence as found in the zero-order correlation matrix in Table 1 for the attitude ratings.Rather, coefficients for sex and age are overall non-significant and small (β = 0.013-0.187),except for CLIMATE (β = 0.239), reflecting females' selection of more left-leaning search queries (borderline significant at just above .05).

Experiment 2: effect of search query political leaning on political leaning of Google search results
The purpose of this experiment was to determine the degree to which the political leaning of search queries, as determined in the previous experiment, affects the political leaning of search results obtained in a typical Google search.This can be seen as a follow-up to the previous experiments, addressing whether ideological bias on the part of the person performing the search also leads to biased search results, that is, the second component of the filter bubble metaphor.

Method
Google searches were conducted using each of the 48 search terms selected in Experiment 1.We conducted the same searches on several computers.However, because differences were minimal we made use of only the results obtained from one computer.For each set of SERPs returned by Google Search, the top six links were saved as a pdf and printed on a separate page (excluding the search query).Raters (N = 15) were recruited through online advertisements, none of whom had participated in the two previous ratings.Each participant was individually asked to order the six sheets belonging to each topic according to their perceived overall relative political leaning, with those relatively more leftleaning to the left and those relatively more right-leaning to the right, with neutral and politically ambiguous pages closer to the middle.

Results
The SERPs ordered from left-wing to right-wing by the participants were assigned numbers 1-6.Overall, ratings were consistent across raters (Cronbach's α = .936).The correlation between the political leaning rating of the search query and the rating of the SERPs obtained with these search queries was 0.60 across all 48 search terms, explaining 36.3% of the variance (p < .00001),suggesting that, indeed, search queries rated as more politically left-or right-wing also yielded search results that aligned politically with those queries.The mean and SD of the SERP ratings is available in Appendix A.

Discussion, limitations and future work
In this study we investigated whether filter bubbles are the result of individuals' choice of search queries.In Experiment 1, one group of participants first rated the left-right leaning of 80 potential search strings, on the basis of which 48 were selected.Another group of participants were then asked to select one of these queries for each of the 8 topics.Overall, participants were significantly more likely to select a query corresponding to their own political leaning, compared to other queries, explaining between 12% and 39% of the variance.Experiment 2 investigated the effect of the political leaning of the same queries on the overall political leaning of Search Engine Result Pages (SERPs) in Google Search.The top six results of each SERP were rated by a third group of participants, explaining 36.3% of the variance across all 48 search terms (p < .00001).Our results suggest a self-imposed filter bubble in which query selection plays a salient role.The first experiment reported above is platform independent and does not involve the use of a search engine.The second experiment, concerning the influence of the political leaning of selected search terms on the political leaning of search results, made specific use of Google Search.It is an intriguing question whether the result of the second experiment would replicate across other commercial search engines.If this is the case, then to the extent that our results indicate that Google users may find themselves in a selfimposed filter bubble, similar conclusions may apply for other search engines as well.In future work, we plan to put this hypothesis to the test by repeating the second experiment for search engines other than Google.
The indication that search results are influenced by user ideology via the choice of search queries suggests a feedback loop by means of which users' ideological and political beliefs are reinforced through search engines like Google Search.Even so, the existence of such a feedback loop does not follow by logical necessity from our results but depends partly on what links in SERPs the user attends to and clicks on, and partly on the psychological effect of consuming the linked contents.For instance, the present results are consistent with users receiving SERPs that have the same overall bias as themselves, for politically charged queries, but nevertheless attending to and selecting links from the other political camp.However, as mentioned in the introduction, Ekström et al. (2022) found that this is generally not the case.In that study, partisan participants (n = 48) were presented with sets of simulated Google Search results on political issues, controlling for the political leaning of each link.The participants spent more time viewing own-side links than other links (p = .037),an effect that was larger for those who identified as right-wing than for those who identified as left wing (p < .001).In addition, we observed a significant effect across the sample, that participants tended to select ownside links (p < .001).Of course, we would expect such effects to be stronger for SERPs with a higher proportion of own-side links, such as SERPs resulting from an own-side query.For instance, we would expect participants to spend even more time viewing own-side links than other links if there is a greater proportion of the former.Further, given that consuming own-side content tends to strengthen prior political attitude (Knobloch-Westerwick & Meng, 2011), the feedback loop extends all the way back to the user, suggesting a filter bubble in which query selection plays a salient role.
Another intriguing issue centers on the observation that the results we get from search engines may change the way we formulate and select search strings in the first place.An obvious way in which they to this is by suggesting potential 'improved' search strings.Thus, search engines to some extent also train or curate the users, which raises the question where the true, genuine Self (agent, subject) begins.Considerations such as these may cast some doubt on the notion of a self-imposed filter bubble if the latter is taken to imply the existence of an internet-autonomous Self.Yet even if there is no sharp boundary between the Self and the internet to be drawn, it seems enlightening to distinguish between filter bubbles that arise mainly or saliently from algorithmic curation and those that arise mainly or saliently from cognitive features and choices of the user.Robertson et al. (2018) noted that (p.148:5) 'how users formulate and negotiate search queries within web search interfaces is an under-researched topic that involves not only the informational retrieval algorithms at play, but also the cultural history of the human interacting with them' (cf.Diakopoulos et al., 2018;Tripodi, 2018).They added that '[t]he lack of up-to-date research on this topic is due, in part, to the difficulty associated with obtaining real user queries' (cf.Borra & Weber, 2012), 'the ever-evolving nature of users' information needs' (cf.Belkin, 1980) and 'the opaque interactions between users and autocomplete algorithms that influence the process of query selection' (cf.Noble, 2018).It is an advantage of our study that it goes at least some way towards studying how users select search queries in a controlled experiment.Our study suggests that, in a political context, search queries are partly selected based on the searcher's political leaning.
The study is, however, limited by the fact that participants were severely constrained regarding what search terms they could choose -both in terms of the topic and the search query to be selected.For each topic, participants were asked to select the query from a list of six possible options.While experimentally feasible, this methodology ignores much of the complexity in the process whereby users formulate search queries, in favor of experimental control (Robertson et al., 2018).Relatedly, earlier studies suggest that own-side information is more thoroughly attended to and processed (e.g., Kawakami et al., 2014;Xiao et al., 2016).In asking participants to choose between politically charged search queries from a list, this mechanism may have been triggered, leading to the choice of a more closely attended (own-side) query, which may or may not coincide with the query that the participant would naturally use.This issue could be studied by measuring the attention paid to each presented search query using eye-tracking technology.At any rate, future experimental work should aim to improve ecological validity by allowing participants greater flexibility in the formulation of queries.
Further limitations became apparent in the process of choosing search terms for our study.Early attempts at selecting appropriate search query terms included consulting Google Trends, with the object of using real-life popular search queries.Two main problems were apparent with this approach.First, most queries obtained from Google Trends were neutral in phrasing and content, for example, 'immigration 2021', or 'what does the social democratic party think about immigration'.Because we, as the larger literature on potential online filter bubbles, were primarily interested in polarizing material, we determined that these terms were inadequate for our purposes.Second, for many of the topics (immigration, abortion, etc.) few search queries were used extensively according to Google Trends.Thus, this result can be interpreted in different ways: less extensive use may reflect either an unpopular opinion or simply that the topic is more rarely publicly covered online.Our solution was to derive our search query terms from coverage of each topic across various online newspapers and publications.We determined that these were more likely to correspond to real-life interactions on each relevant topic.However, at least some of the included search queries were unrealistic (indeed, at least two participants in the second part of Experiment 1 reported that they may not have used any of the presented search terms themselves).We hope, nevertheless, that this approach may serve to inspire future efforts in methodology development including a systematic way of choosing the online newspapers and publications used for coming up with search queries.
Finally, given the less contentious hypothesis to be tested, we used a simplified rating approach in the second experiment.For instance, we asked the raters to order the SERPs according to perceived political content from left-wing to right-wing rather than, say, instruct them to rate each SERP on a numerical scale.Numerical values were assigned to each SERP thus ordered by the experimenter and used for assessing inter-rater reliability.This rating procedure could be improved.However, we have no concrete reason to think that a different rating process would lead to a substantially different result.

Conclusion
It is commonly assumed that personalization technologies used by Google for the purpose of tailoring search results for individual users create filter bubbles, which reinforce users' political views.Surprisingly, empirical evidence for a personalization-induced filter bubble has not been forthcoming.Here, we investigated whether filter bubbles may result instead from a user's choice of search queries.Participants in our experiments tended to select own-side search queries, and using those queries tended to yield own-side search results in the Google search engine.Our results are consistent with the notion of a selfimposed filter bubble in which query selection plays a salient role.

Appendix A.
Pseudo search terms, for each topic, as presented in Swedish and translated to English, and the mean ratings of both the search terms (in experiment 1) and the SERPs using those same search terms (in experiment 2).

Figure 1 .
Figure 1.Political leaning of chosen search queries (dashed lines and open markers), and self-rated attitudes (solid lines and markers) separately for participant who self-identified as left-(circles) or right-leaning (squares).The left ordinate refers to the dashed lines, representing the mean across participants for the eight search terms selected by each participant, which in turn are based on the political leaning ratings given in the search term selection experiment.The right ordinate refers to the solid lines, representing the strength of the positive attitude towards each of the eight topics.

Table 2 .
Linear regressions models for each of the eight topics, with political leaning rating of the chosen search queries as dependent variable, and the Left-Right factor, age and sex as predictors.
Note.R = .484,R² = .234,Adjusted R² = .188 Search terms were rated for political leaning on 7-point scale; higher values indicate more right-wing content, and lower values indicate more left-wing content.SERPs were ranked relative to each other (1-6) such that higher values indicate more right-wing content, and lower values indicate more left-wing content.Cronbach's α = .88for for search terms and .94for SERPs.