Combining complexity-framed research methods for social research

ABSTRACT Research methods with roots in complexity science are increasingly popular in social research. However, they are not widespread and have potential to deliver value more fully and consistently to social research and methodology. One reason for this is that methods are often used alone, or only with traditional social research methods. We attempt to support and catalyse the use of complexity-framed methods in combination in social research, by systematically reviewing which methods framed in the language of complexity (not including traditional social research methods) have been combined, how, and why. We do this to make clear the state-of-the-art of combinations and to consider gaps and potential new combinations. We find many examples of different methods used together, with simulation methods well-represented. Most examples appear in recent years despite the methods, and interest in complexity, being around much longer. We identify four types of combination, seven purposes, and consider future directions.


Introduction
Over twenty years have passed since the 'turn to complexity' in social science (Anzola et al., 2017;Barbrook-Johnson et al., 2020;Byrne, 1998;Byrne & Callaghan, 2013;Byrne & Uprichard, 2012;Urry, 2003). Broadly speaking, this 'turn' has involved the application (and development) of the ideas and methods of complexity science in social research. Complexity science revolves around an understanding of the world as made up for 'complex adaptive systems' which are themselves made up of many interacting and adaptive parts. There is no single or agreed upon mode in which this is implemented but it tends to crystalise in studies which adopt holistic approaches and which seek to understand social interactions, dynamics, feedbacks, self-organisation, emergence and uncertainty in the social world. There are disagreements and variety within this field, and definitions of what 'social complexity' is, are not settled. Byrne and Callaghan (2013), Castellani and Gerrits (Forthcoming), and Boulton et al. (2015) provide excellent introductions and overviews of the field.
There have been some great successes in the application of these ideas and methods, however their use and value are not yet fully realised Castellani & Gerrits, Forthcoming). In some cases, early promise and excitement has waned where methods have been perceived as overly technical or formal (Byrne & Callaghan, 2013), where concepts and language have struck researchers, users, and participants as inaccessible and unhelpful, or where they have simply been confused and misused in the literature (Bruijn & Gerrits, 2018;Rosenhead et al., 2019;Teixeira de Melo et al., 2020). Nonetheless, there is an appetite for complexity in social research, and its use and application has recently increased (Andersson and Tornberg, 2018). Despite this history and interest, definitional issues and debates still arise around complexity in social science (Anzola et al., 2017). These are often conceptual, but there is also a lack of consistency and clarity in the range of methods that exist and how they are used together.
In this paper, we systematically review the use of complexity-framed research methods in combination in social research. By 'complexity-framed', we mean methods that are used in conjunction with the language and ideas of complexity and systems sciences, which can thus include both those methods with roots in complexity and systems sciences, and those which incorporate their ideas and philosophy even if they are not from these traditions originally. We do not include traditional social research methods, such as interviews, focus groups, and surveys in our review (i.e. we do not include studies that combine a complexity-framed method with a traditional social research method). We do this primarily because we are interested in the combination of non-traditional social research methods that are framed by the complexity and systems sciences, and how this can help support the development of this field. It also, more pragmatically, narrows our focus to something more feasible; if we were to include studies which combined traditional social research methods with non-traditional methods, and used a complexity framing, the number of studies that met the criteria would number in the thousands. Our focus is very much on specific methods, not on wider methodologies or concepts and theory. Following a clear search strategy, we collected 102 examples of two or more complexity-framed methods used together.
This review aims to support the continued and expanded use of these methods in social research and highlight the strengths and opportunities for their use in combination. For several complexityframed methods to be used in social research as a source of synergy rather than inefficiency, their combined use should be well-guided. Understanding how methods can be used in combination, where they have been used successfully in the past, and where there may be opportunities for novel and valuable combinations will support their improved use. We hope this paper is a useful resource for those hoping to use these methods together in social research. We use our analysis to reflect on the focus of previous work and suggest possible future directions for research in this area. It is not our aim here to define what is, or what is not, a 'complexity method', or what can, or should be, used when adopting a complexity approach. We do not wish to restrict the often creative approach to using and combining methods, under different ontological and epistemological positions. Rather, we hope our review provides further inspiration and context to the free-flowing and creative use of non-traditional social research methods.
Using research methods in combination has a long tradition in the mixed methods (broadly understood as the use of both qualitative and quantitative methods in the same study) and multimethods (broadly understood as the combination of two or more qualitative methods in the same study) literatures (Anguera et al., 2018;Greene, 2008;Tashakkori & Teddlie, 2010); we cannot overview these large literatures here, but combinations have been used to mitigate against the weaknesses, biases, and gaps in any individual method, create new approaches and perspectives to social and policy questions, and to potentially uncover new insights (Greene, 2008). Most research poses multiple research questions; a single method is rarely sufficient or comprehensive. While we do not explore this assumption in detail, we believe that supporting the use of complexity-framed methods in combination will build on the growing contribution of complexity-framed methods to social science methodology.
Our list of 'complexity-framed' research methods used in social research emerged from our search. As detailed more fully below, the first stage in our search strategy was to search various academic databases for combinations of keywords involving 'complexity' and 'methods'. Anything that came up through this process and that was not a traditional social research method (e.g. interviews, survey), we defined for the purposes of this work as a complexity-framed method. Some researcher judgment was required in this process and this is a potential source of bias, which we were careful to manage. The final list of complexity-framed methods that emerged, in alphabetical order, was: agent-based modelling (ABM), analytic hierarchy process (AHP), Bayesian belief networks (BBN), Bayesian updating, cellular automata, cognitive mapping, concept mapping, fuzzy cognitive mapping (FCM), microsimulation, process tracing, qualitative comparative analysis (QCA), social network analysis (SNA), and system dynamics.
There are existing relevant reviews and overviews of suites of methods and substantive domains, which partially overlap with our aims (in their coverage of methods, rather than focusing on methods in combination). However, previous work tends to either focus on describing a list of individual methods, each in turn (not used in combination), or on a subset or one type of method framed in a different way (e.g. simulation modelling or participatory methods). Relevant reviews of methods include those on participatory modelling (Voinov et al., 2018), complexity science (Mitleton-Kelly et al., 2018), hybrid modelling (Brailsford et al., 2019), modelling and simulation (Badham, 2010), mixed methods (Pluye & Hong, 2014), mental models (Moon et al., 2019), policy evaluation methods (Bicket et al., 2020;Walton, 2014), meta-analysis and systematic reviews (Anderson et al., 2013;Lorenc et al., 2016).
The rest of the paper is structured as follows. First, we outline our methodology, including our search strategy and inclusion criteria. Second, we present the results, including an exploration of which methods are used together and how they are used. Finally, we conclude and suggest possible future research directions.

Methodology
To identify and assess the range of complexity-framed methods used in combination, a literature review of existing studies was undertaken. The review was aimed at finding out which methods are combined, when they are combined, in what domains they have been combined, and how and why they have been combined. The review was inspired by systematic review techniques. Drawing on guidance for systematic reviews (Centre for Reviews and Dissemination, 2009;Petticrew & Roberts, 2006), pre-defined protocols for searching, appraising and including studies were used to facilitate a transparent and comprehensive search and review and reduce the potential for views of the researchers to be reflected in the selection (Stewart et al., 2007).

Search strategy
We searched six databases: ISI Web of Knowledge, Scopus, Science Direct, International Bibliography of the Social Sciences, and Applied Social Sciences, and Google Scholar. Each database was searched with a combination of pre-defined terms; (multi-method OR mixed methods OR methods) AND (complexity OR complex). We used databases' in-built criteria or filters for returning only social science studies (for Web of Knowledge this meant using the 'Web of Science Categories' we believed included social science subjects broadly defined, for Scopus and Science Direct, this meant using their 'subject area' named 'Social sciences', for the International Bibliography of the Social Sciences, and Applied Social Sciences no filter was required, for Google Scholar there are no such filters). The search terms were included in the topic, keyword, title and abstract sections of each individual database search and refined after several trial searches to optimise sensitivity and specificity of the returns.
The search was limited to studies published including and after 1990, to reduce the volume of results to be screened. This date reflects our understanding of when ideas and methods of complexity science began to be used in social science. Where possible, we also used filters to only return results from peer-reviewed outputs (i.e. we did not include monographs or non-peer reviewed results).

Criteria analysis: selection of studies
Inspired by current guidance for systematic reviews, the eligibility of studies for inclusion was framed by the research questions adapted from the PICOS approach to define the Population, Intervention, Comparator, Outcome, Study Design (Centre for Reviews and Dissemination, 2009). This resulted in the following eligibility criteria: • Population: ○ Studies using two or more methods which could not be interpreted as traditional social research methods (e.g. interviews, focus groups, surveys) ○ Studies that use complexity science in a recognisable and meaningful way, or use methods generally understood to have roots in complexity and systems science ○ Studies within social sciences ○ Studies in peer-reviewed outputs ○ Studies written in English • Intervention: N/a • Comparator: N/a • Outcomes: ○ Methods used ○ Year of publication ○ Journal published in ○ Type and purpose of combination • Study design: ○ Studies that describe use of two or more methods in sufficient detail to facilitate characterisation (i.e. position papers and proposals without worked examples were not included) The studies were screened in a two-stage process. First, titles and abstracts were screened. Second, the full texts (of those passing the first stage) were screened. After the search of the six databases using the search terms, 415 studies were returned. After the first stage of eligibility screening, 134 studies were considered eligible, in accordance with criteria set out above. Of the studies that passed the first-stage screening, 78 studies were considered eligible after the second stage of screening. Within these papers we found the 13 methods detailed in the results section.

Secondary search
To gather as many examples as possible of the methods identified from the first search, we next searched for each possible pairing of these methods using the same databases as before, using the method names as search terms. We searched again from 1990, using topics, titles, abstracts, and keywords. This search added 23 papers. As a final check to highlight any omissions or bias, we explained our aims and asked academics and practitioners in the UK Centre for the Evaluation of Complexity Across the Nexus (CECAN) and via Twitter for any examples of research using complexity-framed methods in combination. This process did not identify any significant omissions, and returned only two additional studies, one of which met our inclusion criteria. This brought the total of papers in our review to 102; a full list is provided in Appendix 1.

Limitations
One potential limitation of our approach was for researcher error and/or bias to create gaps in our search. To reduce bias, we did not pre-define a list of complexity-framed methods, but allowed this to emerge from our first keyword search. To reduce researcher error, we used two researchers to conduct the search equally. We did not both cover all the databases, but included some overlap and regular checked each other's work.
The limitation of databases may also have created gaps in our searches. The databases we used bias towards papers in English, and towards journals with more formal academic standing, both of which give a strong Anglo-Western-centric focus. There are strong traditions of complexity and systems science outside this focus, whether they use the language of complexity or not, and we were not able to include these.
A third possible limitation is that our search strategy had an implicit bias towards methods being used in combination right from the start (owing to our use of terms, 'mixed methods', 'multi method', and 'methods'). We chose this approach for three reasons. First, early exploration and testing of keywords suggested this approach returned many examples of methods being used on their own. Second, focusing on use in combination fitted our research objectives. Third, more pragmatically, it returned a more manageable number of results. While we may have missed some methods, we hold reasonable confidence this potential bias is not systematic or significant, because we would have expect to have become aware of it through speaking to other academics and practitioners, reading the related reviews we cite above, and exploring the turn to complexity in the social sciences we describe above.
Finally, it is possible that our search strategy did not catch combinations of methods where authors did not explicitly name one or both of the methods they were using in the title, keywords, topic, or abstract of the paper published. There may be examples where one or both of the methods are not mentioned except in the main body of text, however searching main text returned an impossibly large number of results to screen.

Which methods emerged as 'complexity-framed'?
The list of complexity-framed methods that emerged is summarised in Table 1.
This list includes all of the methods we might expect to see based on recent work applying complexity in social science (e.g. Byrne & Callaghan, 2013;CECAN, 2018), but there were no obvious omissions; however, we found two things surprising. First, there are only a few methods which we might class, or include within the rapidly maturing field of data science (e.g. SNA, BBN). Some machine learning techniques or a larger number of statistical approaches may be expected owing to the increased focus on them in recent years. We do not believe this omission is due to bias in our search strategy, but rather in the way these methods are framed in the literature, typically without using complexity or systems ideas to describe their value. Second, most of these methods have been around at least 20 years. Possibly, this period has provided the methods time to be combined, but as we show below, almost all of the combinations were published in the last 10 years. Why these combinations of older methods have only happened in the last 10 years, when the interest in applying complexity in social research has been present from at least the 1990s, is unclear.

Publications included
In total, we found and included 102 publications, published from 1994 to 2020. Figure 1 shows the count of publications each year, a clear trend can be seen of few papers up until 2009 then an increase towards the present. The count is lower for 2020 because searching was conducted in early 2020. This indicates that combining complexity-framed methods appears to have begun in earnest in the last 10 years, with a lag between the rise of complexity in social science and combining complexity-framed methods.
We also classified the domains of each returned study's journal (see Appendix 1). This process showed 'interdisciplinary environment and sustainability' (23 appearances) and 'computer science' (21 appearances) as the two most common domains of journals. Environment and sustainability studies have often understood the interaction of social and ecological systems through a complexity  lens for a long time, via concepts such as 'socio-ecological systems'. They perhaps are more in need of study using combinations of complexity-framed methods, or simply they have researchers and a research history which favour their combination. Computer science is well-represented likely because of its central role in complexity science, and in the application of methods to applied problems. Many of the computer science papers in the search clearly had a strong focus on the methodological development, as opposed to substantive topic contributions. Engineering (10), topic-specific journals, i.e. focused on applied topics such as cities, safety, and disasters (9), and interdisciplinary social science (7) were the next common domains.

Which methods are combined with which?
Table 2 lists methods in alphabetical order, how many times they appeared in our search, and which methods they are most often combined with. Only five studies involved three methods, and none involved more than three. Where three methods were used together, ABM was involved in four, and a simulation method (ABM, cellular automata, microsimulation, or systems dynamics) was involved in all. Simulation methods are wellrepresented throughout all combinations and tended to be used together with: a different type of simulation (i.e. micro and macro forms of simulation); a data analysis approach (i.e. to analyse simulation outputs); or a static modelling approach (i.e. to inform simulation design). Combinations of two static modelling methods, or data analysis methods, or combinations of a static modelling method and data analysis method, are present, but less common. This suggests that the combination of a simulation approach with another method is often an obvious and fruitful combination; however, combinations of the other types could be being overlooked. Figure 2 provides a visual overview of the combinations of methods found as a network. The network shows the strong representation of simulation methods (orange). There are clear relationships are between ABM and SNA, as well as between AHP and both ABM and system dynamics. The network is laid out using a standard force-directed layout (i.e. nodes with connections to each other are drawn closer together, and those without are moved further away). This algorithm for the layout mostly holds the methods in groups of the same type, the simulation and static modelling approaches are all close. Obvious exceptions are data analysis approaches, split by different uses between AHP and SNA (both used relatively highly) and QCA and process tracing (used less).
It is also worth noting the network is relatively dense (maximum density would be shown if all methods were attached to all other methods); there are many connections and most methods have at least three connections. This reflects the wide range of combinations being made, not the same few combinations, repeatedly. There are no isolated pairs of methods (i.e. not connected with the rest of the network), meaning (in theory) any set of methods could be used together meaningfully (based on existing examples). The longest direct path (i.e. following chains of connections between nodes in the network) between any two methods is also reasonably short at three steps (this is also known as the 'network diameter'). There are three steps between multiple methods and process tracing (e.g. process tracing has been used with QCA, QCA has been used ABM, and ABM has been used with microsimulation). Therefore, although any combination could be made using past examples, they may require the combination of three methods.

How and why are methods combined?
In an iterative process, we coded the examples we found into different types of combinations. We describe first the 'how' of the combination, the way methods are combined, which we refer to as 'type'. Second, we describe the 'why' of the combination, the 'purpose' of combining them, i.e. what it achieves. The categories are described in detail in Tables 3 and 4. These purposes are not mutually exclusive (some combinations may serve two or more purposes, or were difficult to class based solely on published descriptions). However, they do capture the nuances and spectrum of different purposes observed, from increasing coverage of a topic, through to informing, evaluating or cross-referencing each other. This 'spectrum' quality is important. The purpose of combinations appeared rather continuous, with different shades merging between them, rather than neatly defined categorical cases with precisely defined boundaries (like the types of combinations). For example, 'system coverage' and 'parameterise' are two purposes often observed in a combined form. One method would be used analyse one aspect of a system, and then used to set parameters in another method (typically a simulation model), which covered a separate aspect of the system. Table 5 presents the prevalence of each type and purpose we found. 'Feeding inputs once' between methods is the most common type of combination (66 examples), followed by 'fully integrated' (26), 'feeding inputs in cycles' (12), and 'no direct connection' (6). This indicates that combinations are typically only one-time or uni-directional; i.e. not interactive or dynamic combinations in most cases. Table 5 shows 'inform design' (35), 'system coverage' (33), and 'parameterise' (23) are the three most common purposes. We found of a pattern of directional relationships between each pair, where one method plays the role of inputting data, which is then used in the second method to select values or inform design. This type of purpose is covered by both 'parameterise' and 'inform design' and can be achieved with feeding outputs just once.
Appendix 2 contains a large table that presents which types and purposes of combination were observed, in which papers, for each pair of methods. We hope this table provides a useful resource to guide deeper exploration of individual examples both in terms of methods used and the type and purpose of the combination. Considering which types and purpose of combinations are used with different methods allows us to see some patterns. Combinations in which the methods are fully integrated for the purpose of system coverage are common with ABM and cellular automata. In these cases, the two methods are combined closely in one model with the cellular automata often representing the environment or a macro-level phenomena and the agents representing individual decision-making. The use of Cognitive mapping appears to be dominated by feeding inputs once to inform design of another method. This makes sense intuitively, the qualitative causal maps of cognitive mapping can be used to underpin more quantitative causal models like system dynamics of BBN, or even ABM. Another potentially interesting pattern is the use of both QCA and SNA, two relatively different methods; they are often either used in 'feed inputs one way for analysis' or 'feed Table 3. Different types of combinations of methods.

Name Description
Purposes observed (see Table 4 for more detail) A) Two methods used but not directly linked Two methods are used. Outputs and results may be combined once produced, but the two methods do not directly connect or make use of the other.
• Data coverage • System coverage • Cross-reference B) Feed inputs, information, or data one-way or once Two methods are used. Outputs of one, whether they be data, information, ideas, or interpretations of results, are used as inputs (typically only once) for the other method.

C) Feed inputs, information or data in a cycle
Two methods are used. Outputs of both, whether they be data, information, ideas, or interpretations of results, are used as inputs for the other method, often in an ongoing iterative cycle.
• System coverage • Parameterise D) Fully combine or integrate methods into one hybrid The ideas and/or approach of two methods are used but are combined closely so as two produce a new hybrid research method.
• System coverage • Parameterise • Inform design data one way to inform design' modes. These are two of the most common type and purpose pairs. Finally, the two most combined methods, ABM and system dynamics, unsurprisingly have several different types and purposes to their combination with other methods but display a notable variety (especially system dynamics) compared to others. This highlights the ways these methods are combined is not settled or always the same.

Conclusion
In this paper, we have sought to support and catalyse the use of complexity-framed research methods in combination in social research, by systematically reviewing existing examples of such methods used together. We have shown which methods emerged as complexity-framed and are used in combination. This should not be interpreted as any claim of what methods should or should not be used when taking a complexity approach. We have considered patterns in publication by time and journal domains; finding most examples are recent despite the methods themselves, and interest in complexity in social sciences, being around much longer. We described which methods are used together, and found though simulation methods dominate, there are many different combinations in the literature. Finally, we explored how and why methods are combined, identifying four types of combination (no direct connection, feed inputs one-way, feed inputs in cycles, and full integration or hybrid) which serve one or more of seven purposes (data coverage, system coverage, parameterise, inform design, analyse, evaluate, and cross-reference). There are some clear patterns in the types and purposes, and the methods used, but there is also variety, particularly with ABM and system dynamics, the two most-used methods in our list. This paper aims to stimulate wider use of complexity-framed methods in social science by reviewing what combinations of these methods have been used previously and identifying gaps that represent potential for future research. The results reveal many examples of combinations, as well as a range of possible combinations which have not been implemented. The fact that two methods have not been combined is not in and of itself an opportunity for innovation, there may be good reasons for them not being combined; however, of the 'gaps' in combinations, the following may be most interesting or innovative: • Agent-based modelling and Process tracing: ABMs have long been used to explore and enrich understandings of causal mechanisms, with the concept of generative sufficiency (i.e. where being able to 'grow' macro-level phenomena with micro-level rules in a model, allows identification of candidate explanations of phenomena; Epstein, 2006) motivating models. More recently, the concept of inverse generative social science (or iGSS) has developed, where instead of researchers designing micro-rules based on theory and plausibility, many microrules are systematically evaluated in their performance in recreating macro-level patterns. In Table 4. Different purposes of combinations of methods.

Name Description 1) Data Coverage
Analyse or explore different types of data using different methods

2) System Coverage
Analyse different parts of the system with different methods, typically modelling or simulation methods (e.g. micro and macro scale, individual behaviour and ecological or technical system) 3) Parameterise Use outputs of one method to calibrate or parameterise the second method. Where this involves changing individual values of parameters we class it as parameterise, where it reflects a more fundamental change in the method/model structure, we refer to this as 'Inform design' as below.

4) Analyse
Analyse the outputs or results of one method using the other 5) Evaluate Using one method to assess or evaluate how 'good' the other modelling approach was 6) Crossreference Cross-reference outputs/results of two methods to find preferred scenarios/results 7) Inform Design Use the outputs or conceptual approach of one method to fundamentally inform the design of the other either role, ABM has a role in providing additional (simulated) evidence to input into process tracing, potentially complementing empirical and expert opinion inputs. How the relative strengths of these different types of evidence are weighed should be considered; is simulated evidence as valuable or reliable as empirical evidence or expert opinion? • Process tracing and any other method: Process tracing, being a qualitative approach to causal inference, has the inherent flexibility to take input from many types of analysis or modelling. Two studies combined process tracing with QCA (see Appendix 2); other methods could be used in a similar way. • Qualitative comparative analysis and Bayesian belief networks: We see potential in using BBN to extend the description of the configurations of factors produced by QCA. These 'recipes' often require much interpretation and exploration. Using BBN to explore these would add another level to the analysis. A key decision will be in how to ascertain the various conditional dependencies required to quantify the BBN. The existing approach of seeking expert opinion or empirical evidence could be used, which could be supplemented by QCA analysis, e.g. using consistency scores calculated for relationships. While exploring new combinations of methods is an obvious direction, innovation is also likely to come from further development of combinations which have been seen. More examples will be helpful to articulate the value of combining methods, to improve practice, and to show others how it can be done. For more well-established combinations it may be worthwhile formalising these relationships and developing tools to support their combination, e.g. ABM and systems dynamics, or AHP and simulation methods. This could be done by reviewing individual combinations more closely, developing frameworks and guidance for different types of combination of them, and developing software tools to facilitate their use together. However, there is a balance in making methods too easy to combine, so inappropriate connections or analyses might be made. Inappropriate use may be more likely for combined methods as good knowledge in two methods is required, rather than one. Thus, any guidance or tools developed will need to support researchers in not making inappropriate use of combinations.
Only five studies combined three methods, and none used more than three. These combinations deserve further attention to explore whether this low number is representative of meaningful barriers, or pragmatic considerations of time and team expertise.
Finally, as with many research methods, there is a lot of tacit knowledge, or at least underdiscussed knowledge, which is used to make complexity-framed research methods and their use a success. Future work articulating the processes and knowledge needed to combine methods will be valuable. There may be barriers to sharing this knowledge; for individual methods and modelling approaches it can be problematic owing to academic publication criteria and research career incentives. This may be more the case for (potentially niche) combinations. Nonetheless, developing, describing, and sharing this type of semi-tacit knowledge is essential to making the methods we have discussed more usable and valuable, in combination or alone.