The social life of measurement: how methods have shaped the idea of culture in urban regeneration

ABSTRACT Although ‘culture-led regeneration’ has been critiqued as both a concept and practice, it is clear that policy-makers continue to make efforts to use cultural activity of varying forms to achieve ends which could be (and are) described in terms of urban ‘regeneration’. Whilst the idea of culture-led urban regeneration had gained considerable prominence in a range of policy by the early twenty-first century, many questions have remained over how exactly such ‘regenerative’ outcomes could be convincingly demonstrated, despite much activity to attempt such demonstration over the course of preceding years. The desire for convincing evidence can be seen in a continued, and increasing, focus on evaluation, and methods aimed at providing evidence of impact and outcomes. In light of the renewed political focus in recent years on ‘proving’ the effects and value of cultural activity, this paper considers the continuation of practice in this area, and asks what lessons, if any, have been learned in evaluative practice which seeks to demonstrate the regenerative effects of culture. In light of the continuation of apparently problematic practices, the paper seeks to delineate and account for what has been learned, and what has not.


Introduction
Understanding the relationship between social scientific methods and the social world has become a significant component of contemporary research work. Whilst the exact theoretical framework for such study has differing iterations (including performativity, the social life of methods, studies of calculation and valuation, or classificatory struggles), it is clear that an essential part of understanding the contemporary world is to understand the role of social science methods in the construction, existence and replication of that world. Indeed, Journal of Cultural Economy has been the forum for the discussion and development of this agenda, whether in a conceptual (Miller 2008;Butler 2010;Callon 2010;Helgesson & Kjellberg 2013;Law & Ruppert 2013) or applied (e.g. Marres & Weltevrede 2013;O'Brien 2016;Yarrow & Kranke 2016) form. In this context, this paper extends the social life of methods agenda to culture-led urban regeneration, a practice that has reshaped the urban environment across many major towns and cities in the UK, along with urban sites across the rest of the world (most notably in Europe, North America, and South East Asia).
Using a review of the evidence base for the impact of culture-led urban regeneration policy, this paper argues that the methods of evidence gathering, more so than any of the social scientific results of this evidence gathering, have been crucial in the transmission of models of urban regeneration. In particular these models have spread across differing and distinct urban settings that would otherwise have little to render them comparable, and thus appropriate subjects for any form of policy transfer. As the era of formal urban regeneration policy comes to an end (O'Brien & Matthews 2015) the paper demonstrates the contingency of the idea of culture in urban regeneration, a contingency grounded in the social life of methods sustaining that idea. The paper develops a new approach to an otherwise crowded field of study (see Oakley & O'Brien 2016 for a recent summary of the extensive literature) by reframing the questions associated with culture-led urban regeneration around the role methods have played in the creation and continuance of that social and policy phenomena. In taking seriously the social life of methods for culture-led urban regeneration, academic engagements critical of the use of culture in urban settings will find another front for resistance (Lees et al. 2007;O'Brien & Matthews 2015).

Methods and their social life
This paper begins from the perspective that social science research methods are not separate from the social world, but rather they are a constitutive part of that world. This social life of methods perspective develops the well-established concern with reflexivity within the contemporary social sciences. The social life of methods is summarised and elaborated by Law et al. (2011). They use the example of surveys of attitudes across differing European nations to show the interrelationship between methods and the social world. Methods are used within a context that is closely determined by the findings of the methods themselves. In the case of European attitudinal surveys, the idea of Europe itself is produced in close relationship to the survey method and helps to set the circumstances for the activity of the survey. Thus there is a triple movement of survey practitioners and administrators, descriptions of social reality, and the institutional and organisational settings acting on and through the research.
One useful example here is contemporary class analysis, whether in the form of the Great British Class Survey (Savage et al. 2015) or classificatory struggles (Tyler 2015). Class in contemporary Britain is a reflexive set of categories, whereby a social scientific framework used to understand occupational relationships with labour markets (Goldthorpe & McKnight 2004) is also the site for powerful position taking for individual's descriptions of their identities (Savage 2000;Savage et al. 2015), and for revanchist exercises of class power and politics (Tyler 2015). Class is, therefore, as much a technical social scientific tool as it is a lived reality of individuals and communities. Most crucially, it is difficult to separate the social science of occupations, labour relations, life chances, and politics, from the use of class language within individual identity. A second example moves towards the case study of culture-led urban regeneration. O'Brien (2013) has shown how the cultural 'omnivore', itself a product of specific choices in survey instruments, has a powerful place within public policy for culture in the UK, representing the 'ideal-type' consumer of state-sponsored culture. Those who do not fit this pattern are then sites for potential policy interventions to bring their consumption patterns closer to the surveyresearch produced omnivore (Miles & Sullivan 2012). This approach has also prompted an engagement with a specific culture-led urban regeneration policy: Liverpool's 2008 European Capital of Culture. Here the findings of economic impact and community involvement, which for Cox and O'Brien (2012) were specific to a single time and place, were placed at the centre of future, general, public policy. Both methods, and their findings, in this work have a social life. It is here where a more detailed engagement with culture-led urban regeneration is most important.
Historical 'evidence' for the regenerative role of culture There have been attempts to gather evidence on the socio-economic benefits of artistic and cultural activity for many years, at least from the 1970s, if not earlier (cf. Reeves 2002, p. 7;Gray 2007, p. 206;O'Brien 2013, p. 91). Clearly, however, the work of Myerscough (1988) in the late 1980s proved influential, for some serving to rectify an 'almost scandalous lack of empirical research into the operation of the arts sector' prior to this point (Hughes 1989, p. 34). Moving through the 1990s, Gray (2007) identifies a move to focus on a range of instrumental outcomes for cultural activity, and as the 1990s ended, the New Labour administration in the UK epitomised governmental belief in the idea of 'culture-led' urban regeneration, with an intensifying rhetoric around 'evidence-based policy' (Oakley 2008) and an emphasis on gathering 'data' of some form to support the arguments around culture's potentially regenerative role. Nevertheless, much research (e.g. Reeves 2002, p. 17;García 2004, p. 321;Oakley 2004, p. 72;Selwood 2006, p. 45;Gray 2006, p. 111;Lees & Melhuish 2015, p. 253) repeatedly notes a lack of robust evidence as we move into the first decade of the twenty-first century.
The persistence of a pattern?
Given a consistent identification of a lack of robust evidence from a range of research practice up to 2006, but a persistent use of social science (or 'social science-like') methods to prove the case for culture, in 2014 a review was conducted specifically on recent research practice used to assess the role that culture plays in urban regeneration as part of the Arts and Humanities Research Council's 'Cultural Value' project. Material was gathered in relation to cultural policy interventions that were explicitly described as 'regenerative', or which belonged to a 'type' of intervention that is routinely described as 'regenerative', by searching academic sources and grey literature, and through consultation with the cultural sector. The review focussed mainly on data from the UK, but also drew on practice further afield, including significant European cultural regeneration programmes. In total, 151 relevant sources from the period 2006-2014, covering reports from cultural organisations, commissioned research, policy-makers, local government, academics, and media sources, were identified and analysed. 1 Interestingly, Reeves' findings regarding the limitations of evidence produced by methods utilised in the 1990s to demonstrate the impacts of cultural practice (2002, p. 102; echoed by Evans & Shaw 2004), clearly persist in the evidence produced in more recent years. Indeed, in the last decade, nearidentical critiques have consistently been made regarding, for example, the prevalence of ad hoc evaluations which cannot be drawn together to build a substantial evidence base (Ennis & Douglass 2011, p. 10), and evaluation which veers more towards advocacy and which thus cannot entertain the possibility of negative impacts (Belfiore 2006, p. 32). Ennis and Douglass (2011, p. 9) concluded that the highest quality evidence available does not allow the question of whether culture-led urban regeneration is successful to be answered.
It may well be asked, therefore, why after having been identified as problematic for so long, these issues persist. On a very basic sociological level, Schutz (1960, p. 214) argues that 'human activities are only made understandable by showing their in-order-to or because motives'. In this situation, it could be argued that evidence is gathered in similar ways over time, despite the problems noted above, in order to support the narrative around the role culture can play in urban settings, because this fits into a broader discourse on the nature of culture and economy (cf. Campbell 2014), and because a suite of research approaches have become entrenched in this field, structuring it into specific forms. The discussion below attempts to clarify the persistence of the range of motives and constraining circumstances shaped by this broader field, based on a consideration of the material outlined above, and to demonstrate why the evidence base pertaining to the role of culture in urban regeneration has thus become 'stuck' in a particular paradigm, and thus to reveal how and why the methods for evidencing the impact of culture-led urban regeneration have a specific and particular social life.

Persistent explanations
A number of persistent explanatory factors can be suggested to account for the steadfast nature of critiques of evidence-gathering practice.
Short-term data cannot establish long-term outcomes Methods are often employed on behalf of those organising a cultural event to enable impact to be demonstrated to funders and other stakeholders. This is usually a short-term necessity, and so research evidence is most often generated alongside specific programmes or events, and in close proximity to their operation. Once data has been generated, 'evidence' has done its job (for certain actors, at least). There is thus a mismatch between the timescale of methods used and outcomes sought. With reference to a range of developments around new cultural venues over the course of a decade, Hyslop argues that 'in most cases it is too early to judge the success of the long-term impact of these projects on the economic and social development of their communities' (2012, p. 153, emphasis added; see also Barnardo's 2005, p. 16;Ela Palmer Heritage 2008, p. 30;CABE 2008, p. 5). If evidence gathering is repeatedly linked only to time-limited interventions, evidence of long-term impact will, by definition, not be found.
Despite longstanding identification of this problem (Reeves 2002, p. 104;Evans & Shaw 2004, pp. 8, 57), however, it is notable that long-term study remains rare. García and Cox (2013, p. 19), for instance, find 'little to no research' into the long-term effects of the entire European Capital of Culture programme and, in relation to the limited amount of such research that is available, find that 'claims and evidence about medium to long-term effect varies considerably' (p. 24). Relatedly, the 2013 Centre for Economics and Business Research (CEBR) report to Arts Council England and the National Museum Directors' Council (2013, p. 88) finds that many studies do not even seek to establish a baseline from which future impact can be measured, or report findings only relating to a short period of time.
Whilst such short-termism can clearly lead to the lack of robustness in findings, it can also potentially serve a purpose by enabling more optimistic findings than may result from a long-term approach. Sharp (2007, p. 286), for instance, discusses how enthusiasm around the potential impact of cultural projects can initially be very high, but also outlines how this can easily change dramatically; a change uncaptured by short-term studies. Similarly, García and Cox (2013, p. 133) demonstrate that developments in physical infrastructure can be well-used during a major cultural programme, and that tourism levels can be boosted in the short-term, but that these changesif measuredcan prove unsustainable. This potential complexity is 'helpfully' reduced by brief studies which focus on initial 'good news', and inappropriately inflate the role that culture plays in broader regeneration. Once this evidence of 'success' has been achieved, often there is no political impetus to consider whether effects over the longer-term could or should be examined. Evans (2011, p. 6) succinctly describes how this kind of practice will not enable us to adequately contextualise the size and nature of the role that cultural programmes may appropriately play in urban settings: The culture and regeneration story requires a historical analysis that also maps change and effects over a much longer time period, within which events form only a relatively small (financial and strategic) part. Investment in housing, retail, transport, education and local amenities are likely to have a more lasting legacy and impact. This will also be important in order to consider how culture might better contribute to the regeneration process, as opposed to simply being corralled into a 'festival event' or 'year'.
There is little evidence, however, of the adoption of such analysis. Also, whilst there is much that is specific about regeneration programmes using culture, it should be noted that this evidence deficit applies much more broadly. Ennis and Douglass pointedly state the continuing necessity for longerterm study, and the production of longitudinal data, without which 'it may not be possible to determine conclusively whether culture-led regeneration, or indeed any regeneration, works' (2011, p. 2, emphasis added). This form of study, however, may be increasingly unlikely in the context of a 'fast policy' world, in which 'rapid results', and positive results at that, are required (Van Heur 2010, p. 190).
Culture may be a low priority, or have limited resources A further reason for an absence of longitudinal methods is likely to be the expense of carrying out such research. Belfiore (2002, p. 98) rightly notes the complexity and expense of such assessment (see also Evans & Shaw 2004, p. 58;Ennis & Douglass 2011, p. 10), which is likely to be beyond the capabilities of all but the very largest institutions (Ela Palmer Heritage 2008, p. 30). Evans and Shaw's conclusions from 2004 (p. 21) specifically regarding economic impact thus still seem to apply a decade on, and will likely persist: The level of primary survey research required to measure economic and distributive effects outside of the cultural project itself (and even here, distributive data is hard to capture) is felt to be prohibitive and hard to justify, unless motivated by a funding or other imperativelongitudinal studies of effects even more so.
To adequately back up the claims made for culture takes financial resources which, increasingly, the cultural sector does not have (Lees & Melhuish 2015, p. 256). Besides limited financial resources, we can also consider how limitations in terms of other forms of resource affect evidence-gathering practices. Generating evidence of regeneration is an inherently difficult task, but it is also only one task amongst many for those working in the cultural field (Hesmondhalgh et al. 2015, for example, describe a proliferation of performance management indicators over recent years). For many cultural organisations, this kind of activity is also likely to be significantly outside the skills and capacities of the governance and administration of the organisation. The kinds of evidence sought regarding 'regeneration' can also often relate to outcomes that are not necessary a fundamental part of core operational or charitable objectives; evidence which can be used for advocacy may seem worth generating, evidence which cannot may seem to be more limited in its 'use' to many organisations. Relatedly, it is instructive to consider the following point from one of the reports reviewed regarding the competing pressures which apply not just to cultural organisations, but to potential external partners in research: The evaluation methodology was predicated on involving local authority data collection officers in gathering the quantitative data. Despite extensive efforts it proved impossible to ensure the involvement of the officers in gathering data to demonstrate the role of culture within regeneration. If Arts Council England is to pursue an emphasis on 'hard evidence' it may wish to consider making funding conditional on the involvement of such officers. (General Public Agency 2008, pp. 8-9) The resources which evaluation can draw on to establish a firm case may, therefore, continue to be limited in a number of senses. Vickery (2007, p. 21) notes in the context of local authority practice how culture is a relatively low priority and, whilst it is certainly true that many recent regeneration initiatives have found a role for arts and culture as part of wider investment programmes, one key continuity in recent practice is the somewhat marginal position of cultural activity in such programmes.
Vickery (2007) also notes that up until the 1990s, urban regeneration was largely led by property development (p. 29), and that during the 1990s, the role of culture in urban regeneration was still often linked strictly to physical or spatial development, focussed mainly either on altering the design of the urban environment, or on installing public artworks (pp. 18-19). Recently, initiatives which can be identified as positing a role for culture in processes of regeneration include the Single Regeneration Budget, which helped, for instance, to fund public art (Public Art Leicester 2005; Gateshead Council 2006), community music projects (Dhamak Beats 2012) and artists' studios (ACAVA 2014); the European Regional Development Fund, which has invested in the renovation and construction of new cultural facilities (European Commission 2013), business assistance for cultural enterprises (DCLG 2013) and a centre for carnival arts (UKCCA 2010); the New Deal for Communities, which, in certain areas, used arts and culture-related activities to attempt to tackle mental health inequalities (Blank et al. 2004); and the Housing Market Renewal scheme, which helped fund artists' studios (NFASP 2010), public art (Pendle Borough Council 2014) and artist-led participatory projects (Media and Arts Partnership 2008; Arts Council England 2009). Whilst such initiatives fund cultural projects, and so propose that these may contribute in some way to wider regeneration goals, the peripheral position of this activity proves problematic in terms of dealing with the persistent evidence gap outlined above, as there remains typically little or no discussion of the role of culture in relation to overall findings regarding regeneration (e.g. DCLG 2007; DCLG 2009; DCLG 2010; Audit Commission 2011; Leather et al. 2012). So long as culture remains a low priority in urban policy broadly speaking, we can therefore expect the evidence base available to reflect this with the 'social life' of the methods shaped by the level of importance given to such activity. It should also be noted that, even when an explicitly regenerative programme's focus is entirely cultural such as in the case of the European Capital of Culture programmethat the scale of investment is often still small in comparison with other activity seeking urban regeneration outcomes (Cox & O'Brien 2012, p. 97).
A lack of clarity in the terms of research The circumstances delineated above may thus help us to understand the persistence of a lack of clarity about what research methods are actually attempting to achieve, and therefore how these achievements could be appropriately understood. Often the starting point of research is to position cultural activity as a known benefit with known effects, already revealed by previous evaluation practice, which can thus be continued.
If, for instance, cultural activity aims to promote 'creative industries', what is the mechanism by which this will occur? Often, it seems as though this question need no longer be asked, even though Knell and Oakley (2007, p. 21) note that whilst discussion of creative industries has become prominent, funding of arts activity has often carried on in much the same manner as it did prior to this prominence, with links being assumed rather than made explicit. Correlation, of course, need not imply causation, and simply monitoring and reporting what happens at the same time as a cultural event does not necessarily provide us with evidence for what this event has itself achieved. By what means would creative industries be promoted, if indeed they would?
On this subject, specifically in relation to the recent years of the European Capital of Culture programme, García and Cox (2013, p. 142) point out a continued articulation of an assumed relationship between this programme and the development of creative industries, rather than any kind of clear planning to promote such a relationship. Again, when evaluating the achievements of culture, a search for 'good news' rather than an attempt to establish solid evidence seems to characterise practice. Without the articulation of a mechanism by which culture has a role to play, evidence gathering can involve a 'sweeping up' of any available information that can add weight to the argument for the beneficial role of culture, veering away from research and towards advocacy. This leaves us in a position where a role for culture is asserted (e.g. New Economy 2013, p. 27), suspected (THRU 2013, p. 259), or where research proves to be merely descriptive rather than analytical (Markusen & Gadwa 2010, p. 382). By contrast, where a relationship seems to be established in a more concrete fashionfor instance, between the concentration of cultural institutions and house pricesthe direction and nature of this relationship is not necessarily clear (CEBR 2013, p. 113).
Without clarity on how, and by what means, culture is expected to contribute to regeneration, most evidence produced will be of little relevance beyond its own specific context. In recent years, Evans (2011, p. 13) has astutely tackled this issue: Whilst there is no shortage of 'evidence', techniques and methods, how these relateif at allto the governance and regeneration regime, and where power over which and whose culture is 'invited to the festival' resides, is not apparent or at least not part of the evaluation or impact study process. The extent to which this accumulating evidence on the wider effects of hosting and delivering such mega-events has and may be used in the future to inform future events and both cultural and regeneration strategies, is at best marginal.
Nevertheless, such accumulation continues to occur.

Measurement is not always feasible
Clearly, many of the effects that cultural activity can have are difficult, if not impossible, to capture. This does not mean, however, that there is not room for improvement in practice in this area. In a review of 'evaluations by English RDAs […] that deal with culture and regeneration', Ennis and Douglass (2011, p. 8) find that whilst evidence which is related to economic outcomes, such as number of 'safeguarded jobs', can be firmly established, other areas are left to 'qualitative judgment'. They continue, echoing the points above, These qualitative assessments are in some cases little more than hopeful statements, lacking the specific evidence that has long been needed. For example, countless evaluations point to improved perceptions of a neighbourhood, but do not demonstrate how this connects to the long-term goals of regeneration. Similarly, projects are said to improve confidence amongst residents, based on small surveys taken shortly after the completion of the project. Since evaluations are generally carried out shortly after completion, there are none showing the long-term impact that really matters most for regeneration.
Labadi also notes (2008, p. 107) that qualitative data on the impact of regeneration schemes can be tainted by 'optimism bias', referring to one study where, the six people who agreed to be interviewed and had overall positive opinion of the scheme were all deeply involved in the regeneration and therefore could not really have the necessary critical detachment.
Again, the methods adopted to demonstrate regeneration impacts may veer too close to advocacy for cultural organisations for it to give an accurate picture of what has been achieved, seeking primarily to show positive impact rather than ascertain if it exists (e.g. General Public Agency 2008, p. 11;DC Research 2011, p. 2;LARC 2011, p. 9). There are many ways in which such practice could theoretically be changed, but there is limited evidence from this review that these changes have taken place in recent years. For instance, Reeves (2002, p. 104) noted the lack of comparative studies, or the use of control groups to establish the effects of cultural activity over a decade ago. In the course of this research, we did not identify any evidence of such studies having taken place in the more recent past to provide evidence for the value of cultural activity, and broader problems with the evidence base in this area have been noted (Belfiore 2006;What Works Centre 2014). O'Brien (2013, p. 47) when discussing methods to assess the impacts of cultural engagement notes that 'there has been little progress over the past decade of evaluation and research', and Lees and Melhuish (2015, p. 243) note repeated difficulties with evaluating arts projects linked to forms of social regeneration. Current methods of evaluation cannot acknowledge that the arts and culture are, as Gray (drawing on Gallie) puts it, 'essentially contested concepts ' (2008, p. 212), and this may lead to an emphasis on a 'simpler' area in which to identify resultsthe economic.

An over-emphasis on the economic
The drive to demonstrate economic impact continues to shape research methods. The necessity of some economic component to urban regenerationas well as a broader research agenda which emphasises the necessity of fostering 'economic competitiveness' (Belfiore 2015, p. 97)also helps to explain evidence production skewed towards economic indicators and economic impact. Indeed, whilst non-economic impacts may be poorly established due to the difficulties outlined above, they may also be poorly established due to a lack of interest in such issues. The goals of those commissioning and funding cultural activity may not match those of cultural institutions, or indeed those of the wider public (cf. García 2005, p. 846;Holden 2007, p. 32;Shin & Stevens 2013, p. 644). In their discussion of the specific use of heritage-related programmes to achieve regeneration goals, Ela Palmer Heritage note a lack of evidence relating to social impacts: The lack of requirement for social impact evaluation has stemmed from a focus on the economic impacts of regeneration, mainly concentrated on market value, job creation and the desirability of the area for business. (2008, p. 30) This focus primarily on the economic may also be explained by the fact that, once economic matters are dealt with, social impacts continue to be either taken for granted or expected to naturally 'trickledown'. Clearly this is controversial with respect to cultural programmes (Colomb 2011, p. 77, see also Evans & Shaw 2004, p. 58), as well as regeneration programmes more broadly (Jones & Evans 2008, pp. 72-73).
Even if figures relating to direct economic impact can be established, these can be difficult to disentangle from the impacts of other investments and programmes (Cox & O'Brien, 2012). What is perhaps more problematic, however, as O'Brien (2013, p. 12) points out, is that 'our choice of economic valuation methods may then lead us to actually miss the importance of what it is we are attempting to value'. Nevertheless, the prevailing circumstances resulting in a reliance on economic valuation methods continue to intensify. This does not mean that there is not room for improvement in the methods applied in this area, however. In earlier reviews of methods used in this field, Reeves (2002, p. 42) noted the reliance on economic indicators but also the following problems: • a reliance on narrow economic values and economic indicators considered inadequate for measuring 'difficult-to-quantify' outcomes • failure to take account of displacement and leakage of spending from the local economy • failure to distinguish between distributional effects and aggregate income effects of arts spending These criticisms can also be identified long before this list of Reeves (cf. Hughes 1989, p. 38), and continue to shape the research field. In considering a wide range of cultural research, Markusen and Gadwa argue that many studies, … are plagued with unwarranted assumptions and inference problems. These include: 1) not adequately demonstrating that the arts are an export base industry; 2) treating all spending as new spending, as opposed to factoring out expenditures that would otherwise have been made elsewhere in the local economy; 3) not acknowledging that non-profit arts expenditures are directly subsidized by the public sector through both capital and operations support; and, 4) failing to count the foregone tax revenues that non-profit status confers. (2010, pp. 380-381) These points are also echoed in consideration of specific studies (e.g. Labadi 2008, pp. 28-29). Clearer guidance on how to appropriately apply such methods has become available in recent years, such as the English Partnerships 'Additionality Guide' (2008), and guidance is also available in the UK government's 'Green Book' on evaluation regarding leakage, deadweight and displacement (HM Treasury 2003, p. 53), but with the following caveat: In some cases, the best source of information for assessing additionality may be from those who clearly have an interest in the outcome of the decision. In these circumstances, the information and forecasts should be confirmed by an independent source.
In recent cases relating to the cultural field (e.g. GHK 2009, p. 28), it is difficult to ascertain whether this has been attempted, but it seems rather that 'traditional' methods persist which do not heed such advice. Even studies which do take these issues into consideration somewhat can leave questions unanswered. For instance, English Partnerships (2008, p. 14) caution against assuming zero deadweight when assessing interventions, but this assumption can be seen to be made in practice (e.g. Roger Tym and Partners 2011, p. 27). More often, though, it seems that issues such as these are simply not considered at all in extant research (CASE 2011, p. 63). A recent evidence review by the What Works Centre (2014) did not consider any studies attributing economic impact to cultural interventions as a result of visitor numbers, on the basis that the available studies were insufficiently robust, particularly in respect to identifying the net economic benefits of interventions. Continual inappropriate application of methods has led some economic impact figures to be characterised as no more than 'wild guesses' (Labadi 2008, pp. 34-35). If the adoption of more robust methods were to lead to a diminution of economic impact figures, however, then less robust methods may be favoured in order to increase the level of 'good news', or because they are simpler (and cheaper) to apply.
What should also be noted on this point, however, is that even if these factors were considered there is also robust criticism of the use of economic impact studies per se. Rushton (2015) argues that 'there are no insights to be gained, no policy implications' from such studies, which are 'expensive to produce, useless and forgotten as research'. Little, therefore, seems to have changed since Madden considered these techniques in detail in 2001 and found that, despite their proliferation, 'nearly every economist who reviews "economic" impact studies of the arts expresses concern over the technological and practical limitations of the methodologies' (p. 165), agreeing that no policy implications can be drawn from such studies (p. 161), which, indeed, are not designed to serve such a purpose, but which can be used to inappropriately overstate contribution to a local economy (p. 168). When it comes to the matter of what evidence can be used to make the case specifically for the role of cultural activity, the following point of Madden's is especially instructive: By their very nature, 'economic' impact studies are poor at accounting for intangibles. And it is worth pointing out that governments are prone to put intangible considerations before financial considerations. For example, governments often intervene to depress the drugs, prostitution and pornography industries despite their sometimes substantial financial flow-on effects. (2001, p. 170) Given their persistence, however, Madden's view that despite their inappropriateness in many ways, economic impact studies of cultural activity 'can be seen as a perfectly rational response to political demand' (p. 174) is also instructive, a point reinforced by Crompton (2006). As with a range of methods, we may not be led to a clearer understanding, but achieving such an understanding may not be the dominant goal of evaluation practice.
'When the legend becomes fact, print the legend' The body of research regarding cultural regeneration thus seems particularly susceptible to the generation of myths or legends (common in many areas of research, cf. Rekdal 2014), whereby a piece of data, often a figure or set of figures, becomes a 'known' truth, repeated in multiple sources, free of contextual information, and thus becomes a desired point of replication for future study. For instance, as a very brief illustration of the disconnect between evidence and its provenance, Arts Council England (2014, p. 19) note the 'estimated £10 million' contribution to the local economy of the Hepworth Wakefielda figure sourced via Local Government Association's (LGA's) 2013 report. This second report gives limited information regarding the derivation of the figure (p. 6), but no further reference to enable the source of this information to be followed up. Further investigation reveals that this figure seems to have been sourced from the gallery's annual report (Hepworth Wakefield 2012, p. 80), which notes that the figure is based on '511,781 visitors' spending 'an average of £21' each. The caveats above regarding economic calculations clearly apply here, but this would not be clearly known from either of the initial reference points, which are likely to have a higher visibility, helping to perpetuate this figure as a 'known' truth. O'Brien (2013, p. 111) similarly demonstrates how a range of research on Liverpool's tenure as European Capital of Culture is 'debased' into a set of key headline figures, which help to form a 'mythology' around the nature of culture-led regeneration. If such 'mythologies' persist, then they themselves likely help to perpetuate the patterns of research identified above.
Where to next?
What can be seen in the case of evidence for culture-led urban regeneration, then, is the persistence in recent years of the utilisation of methods that continue to evince the problems identified in the field historically, and so a somewhat problematic social life for the methods used to justify cultural interventions in association with regeneration. As the underlying factors shaping evidence gathering persist, so too do the kinds of evidence produced. These methods thus help to constitute an object of study and a field of practice which, having been well established in the early 2000s, and found to persist in the early 2010s, is likely to continue for the foreseeable future, and perhaps intensify in its drive towards quantification. There also seems to be increasing enthusiasm (e.g. Markusen & Gadwa 2010, p. 382; Arts Council England 2014, p. 6) for the use of statistical analysis of secondary data to ameliorate some of the problems with evidence discussed above, perhaps inspired by recent enthusiasm for the concept of 'big data'. There is good reason to exercise caution here. The recent 2014 Arts Council review of evidence of the value of culture (p. 6) points to 'promising work using […] logistic regression techniques', but the work referred to actually enables us to conclude relatively little from its statistical findings. Before methods move further in this direction, it may be wise to note Gorard's warning on the use of statistical analysis more broadly: The danger of spurious findings is a general one, and cannot be overcome by using alternative forms of regression […] In fact, more complex methods can make the situation worse […] Complex statistical methods cannot be used post hoc to overcome design problems or deficiencies in datasets. (2006, pp. 82-83) This approach also raises the question of what can realistically be known with available secondary data. Whilst secondary data can deal with some of the problems around data gathering outlined above, the Culture and Sport Evidence (CASE) report into this very issue (2011) concluded that current data and methods were mainly helpful when considering business and property-related data (p. 71), and that current approaches still often do not effectively consider issues of economic displacement (p. 4), appropriate comparators (p. 17), or direct attribution to specific interventions (p. 80). When data is available to undertake such methods (which is rarely), only relatively large interventions can be adequately assessed. As such, this does not seem like a trajectory for research that will necessarily avoid the difficulties noted above.
In addition, given the conditions outlined above, it seems unlikely that broader factors constraining the production of robust evidence on culture's role in regeneration will disappear. What is perhaps more pressing is whether the very idea of culture-led regeneration is one which has now reached its peak (Evans 2011, p. 15;Lees & Melhuish 2015, p. 243). Nevertheless, the importance of establishing a robust evidence base around these issues continues to have some prominence , and arguments around regenerative potential remain familiar: There are five key ways that arts and culture can boost local economies: attracting visitors; creating jobs and developing skills; attracting and retaining businesses revitalising places; and developing talent […] There is strong evidence that participation in the arts can contribute to community cohesion, reduce social exclusion and isolation, and/or make communities feel safer and stronger. (pp. 7-8) That said, just as the arguments made around the value of culture seem to have an impressive stability, so now do the critiques made of these arguments. Arts Council England's review of evidence gathering since 2010 finds that 'most of the studies reviewed cannot establish causality between arts and culture and the wider societal impacts' (p. 8), again highlighting the need for longitudinal study. Nevertheless, such a document continues some of the circular reasoning which partially accounts for the lack of progress highlighted above. After stating that no causal relationship has been established between cultural activity and broader social impact, we find this statement: We know that arts and culture play an important role in promoting social and economic goals through local regeneration, attracting tourists, the development of talent and innovation, improving health and wellbeing, and delivering essential services. (p. 11) Thus the social life of these methods continues even as it is being critiqued: we cannot establish the broader causalities suggested by policy rhetoric, but we (the Royal policy 'we') know that culture plays a role, or at least believe this. Despite the limitations outlined in this paper, the methods currently in use serve sufficiently to support this belief, and to provide ways in which the belief can be articulated, framed and developed further in future policy interventions. It is, on the whole, unclear what the role of 'better', and particularly of more fine-grained and/or longitudinal data, might be in policy development. What research may usefully now ask, therefore, lest these issues persist, is not what can be achieved by cultural activity, but rather what forms of social science are needed to break the hold of the current social life of methods in culture-led urban regeneration? Note 1. A full list of sources consulted can be found in Campbell et al. (2015).

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
This work was supported by the Arts and Humanities Research Council [grant number AH/L014505/1].

Notes on contributors
Peter Campbell is a lecturer in Sociology and Social Policy at the University of Liverpool. His research to date has focussed upon cultural policy, and on attempts to align culture and creativity with processes of socio-economic urban regeneration. In addition to doctoral research studying political discourse and activity regarding the 'creative industries', and their broader social role, he has worked on evaluation of the European Capital of Culture research programme, and the London 2012 Cultural Olympiad.
Tamsin Cox is Head of Policy and Research at DHA, and an Honorary Research Fellow in the School of Sociology, Social Policy and Criminology at the University of Liverpool. She has undertaken a range of work with funding bodies, local authorities, organisations, networks, and events.
Dave O'Brien is a Chancellor's Fellow in Cultural and Creative Industries at Edinburgh College of Art, University of Edinburgh. He has published extensively on cultural policy, cultural and creative work, cultural value, and culture-led urban regeneration. His most recent books are the edited collections Routledge Companion to Global Cultural Policy, After Urban Regeneration, and the monograph Cultural Policy.