The limits of policy labs: characteristics, opportunities and constraints

Abstract Policy labs are promoted as providing supportive structures and processes for innovation. Their contributions to policy advisory systems are seen as residing in developing creative policy solutions “outside” traditional bureaucratic structures, and in providing experimental sites for solving problems. This paper examines the characteristics of policy labs in terms of their organizational forms, size, focus and the methods that they employ. It then analyses the opportunities and constraints that labs have in relation to policy design. Labs can be government-controlled, government-enabled, government-led or independently run. They are typically small and tend to be short-lived. Labs often focus on “design” methods. Their autonomy and close connection to citizens and communities provide opportunities, and design-led approaches are helpful in reframing policy problems and finding a broader set of potential solutions. While a key strength is flexibility, labs are comparatively easy to shut down, defund, or ignore, and their survival depends on political patronage. Labs also face constraints in terms of operational capacity and their favored (design) methods, which clash with standard policy processes and bureaucratic structures. Policy labs certainly provide capabilities for improving the design of public policies. However, labs reside in broader policy advisory systems and alone, they cannot provide the solution to all policy design challenges.


Introduction
Over the last few decades, many policy labs have been created by governments, universities, not-for-profit and for-profit organizations. These aim to address challenges that appear to be intractable in the face of the routine ways of approaching societal problems. They are promoted as a means for answering the need for innovation in the public sector, where stability, bureaucracy, and accountability are paramountfor good reason. The resulting growth in all kinds of labs as a global trend has created interest in testing different methods for designing policy. This trend has led to a number of questions about the characteristics of these labs, and what opportunities and constraints they face.
With the incredible rise and rise of policy labs around the globe, an important starting question is: Why have labs become so popular? The pressing problems that the world and many governments face have concentrated attention on the need for innovation and new forms of organization to deal with multiple and challenging issues. Fashion and mimicking, as well as the diffusion of ideas through networks of experts, are also related to the steep increase in the number of policy design labs (including public sector innovation labs, government labs, policy labs, innovation units and teams, and a range of other terms)be they inside government, outside but steered and funded by government, or wholly independent entities. There is a broad consensus that new approaches and organizational structures are required in the public sector, to address large and pressing societal problems. There is also a broad consensus that one mode of achieving this novelty is through the establishment of labs. As Kieboom (2014) observed, to "lab" complex issues is a recent trend in addressing global challenges. Labs hold out the promise of being more open and agile than large bureaucracies.
Given the rise of labs, and a widespread acceptance of their ability to tackle policy problems, relatively little is known about them and their place in policy advisory systems. This paper asks: What characterizes these labs, in terms of organizational forms, focus and methods? And what opportunities and constraints do labs have in relation to policy design?
To answer these questions, I have used the findings of three interrelated studies of labs, undertaken in the last few years: A desk study of 20 well known labs from around the world (McGann, Blomkamp, and Lewis 2018); A survey of 52 labs in Australia and New Zealand (McGann, ; and In-depth case studies of five labs in Australia and New Zealand (McGann, Wells, and Blomkamp 2019).
These empirical studies provide a broad overview of the international landscape of labs, and also more detailed case studies from two nations, which shed light on how these labs are structured and what they focus on. Recent surveys of labs in other countries (e.g. CPIPE's 2018 survey in Canada) indicate that the Australian and New Zealand labs are typical of those in other liberal democratic nations, in their characteristics.

The characteristics of labs
What organizational forms do labs take? Policy labs come in many different forms and have varying relationships with governments. In our study of 20 well-known and long-standing labs from around the world, we found a variety of levels of government control over their funding, and different amounts of direct government oversight . Following on from that study, our survey of 52 labs in Australia and New Zealand (McGann, Lewis, and Blomkamp 2018) uncovered four main types: 1. Government-controlled units that are based within or owned by a government department(s) or public sector agency and wholly funded by government 2. Government-led units that are based within or owned by a government department(s) or public sector agency but only partly funded by government 3. Government-enabled units that are based within or operate as a non-government organization but rely to a significant extent on government funding (mainly through contracts) 4. Independently-run units that are based within the private or third sector and which receive no government funding (these types of innovation units are most analogous to think tanks that seek to influence public sector innovation and reform through independent research, advocacy, and the promotion of ideas).
Most of those in our survey could be classified as either government-controlled or government-enabled labs. While some of these labs were quite large, about half of those we surveyed employed five staff or less, and one-quarter employed two or fewer people (McGann, . They were also relatively "young" organizations, with more than half of those surveyed having been established within the previous two years. Like other small organizations that work closely with government, labs have an emphasis on organizational autonomy and their capacity to provide expertise and legitimacy to the public sector. Most work across government agencies and departments, traverse multiple policy sectors, are rarely subject to specific performance measures or strenuous evaluations, and operate with high levels of autonomy (Williamson 2015b;Tõnurist, Kattel, and Lember 2017). These characteristics have led them to be described as new boundary-crossing organizational forms (Williamson 2015a), designed to overcome barriers that make innovation and cross-cutting coordination difficult within public sector bureaucracies. These barriers include the policy sector specific nature of the public service, both administratively and horizontally, and the structure of public bureaucracies which fosters risk aversion and resistance to change. Hence, labs' contributions to policy systems lie in their capacity to develop creative policy solutions with a flexibility that traditional public sector bureaucracies do not have. They are experimental sites for solving social and public problems in their organizational forms, but also in their approaches and methods (Lewis, McGann, and Blomkamp 2020).
What is a typical size of labs?
The diversity of size in labs is enormous. In our survey, the smallest had two staff and the largest had more than 80 in multiple offices (McGann, . Almost a quarter of the PSI units based within government (6 out of 26) had been in existence for 12 months or less, and three quarters (20 out of the 26) had been established within the past 3 years. By comparison, the non-government or independent PSI units tended to have a more established organizational history with almost two thirds (15 out of 23) indicating that they had been operating in their present form for three years or more. Further, we found no clear relationship between the size of labs and their longevity.
In addition, it is characteristic of many labs that people might be seconded into them for short periods of time to "lab" ideas on particular issues. Lab staff might also be seconded out of them to work in other organizations on issues. And interns and contracted consultants are sporadically employed in labs to work on particular projects, particularly in independent labs. In the 6 months prior to our survey, governmentbased PSI units each hired under four consultants or contractors on average while units based outside government (independent PSI units) hired over six consultants or contractors on average to carry out work on their behalf. More consultants/contractors than employees carried out work for independent PSI units in the six months prior to the survey (McGann, .
The picture that emerged from our survey of labs was that there is no typical or optimal size for a lab. Similarly, there is no typical or optimal location in regard to being situated either within or outside of larger public sector organizations, or whether located in the public, private or community sector. But many labs did report having a small core group of staff, supplemented by short-term contractual staffing arrangements. The key attribute seems to be more about flexibility rather than size, although in comparison with government departments even the largest of these labs are indeed small.

What do they focus on and what kinds of methods do they use?
The 52 labs in our survey of Australia and New Zealand undertook work across a number of policy sectors, although some sectors were more heavily represented than others. More than half of them worked in the area of "Social issues, housing and welfare." "Public administration and governance," "Education," "Health," and "Indigenous and Maori issues" are other prominent sectors that they worked on. More than a quarter work on "Transport" and around one-quarter undertake worked on "Policing, crime, and the justice system." We also discovered three distinct domains of innovation that labs were involved in: Policy development and reform; Evaluation and systems improvement; and User and customer-experience. Many labs undertook work in all three of these domains of innovation, although activities in User and customerexperience, and Policy development and reformthe "front end" of policy designwere their predominant focus (McGann, . In our study of 20 prominent labs (McGann, Blomkamp, and Lewis 2018), we found that an array of approaches and methods were being used. However, "design thinking" was clearly favored (being reportedly used in around half of the labs), as was a focus on identifying and testing proposals and solutions as a way of contributing to the policy process. These findings were replicated when we surveyed Australian and New Zealand labs (McGann, . Behavioral insights teams (BITs) were not included in our study of prominent labs, because we only included labs that were mentioned in two or more previous inventories of labs. Some of our survey respondents in Australia and New Zealand were BITs or focused on behavioral insights in their approaches. BITs often constitute a separate and specific (non-design) method group that has also proliferated rapidly around the world. But a number of the labs we studied reported using behavioral insights in combination with other approaches.
A high proportion of the labs we surveyed reported that they employed people with formal qualifications in design disciplines such as "graphic design" and "service or user experience design." Almost 70% of the independent labs reported that some, many or almost all their staff had prior experience working in a design agency. This compared with only eight (out of 22) government-based labs that reported having recruited some or many staff from design agencies. Government-based labs/units appeared to be heavily reliant on recruiting people already within the public sector.
The methods that PSI labs in Australia and New Zealand reportedly use converged around three main frameworks: A Human-Centered Design framework: using interviews and/or empathy conversations; focus groups; ethnographic methods; citizen/stakeholder engagement through workshops, walkthroughs, and other collaborative approaches; user testing/prototyping; and systems thinking or mapping; An Evidence-based framework: using RCTs (randomised control trials); behavioral insights; Survey research; Research/evidence reviews; and Analysis of existing (big) data sets; and An Agile methods framework: using design sprints; agile or lean project management; and challenge prizes, awards, and open innovation programs.
Of these three, Human-Centered Design (HCD) was the methodological framework most frequently employed by those surveyed (McGann, . It is worth noting that what this means in practice is likely to be highly elastic, as HCD has become extremely fashionable with governments. Large consulting firms have established (and sometimes bought in) units that have HCD as a central part of their approach, in order to win tenders with government departments. This trend has led some of the smaller and more community and place-based focused firms to question whether they are really using HCD or simply paying lip-service to it because it is fashionable. It has also generated some of the frustration expressed by labs working with government departments that want "co-design" without understanding the length and depth of engagement it requires (McGann, Wells, and Blomkamp 2019).

Opportunities and constraints
Having provided some background on the characteristics of labs, I now turn to an examination of the opportunities and constraints that labs have in relation to policy design.

Opportunities
A range of opportunities are afforded to labs by their form, size, approach and methods. Those who work in these labs see them as distinct organizational forms that have an outsider position, nontraditional structures, and more fluid ways of working (McGann, Wells, and Blomkamp 2019). Their small size means they have a degree of agility that allows them to act as change agents. This small size also means that they tend to work without the high level of oversight and accountability requirements that public sector organizations generally face, which allows them to take more risks and be more agile.
Our survey of Australian and New Zealand labs lends support to the notion that labs and their use of design approaches is providing some novel methods for governments to address challenges, particularly through the use of (re)framing problems and incorporating more voices into the construction of both problems and potential solutions. This is related to "design thinking" using a form of reasoning that moves beyond the analysis and problem solving often associated with the policy process to create the end value desired. It can be likened to an analysis where complex situations are distilled through a process of insightful invention, discovery and disclosure (Dorst 2011). This orientation implies that policy making should be guided by the values of "empathy" and "curiosity," along with "rationality" (Torjman 2012) and a focus on "crafting new solutions with people, not just for them" (Carstensen and Bason 2012, 6).
Policy labs can help drive a more participatory and design-oriented approach to public service innovation (Lewis, McGann, and Blomkamp 2020). This appears to be a major strength of labs and one that holds promise for improving policy design. However, these collaboratively proposed ideas still need to be diffused into the larger policymaking process and "sold" to decision-makers, within a broader institutional and cultural context that shapes the best fit for a policy purpose. This can only occur when the policy advisory system is supportive, and where labs are explicitly recognized as a means for empowering citizens in driving public and policy innovation.
The independent labs we surveyed reported engaging with citizens and community interest groups on a very frequent basis (McGann, . When the benefits of involving citizens throughout all stages of the design process are recognized, this supports a strong role for labs. Non-government labs tend to see themselves as providing an alternative to consulting firms in providing new approaches to problem solving that is centered on the lived experiences of service users. They are uniquely positioned to provide a conduit to government for those who would not otherwise have a voice in the decisions that directly impact them (McGann, Wells, and Blomkamp 2019).

Constraints
Although labs do present opportunities for policy design, they also face challenges in speaking truth to power. In particular, there are few concrete examples that demonstrate how design thinking's methods can be standardized and scaled up (Lewis, McGann, and Blomkamp 2020). Moreover, the development and dissemination of design capabilities both within and by labs remains a real challenge for the public sector. These new logics and practices require significant cultural change and capacity building if they are to be embedded in policy making.
As has already been noted, the methods that labs use include new ways of working with citizens, using ethnographic and interview research that is then taken back to the commissioning government agency (McGann, Wells, and Blomkamp 2019). The prototyping and iterating that should follow on from this, as an integral part of a design approach, would take this beyond the more usual and formal paper-based reports. But in practice this seldom happens, begging the question of how much added value there is from these methods, when they are forced to conform to more traditional public sector bureaucratic processes and compressed government timescales. This also raises concerns that false promises are being made to the citizens and stakeholders participating in design research and other activities led by labs, if their contributions are not actually leading to changes within government policies and programs. Governments might use citizen participation in unhelpful and non-genuine waysparticularly as a means of education (and manipulation) or involvement (and placation) rather than genuine participation (Damgaard and Lewis 2014). In such cases, the application of design principles by labs will not always be about gaining meaningful input.
The 52 labs in our survey identified a number of operational challenges. These included a lack of operational capacity within the organization, risk aversion within the organization, funding constraints, a lack of commitment to driving innovation or change from senior internal/external decision makers, a lack of capability and skill-sets within the organization, and the difficulty of attracting high quality staff. Governmentbased labs also identified the challenge of securing endorsement from the unit or team's host organization's leadership as an important constraint.
Another constraint is related to their methods of choice. Labs use design approaches as a means to move the focus toward creativity. It follows that the evidence for action that is generated by design's reasoning styles differs fundamentally from the ideal evidence base required for policy development (O'Rafferty, de Eyto, and Lewis 2016) in more traditional policy design. Labs face challenges in working with government departments because of this tension (Lewis, McGann, and Blomkamp 2020). It is also true that policy proposals generated by these or any other methods are set within a particular context and set of institutional constraints and opportunities and must provide a reasonable "fit" if they are to be advanced.
To date, there is little evidence that design methods are being standardized and scaled up to an entire policy sector, or government, over long periods of time. Design thinking (and the work of labs) may be treated simply as the latest fashion for generating policy relevant knowledge and increasing the pool of ideas available to decisionmakers (Lewis, McGann, and Blomkamp 2020). And as was noted earlier, claims made about using co-design and HCD might simply be a tactical re-labelling of a process that is not so very different to alternative ways of reaching solutions. A more critical reading of this situation would extend this to the possibility that design approaches and labs, when they are the methods and organizations hosting the design of policy, are more important for signaling a government's innovation credentials than for doing anything novel.
An in-depth study of five labs in Australia and New Zealand, based on interviews with staff working in them, found that they face a loss of political support via ministerial changes and departmental staff turnover. They also report a resistance to new ways of working from mid-level civil servants, and an important tension between the resources needed to (genuinely) use design approaches to generate solutions and the political pressures to deliver these quickly (McGann, Wells, and Blomkamp 2019). They described how the high turnover in both political and administrative positions promoted short term actions and created an environment where staff were impatient for change and keen for (fast) innovations that ministers can announce. They also reported that public bureaucracies have powerful traditional administrative traditions that stop new ways of working, particularly via the "permafrost" of middle managers who are able to block change.
Finally, the uniqueness of labs in their organizational form makes them comparatively easy to shut down (for internal labs), defund, or ignore (for external labs) compared with large and established public sector organizations. Their survival depends on ongoing political patronage, as was illustrated by the closure of the longstanding Helsinki Design Lab in Finland, and MindLab in Denmark. They need the support of politicians and high-level civil servants to act as champions if they are to survive, and their smallness (limited authority, few staff, low budgets, minimal oversight) makes them an easy target for closure (Tõnurist, Kattel, and Lember 2017). Big labs, on the other hand, run up against existing structures and standards and central control. Whether they are large or small, or internal or external to government, they do appear to struggle to survive long term.

Conclusion
Labs clearly face a set of constraints, despite the potential opportunities that they afford for policy design. Many of these relate to the broader context they are located within. Beyond the "labification" of public problems which is the signature of labs, it is useful to consider the notion of innovation bureaucracies (Kattel, Drechsler, and Karo 2019). These authors argue that public sector innovation is stuck between the need for change and the need for stability. Public bureaucracies must somehow succeed at the balancing act of unleashing innovations, while also maintaining socio-political stability. The answer, they claim, is for states to support "innovation bureaucracies"constellations of public organizations that are capable of delivering agile stability. If governments create new organizations (like labs) which are led by charismatic outsiders, but these people and their networks do not become part of the routine of government, innovation in the public sector will not be sustained (Kattel, Drechsler, and Karo 2019). This systemlevel approach is useful in taking the conversation beyond the idea that organizations such as labs are capable of providing the single solution to policy design challenges. It serves as another reminder that labs are one piece of the larger puzzle of any policy advisory system, and as such, they alone cannot be expected to change whole systems.
As this examination of labs has shown, these small and agile units with a new (design) toolkit bring some promising creative methods to policy design challenges which are complementary to other (rational and participatory) policy design approaches (Lewis, McGann, and Blomkamp 2020). But labs are unlikely to be capable of acting as a panacea for the difficult terrain of coming up with and implementing solutions in a publicly funded and publicly accountable space. As other system-level studies of public sector innovation have argued, what is needed is the tilting of whole systems toward new ways of working that will make them more open to innovation (Lundvall 1992). This and other studies indicate the added value of thinking about innovation in terms of networks or systems and beyond single organizations, in order to build robust systems that can create and sustain public sector innovation. One study which compared three European city governments, demonstrated the importance of each of governance structures, networks, and leadership styles in supporting innovation capacity (Lewis et al. 2017).
In summary, while labs are currently in vogue and do have an important part to play in the generation of innovative solutions to problems, they alone will never be able to solve every societal challenge. Labs do offer an alternative and complementary form of organization, and they have the ability to bring different people into the discussion, who would not necessarily be involved in policy making. At the very least, labs appear capable of adding some agility and new perspectives and ideas to the designing of public policy. But their impact on public advisory systems, and any larger scale impact on policy design, is much less certain, and clearly related to the contours of the specific context that they are located within.