The challenges of monitoring national climate policy: learning lessons from the EU

ABSTRACT One of the most central and novel features of the new climate governance architecture emerging from the 2015 Paris Agreement is the transparency framework committing countries to provide, inter alia, regular progress reports on national pledges to address climate change. Many countries will rely on public policies to turn their pledges into action. This article focuses on the EU’s experience with monitoring national climate policies in order to understand the challenges that are likely to arise as the Paris Agreement is implemented around the world. To do so, the research employs – for the first time – comparative empirical data submitted by states to the EU’s monitoring system. Our findings reveal how the EU’s predominantly technical interpretation of four international reporting quality criteria – an approach borrowed from reporting on GHG fluxes – has constrained knowledge production and stymied debate on the performance of individual climate policies. Key obstacles to more in-depth reporting include not only political concerns over reporting burdens and costs, but also struggles over who determines the nature of climate policy monitoring, the perceived usefulness of reporting information, and the political control that policy knowledge inevitably generates. Given the post-Paris drive to achieve greater transparency, the EU’s experience offers a sobering reminder of the political and technical challenges associated with climate policy monitoring, challenges that are likely to bedevil the Paris Agreement for decades to come. Policy relevance The 2009 Copenhagen summit ushered in a more bottom-up system of international climate governance. Such systems typically depend on strong monitoring approaches to assess past performance and estimate future national contributions over time. This article shows why decision makers at multiple governance levels should pay serious attention to empirical data on the experiences and challenges that have emerged around monitoring in the EU, a self-proclaimed climate leader. The analysis highlights key political and administrative challenges that policy makers will likely encounter in implementing climate policy monitoring and ensuring transparency in the spirit of the Paris Agreement.


Introduction
The 2015 Paris Agreement (UNFCCC, 2015) marked a shift towards a more bottom-up form of climate governance (Jordan et al., 2015), through which countries make emission reduction pledges known as Nationally Determined Contributions (NDCs). A key component of the Agreement is a 'transparency framework' (Article 13) to ensure tracking of the measures that countries undertake to honour their pledges. As national public policies will be important in implementing NDCs, the demand for knowledge about past and anticipated climate policy effects is growing fast. This requires monitoring, which is often understood as the routine collection of data on specific indicators of policy performance (Dahler-Larsen, 2011, p. 65;OECD, 2002).
Few scholars have analysed existing climate policy monitoring systems to understand the political and technical challenges that have emerged and are likely to persist in the future. Therefore, this article explores the challenges that the Paris Agreement's transparency framework is likely to encounter. It does so by drawing on a detailed and original empirical analysis of the EU's existing, but little studied, system for monitoring national climate policies. The EU is worthy of deeper analysis because its system is perhaps the most mature in situ example of the bottom-up monitoring of national climate policies and measures in existence. 1 While the EU exhibits some features of a state-like system of supranational governance, its climate policy monitoring system is similar to the country-by-country reporting provided for in the Paris Agreement. These similarities make the EU an apt (but by no means unproblematic) source of lessons for the evolving international regime.
The remainder of this article unfolds as follows. The second section describes the emergence and key elements of the Monitoring Mechanism. The EU has repeatedly upgraded its monitoring system, which emerged in 1993 and at first focused on monitoring GHGs (Haigh, 1996). A focus on policies and measureson which this article centresonly emerged in the late 2000s. 2 Our analysis focuses on the challenges that have emerged as the EU has tried to construct a monitoring system that generates complete, consistent, comparable, and transparent data on climate policies. These reporting quality criteria 3 derive from international negotiations 4 and Intergovernmental Panel on Climate Change (IPCC) guidance (1996). They are also directly informed by academic thinking on comparative policy analysis (e.g. Purdon, 2015), policy monitoring and evaluation (e.g. Schwartz & Mayne, 2005), and transparency (e.g. Gupta, 2010).
The third section explores how far these four criteria are reflected in the EU's Monitoring Mechanism. Although the mechanism regularly produces data, to our knowledge no longitudinal academic analyses of its outputs on policy monitoring have been produced. In our analysis, we therefore draw on, for the first time, the first three available quantitative data sets (from 2009, 2011, and 2013), 5 as well as qualitative reporting documents submitted by each country. We also consulted a range of published documents and other primary sources related to the Monitoring Mechanism, including minutes of committee meetings, as well as debates in the European Parliament. And we conducted interviews with five European Environment Agency (EEA) staff, who have been closely involved in the design and operation of the Mechanism. The fourth section presents our discussion and conclusions.

The EU's Monitoring Mechanism
The roots of EU climate policy monitoring lie in the UNFCCC. Following an EU proposal, 'regular reporting … [became] the backbone of the climate change regime' (Yamin & Depledge, 2004, p. 327). At the time, negotiating parties agreed to focus on ex ante (i.e. forward-looking) estimates of policy effects as the 'main goal should be to help parties come into compliance with the Convention rather than to adjudicate blame or impose sanctions' (Bodansky, 1993, p. 451).
As a signatory of the UNFCCC, the EU duly created an EU Monitoring Mechanism in 1993 in order to compile annual data on GHG emissions. Much later (in the late 2000s), the Mechanism also started politically more sensitive monitoring of individual policies (Haigh, 1996;Hildén, Jordan, & Rayner, 2014;Hyvarinen, 1999). The system is, like the UNFCCC, essentially bottom-up in the sense that states collect data on policy effects and then submit them to the European Commission. The 1993 Council Decision established a technical committee consisting of representatives of Member States to assist the Commission (renamed the 'Climate Change Committee' [CCC] in 2004) in developing the implementation of the climate policies, including monitoring. However, there are some hierarchical elements: the European Commission can legally enforce the submission of reports. 6 Indeed, in 2007 and, the European Court of Justice found Luxembourg to be in breach of its reporting obligations (Farmer, 2012). 7 In practice, the EEA runs the Monitoring Mechanism. The EEA's role is potentially sensitive as some in the EU wish to see it focus only on data collection, whereas others emphasize the importance of it taking on more analytical and evaluative roles (see Martens, 2010). EEA staff stressed that by engaging in policy monitoring activities, the Agency gains political capital to carry out more sensitive evaluations. 8 Several studies have noted problems with data quality earlier (European Commission, 2003;Farmer, 2012;Gupta & Ringius, 2001;Wettestad, 2000). 9 However, to gain a deeper understanding of monitoring challenges, one should also look into how these inescapably political and technical aspects interact.

The challenges of monitoring climate policies in practice
This section explores how the EU has interpreted each of the aforementioned quality criteria, how its interpretation has affected data outputs, and which alternatives to the status quo have (not) been considered.

Completeness
The Monitoring Mechanism's Quality Assurance and Quality Control procedure (EEA, 2013, p. 9), as informed by the IPCC (1996), states that: Completeness means that information is reported for all national policies implemented in response to [EU] policies. For each specific policy, all mandatory information should be reported as specified in the […] reporting guidelines, including information on the affected gases, sectors and instrument types. Completeness also means full geographic coverage of policies and measures […].
At least in principle, the Monitoring Mechanism should generate both ex ante (forward-looking predictions) and ex post (backward-looking) monitoring data. However, while EU law requires quantitative ex ante estimates, ex post quantitative estimates are only recommended as per a UNFCCC precedent (see Bodansky, 1993;Hildén et al., 2014). An EEA report found that the 2011 submissions were relatively complete (ETC/ACC, 2012). However, the interpretation of this criterion does not encourage Member States to submit ex post data. As Hildén et al. (2014, p. 898) note, '[…] less than 10% of the entries in the 2011 reporting cycle included [ex post] quantitative data […]'. Our own, much fuller, analysis suggests that this proportion has hardly increased; ex post estimates only grew slightly from 0% (2009) to 5% (2011), and 5% (2013). 10 By contrast, the share of quantitative ex ante estimates grew over time: Member States reported ex ante estimates on average on 43% of the policies reported in the Monitoring Mechanism in 2009, 53% in 2011, and 65% in 2013. According to the EEA, it is likely that ex post estimations will remain rare. 11 Member States have often fiercely resisted efforts to insert additional reporting requirements into the Monitoring Mechanism. For example, the Commission was forced to accept a softening of its proposals to reform the 2004 Regulation governing the Monitoring Mechanism (e.g. information is only to be provided 'where available ' -Hildén et al., 2014, p. 889). Our interviews revealed that the perceived 'administrative burden' of monitoring is one important argument behind Member States' reluctance. According to the EEA, Member States were particularly concerned about reporting costs 12 and insisted that the revised Monitoring Mechanism Regulation should not lead to additional reporting burdens. 13 In fact, the latest regulation (revised in 2013) now specifically addresses this in preamble 12/13. But is such a strong emphasis on costs justified? The Commission's impact assessment of the new regulation estimated that administrative costs were around one million Euros p.a., with 300,000 Euros in additional one-off set-up costs (European Commission, 2011, p. 46). This estimate does not include the actual costs of producing and submitting the monitoring data, which are borne by the Member States. While Member States do not reveal the full costs of policy reporting, costs alone cannot fully explain the resistance to ex post monitoring.
The EU's Monitoring Mechanism thus illustrates that there are many obstacles to completeness. Tensions over cost and political control have prevented the EU from moving beyond the UNFCCC's relatively minimal requirements. Despite pressure from the Commission and the European Parliament, broader interpretations, not only starting from a stronger focus on ex post data, but also considering other metrics, such as the share of national emissions accounted for by monitored policies, have so far not been incorporated into the Monitoring Mechanism. Limited reporting means that the Monitoring Mechanism may be unable to provide the data needed to generate in-depth policy evaluations, but to date there has been no research on the extent to which existing climate policy evaluations (see Huitema et al., 2011) draw on the EEA's data.

Consistency
The Monitoring Mechanism's Quality Analysis and Quality Control procedure (EEA, 2013, p. 14) provides a technical definition of consistency as follows: Consistency means that reported information on policies and measures is internally consistent, including between years. Consistency also relates to the reporting of information for analogous policies within the same sector, in different Member States. Estimated emissions savings are consistent if the same methodologies are used for the base and all subsequent years, and if consistent data sets are used to estimate emissions savings from all policies […] The EEA's report details inconsistencies in the 2011 data from seven Member States, such as reporting 'zero' for both no emissions reductions and missing information (ETC/ACC, 2012). In order to shed further light on what a broader interpretation of consistency might mean for reporting, we compared the number of policies reported across the reporting cycles (see Figure 1). Given significant experience with reporting on policies, one might expect to find a commensurate increase and convergence in the number of policies. Instead, Figure 1 reveals considerable variation, with little convergence over the three reporting cycles. The total number of policies reported rose from 1274 (2009)  The Monitoring Mechanism also misses other important policy dimensions associated with consistency. For example, research has shown that technological innovation requires policy consistency (e.g. Blyth et al., 2007). Yet without ex post evaluation, it is difficult to know if policies genuinely deliver. A much broader interpretation of consistency could conceivably examine how particular policies contribute to global targets, as highlighted in global emissions gap analyses (UNEP, 2014) and tracker exercises (Climate Analytics, ECOFYS, New Climate Institute, & PIK, 2015).

Comparability
The Monitoring Mechanism's Quality Assurance and Quality Control procedure (EEA, 2013, p. 13) defines comparability to mean that 'style and format of information reported on policies and measures is comparable among Member States […]'. Data presented in the previous two sections indicate high levels of heterogeneity, which restricts comparability. An additional way to understand implications for comparability is to consider reporting information on effects across sectors (such as agriculture and energy). As Figure 2 shows, there is considerable variation in reporting practice, with some Member States on average assigning nearly twice as many sectors per policy than others. However, the data do not indicate in which sectors emissions reductions occurthat is, only the 'primary' one of a policy or across multiple sectors? This omission severely hampers comparability.
There are also additional differences in the interpretation of what to report that impede comparability. For example, the report on the 2011 submissions (ETC/ACC, 2012) highlighted diverging methodologies for ex ante and ex post emissions estimates. But challenges may be even more fundamental: because the Monitoring Mechanism does not specify what a 'measure' is, Member States may have variously interpreted it as a single policy instrument (be that a regulation, tax, or subsidy) or a package of instruments. The fact that the number of policies reported (see Figure 1) and the sectors assigned to them vary significantly ( Figure 2) suggests that this may be inhibiting comparability.
Another issue arises over the geographical focus of policy monitoring. Consider, for example, the well-known distinction between consumption and territorial emissions (e.g. Davis & Caldeira, 2010). Currently, reporting on policies follows the logic of the emission inventory and provides only a territorial perspective (i.e. emissions generated within the borders of a country). This has limited the uptake of consumption-based monitoring, which is politically and technically more demanding as it must combine comparable data across countries (both within and outside the EU). EEA staff stressed that new formal requirements to include these (or indeed other additional metrics) in the Monitoring Mechanism are unlikely as Member States generally oppose additional 'reporting burdens'.
In principle, policies could also be evaluated in relation to their effect on carbon prices, energy prices/taxes, or overall spending (Aldy & Pizer, 2015). In some cases, actors have pushed for alternative options; for example, the Bush administration attempted to introduce (production-based) emission intensity indicators, because these do not constrain emissions-driven economic growth (Aldy & Pizer, 2015). In short, comparing policies via indicators is certainly not a 'neutral' activity because indicators always embody underlying values and (hence political) choices (see Gudmundsson, 2003;Lehtonen, 2015).
The most common reasons for adopting a narrow framing of comparability relate to cost, the perceived usefulness of additional information, and ongoing resistance from Member States against providing more comparative information on policies, lest this afford EU institutions too much control. 15 Instructively, recent attempts by the Commission to revise the Monitoring Mechanism led to contentious debates on alternative indicators. The Commission's proposal appeared on the CCC's agenda four times, 16 which according to our interviews 17 indicates it was politically contentious. 18 Our own analysis of the committee's minutes suggests that Member States fought to the very last meeting to remove some of the newly introduced climate-related financial reporting from the implementing legislation 19 and to delay reporting deadlines. During the final debate 20 in the European Parliament, the Parliament's Rapporteur, Bas Eickhout, identified Germany and the UK as the main obstructers. Former climate Commissioner Connie Hedegaard lamented that the frequency of reporting was lower than she had hoped, but many Members of the European Parliament stressed the importance of minimizing additional administrative burdens. Thus, agreeing on a new monitoring system to improve comparability was anything but straightforward.

Transparency
In the context of the Quality Assurance and Quality Control Procedure (EEA, 2013, p. 8), transparency mainly refers to the source of monitoring information: Transparency means that key information on policies and measures should be clearly identifiable. The assumptions and methods used to estimate the emissions savings from policies should be clearly explained. The sources of all data used, including any parameters reported, should be clear to users. […] In order to shed further light on how the EU has interpreted this, we used the number of references to other sources in qualitative and quantitative national submissions as a proxy measure of transparency, especially because many reports are too brief to assess methods and assumptions in depth. As Figure 3 details, the majority of Member States provide a relatively low number of references (1)(2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20) in their submissions. 22 There are, however, a few notable outliers. Finland, for example, provided no references whatsoever across three reporting cycles. By contrast, Germany provided a consistently large number of references. A little over a third of Member States provided reference lists in their qualitative submissions, 23 with Germany again being a prominent example. By contrast, the UK and Spain provided no reference lists at all. Thus, Figure 3 reveals that on this particular measure, countries interpret transparency differently.
In addition to these essentially technical issues, the EEA has also endeavoured to increase general reporting transparency through improving the usability and accessibility of the data. 24 This has mainly been done through improving the EEA's database on mitigation policies. 25 However, retrieving Member State submissions from the public European Environment Information and Observation Network (EIONET) website 26 is still challenging, requiring a detailed knowledge of specific reporting obligations. The EEA's repository lists some submissions more than once, or in different versions, and some are not available or locked for countries that have chosen not to make their reports publically available. 27 Finally, transparency also implies that someone actually uses these data. The EEA staff stressed that they perceive policy makers to be the main users of the Monitoring Mechanism, 28 but they lamented low interest among academics. 29

Discussion and conclusions
One of the key features of the Paris Agreement is the new transparency mechanism to track progress every five years (Article 13). Because many countries will rely on public policies to deliver on their NDCs, reporting will need to include information on policy performance. This study draws on the EU's long and, for the most part, unexplored experience with climate policy monitoring to gauge what challenges lie ahead for effective monitoring in the UNFCCC after Paris.
What lessons emerge from the EU's experience? The first thing to note is that the Monitoring Mechanism has evolved in a situation of relatively high institutional trust and political buy-in; that is, the EU is a system where some aspects of reporting can be legally enforced and hence is a 'more likely' case of sophisticated policy monitoring. But even in this relatively propitious context, the Commission and the EEA have had to operate under considerable political and technical constraints. The strong imbalance between ex ante projections and ex post reporting is especially instructive. This imbalance is in line with issues in other 'policy tracking' initiatives (Fransen & Cronin, 2013). It strongly suggests that the persistent challenges in the EU are likely to emergeperhaps even in an exacerbated formin other fora, notably in the UNFCCC, where levels of trust and cooperation are much lower and countries retain greater control over national climate policy target-setting and monitoring. The EU's experience of governing monitoring suggests that the UNFCCC must likely expend significant and sustained political effort to address these challenges.
Second, our results reveal that there is an important interaction between the EU's internal monitoring activities and international developments. The international influence has contributed to the deeply path-dependent, emissions-centred understanding of EU policy monitoring, which originated from the IPCC's efforts to track GHG fluxes. Our findings suggest that this partly hampers the generation of more comprehensive knowledge on specific policy performance (see also Yamin & Depledge, 2004, Chapter 11), including the difference between territorial and consumption emissions. With time this may, however, change as the Paris Agreement recognizes that 'sustainable lifestyles and sustainable patterns of consumption and production, … play an important role in addressing climate change' (UNFCCC, 2015, p. 20). Agreeing upon new review practices in the Paris transparency mechanism is likely to be very challenging.
Third, it is curious how something seemingly so technical as monitoring appears, with the benefit of hindsight, to have been so deeply contentious. Conflict has flared not only over 'administrative burdens', but also over issues in relation to who should monitor climate policy and the perceived usefulness of different types of monitoring data. There is an ongoing struggle to coordinate national monitoring systems that are at very different stages of development. In the EU, the very name 'Monitoring Mechanism' seemed suitably anodyne a topic that could be pigeonholed in the category of mere technicalities. This may explain why scholars have so far paid little attention to policy monitoring in the EU. While Paris may have established the base for new monitoring institutions and procedures at the international level, the technicalities and politics of monitoring in practice are likely to remain contentious.
Fourth, focusing on emissions implicitly contains a normative vision that absolute emissions reductions within a country matter much more than other policy effects, including effects on, for example, consumption. Our results reveal that an approach borrowed from measuring fluxes of GHGs remains narrow when applied to specific policies. When the main purpose is checking data for a central database, the criteria of complete, consistent, comparable, and transparent information advance limited learning. The resulting information largely fails to reflect complex political choices that arise from systems of sovereign (i.e. state) decision making. The EU's interpretation thus misses an opportunity to engage in climate leadership.
Our findings suggest that the technical focus, coupled with the fact that the Mechanism functions in an essentially bottom-up manner, has led individual Member States to interpret reporting requirements very differently. The result has been, despite the criteria, highly heterogeneous data that are difficult to compare across countries and over time without substantial additional information. If replicated at the wider, international level (and in related fields, such as private climate governancesee Widerberg & Stripple, 2016), such features of the data will make it difficult to gauge the size of national contributions to limit climate change in accordance with the Paris Agreement, let alone judge the relative importance of policy versus non-policy effects. One way to address this might be for the UNFCCC to reframe monitoring from being a routine collection of information on compliance to become an exercise in generating policy learning and insight.
New research on EU monitoring could support the emerging international climate governance system. First, the qualitative country submissions to the Monitoring Mechanism could be analysed in more detail. An in-depth analysis of the views of Member States involved in preparing the submissions could support identifying incentives for better reporting. This would help identify potential strategies to harmonize submissions and thus increase comparability and transparency. Second, if data quality on climate policies and measures did improve, the Monitoring Mechanism could in turn facilitate deeper evaluation work in order to support conclusions by tracking exercises on why progress is being made (or notsee also Jordan et al., 2015). For example, what are the common characteristics of the most effective policies? Are countries that report more policies better at reducing their emissions and what are the most significant policies in terms of GHG reductions and other effects (see Hildén et al., 2014)? Third, the implementation of the new Monitoring Mechanismstarting with the most recent (2015) datamerits careful analysis, particularly with a view to checking if the quality of reporting has evolved. It would also be useful to study how newer Member States and accession states (which are generally perceived to have much weaker evaluative capacitiessee Furubo, Rist, & Sandahl, 2002) are engaging. Such an analysis might offer some useful insights into understanding the challenges that even poorer countries may encounter after Paris. 30 We conclude that some of the high hopes vested in bottom-up governance may be based on a range of less commonly addressed but potentially problematic assumptions, not least in relation to the capacities and political desire to undertake monitoring and evaluation. Many of the political challenges identified by Thompson (2006) a decade ago appear to persist even in the EUa self-declared climate leader with one of the most advanced systems for climate policy monitoring. These include the ability and willingness of political actors to provide information on policies, which at present even in the EU appears to be severely limited. But other contexts can also provide helpful lessons on monitoring for the UNFCCC, as we have shown by drawing on the US-based discussion on similar issues (e.g. Aldy, 2014;Aldy & Pizer, 2015;Aldy et al., 2016;Feldman & Wilt, 1996). Looking forward, a key question is what governance innovations (Jordan & Huitema, 2014) may move countries to monitor and evaluate their policies more effectively. A starting point may be the growing interest in national climate policy (Jordan et al., 2015), reflecting interest group pressure and a deeper concern with policy effectiveness.
Ultimately, the implementation of the Paris Agreement will hinge on whether political actors can successfully navigate these monitoring challenges. The EU's experience shows that incorporating policies into NDCs should be seen as only one step in a long journey towards better knowledge of climate policies, requiring ongoing political and administrative support, both at the national and the international level.