Monitoring and evaluating the payment-for-performance premise of REDD+: the case of India’s ecological fiscal transfers

ABSTRACT Introduction: The central premise underlying international payments for Reducing Emissions from Deforestation and forest Degradation (REDD+) is that offering governments ex post payments for verified success in reducing emissions will motivate them to protect and restore forests. However, the extent to which performance-based payments motivate governments to protect and restore forests has yet to be evaluated quantitatively. Researchers have only quantitatively evaluated performance-based payments to non-governments for forest outcomes (e.g. payments for ecosystem services) and to governments for non-forest outcomes (e.g. results-based aid). Methods: We describe how researchers now have an opportunity to more easily evaluate performance-based payments to governments for forest outcomes thanks to India’s new ecological fiscal transfers (EFTs), which provide $6-12 billion per year to Indian states in proportion to their forest cover. Discussion: India’s EFTs differ from REDD+ programs in that they pay for states’ stock of forest area in the recent past rather than reductions in the rate of forest carbon loss in the near-future. Nevertheless, India’s EFTs focus on a single outcome and have many recipient governments, significant financial scale, lack of contemporaneous confounding policy changes, universal participation, and long-term data collection. Conclusion: These features make India’s EFTs especially useful for testing the payment-for-performance premise of REDD+.


Introduction
The central premise underlying international payments for Reducing Emissions from Deforestation and forest Degradation (REDD+) is that offering governments ex post payments for verified success in reducing emissions will motivate them to increase protection and restoration of forests above baseline levels. If the offer of performance-based payments has this incentivizing effect, then REDD+ payments represent an opportunity to mobilize a high-volume, low-cost source of emission reductions while making efficient use of public or private funds. REDD+ has been discussed and analyzed for more than a decade (e.g., Angelsen et al., 2008;Corbera and Schroeder 2011;Seymour and Busch 2016). It was successfully included in the Paris climate agreement (UNFCCC, 2015). More than $10 billion in finance has been pledged for REDD+, of which nearly half is resultsbased (Norman and Nakhooda, 2014). Multiple finance mechanisms for REDD+ are in operation or in development (e.g., bilateral and trilateral agreements, multilateral facilities such as the Forest Carbon Partnership Facility and the Green Climate Fund, and private voluntary carbon markets outside the auspices of the UNFCCC). But despite these developments, the premise that performance-payments motivate governments to conserve forests has yet to be evaluated quantitatively. Now, as I describe in this essay, a new financial mechanism enacted by India offers the opportunity for researchers to more easily monitor and evaluate the payment-for-performance premise of REDD+. First I discuss what can be learned about REDD+ from monitoring and evaluating performance-based payments in "adjacent" settings: payments to non-governments for forest outcomes and payments to governments for nonforest outcomes. Next I summarize India's new ecological fiscal transfers (EFTs), drawing heavily from a previous paper that describes the EFTs in greater detail (Busch and Mukherjee 2017). Then, I list features that make programs useful for monitoring and evaluating REDD+, and how India's EFT fares on those features relative to other REDD+ programs. Finally, I discuss four aspects of REDD+ that monitoring and evaluation of India's EFTs can help illuminate.
Monitoring and evaluating payments for performance in adjacent settings challenging for a few reasons. For one thing, there are not many such agreements. Currently in operation, there are only seven Norwegian-funded governmentto-government bilateral agreements (Brazil, Guyana, Indonesia, Peru, Liberia, Ecuador, and Colombia) as well as Germany's REDD Early Movers program with Ecuador, the state of Acre in Brazil, and the Amazonian region of Colombia. Multilateral performance-based funding programs for REDD+, e.g., the Carbon Fund of the Forest Carbon Partnership Facility and Results-Based Payments for REDD+ through the Green Climate Fund, have not begun implementation, let alone advanced to a stage where retrospective evaluation is possible. For another thing, these REDD+ programs make the delivery of funding contingent upon multiple elements in addition to forest-related performance. Requirements include upfront investments in monitoring and planning, reporting on social and environmental safeguards, and benefit-sharing provisions. Furthermore, the agreements take place in the context of other potentially confounding policy changes. The same political will that led a forest country to sign a REDD+ agreement may have concurrently motivated other policy changes aimed at reducing deforestation. Brazil is case-in-point; its Amazon Fund was only a small component of a larger policy push to reduce Amazon deforestation (Nepstad et al., 2013).
Because of these challenges, analyses of the impact of REDD+ agreements to date have been limited to qualitative program assessments (e.g., Birdsall and Busch 2014;Birdsall, Savedoff, and Seymour 2014;Laing 2015;Seymour, Birdsall, and Savedoff 2015;Laing 2018). There has yet to be a study rigorously comparing observed forest outcomes to what would have happened in a counterfactual scenario without a REDD+ agreement, let alone a counterfactual scenario in which finance for forest conservation was channeled through traditional ex ante Official Development Assistance.
In the absence of direct evaluations of the effectiveness of performance-based payments to governments for REDD+, suggestive evidence is available from studies of payments in two adjacent settings: payments to non-governments for forest outcomes and payments to governments for non-forest outcomes. The premise that ex post performance payments can motivate non-governments (e.g., households or villages) to undertake forest conservation is the idea underlying payments for ecosystem services (PES) programs in Costa Rica, Mexico, Ecuador, Vietnam, and elsewhere. Evaluations of the effectiveness of PES have been mostly quasi-experimental (e.g., Sanchez-Azofeifa et al. 2007; Honey-Roses, Baylis, and Ramirez 2011) or hypothetical (e.g., Andersson et al. 2018), with the recent exception of Jayachandran et al. (2017) who used randomized controlled trials to test the efficacy of PES in Uganda. Evaluations of PES programs have been the subject of several review studies (Miteva, Pattanayak, and Ferraro 2012;Wunder 2013;Pattanayak et al., 2014;Busch and Ferretti-Gallon 2017), with one review concluding that PES is most effective when targeted to higher threat areas, enrollment is high, recipients are monitored for compliance, and the link between land use and the desired ecosystem service is well established. (Pattanayak, Wunder, and Ferraro 2010). Evaluations of the effectiveness of project-level REDD+ initiatives have been conducted by the Center for International Forestry Research's Global Comparative Study on REDD+ using Before-After-Control-Impact methods (e.g., Bos et al. 2017;Larson et al. 2018). Evaluations of incentives to non-governments for non-forest outcomes (e.g., Gertler 2004;Duflo, Glennerster, and Kremer 2007) are numerous and beyond the scope of this paper to synthesize.
However, even if households or villages might respond to the offer of payments for performance, governments might not. Thus, it is also useful to learn from the experience of payments to governments for non-forest outcomes, i.e., results-based aid (alternatively: cash-on-delivery aid (Birdsall and Savedoff 2010), payment-for-results, or performancebased aid), in which ex ante payments are offered for performance in other sectors, e.g., health, education, and infrastructure. Results-based aid has been theorized to motivate governments not only through their pecuniary interests, but also by enhancing the visibility of results, establishing accountability to constituents, and giving recipients discretion to engage in local problem-solving (Perakis and Savedoff 2015). On the other hand, "a major conclusion from earlier research [on results-based aid] is that aid cannot buy policy reforms" (Angelsen 2013).
Quantitative evaluation of the impacts of resultsbased aid is nascent. Olken, Onishi, and Wong (2014) found that villages in Indonesia that were randomly selected to receive performance incentives for health and education saw improved preventative health outcomes and increased school enrollments relative to villages without performance incentives. Gertler, Giovagnoli, and Martinez (2014) found that Argentina's health program Plan Nacer, which provided funds to provinces based on their enrollment of beneficiaries with performance incentives for better health outcomes, increased the use and quality of prenatal care. More recently, a study of the El Salvador health system that randomized the mode of delivery of financial support for health systems across 98 municipalities found that results-based aid modalities increased efficacy two-and-a-half times relative to conventional aid (Bernal, Celhay, and Martinez 2018).
A new mechanism paying governments for forest outcomes Now, efforts to evaluate the central premise of REDD+that offering performance-based payments to governments can motivate them to protect and restore forests above baseline levelswill benefit from a new policy in India: EFTs for forest cover. As described in greater detail in Busch and Mukherjee (2017), in February 2014, India's fourteenth Finance Commission added forest cover to the multi-element formula that determines how much tax revenue India's central government distributes annually to each of India's 29 states (Government of India 2014). From 2015 through 2019, India's central government will devolve approximately US$6-12 billion per year to states in proportion to their 2013 forest cover. This funding has no conditions besides forest cover and goes into the state's general budget where it can be spent on any purpose. A handful of other countries had previously introduced EFTs to pay for protected areas (including Brazil and Portugal, where such EFTs were found to incentivize municipalities to designate additional protected areas (Droste et al. 2017a(Droste et al. , 2017b), and EFTs for forest outcomes had previously been proposed (e.g., Irawan, Tacconi, and Ring 2014), but India's EFTs are the first for forest outcomes to be enacted.
India's stated motivation for enacting EFTs for forest cover was primarily fiscal and only secondarily environmental (Government of India 2014). The transfers are intended to compensate states for the "fiscal disability" caused by forgone opportunities to convert forests to other uses by rewarding the maintenance of existing forest cover. They are not designed to maximize incentivizes to generate additional forest cover. Even so, the transfers will certainly have some incentive effect on states that is worth evaluating.
India's EFTs are not billed as REDD+, and there are some notable differences between the two. For one thing, the EFTs pay for stocks of forest cover, while REDD+ mechanisms pay for reductions in the rate of forest carbon emissions relative to a baseline reference level. For another, EFTs between 2015 and 2019 are based on the state of forests in the recent past, as monitored in 2013, rather than at the end of a near-future performance as in REDD+. Thus, the EFTs will only give governments an incentive to undertake policy changes if they believe that nearfuture changes in forest stock will be reflected in payments after 2020, i.e., that the fifteenth Finance Commission will include an updated measure of contemporaneous forest cover in the revenue-distribution formula in five years. It is still uncertain that this will happen, though there are several reasons to expect that it might (Busch and Mukherjee 2017). The prospect of uncertain funding several years into the future is likely less motivating to politicians than REDD+ funding that can be used during their term in office.
Despite these differences, India's EFTs and REDD+ share some important commonalities. Both offer recurring payments to governments in proportion to their performance in achieving a forest-related outcome. Both operate at the scale of state governments (in this regard, India's EFTs are most similar to REDD + in its state-level formulations: e.g., the Forest Carbon Partnership Facility, Germany's REDD Early Movers program, or the voluntary Verified Carbon Standard's Jurisdictional and Nested REDD+ program). And both seek to operate on the scale of billions of dollars per yearfunding through the EFTs works out to around $174-303 per year of each hectare of forest.

Monitoring and evaluating REDD+: advantages and drawbacks of India's ecological fiscal transfers
In numerous aspects that make a program suitable for evaluation, India's EFTs have advantages relative to REDD+ programs. First, most REDD+ programs make payments contingent on multiple facets beyond performance in achieving a single forest-related outcome. Norway's bilateral REDD+ agreements, Germany's REDD Early Movers Program, the Carbon Fund of the Forest Carbon Partnership Facility, and the Green Climate Fund all have requirements about how outcomes should be achieved and how funds are spent. These requirements include strategies and workplans, upfront investments in monitoring, reporting on social and environmental safeguards, and benefit-sharing plans. While such features are often included for good reasons related to program design or equity, these additional requirements for receiving funding nevertheless make program evaluation more challenging. It is more difficult for researchers to attribute changes in outcomes to the payments for performance rather than other facets of the program. In contrast, India's EFTs are essentially a "pure" payment-for-performance instrument. The only condition for receiving payment is the level of forest cover, with no additional requirements about how the outcome is produced or funds are spent.
Second, REDD+ programs often take place in the context of other contemporaneous (and potentially confounding) forest conservation policy changes. For example, in Brazil's pay-for-performance agreement with Norway, performance-based contributions to the Amazon Fund took place amidst successive waves of new anti-deforestation policies (Nepstad et al. 2014). Indonesia's pay-for-performance agreement with Norway was accompanied by a moratorium on new agricultural and logging licenses on primary forest and peatland (Busch et al. 2015). Such contemporaneous policy changes make it challenging for researchers to isolate the effects of payments for performance from the effects of other policy changes, or from an increase in political will responsible for both. In contrast, India's EFTs originated outside the forest conservation sector and were not accompanied by other changes to forest policies.
Third, the financial scale of most REDD+ programs is small relative to the size of the economies in which they operate. Offers of $1 billion in Brazil or Indonesia are small relative to the economies and government budgets of those countries. States participating in the Carbon Fund may expect to receive only $50 million or so. In contrast, India's EFTs amount to around 2% of states' budgets, with a higher percentage in more-forested states (Busch and Mukherjee 2017). While this scale of funding is still modest, it is plausibly enough to motivate state governments.
Fourth, participation in other REDD+ programs is voluntary by design. Commonly, voluntary programs face issues of selective participation in which some potential recipients select to enroll before or instead of others. The characteristics of those that enroll may differ systematically from those who do not. For example, households that self-select to receive PES may be more favorably inclined toward conservation anyway (Sanchez-Azofeifa et al. 2007). To evaluate voluntary programs facing selection issues, researchers must find ways to compare households or sites that would have been equally likely to conserve in the absence of the program. They commonly do so by randomizing participation experimentally or by quasi-experimentally matching participating and non-participating households that have similar pretreatment characteristics. But for researchers of India's EFTs, neither technique is necessary since participation by states in India's EFTs is universal.
Fifth, most REDD+ programs only have a few recipients. Norway's bilateral agreements have seven; Germany's have three. The Carbon Fund has none yet though it may eventually have more than a dozen; the Green Climate Fund may provide results-based payments for REDD+ to two to three countries in its first tranche. Not only are the recipient countries or states few in number, but they vary widely from one another. They can be assessed individually but cannot be analyzed as a group with any expectation of statistical power. India's EFTs, on the other hand, involve dozens of states. While India's states are not homogenous, states within a single country are arguably more comparable than states in multiple countries. They can more easily be compared in multistate analyses, as in the above-referenced study of cross-state health outcomes in El Salvador (Bernal, Celhay, and Martinez 2018).
Finally, most countries with REDD+ programs lack their own time-series data on forest outcomes spanning the periods before and after the treatment. Frequent and consistent monitoring of data across time and space is necessary and non-trivial for program evaluation. Brazil has long maintained its own high-quality data on deforestation, but most other countries have not. This means that research on the effect of payment-for-performance offers in these countries would depend on externally collected data, e.g., Landsat-derived maps of annual forest-cover loss from 2000 onward (Hansen et al. 2013) that are now freely available on the online platform, Global Forest Watch. However, the India Forest Service has been monitoring forest cover biennially since 1987. This means that an evaluation can use both domestic data (for internal legitimacy) and externally collected data (for independent validation).
Despite attractive features that make it easier to evaluate, India's EFTs also have features that make them more difficult to evaluate relative to other REDD+ programs. One such feature is variation in the enrolment, timing, or magnitude of potential incentive across states within a country. The presence of such variation makes it easier for researchers to compare results across treatment and control states, or across states with different levels of treatment. With the Carbon Fund's state-level programs, for example, some states will receive an offer of REDD+ payments while nearby states will not, potentially enabling plausible control groups to be constructed from among untreated states. With India's EFTs though, all states automatically receive fiscal transfers in proportion to their forest cover, with an equal level of payment-per-hectare among states. Thus, researchers must infer program effects by constructing a synthetic control group from countries or states outside India (as in Sills et al. 2015), or by using variation in states' characteristics that might plausibly produce a different outcome in response to identical treatment (as in Busch and Mukherjee 2017).

Monitoring and evaluating multiple aspects of REDD+
There are multiple aspects of REDD+ that are worth monitoring and evaluating, and for which India's EFTs provide attractive opportunities to test relative to other REDD+ programs. One is the central premise of REDD+ mentioned above: that the offer of payments-for-performance will produce financial incentives that lead state governments to protect and restore forests more than they otherwise would. This premise was tested preliminarily by Busch and Mukherjee (2017). They found no significant correlation across India's 29 states between improvements in forest cover from 2011-2013 to 2013-2015 and the size of the forest-proportional transfer as a share of total state revenue (r = −0.03; p = 0.89). However, they caveat that it's likely too soon to expect the EFTs to have had an effect and propose conducting a more rigorous evaluation at least five years after the reform. Updating the analysis of Busch and Mukherjee (2017) following the release of the 2017 India State of Forest Report (MoEFCC, 2017), I find that there is still no significant correlation between improvements in forest cover from 2011-2013 to 2015-2017 and the size of the forest-proportional transfer as a share of total state revenue (r = −0.06; p = 0.77). The same caveat to the previous analysis applies here as well.
Beyond the direct impacts of the offer of performance-based funding on forest cover, India's EFTs have advantages for studying other aspects of REDD+ as well. One such aspect is the effect of social and environmental safeguards. The UNFCCC agreed on the Cancun Safeguards for REDD+, which are intended to prevent undesired consequences such as unjust exclusion of local people from natural resources, or increased reforestation with non-native tree species in biologically important grasslands. For REDD+ programs that have these safeguards, a research challenge is comparing observed social and environmental outcomes (e.g., evidence on land disputes or the species composition of reforested areas) to what would have happened in a counterfactual scenario without safeguards. For India's EFTs, the research challenge is the reversecomparing observed outcomes to what would have happened with safeguards. This makes evaluation of the social and environmental effects of India's EFTs an informative complement to studies of other REDD+ programs.
It is also worth studying what motivates governments to respondor notto incentives from payments for performance. Perakis and Savedoff (2015) identify four potential reasons that resultsbased programs might motivate governments more than alternative aid instruments such as grants. These include pecuniary interest (the potential increase in state revenue); attention (making results more salient to politicians and managers); accountability to constituents (through transparent reporting on outcomes); and discretion (giving recipients the ability to problem-solve rather than prescribing methods). Research on motivations would potentially entail interviews with state legislators and bureaucrats, e.g., as conducted for the Salud Mesoamerica Initiative for results-based aid for health outcomes (El Bcheraoui et al. 2017). Since this type of qualitative research would require the participation of state government officials, it could be helpful for the central government or a consortium of state governments to commission the data collection. India's EFTs are advantageous for this type of research due to the relatively large number of recipient governments from which to identify potential interviewees.
Finally, it is worth studying the policy channels through which state governments choose to respond to financial incentives to protect and restore forests. Do they increase budgets for forest conservation, seek to improve forest management, make changes to forestrelated zoning, redesign infrastructure projects, or pass along payments to local households? Do they enlist the participation of citizens, as in Guinness Book of World Records-setting tree-planting initiatives in Uttar Pradesh (National Geographic 2016) and Madhya Pradesh (National Geographic 2017)? Do they attempt to increase forest cover by slowing deforestation, accelerating reforestation, or some combination of the two? With this type of research, researchers must make assumptions about whether activities that take place after the offer of payments for performance can be attributed to the offer, or would have happened even independently of the offer. Again India's EFTs will be advantageous for this type of research due to their relatively large number of recipient governments.

Conclusion
How effectively payments for performance can motivate governments to protect and restore forests is important to the future form of efforts to finance reductions in emissions from deforestation. Research on any and all REDD+ programs, as well as on "adjacent" program types, is useful. India's EFTs are not labeled as REDD+ and have important distinctions from other REDD+ programs. However, India's EFTs have a large number of recipient governments, focus on a single outcome, lack of contemporaneous confounding policy changes, significant financial scale, full and mandatory participation, and long-term data collection. All of this makes India's EFTs especially useful as a test case for the payment-for-performance premise, as well as for other REDD+-related questions regarding safeguards, motivations, and policies.