Reply to Santos and Colleagues ‘The Importance of Reliability in the Multidimensional Poverty Index for Latin America (MPI-LA)’

Reply to Santos and Colleagues ‘The Importance of Reliability in the Multidimensional Poverty Index for Latin America (MPI-LA)’ David Gordon & Héctor E. Nájera Catalán To cite this article: David Gordon & Héctor E. Nájera Catalán (2019): Reply to Santos and Colleagues ‘The Importance of Reliability in the Multidimensional Poverty Index for Latin America (MPI-LA)’, The Journal of Development Studies, DOI: 10.1080/00220388.2019.1663178 To link to this article: https://doi.org/10.1080/00220388.2019.1663178


(Original version submitted August 2019; final version accepted August 2019)
We thank Santos and her colleagues for their comments and welcome debate about the best ways to improve the measurement of multidimensional poverty in Latin America. While we agree with many of the points that have been raised, there remain some fundamental areas of disagreement. Four main issues have been raised regarding our work, so we will first summarise our position and then address each issue in turn.
We fundamentally disagree with the argument that poverty can be measured directly, that latent variable modelling is inadequate, and our conclusions are incorrect. Poverty is a concept (i.e. an idea). The statistical term for a concept is a Latent Variable, i.e. a concept/construct which cannot be measured directly but can be measured/estimated indirectly using indicatorsdata we can collect/ observe on related variables. Thus, concepts like poverty, human rights or mathematical ability can only be measured indirectly by using suitable indicators and adopting a latent variable approach. The MPI-LA uses deprivation indicators which indirectly estimate multidimensional poverty as a latent variable (i.e. the MPI-LA aggregates across its dimensions to produce a single estimate of multidimensional povertyan adjusted headcount number).
We are in agreement that the MPI-LA is a measurement instrument, however, all measurement instruments depend on an explicit or implicit measurement model, i.e. to measure length you need both a ruler (a measurement instrument) and a concept and definition about the size of a centimetre (a measurement model).
There is, of course, no such thing as error free measurement and all poverty measures will be affected by some amount of random and systematic error (Loken & Gelman, 2017). Santos and Villatoro's (2016) methods were not devised to estimate error. However, over the past one hundred years, across both the natural and social sciences, standard methods have been developed to assess the degree of measurement error and identify acceptable and unacceptable levels of such error (Brennan, 2006). It is important that poverty measures are developed which conform to these fundamental scientific standards and methods. This is true irrespective of the other properties of a poverty measure, e.g. if it is axiomatic, elegant, well liked, etc. Hence, both reliability and validity are necessary to fulfil Principle 4 of the Commission on Global Poverty (Atkinson, 2017).

The reliability of the MPI-LA results and correlation
We are in agreement with Santos and colleagues about the utility of using a range of different approaches to assess reliability and that Omega is a better measure of reliability than the widely used Cronbach's Alpha. Thus, in our paper, we used a range of Classical Test Theory measures (Alpha, Beta & Omega) and also Item Response Theory measures to provide a broad assessment of the reliability of the MPI-LA. We had to approximate Omega with and exploratory model due to the fact that the MPI-LA does not fit the data with a confirmatory one. The reported Omega is the highest possible reliability value for the MPI-LA. The literature cited by Santos states that these tests are based on Spearman's theory (1904)observed deprivations must be caused by the same phenomenon (i.e. caused by poverty). We found that the MPI-LA is unreliable because the indicators are caused by multiple phenomena and not primarily by multidimensional poverty. All the broad range of methods we used lead to the conclusion that the MPI-LA is unreliable.
The argument that the bootstrapping results of Santos and Villatoro (2016) demonstrate that the MPI-LA is a reliable measure is simply incorrect. Bootstrapping is a method for producing robust standard errors for the poverty estimates. It is not a method which can be used to determine the reliability of the poverty measure per se. In technical terms, bootstrapping resampling does not provide an estimate of the population distribution, it is done to provide an estimate of the sampling distribution of the sample statistic (e.g. the poverty rate, poverty depth, etc.). This means that bootstrapping results cannot be interpreted to mean that the same poverty headcount would be found if the survey was repeated, i.e. the reliability of the poverty headcount measure.
We believe that all poverty measures need to be reliable and valid if they are to be useful for policy, monitoring and evaluation purposes. Reliability is the amount of random error affecting a poverty measure and validity is the amount of systematic error or bias. We consider it to be an incontestable fact that a poverty rate of 20 per cent ± 1 per cent is of greater use than a poverty rate of 20 per cent ± 30 per cent. Similarly, an invalid poverty measure which incorrectly identifies certain groups of people or geographic areas as 'poor' is of little use for policy making and may result in the misallocation of money and efforts.

The selection of indicators of poverty
We agree with Santos and Villatoro (2016) on the utility of adopting the Latent Variable Method Exploratory Factor Analysis to help select the MPI-LA indicators. We also agree that there are few poverty indicators that are measured in a consistent manner in all countries. However, this is not a reason to only include the lowest common denominator sub-set of variables in the MPI-LA poverty measure, irrespective of their country specific validity and reliability. Although it may seem counter intuitive, a reliable index with measurement invariance permits direct comparisons of poverty measures using the best sub-set of indicators in each country, as long as a few common variables are also retained (Meredith, 1993).
Confirmatory Factor Analysis (CFA) is a Reflective Measurement Model where the indicators are causally related to the latent variable, i.e. when the latent variable changes (e.g. poverty gets better or worse) then the indicators will change (Brown, 2006). This approach has many useful properties: the indicators are substitutable, it is not necessary to include all possible indicators to achieve a good measure of poverty and you do not need to use an identical set of indicators in each country/context, i.e. you can use the most appropriate set of indicators in each country and still achieve a comparable measurement of poverty. Both the IRT the CFA model results identify the same problematic indicators in the MPI-LA. It is a good idea not to mix cause and effect indicators, not just for important policy reasons but also for important statistical reasons (Booysen, 2002), i.e. to produce a valid and reliable measure of multidimensional poverty.

Reply to Santos and colleagues 1791
The selection of deprivation indicators is a key problem in poverty research but it is a problem which has been largely solved. In 1995, Gordon argued that 'many deprivation indices seem to be composed of combinations of variables that the authors think measure something "bad". Although, what this "bad" thing is often unclear' (p. 39). Over, the past twenty-five years standard index construction methodologies have been applied in poverty research to determine if deprivation indicators are suitable, valid, reliable and additive and thus identify the optimum sub-set of deprivation indicators, i.e. which indicators to keep and which to drop (see, Guio, Gordon, Catalan, & Pomati, 2017;Guio, Gordon, & Marlier, 2012;Guio, Gordon, Marlier, Najera, & Pomati, 2018;Guio et al., 2016;Rio Group, 2006). These poverty indicator selection methods were extensively reviewed by the Eurostat Task Force on Material Deprivation (MD) who argued that 'The work is considered technically as providing a "gold standard" for the list of MD variables and indicator's construction and has unanimous support' (Eurostat, 2012, p. 7) 3. The meaning of dimensions in multidimensional poverty measurement If multidimensional poverty is not a single concept but many different unrelated concepts, then by definition, it would be impossible to calculate a multidimensional poverty rate or score. You cannot meaningfully sum 'apples and pears', 'fish and fowl', 'love and GDP', etc. Similarly, if the conceptual dimensions have no empirical reality (i.e. they don't exist in the data used) then applying a differential weighting scheme to the indicators in each dimension will introduce systematic error into the estimation of multidimensional poverty. We did not find evidence to support the validity of the MPI-LA dimensions.
Santos and her colleagues argue that it is not problematic that the MPI-LA indicators do not adequately measure the dimension in which they are grouped. They argue, 'there is no formal construct that needs to be confirmed' in their MPI-LA model. But their instrument has an underlying measurement model. Since their dimensions are conceptually distinct (e.g. 'housing and education' are conceptually different), they argue that it does not matter that the indicators are not empirically adequate measures of these conceptual dimensions. This rather 'begs the question' of why bother to sub-divide the indicators into dimensions at all (and differentially weight them) if they cannot be used to adequately measure the underlying dimensional concepts. That is, if the indictors cannot be used to determine if housing or education deprivation are the most prevalent problems for poor people in a country, then the MPI-LA dimension results cannot be used to provide policy advice about the extent and nature of education and housing problems or to monitor the effectiveness (or otherwise) of anti-poverty policies which aim to improve education and housing conditions.

The importance of the normative perspective
We agree with Santos and colleagues that all poverty measures should have a normative basis, indeed all measures in the social sciences arguably should be based upon a normative theory and/or framework. A normative theory allows a researcher to select a meaningful set of indicators to measure a concept like poverty. However, theories can be wrong, and the decisions made by researchers need to be tested. For policy, evaluation and monitoring purposes, a poverty measure needs to be valid and reliable.
Over the past fifty years, researchers at the University of Bristol and their colleagues have consistently argued for the application of scientific methods to improve the measurement of multidimensional poverty (see Townsend, 1993). In order to produce effective and efficient anti-poverty policies and evaluate their effectiveness, policy makers require accurate and precise measurements of the extent and nature of multidimensional poverty. The methods we used to evaluate the MPI-LA were not confined to psychometrics but were also developed by researchers in the fields of agriculture, biology, education, environmental science, medicine, sociology and applied statistics. They have been used extensively across the natural and social sciences, including by poverty researchers in many countries (for example, see Betti, Gagliardi, Lemmi, & Verma, 2015;Cappellari & Jenkins, 2007;Fusco & Dickes, 2008;Goldfeld et al., 2018;Martini & Vanin, 2013;Szeles & Fusco, 2013;Whelan, Layte, Maitre, & Nolan, 2001).
Poverty measures can of course be used for many different purposes, such as advocacy, and we welcome and applaud the excellent work of Dr Santos and her colleagues at the Oxford Poverty and Human Development Initiative in using the MPI to highlight the importance of multidimensional poverty. However, the Latin American MPI is so unreliable and invalid that the results produced are highly likely to be both wrong and misleading. Thus, we cannot recommend the use of the MPI-LA for policy making or academic research purposes in Latin America unless its reliability and accuracy can be radically improved.