Does Refusal Bias Influence the Measurement of Chinese Political Trust?

Abstract Measurements of Chinese political trust may be inaccurate due to ‘refusal bias’, resulting from unwillingness of people with certain attitudes to take part in surveys. Such bias is especially problematic because researchers usually have little or no information about refusers. Nevertheless, techniques have been developed which allow correction of refusal bias by extrapolating from reluctant or difficult respondents on the basis of various measures of response propensity. Using data from a nationwide survey conducted in China in the winter of 2012/13, this article shows that this type of correction procedure improves the accuracy of measurement of the Communist Party membership rate, and produces significantly lower estimates of trust in the central government/Party leadership, trust in local government and support for the current system of government. Refusal bias is likely to result from the social desirability of expressing political trust and support under authoritarian conditions.


Introduction
To address the criticism that China's reported levels of political trust, especially trust in the central government, are 'too good to be true' , 1 a number of quantitative studies have been carried out. Tianjian Shi looked at the correlations of political caution with political trust, and found a relatively low correlation. 2 In another study he showed that political caution had smaller correlations, in absolute value terms, with don't know and no reply responses to trust questions than education and interest in politics. 3 Xuchuan Lei and Jie Lu conducted experiments using local surveys in Chengdu where the interviewers variously identified themselves as conducting academic or government surveys and wore or did not wear a CCP badge. 4 They also included a measure of political caution. The use of CCP badges, contrary to expectations, seemed to embolden respondents to voice criticisms of the CCP and lower their levels of reported trust. Political caution had no significant influence. These authors thus concur with Shi that dissimulation bias has only a very small effect on political trust measurements in China. However, interest in politics. 10 Walgrave et al. found in the course of surveys of demonstrators in six European countries that refusal bias, resulting from failure to return a follow-up questionnaire, was associated with younger and less educated demonstrators who were less motivated to participate, less politically efficacious and less likely to have been mobilized through the mass media. 11 Generally, the literature suggests that response mechanisms in political surveys are related not just to age and education but also to engagement with the public sphere. 12 Using nationwide survey data collected in China in late 2012 and early 2013, this article deploys the 'Variable Response Propensity Estimator (VRPE)' method for detecting and correcting for unit nonresponse bias. 13 This method belongs to a class of approaches whose basic assumption is that it is possible to extrapolate from reluctant or difficult respondents to non-respondents using measures of response propensity. 14 The VRPE method uses simultaneous equations for response propensity and the outcome variable, regressed on a series of socio-demographic variables whose population distribution is known, to measure refusal bias and to produce corrected estimates of the population proportion having the outcome characteristic. The principal advantage of such approaches is that they do not require additional fieldwork beyond the survey itself. Their principal disadvantage is that there is no inherent way of knowing that the extrapolations are correct. To get around this problem, researchers often attempt extrapolation using an outcome variable whose distribution is known in the population from other sources. If the extrapolation brings estimates closer to the true values, the method is judged to perform well.
To validate the method, this article compares self-reported Communist Party membership rates with a rate calculated from official statistics. Assuming that the response mechanisms for self-declared Communist Party membership and political trust and system support are similar, it then applies the method to measures of trust in the central government/Party center, trust in the local state and political system support. As measures of response propensity, this article uses interviewer assessments of respondents' attitude to the interview and the percentage of questions for which the respondent says 'don't know' or gives no answer. It also conducts supplementary analyses to check whether how refusals are counted has a substantial effect on the corrected estimates.
Results suggest that refusal bias is non-ignorable and substantive. Corrected estimates of the Communist Party membership rate are closer to official figures than the uncorrected estimates. In addition, there is statistical evidence of significant refusal bias across all three measures of political support and trust. The discrepancy between corrected and uncorrected estimates is between 1 and 6% for trust in central political institutions, between 3 and 8% for political system support, and between 2 and 7% for trust in local government. 15 15 there are four levels of sub-national government in China: province, prefecture, county and township. the meaning of the term 'local government' (difang zhengfu) is context-dependent, and could refer to any one of the four sub-national levels. and the less educated are more likely to refuse to be interviewed, but these socio-demographic factors do not fully account for selection into the sample. There are significant correlations between the error terms of the response equations and the equations for political trust and support. This implies the existence of unknown factors which influence response propensity and trust/support simultaneously. Most likely, these factors are related to the social desirability of expressing trust and support under authoritarian conditions. The next section explains the study design and method of analysis and introduces the data and outcome variables. The third section presents results, starting with Communist Party membership as outcome variable before moving on to trust in the central authorities and the local state and political system support. The penultimate section discusses the findings, and the article concludes with a summary and recommendations for future research.

Method, Study Design and Data
The VRPE method relies on the assumption that it is possible to extrapolate from reluctant or difficult respondents to non-respondents on the basis of response propensity. This may sound like a bold assumption, but it is in fact a cautious approach when the relationship between response propensity and the outcome variable is unknown. 16 The method assumes that the correlation between response propensity and the outcome variable is worth measuring. If the correlation is found to be close to zero, the method will produce an estimate of the population proportion nearly identical to the sample proportion, but if the correlation is significantly different from zero, it will produce an estimate which is higher or lower than the sample proportion. The method requires an algorithm and computer program to estimate simultaneous equations for a dummy outcome variable and response propensity by regressing them on a set of demographic variables whose distribution in the population is known; the equations can then be used to produce estimates of the population proportion which are correctly weighted both for demographic characteristics and for the likelihood of response. 17 The correlation between the error terms in the two equations is an indicator of bias. A zero correlation implies that the demographic variables fully control for selection into the sample, or that the refusers represent data which are missing at random, whereas a significant correlation implies that there is non-response bias.
The main data file analyzed here comes from the China National Health Attitudes (CNHA) survey. This was a nationwide survey of adults age 18+ in residential dwellings, conducted from 1 November 2012 to 17 January 2013. 18 Fieldwork was carried out by the Research Center for Contemporary China (RCCC) of Peking University. The target population was mainland citizens aged 18-70 residing for more than 30 days in family dwellings in all 31 provinces. The survey used the GPS Assisted Area Sampling Method. 19 Stratification took place in stages. At the first stage, the country was divided into three official macro-regions, Eastern, Central and Western, and each macro-region was divided into urban and rural administrative areas, giving six layers in total. Sixty primary sampling units (PSU) corresponding to county-level administrative divisions were selected at random across the six layers with probability proportionate to population. Within each PSU, three half-square minutes (HSM) of latitude and longitude were chosen with probability proportionate to population density, and within each of these, again proportionate to population density, a number of spatial square seconds (SSS) corresponding to 90 m×90 m squares were selected at random. Within each SSS, all dwellings were enumerated, and 27 were selected in each HSM by systematic sampling. The result was a sample of 5,419 valid addresses. 20 16 peress, 'Correcting for survey nonresponse using variable response propensity' , pp. 4ff. 17 Details of the algorithm are in Ibid. professor peress provided a copy of his C++ program and replication data which come from the american national election Surveys. after replicating his results, the author modified the program to get it to work with the Chinese data sets analyzed here. 18 Distributed by uK Data Service, Sn852091, available at: https://discover.ukdataservice.ac.uk/ (accessed 28 July 2016). Having made contact at each dwelling, interviewers attempted to complete a short pre-selection questionnaire and then identified respondents by the Kish method. Refusals were the largest category of non-respondents. There were 712 refusals at the address, including those where the interviewer was unable to enter the community or apartment block or was turned away by an unknown person, and 287 refusals by the person selected for interview. Non-contacts occurred at 610 addresses where no one answered and 92 addresses where the selected interviewee was not at home after several call-backs. There were 38 cases where physical and language barriers due to old age prevented any interview taking place. The result was a total of 3,680 valid interviews, which equates to a response rate of 67.9%. Post-stratification weighting by age and gender was conducted to match closely the 2010 census (see Appendix Table A1 for details). The survey had a margin of error of no more than 3% at the 95% confidence level.
As we have seen in the data analyzed here, refusals are documented in two varieties, 'refusals at the address' and refusals by the person selected for interview. In the latter case, it is apparent that the interviewer has managed to make contact with the household and succeeded in enumerating its members, only to be refused by the person selected. In the former case, we don't know who made the refusal. The author conducts sensitivity analyses to test the effect of counting refusals at the address as well as personal refusals, or just personal refusals in estimating the effects of refusal bias (see Appendix Table A2).
As with most surveys which are not explicitly designed to study non-response, the CNHA survey has very little paradata which were collected for both respondents and non-respondents, but one datum available for both is the location of the intended interview. This allows us to determine that, excluding non-contacts and cases where physical and language barriers prevented any interview taking place, the refusal rate in urban areas was 23%, and among these 17% were refusals at the address and 6% refusals by the person selected for interview; in rural areas, the refusal rate was 19%, among which 13% were refusals at the address and 6% by the person selected. This information is consistent with anecdotal evidence that it is harder to get interviews in urban areas, because of the existence of gated communities, secure entry systems for apartment blocks and other reasons.
Non-contacts are treated differently to refusals for the following reasons. Survey researchers often assume that post-stratification weights can partially mitigate the effects of non-contact because its likelihood is related to age and gender. For example, young people are less likely to be at home due to lifestyle habits and work patterns. If they are under-represented, they can be weighted up through post-stratification weighting to match census data. Crucially for this study, non-contacts do not choose to be non-respondents, nor does any person connected with them choose on their behalf, so there is no basis to assume that non-contact shares similar causes with difficult or reluctant response. In this study the author assumes that the socio-demographic variables of age, gender, education and urban or rural residence fully control the determination of contact.
In the VRPE approach, response propensity is measured by an ordinal variable which is coded so that the minimum value indicates the most willing respondents, and the maximum value indicates the most reluctant respondents. Refusal is assumed to correspond to the maximum plus one. This analysis deploys two different measures of response propensity, one based on the interviewer's assessment of the respondent's attitude to the interview, 21 and the other based on the item non-response rate across all questions, that is, the number of times the respondent said 'don't know' or gave no reply, divided by the total number of questions.
Interviewer-assessed response propensity is based on a set of five questions. Interviewers were asked to rate the interviewee on a five-point scale in terms of their level of cooperation, understanding, doubt before the interview, trustworthiness of replies and interest in the interview, coded so that high values indicated difficult respondents. An exploratory factor analysis showed that all questions loaded on a single factor. 22 The author computed the average of all five scores and divided the resulting scale into  For each question concerning the respondent's attitude toward the interview, if the interviewer gave a negative rating, she or he was asked to record the reason for the negative evaluation as a short open-ended text. Looking at the reasons gives us a flavor of the types of encounters interviewers have with reluctant respondents. 23 One interviewer noted that the respondent kept 'asking repeatedly where we were from, appearing very busy'/'老问你们是哪的, 对方很忙'). Another found that the respondent 'thought we were selling insurance, felt the survey was pointless'/'认为是卖保险的, 总觉得没什么用' . An interviewer observed that 'the expression in their eyes suggested they did not answer very sincerely, and other relatives of the respondent appeared'/'眼神闪烁回答问题不太认真, 接待人的其它亲人出现'). Sometimes the reasons for insincerity were clearly political. For example, the respondent 'didn't want to answer some questions like C20 [on political trust] or C27 [on political attitudes]'/'有的问题如C20，C27就不想回答'). More often the reason had to do with economic insecurities. For example, 'the household had a lot of pigs, yet they said their income was very low'/'家里养好多猪，收入却很少' . The questionnaire did not probe the reasons for lack of interest in the interview.
To derive a behavioral measure of response propensity, the author computed the proportion of questions for which the respondent answered 'don't know' or gave no answer from across all questions in the survey which were asked of all respondents. This included 53 questions about the respondent's life circumstances, 119 questions about health and health care experiences, 119 questions on social, economic and political issues, as well as 20 questions on income and expenditure. Each time the respondent chose 'don't know' or gave no reply was counted as an item non-response. The total number of item non-responses was divided by the total number of questions, and, to avoid outlier effects, the top 2% on the item non-response rate distribution was collapsed into a single category. The result was a continuous measure of item non-response ranging from zero to 40 with a mean of 11.8% and standard deviation of 8.8%. To match the scale based on interviewer-assessments, the author divided this scale into seven roughly equally sized ordinal categories. The distributions of the interviewer-assessment and item non-response ordinal scales across all valid addresses in the sample are shown in Table 1.
Political trust was measured using responses to the following question: Please indicate to what extent you trust the following institutions to operate in the best interests of society. If you don't know what to reply or have no particular opinion, please say so. The scale had four points from 'don't trust at all' to 'trust a lot' but to achieve a dichotomous variable, which is required by the VRPE method, all respondents who answered 'trust a lot' or 'trust somewhat' were coded one and all others zero. The central government 23 the full text of all the explanations is available from the author. and the Party centre (党中央) were listed separately, but since the two overlap and their trust measures correlate at 0.87, they are treated as measures of a single concept, trust in the central authorities. Trust in local government was treated as a separate outcome variable. Finally, as a way of triangulating findings in relation to political trust, the author included a measure of system support based on the following proposition: Whatever its faults may be, our form of government is still the best for us. All who agreed with it were coded one and all others zero. Uncorrected and corrected estimates for these outcome variables are presented in the Results section, below. Before considering those estimates, however, it is useful to first validate the method by comparing survey-based estimates of the Communist Party membership rate with official statistics.

Results
It can be shown that in 2012 the number of Party members as a proportion of the adult population was around 8.5%, the 95% confidence interval (hereafter, C.I.) being between 8.3 and 8.7%. 24 Studies of the census suggest that official statistics tend to undercount the population because of attempts to evade birth planning and migration controls, 25 so our calculation of the proportion of Party members is likely to be on the high side.
In the unweighted sample, 12.6% of respondents claimed to be Party members, (C.I. 11.5-13.7%, Table 2). Post-stratification weighting by age and gender brings the proportion down only slightly to 12.3% (C.I. 11.2-13.4%). Applying the VRPE method with interviewer assessment as the measure of response propensity produces a corrected estimate of 9.8%. We can compute standard errors for the VRPE estimates using the delta method, 26 which gives us the C.I. as between 9.2 and 10.5%. Using the item non-response measure of response propensity makes little difference: the VRPE estimate is 9.9% (C.I. 9.3-10.6%). The VRPE estimates are much closer to the Party membership rate calculated from official statistics than the simple estimates. We can conclude that the VRPE method is likely to improve the accuracy of measurement for this outcome variable.  With the item non-response measure of propensity, the estimate is 46.2% (C.I. 44.0-48.3%). Again, the VRPE estimates are substantively lower than the simple estimates, albeit with a small loss of precision.
As regards support for the current system of government, the unweighted sample estimate is 61.8% (C.I. 60.2-63.4%) and the weighted sample estimate is 61.5% (C.I. 59.9-63.1%). The VRPE correction for refusal bias produces an estimate of 58.8% (C.I. 55.8-61.7%) when the interviewer-assessed measure of propensity is used, and 54.1% (C.I. 51.4-56.7%) when the item non-response measure of propensity is used. Thus the VRPE method produces lower estimates for all three attitudinal measures of trust and support.
Looking at the correlations of the error terms in the paired equations which produce the corrected estimates allows us to examine the extent of refusal bias (Table 3). For each outcome variable, the table shows the correlation (rho) between the error terms of the two equations. Rho can be interpreted as a measure of refusal bias, since it tells us the extent to which unspecified non-socio-demographic factors influencing selection into the sample also influence the outcome variable. 28 A positive rho correlation indicates that respondents are more likely to be positive on the outcome variable than refusers, for reasons other than their socio-demographic characteristics.
We can see from the correlations that there is significant and non-ignorable refusal bias across all the outcome variables included in the analysis. Trust in the central authorities and support for the current system reveal more bias than trust in local government. Thus, using the item non-response measure, trust in the central authorities has a rho of 0.43***, support for the current system has a rho of 0.39*** and trust in local government has a rho of 0.33***. Using the interviewer assessment measure, the corresponding rho are 0.14***, 0.14*** and 0.07*. It is noteworthy that for the attitudinal questions the item non-response measure of propensity detects more bias than the interviewer assessed measure. For Communist Party membership, the amount of bias detected is similar whether response propensity is measured by interviewer assessment (rho 0.09**) or by item non-response (rho 0.08*).
Looking at the response propensity equations tells us the socio-demographic correlates of response propensity using the two measures (Table 4)  measure of propensity is used, and which outcome variable we look at, older and more educated people have greater response propensity, as shown by positive and significant coefficients for all age categories over 40, and for all those educated at least to senior high school level, including technical or vocational qualifications. Gender and rural residence matter in different ways, depending on how response propensity is measured. If we use interviewer assessments, it appears that rural dwellers have lower response propensity (beta -0.12**). However, if we use the item non-response scale, it appears that women have lower response propensity (beta -0.11**) and rural residence is insignificant. The response mechanism seems to differ slightly depending on how response propensity is measured. The regressions on the outcome variables (Table 5) demonstrate that, controlling for the likelihood of selection into the sample, position in the social structure matters. For Party membership, the dummy variables for older and more educated categories all have significant positive coefficients. It is noteworthy that female gender is a negative influence on Party membership (the coefficient, alpha, is -0.38*** or -0.39*** depending on the measure of response propensity). Rural residence is a marginally negative influence on Party membership (alpha -0.14*).
As regards trust and system support, older and more educated respondents are generally more trusting and supportive, although the precise way in which education matters appears to vary with the outcome variable and the measure of response propensity used. For trust in local government, only the university-educated stand out as more trusting, regardless of which measure of propensity is used (alphas, 0.31*** and 0.38***). For current system support, those with senior high school education or above are more positive. Female gender is a negative influence on trust in the central authorities (alpha -0.13* or -0.12*), but not on the other outcome variables. Rural residence reduces trust in local government (alpha -0.10* or -0.12**) but has no effect on trust in the center or system support. In sum, these results suggest higher status socio-demographic groups are more likely to trust and support the regime. Figure 1 depicts the relationships between outcome variables and response propensity in the models reported above. The data points are the 80 socio-demographic categories corresponding to the matrix of dummy variables mapping all combinations of gender, urban or rural residence, age decile and level of education for which census data are available. As can be seen from the figure, the relationships between propensity and the outcome variable are always monotonic but not always linear. Communist Party membership has what looks like an exponential relationship with response propensity: it increases slowly at first and then rises sharply. For the attitudinal outcome variables, the scatter plots produce   roughly linear patterns, although the item non-response plots feature a kink in the middle. All the scatter plots suggest a positive relationship between the outcome variable and response propensity.

Discussion
The source of bias lies in the relationship between the outcome variables and response propensity. The models presented here control for the differing likelihoods of demographic categories to be selected into the sample. So, although higher status people are more willing to answer questions, perhaps because they have more self-confidence, this is not the source of bias. Similarly, although low status individuals may de-select themselves for reasons which have nothing to do with their political views, neither is this the source of bias.
In the absence of follow-up surveys of refusers, the reasons for refusal bias can only be inferred. Some unmeasured factors, other than socio-demographic characteristics, influence selection into the sample as well as political trust and system support. Because the survey over-estimates Party membership in a similar way to political trust and system support, and becoming a Party member is a behavioral expression of support, it seems reasonable to conclude that those unmeasured factors include political and/or social desirability. This inference is consistent with the fact that expressed trust in the center and system support exhibit more bias than trust in local government. It is also consistent with how respondents answer questions about politically and socially approved attributes and in other contexts. 29 Because trust and support are approved, there is pressure on respondents to exaggerate these attitudes.
The differences made by refusal bias are substantive for some variables. For local government trust, where the lower end of the confidence interval for simple survey estimates is close to 50%, taking refusal bias into account may push the estimate below a majority. Support for the current system of government, likewise, can move from the low sixties to the low fifties when refusal bias is taken into account. Only for the central authorities is trust high enough that refusal bias, which discounts up to 5% of the 85% who expressed trust in the weighted sample, makes little difference.
Differences in the determinants of the two measures of response propensity imply that the response mechanism is slightly different for each. The subjective element in interviewer assessments may make them susceptible to contamination by the social distance of the interviewer from the respondent. 30 For this survey, all we know for sure about the interviewers is that they were local university students age 22-26. 31 Amongst respondents, 75% of the university-educated live in urban districts, so it is a reasonable assumption that the interviewers, too, were predominantly urban residents. It seems possible that social distance at least partly accounts for the lower levels of cooperation these interviewers encounter in rural areas. The item non-response measure avoids subjective interpretation of respondents' behavior. In this respect, item non-response rate may be a better measure of response propensity than interviewer assessments. The fact that the available paradata showed a higher refusal rate in urban areas is another reason to be cautious about the validity of the interviewer-assessed measure of response propensity. It is thus quite likely that the interviewer-assessed measure of response propensity is contaminated by social distance effects. By contrast, the item non-response measure of response propensity shows no significant effect of rural residence, which suggests it is not contaminated in the same way.
The shapes of the relationships between response propensity and the outcome variables suggest that the response mechanism for Party membership differs from the attitudinal questions. The exponential relationship found in the case of Party membership most likely reflects the existence of Party membership criteria. Even though response propensity increases in linear fashion with status, the likelihood of Party membership does not, because there is a status threshold above which this likelihood increases rapidly. For political trust and system support the fact that the relationships of the outcome variables to response propensity more closely approximate linear relationships may reflect the fact that there are no status requirements for attitudinal expressions of support.
The assumption that personal refusals and refusals by another person have much the same implications for the measurement of refusal bias is substantively important. If we adopt a stricter standard, so that only the 287 personal refusals by the selected interviewee count as refusals, the corrections are much more modest (see Appendix Table A2). If the interviewer did not have time to enumerate household members, there is no way of knowing who exactly refused the interview and so it is assumed that all refusals are alike, but this is clearly an oversimplification of a complex situation on the ground.
The positive effects of age and education and the negative effects of female gender and rural residence on the likelihood of Party membership are consistent with previous research. 32 The age and education parameters reflect two trends since the 1980s: people have been joining the Party later in life, which makes membership amongst 18-29 year olds less likely than amongst older age groups, and educational qualifications are increasingly important as criteria of recruitment. The under-representation of women may result from traditional gender stereotypes. The decay of Party organization in rural areas may explain why rural residents are less likely to be members.
As regards the attitudinal outcome variables, the positive effects of age are consistent with what previous research has found. 33 However, whilst education is clearly a positive influence on trust and system support in this study, previous studies have produced somewhat inconsistent findings. A number of authors found that education is a weak negative influence on political trust. 34 It seems unlikely that the positive effect of education found here is simply an artifact of the method used. 35 It is possible that the relationship of education to trust and system support is dynamic. Longitudinal analyses suggest that the relationships of education to the outcome variables differ over time. 36 The patterns of relationships of education to trust and system support may also not be linear. 37 This study has a number of limitations. Because it relies on specialist software which was written to evaluate simultaneous equations for two dummy variables, the approach adopted here is not applicable to outcome variables measured on ordinal or continuous scales. The study has assumed that the sources of refusal bias are similar for Party membership and political trust/support. However, the results have shown the relationships between response propensity and trust or support are not the same shape as the relationship between response propensity and Party membership. This implies that our instruments for detecting and correcting bias are still rather crude. Finally, we do not yet have a method for combining the correction of different kinds of bias in a single measurement.

Conclusion
This article has shown that refusal bias is a serious and non-ignorable problem for the measurement of political trust and system support in China. It causes over-estimates of mean trust and system support which cannot be avoided through the usual post-stratification weighting techniques. Because response propensity differs across social structural characteristics, inferences about the relationships between such characteristics and political trust and system support are also likely to be affected. Researchers should probably routinely control for response propensity when developing models of socially or politically sensitive outcomes. The implications of this article extend beyond the problem of measuring political trust, and touch on other sensitive areas such as experience of corruption, sexual behavior, income and so on. Where it is important to establish an accurate point estimate of some phenomenon, it is advisable to correct for refusal bias using procedures like those demonstrated here. Where it is important to estimate some parameters, such as the effects of independent variables on a sensitive outcome, it would probably improve accuracy to include some measure of response propensity in the model.
Although the present study has focused on a technical solution to problems of measurement and design, this is not to deny the importance of employing appropriate interpretive devices to avoid naïve readings of survey data. 38 The elaborate lengths to which survey researchers go to make their data as objective and 'scientific' as possible can at times obfuscate the political nature of their research practice. It is important that quantitative researchers are able to empathize with the range of feelings which their research may elicit in its subjects, and to think through the implications of such feelings for the researcher's claims to objectivity and truth.
The questions of how to measure response propensity and how the different measures relate to socio-demographic categories require more detailed exploration. There is broad scope for comparative research, examining variations in refusal bias across different social and political contexts cross-nationally, as well as variations over time. To explore varieties of refusal bias, future studies will need to turn to more fieldwork-intensive techniques, including follow-up surveys for non-respondents and experiments with different ways of administering surveys.

Disclosure statement
No potential conflict of interest was reported by the author.