Reliability and validity of a Mental Health System Responsiveness Questionnaire in Iran

Background The Health System Responsiveness Questionnaire is an instrument designed by the World Health Organization (WHO) in 2000 to assess the experience of patients when interacting with the health care system. This investigation aimed to adapt a Mental Health System Responsiveness Questionnaire (MHSRQ) based on the WHO concept and evaluate its validity and reliability to the mental health care system in Iran. Design In accordance with the WHO health system responsiveness questionnaire and the findings of a qualitative study, a Farsi version of the MHSRQ was tailored to suit the mental health system in Iran. This version was tested in a cross-sectional study at nine public mental health clinics in Tehran. A sample of 500 mental health services patients was recruited and subsequently completed the questionnaire. Item missing rate was used to check the feasibility while the reliability of the scale was determined by assessing the Cronbach's alpha and item total correlations. The factor structure of the questionnaire was investigated by performing confirmatory factor analysis (CFA). Results The results showed a satisfactory feasibility since the item missing value was lower than 5.2%. With the exception of access domain, reliability of different domains of the questionnaire was within a desirable range. The factor loading showed an acceptable unidimentionality of the scale despite the fact that three items related to access did not perform well. The CFA also indicated good fit indices for the model (CFI=0.99, GFI=0.97, IFI=0.99, AGFI=0.97). Conclusions In general, the findings suggest that the Farsi version of the MHSRQ is a feasible, reliable, and valid measure of the mental health system responsiveness in Iran. Changes to the questions related to the access domain should be considered in order to improve the psychometric properties of the measure.

screening and active follow-up of the patients, especially in rural areas, were developed (12). Although the service coverage in urban areas was still around one-third of the population, by 2006, evaluations showed that mental health program coverage reached 82.8% of the rural population (10). Despite this considerable effort, there is still need for improvement in terms of evaluation and monitoring of the quality of services provided (11). One of the areas in need of expansion is the focus on service users' experiences, as this can offer reliable and valid information that can help to achieve better quality of care (13).
To relate patients' experiences to an operationalized and comparable framework, in 2000 the World Health Organization (WHO) introduced the concept of responsiveness (7). Responsiveness has been defined as a measure of how well the health system responds to the population's non-medical expectations when interacting with the system (8). Although many studies have evaluated health care responsiveness as part of the WHO Multi-Country Service Study (MCSS) (7), to our knowledge the application of the concept of responsiveness specifically to the mental health care system has been limited (13). Given the context specific organization of health systems and the relevance of cultural norms in mental health, it is important to have a valid instrument that is easy for mental health service providers in Iran to both understand and use. The aim of our study was to adapt the original form of the Health System Responsiveness Questionnaire, developed by the WHO, to the mental health care system in Iran, by determining the validity and reliability of this new version.

Scale development
To evaluate the general health care system responsiveness on a national level, WHO developed and validated a questionnaire by using a comprehensive review of existing instruments and field tests of new and adapted items. The questionnaire measures responsiveness for general inpatient and outpatient care in eight domains (7,14). The English version Health System Responsiveness Questionnaire was translated into Farsi by the first author. The translated version was adapted based on the findings of our previous qualitative studies in which we evaluated the applicability of the health system responsiveness concept to the Iranian mental health system (15,16). As a result, a new domain of effective care was added, the domain of prompt attention was divided into two new labeled domains Á access to care and attention Á moreover, the domains choice of health care providers and autonomy were integrated, and some new questions were added to existing domains.

Setting and design
In Tehran, Iran's capital, mental health services are organized in terms of catchment areas. Each of the four public medical universities is responsible in terms of providing and supervising mental health services for a defined catchment area with specific geographical boundaries. The corresponding public medical university also supervises the existing private mental hospitals and outpatient clinics. The approved Farsi version of the Mental Health System Responsiveness Questionnaire (MHSRQ) was tested in a pilot study carried out in two outpatient centers with 20 participants. Based on these findings, the wording of several items was revised for clarification. The final questionnaire consisted of 40 questions representing eight domains. The domain 'access to social support' was excluded from the questionnaire because inpatient cases were not included in this study.
Between January and April 2013, a cross-sectional survey was implemented in all nine outpatient public mental health clinics, distributed in different city regions (north, south, east, west, and central); private psychiatric clinics were not included. A non-random sample of 500 mentally ill patients attending the selected clinics was recruited. The number of participants was calculated using the number of items entered into the factor analytic procedure. As a general rule, 10 subjects are necessary for each variable in factor analysis (17). The number of participants assigned to each clinic was proportional to the total volume of patients attending the clinics during the previous 3 months. All participants were diagnosed as mentally ill based on a professional psychiatric evaluation. The inclusion criteria for participating in the study were 1) being an adult (18Á65 years old), 2) receiving outpatient care during past 12 months, and 3) according to their clinical record, being in remission phase of their disorder and mentally capable to follow the interview. The type of participants' mental disorder was not considered as inclusion criteria because health care experiences relate more to the health services functioning than to the patient's current diagnosis (13,18).

Data collection procedure
Through participation in a 4-hour training session, 10 interviewers with a bachelor degree in psychology learned about the background and objectives of the study. In addition, the respondent selection procedures and interview process were explained to participants. On the basis of the pilot study, it was decided that the interviewers would read the questions to those participants with 5 years or less of formal education.
Patients attending the public mental health clinics were recruited after being approached by interviewers and asked for their consent to participate. All participants were interviewed in mental health clinics and each interview lasted approximately 45Á50 min. Before the interview, each participant was informed about the objectives of the study, explaining that the completion of the questionnaire was voluntary and their identification would be protected, as the data files were anonymous. The Ethical Committee and Research Council of the University of Social Welfare and Rehabilitation Sciences, Tehran approved the study protocol.
Failure to include all participants' data in the analysis may bias the results. Our first approach was to investigate the missing data and assess whether respondents had substantial difficulties in answering the questions. This was done calculating the item-missing rate, as the percentage of non-response to an item and the average across sections of the questionnaire. A missing rate of 5% or less was considered ignorable, whereas items with more than 20% (19) missing were considered problematic.
Reliability of the questionnaire was checked with internal consistency assessment methods. Consistency of the entire scale was assessed using the Cronbach's alpha coefficient. For each item, the alpha is given if the item is deleted. Other internal consistency assessment methods included the item-test correlation (the correlation of the item score with the average of items within a domain) and the item-rest correlation coefficients (the correlation of the item score with domain average that excludes the item from the equation). Other studies suggested that the itemtest correlations exceed 0.5 and Cronbach's alpha exceed 0.7 (20).  To evaluate the construct validity, we focus on the internal structure of the questionnaire, particularly on the dimensionality and homogeneity of items (questions) hypothesized to represent one domain. As the WHO (7) had already established the factor structure of the instrument, confirmatory factor analysis (CFA) was used to assess the construct validity of the new instrument. CFA followed Jö reskog's guidelines for the analysis of ordinal data (21). Diagonal weighted least-squares estimation was applied to polychoric correlations that were based on the asymptotic covariance matrix. Although, according to the WHO, there is no strict cut-off to describe the power of the association of the variance, the closer to (1 or '1, the stronger the unidimensionality of the construct (7). However, Hair et al., revealed that for a practical significance, loading factors 90.3 are of minimal significance, loadings 90.4 are considered important, and 90.5 indicate significant loading (22). The models were evaluated by means of BentlerÁSatorra chi-square score, root mean square error of approximation (RMSEA) (23), goodness-of-fit index (GFI), and adjusted goodness-of-fit index (AGFI), where the values of RMSEA less than 0.05 indicate a close fit, values in the range of 0.05 to 0.08 indicate fair fit, and that values above 0.1 indicate poor fit. For GFI, AGFI, and comparative fit index [CFI], values exceeding 0.90 indicated a good fit of the model to the data. CFI and incremental fit index were also reported (24,25), where values equal to or greater than 0.90 denote an acceptable fit to the model (26,27). Confirmatory factor analyses were performed using LISREL 8.8.

Results
The results of the descriptive statistics of demographic characteristics of participants showed that of the 500 patients enrolled in the study, 38% were female and 62% were male. The majority of participants were in the 25Á35 year old age group (33.4%). About 24% had 5 years or less of formal education and 28.7% were unemployed. All participants revealed that they had more than one time experience of using the services during past 6 months, and 96% more than two times. The majority of partici-pants (52.7%) revealed that they belong to the middle social class and 92.8% of participants had access to medical insurance.

Response rate and missing
The item-missing rate is reported in Table 2. All items met the pre-established criteria for feasibility of less than 20% missing. The access domain and its related questions showed the highest average item-missing rate (5.2%).

Reliability
The findings presented in Table 3 show the item-test correlation, the item-rest correlation, and the Cronbach's alpha coefficient (consistency of the entire scales). Overall, the results of the item correlation test were in the acceptable range with a Cronbach's alpha coefficient 0.70 for six of the eight domains. Access to care (a 00.56), and effective care (a00.66) were the worst performing domains. Figure 1 presents the results of the factor analysis based on the responses of the participants in the survey. The numbers indicate the factor loadings on the latent variable that represent the amount of variance that an item has in common with that latent variable. Three items related to the domain of access did not perform well: 1) the item on 'the length of time from requesting care to receiving it', 2) the item about 'length of time staying in waiting room before receiving the needed mental health service', and 3) and the item regarding the 'distance to reach the mental health care service'. Chi-square was calculated as 79.3 (df0566) which was not significant. Modification attempts were conducted in a step-wise procedure; at each step the items that had higher residuals than 0.8 (Theta-delta) were identified and that with the highest residual was eliminated. Two items, namely 6,151 (residual 00.894,) and 6,162 (residual 00.88) were removed based on this criteria. Correlations among eight factors are given in Table 4. Fit indices for this model were all indicative of an acceptable model (CFI 00.99, GFI 00.97, IFI 00.99, AGFI 00.97).

Discussion
The findings of this study show acceptable reliability and validity properties for the Farsi version of the MHSRQ. Fit indices of the overall model were good (GFI, AGFI, CFI0.9), although the domain of access did not perform well in the psychometric evaluation.
With the exception of the access domain (5.2%), the missing rates reported were lower than 1% for all domains. However, even the missing rate for the access domain was lower than the pre-established cut-off level of 20% (19). The worst performing items of this domain were those concerning the time it takes to reach a mental care clinic. Problems with these items have also been noted in the short form of the MCSS questionnaire for general health patients (7). One reason for problems with this domain is that the questions in this domain might have still been difficult for respondents to understand.
Mental health users usually attend care services several times during 1 year. Therefore, they might have found it difficult to remember the waiting time at the different visits. Technical modifications and wording revision of these items might be useful to overcome this problem.
The internal consistency of the questionnaire was good. The figures are similar to the classical psychometric assessment for the original responsiveness instrument (7), reinforcing its reliability. Our findings show that the responsiveness domains with the highest Cronbach's alpha were communication, attention, and quality of basic amenities. The latter showed a high alpha coefficient in the original version of the responsiveness instrument as well (7). The high Cronbach's alpha might indicate that the questions related to the domain were referring to similar issues and measuring the same aspects of the domain. Access (a 00.56) was the worst   The validity of the questionnaire was tested by focusing on the internal structure, in particular the dimensionality of the questions representing a domain. Although the results generally confirmed the structure of the responsiveness domains, three items related to the access domain showed loading factors less than the acceptable level of 0.3 (22). These items dealt again with time and distance in which numerical responses were more appropriate for reporting them. Because our analytical approach was suitable for analyzing ordinal responses (21), the continuous responses were transformed into categorical variables. Accordingly, this might explain the low correlation between these categorical responses and the actual continuous time/distance variables originally reported by respondents.
This study also includes certain limitations. The psychiatric diagnosis of participants was ignored arguing that patients' experience with the mental health system is not related to their current diagnosis. Although, we are aware that there are some literature suggesting that patient satisfaction with the system could be affected by their diagnosis (29,30). Following WHO instructions, first time patients were included. However, few times of experiencing mental health services could make it difficult for patients to give an accurate answer to some questions. Because there was not a previously validated instrument in Iran, it was not possible to make a direct comparison. Inpatient mental health users were not included because of the difficulties in accessing this group while they are in the remission phase of their illness and thus cognitively capable to participate in the study. Therefore, the domain related to access to a social support network, which is only relevant to inpatient care, was not included in the study.

Conclusions
This study has reported the feasibility, reliability, and validity of the WHO instrument used to assess mental health system responsiveness in Iran. A low item missing rate indicates that it is feasible to apply the instrument in Iran. The reliability and internal consistency of the questionnaire was acceptable in general, although some items showed lower item correlation than others. With exception to the access domain, a validity investigation also showed good results for all domains of the questionnaire and consistent responses in general. Further steps will include additional research to overcome some of the limitations of the present study. The future application of the Farsi version of the MHSRQ will positively contribute to mental health system improvements in Iran.