Content validation of Mental Health Literacy Scale (MHLS) for primary health care workers in South Africa and Zambia ─ a heterogeneous expert panel method

ABSTRACT Background: The lack of public knowledge and the burden caused by mental-health issues’ effect on developing and implementing adequate mental-health care for young and adolescent in low- and middle-income countries (LMIC). Primary health care could be the key in facing the challenge, but it suffers from insufficient resources and poor mental health literacy. This study’s aim was to adapt the content validity of the Mental Health Literacy Scale (MHLS) developed by O’Connor & Casey (2015) with researchers and primary health-care workers in low- and middle-income contexts in South Africa (SA) and in Zambia. Objectives: The study population comprised two expert panels (N = 21); Clinical Experts (CE) (n = 10) from Lusaka, Zambia and Professional Research Experts (PE) (n = 11) from the MEGA project management team were recruited to the study. Methods: MHLS was validated in a South African and a Zambian context using a heterogeneous expert-panel method. Participants were asked to rate the 35 MHLS items on a 4-point scale with 1 as not relevant and 4 as very relevant After the rating, all 35 MHLS items were carefully discussed by the expert panel and evaluated according their relevance. The data were analyzed using an item-level content validity index (I-CVI) and narrative and thematic analyses. Results: All 35 items ranked by the PREs met the cutoff criteria (≥0.8), and ten (n = 10) items were seen as relevant by CE when calculating I-CVIs. Based on the results of ratings and discussion, a group of sixteen (n = 16) of all items (n = 35) were retained as original without reviewing. A total of nineteen (n = 19) items were reviewed. Conclusion: This study found the MHLS to have sufficient validity in LMICs’ context but also recognized a gap between professional researchers’ and clinical workers’ knowledge and attitudes related to mental health.


Background
Over 85% of the world's population live in low-and middle-income countries (LMIC), and represent the great majority (>80%) of people with mental health disorders [1,2] without access to proper treatment [3]. These numbers clearly point out the burden of inadequate mental health care especially in LMIC countries. One main barrier for adequate care is the lack of trained health-care professionals; the World Health Organization (WHO) [4] estimates there is only one trained mental health worker serving more than 200.000 people in LMICs.
Understanding cultural needs in terms of mental health is a core component of health promotion [2,5]. The WHO [6] acknowledges the public's lack of knowledge about mental disorders in children and in adolescents as one of the main barriers to adequate mental health care worldwide. Considering the cultural context of mental health in LMICs, complementary practitioners and traditional healers are still the number one choice for help [2,7]. A lack of knowledge, stigma related to mental health issues, and missing resources are widely recognized barriers to implementing and accessing mental health care [2,5,8].
On a policy level, mental health care's integration into primary care is recommended [4,7]. According to Atilola [5], training community health workers on mental health and psychiatric care topics would also greatly benefit mental health education programs. Thus, psychiatric nurses [9] and primary health-care workers [5] would have a key role in providing mental health treatment and services.
With mental-health services already scarce in developing countries, poor mental health literacy among primary health-care professionals contributes to the disease's burden and the indequate treatment of those in need [7]. The concept of 'mental health literacy' can be defined as 'knowledge and beliefs about mental disorders which aid their recognition, management or prevention' [10]. Previous studies have shown that training health-care workers effectively improves knowledge in mental health literacy [7,11,12], which relieves the burden of mental health disorders [7].
With mental health literacy (MHL), the recognition, knowledge and attitudes [13] toward mental health can be assessed from a professional perspective [12,14,15]. Previous studies lacked an assessment of all MHL attributes, including recognition, knowledge and attitudes towards mental health [13,14]. The new scale-based measure, the Mental Health Literacy Scale (MHLS) includes all MHL attributes. It was originally developed and tested in Australia by O'Connor & Casey [14]. The scale has demonstrated excellent methodological quality in psychometrics for internal consistency, content and structural validity [16] and internal and test-retest reliability [14].
This study aimed to adapt the content validity [17] of the MHLS with researchers and primary health care workers in low-and middle-income contexts in South Africa (SA) and in Zambia.

Study design
In this study, MHLS was validated in SA and in Zambia using a heterogeneous expert-panel method [17]. Research experts and clinical experts evaluated the instrument's content validity in two phases from April to May in 2018.
The study is a part of a larger European Union-funded project: 'MEGA-Building capacity by implementing mhGAP mobile intervention in SADC countries' (funding number 585,827-EPP-1-2017-1-FI-EPPKA2-CBHE-JP) [18]. In the MEGA project, primary health-care workers will be trained to screen young and adolescents' mental health problems using a new mobile application.

Study setting and population
The study population comprised two expert panels (N = 21) divided into clinical experts (primary healthcare workers) and professional research experts (MEGA project researchers) [17]. Ten clinical experts (CEs) (n = 10) from a primary health clinic in Lusaka, Zambia were recruited by The University of Zambia in April, 2018. The CEs were invited to secure a cultural understanding of mental health in low-and middleincome contexts in Zambia. Currently, screening for mental health issues among young and adolescent is over-burdened as five psychiatrists are responsible for approximately 13 million people [19]. Thus, screening and care is provided mainly by specially trained nurses [9,18]. In this study, the participating primary healthcare workers had to meet adolescents regularly in their clinical practice. Eleven (n = 11) professional research experts (PREs) from the MEGA project's management team were recruited to secure a theoretical understanding of mental health literacy. PREs (i.e psychiatrists, psychologists and psychotherapists) had scientific backgrounds and a long-term understanding of young and adolescent mental health in low-and middle-income contexts.

Data collection
The first expert panel, consisting of PREs, was organized in University of Free State, South Africa in April, 2018. The second expert panel, with the CEs, was held in the local hospital in Lusaka, Zambia in May, 2018. The investigator informed participants on the expert panels via an informative letter. Every participant was asked for a written informed consent prior to the study. Participating in the panel was voluntary. After the informed consent procedure, participants were asked to rate each of the 35 MHLS items on a 4-point scale [17,20] with 1 as 'not relevant,' 2 as 'unable to assess relevance without revision of the item,' 3 as 'relevant but needs minor alteration' and 4 as 'relevant to the process of measurement of the MHL.' After the rating, all 35 MHLS items were carefully discussed within the expert panel and evaluated according their relevance. The instrument went through each item and experts could state whether an item is relevant or not or if it needs revising. At the suggestion of the original MHLS author to meet given changes in DSM-5, the terms of the disorders were modified on two scale items before the data collection.
At the beginning of the expert panel, participants were asked for background information of country, working region and experience, age, gender, education and current profession. Both discussions in expert panel were audio recorded for transcription and analyses. The researcher J. Korhonen was responsible for leading discussions, and the other researcher took field notes about the discussion to support analyses [21]. A total of 23 transcribed pages (A4, Times New Roman, font size 12 and single line spacing) consisting of the field notes (8 pages) and three hours of audio recordings were analyzed with narrative and thematic analyses [22,23].

Data analysis
The data were analyzed using an item-level content validity index (I-CVI) [20,24]. A CVI is commonly used when deciding whether to delete, revise or retain research scale items [20]. I-CVIs for (1) PREs (I-CVI-PRE, n = 11), (2) CEs (I-CVI-CE, n = 10) and (3) the total group of experts (I-CVI-ALL, n = 21) were calculated for each of the instrument's 35 items. On a 4-point scale, the best overall item score is 1.00. A cutoff point (≥0.8) was used in analyzing the relevance (Schilling et al. 2008). If the I-CVI-PRE of an item met the cutoff of ≥0.8 but I-CVI-ALL did not, the item was evaluated critically, and a decision was made to retain, eliminate or revise the item based on experts' suggestions and ratings during the expert panel discussions [17]. The item rating (1 to 4) was signed as '*' if it was unclearly marked by the experts. The I-CVI-Clinical was reported separately only for a better understanding of phenomena in the research. The final decision was made according to the PREs' suggestion. The average CVIs for the entire scale (S-CVI/Ave, 35 items) were calculated for both expert panels with a cutoff criteria of (≥0.9) for entire scales [17,20].

Characteristics of participants
In line with the protocol, both PRE and CE experts, totaling 21, participated in the study (N = 21). Eleven (n = 11) of these were PREs, and ten (n = 10) were CEs, representing the target population of the main research. Female and male experts were involved from four (n = 4) different countries. Participants' backgrounds in professional education varied from a certificate to a PhD. When asked for work titles or current professions, PREs had both clinical and research backgrounds. None of the CEs reported research background in a professional health category. The majority (n = 8) of all PREs reported a work experience of 15 years or more, while half (n = 5) of the 10 CEs reported their working experience as up to 5 years. A full demography of participants is presented in Table 1.

Content validity of the MHLS
The item-level content validity index (I-CVI) for all 35 MHLS items was rated by eleven (n = 11) PREs and by ten (n = 10) CEs. The I-CVIs for the 35 items ranked by the PREs varied from 0.82 to 1.00, and the CEs' rankings ranged from 0.1 to 1.00. Among PREs, all the MHLS items (n = 35) met the cutoff criteria of ≥0.8 for the I-CVI. The average of the I-CVIs (S-CVI/ Ave) for all items on the scale within PREs was 0.95, meeting the desired cutoff criteria (≥0.9). The 35 item ratings by PREs on a 4-point relevance scale (Polit & Beck 2006) are shown in Table 2.
Among CEs, ten (n = 10) items out of all I-CVI items (n = 35) met the cutoff criteria (≥0.8). The average I-CVI (S-CVI/Ave) for all items on the scale rated by CEs was 0.62. When calculating mean validity index for the both expert groups, PREs and CEs of experts, S-CVI/Ave was 0.8. Ratings on 35 items by CEs on a 4-point relevance scale (Polit & Beck 2006) are shown in Table 3.

Suggestions for relevance and clarity of MHLS in expert panel discussions
Expert panel discussions were held after both ratings. In the PREs' panel (n = 11), twenty-nine (n = 29) out of all (n = 35) items were considered relevant. In four cases, items were considered relevant with minor alteration (n = 4), and two (n = 2) items were considered unclear. In the expert panel discussion for CEs (n = 10), twenty-six (n = 26) items out of all (n = 35) items were seen as relevant, and six (n = 6) were considered relevant with minor alteration. Three (n = 3) items were considered unclear.
Following both expert panels, and based on the results of ratings and discussion, a group of sixteen (n = 16) out of all items (n = 35) were retained as original without review. A total of nineteen (n = 19) items were reviewed. Eleven (n = 11) of these reviewed (n = 19) items were modified, but eight (n = 8) items (4,12,21,26,27,28,30 and 33) were retained as original after careful consideration by the researchers. Five (n = 5) (4, 12, 27, 28 and 33) of these eight (n = 8) retained items were not modified as there was no specific suggestion by the experts; PREs rated and CEs discussed these items as relevant. For three (21,26 and 30) of these eight (n = 8) reviewed but retained items, the comments reflected CEs' attitudes and knowledge toward mental health illnesses rather than the evaluation of the items' relevance, so they were retained as original. Three (n = 3) items (6,7 and 14) that met the cutoff point of 0.8 (I-CVI/ PRE, I-CVI/ALL) were modified for better conceptual clarity, as suggested during the expert panel discussions. Both groups agreed that the MHLS is relevant overall for measuring mental health literacy among primary health-care workers. The ratings, consensuses and rationales of expert panels for 35 items are shown on Table 4.

Discussion
Our purpose was to explore the content validity [17] of the MHLS developed by O'Connor & Casey [14] in lowand middle-income contexts using a heterogeneous expert panel. Mental-health literacy is widely recognized as a key concept for better mental-health knowledge among researchers [7,11,12]. As stated before, primary health-care workers have a key frontline position to render mental-health service [5], but many of them still have an insufficient understanding of mental health [25,26]. The MHLS has shown excellent methodological validity in previous literature, and this study found that the MHLS also has sufficient validity in lowand middle-income contexts. Nevertheless, a gap between professional research experts' and clinical experts' knowledge on mental health was recognized. For the first time, this study explored the content validity of the MHLS in this context with two expert panels.
As discussed in previous studies [17], using a heterogeneous panel with experiential experts and professional research experts is fairly uncommon. By using CEs in this study, the researcher wanted to hear CEs' voices on improving the context-specific relevance of the MHLS. However, PREs' and CEs' opinions differed remarkably in the study. All 35 items ranked by the PREs met the cut-off criteria (≥0.8), but only ten (n = 10) items were seen as relevant by CEs when calculating I-CVIs. This reveals that CEs were not familiar with this method or did not have adequate knowledge to evaluate the MHLS. During the expert panel discussion, however, CEs reached a consensus of relevance (relevant or relevant with minor changes) in 32 out of the 35 total items. In addition, a validity index for the overall scale (S-CVI/ Ave) rated by PREs easily met the preferred criterion, but CEs' mean rating for the scale stayed well below the cutoff line. However, even differences between the two groups were notable. After the expert panel, only eleven (n = 11) MHLS items were modified. As a result of the expert panel discussion, CEs agreed  CEs had more difficulties in separating the rating of the items' relevance from answering the mental-health questions on the scale itself. During the panel, the CEs were obviously reflecting their own knowledge and attitudes toward mental-health topics. This strengthens previous findings of primary health-care workers' insufficient mental-health knowledge and negative attitudes toward mental-health issues [25,26]. As an explanatory factor, Kapungwe et al. [26] reported that most of their Zambian study population was young (from 25 to 45 years old). Findings also revealed relatively young ages and work experience within the CE group. The demography showed that the half (=5) of participating CEs (n = 10) had work experience of up to five years, and none had done master's level studies. However, most PREs had 15 years or more experience in the mental-health field. This finding again supports previous studies [7,9,27] on the need to train primary health-care workers on mental-health issues in LMICS. This also aligns with the country's Mental Health and Poverty Project (MHaPP) report [28] stating that mental-health workers have not received training in human rights nor any refresher courses in the last five years in Zambia. This should be considered when training the local primary-care workers in the future.

Strengths and limitations
This study has several methodological limitations that must be considered when interpreting the results. Firstly, the expert panel discussion was key when deciding to retain or revise the MHLS items. During the expert panel discussions, the attitudes of CEs were strongly represented in the mental health topics' terms. This obviously affected the experts' decisions and reasoning. Researchers needed sensitivity in understanding an accurate consensus, even if some participants were reflecting their own attitudes or knowledge rather than evaluating the item's relevance, which was this study's purpose. For better reliability in interpreting the results, the authors suggest monitoring expert panels with a co-researcher with field notes and audio recordings.
Secondly, the study included a fairly big group of experts in the both groups, which isn't necessary when using a CVI with the expert panel method [17,20].The more experts involved in study, the more complex total   (Continued ) (Continued )  [20]. Using focus groups in the expert panels may especially help to understand the phenomenon as common in explanatory studies, but may also lead to different interpretations with different group sizes. This can partly be avoided by following a suggestion by Polit & Beck [20] to use a more relaxed calculation, such as S-CVI/Ave, for a content-validity index instead of analyzing experts' individual behaviors. Finally, this study did not involve re-study phase [20] or explore the effect of revising and modifying MHLS items for content validity. Nevertheless, the final decision of whether to retain the item was based on PREs' ratings. However, CEs' suggestions for revising items regarding the relevance of the scale were carefully considered in the expert panel discussions. In the future, there is a need for psychometric testing of the validated MHLS version in LMICs' contexts.

Conclusion
Using expert panels can be a useful method to culturally and contextually validate an instrument in LMIC settings. The MHLS seems to hold a strong content validation in South African and Zambian contexts. However, more evidence is needed to show the reliability of the MHLS in the previously mentioned context. Therefore, more research needs to be done among primary health workers to show their level of understanding related to MHL. Currently, studies reporting all the Mental Health Literacy components in LMICs are lacking, especially from a professional perspective. More studies are needed to explore and to fill this gap.