Improving a web-based employability intervention for work-disabled employees: results of a pilot economic evaluation.

Abstract Purpose The purpose of this study is to improve web-based employability interventions for employees with work-related health problems for both intervention content and study design by means of a pilot economic evaluation. Methods Uptake rate analysis for the intervention elements, cost effectiveness, cost utility and subgroup analyses were conducted to identify potential content-related intervention improvements. Differences in work ability and quality-adjusted life years and overall contribution of resource items to the total costs were assessed. These were used to guide study design improvements. Results Sixty-three participants were a-select allocated to either the intervention (n = 29) or the control (n = 34) group. Uptake regarding the intervention elements ranged between 3% and 70%. Cost-effectiveness and cost-utility analyses resulted in negative effects although higher total costs. Incremental effects were marginal (work ability −0.51; QALY −0.01). Conclusions The web-based tool to enhance employability among work disabled employees requires improvements regarding targeting and intensity; outcome measures selected and collection of cost data. With respect to the studies of disability and rehabilitation, the findings and methods presented in this pilot economic evaluation could guide the assessment of future assistive “e-health” technologies. IMPLICATIONS FOR REHABILITATION The methods presented in this pilot economic evaluation have large potentials to guide the assessment of future assistive e-health technologies addressing work-disabilities. The findings show that the web-based tool requires content related improvements with respect to targeting and intensity to enhance employability among work disabled employees. The findings show that the web-based tool would benefit from improvements related to the study design by more adequately selecting and collecting both outcome measures and cost data. The burden attributable to large-scale studies and implementation issues were prevented as the outcomes of the pilot economic evaluation did not support the implementation of the web-based tool.


Introduction
Since 2000 research has established that sickness absence results in considerable negative effects on work for employees, employers, and society as a whole. The absence from work due to sickness absence constitutes a large economic burden, especially for those on long-term sickness absence. [1,2] Employees with health complaints, chronic illness or disability often worry about the sustainability of their employment. Due to their health complaints, problems can arise at work, primarily with respect to either obtaining work or remaining at work. Franche et al. and Amick et al. in 2002 and, respectively, provided evidence regarding the readiness for return to work and predictors for successful work functioning. [3,4] Both studies found significantly improved self-efficacy, or self-perceived work ability, among supportive organizations that acknowledge the importance of psychosocial management [4] and the interplay among care, workplace, and insurance. [3] Additionally, the importance of improving an employee's self-efficacy to maintain successful work-role functioning and improve work ability has been established. [3][4][5][6] More recently, Hoefsmit et al. [7] found that in order to improve work ability, cooperation between employe and employer is essential. Likewise, employees with higher levels of self-efficacy are better able to stay at work despite their health problems and may even return to work faster after reporting sick. [7] Furthermore, a recently published study showed that return-to-work self-efficacy was a significant predictor of reintegration to work among very long-term sick-listed employees. [8] Thus, improving self-efficacy may empower an employee to handle difficulties in relation to his or her health problems. [9] In recent years, the Internet is being used more often to locate information on a broad range of topics, including work-and health-related issues. Previous research has investigated the reach, compliance, and perceived effectiveness of interactive websites or etherapies aimed at empowering work-disabled employees. However, these interventions occurred either prior to the disability assessment undertaken by insurance physicians [10,11] or focused solely on information provision and helping patients to formulate questions to ask their physicians. [12,13] Because of the increasing number of Internet users, a growing number of interactive interventions are delivered online.
To increase self-perceived work ability (i.e., selfefficacy) among work-disabled employees who experience problems in relation to their employability, a web-based employability intervention was developed. The web-based intervention consisted out of four different intervention components: (i) a knowledge website, (ii) personal advice, (iii) a feedback session, and (iv) a forum. [14] Previously, the effectiveness of a web-based employability intervention was evaluated among unemployed persons. Unfortunately, among an unemployed study population, no overall effects were found and only one intervention component appeared to be effective on the degree of self-control, empowerment, or return to work. [15] Before assessing the impact of a web-based intervention in another study population, i.e., employees at work hampered by health complaints, it was assumed very valuable to first critically assess both the study design and the content of the web-based employability intervention before conducting a full and extended assessment of the web-based tool. Smaller pilot trials can inform about the feasibility to conduct (cost-) effectiveness trials, which can be costly and timeconsuming. [16] This saves valuable resources being unnecessarily spent on large scale studies that may not be feasible.
The purpose of this study is therefore to detect how to improve web-based employability interventions for work-disabled employees for both intervention content and study design by means of a pilot economic evaluation. The aim of this pilot economic evaluation was two folded: (i) To evaluate whether, based on the results of the pilot economic evaluation, the content of the web-based employability intervention requires improvements to achieve widespread implementation. Insight in the uptake of the various intervention elements and the effectiveness in diverse employee groups are therefore required. (ii) To evaluate whether, based on the results of the pilot economic evaluation of the webbased intervention, the study design requires improvements to guide future research.

Materials and methods
The data collection for this pilot economic evaluation was conducted in the Netherlands between June 2013 and March 2014 with a follow-up period of 3 months. The Medical Ethics Committee (METC) was consulted and declared no ethical approval was needed. The research was conducted conforming The Code of Ethics of the World Medical Association (Declaration of Helsinki).
The web-based employability intervention was compared with regular trade union support among participants with work-related health problems. Members of the trade union could call or use the instant messaging service via the website to ask questions about work, health, and disabilities. They were further attended to a union consultant who answered the questions or referred the caller to a disciplined expert. Participants were approached via the participating employees trade union in the public sector. Members of this union predominately worked for privatized government agencies or in the care and welfare sector. Eligible participants were adults aged between 18 and 65 years. Participants either had concerns regarding their ability to continue their work responsibilities, were at a crossroad of calling in sick for a longer period of time, or were absent for more than 6 weeks and were feeling unable to return to work in the near future. Those who were absent from work for more than 2 years were excluded from the study. Advertisements in the trade unions' newsletters targeted the participant group and based on individual's own intrinsic motivation could be recruited into the research. After the completion of the first questionnaire, participants were randomly divided into either the control group or the intervention group. Participants and researchers were not blinded for either the treatment allocation or intervention uptake. In the control group, a reward of E10 was provided to those who completed both questionnaires. In the intervention group, participant received free usage of the web-based employability intervention for three months.

Intervention and control condition
Participants from the control group could contact the participating trade union for regular support (e.g., information on website, telephone contact, etc.) regarding their work-and health related concerns. The control group did not have the access to the web-based employability intervention.
The web-based employability intervention focused on improving the self-efficacy of work-disabled employees by encouraging and motivating them to take control of their own work context and enhance their work ability. The web-based employability intervention focused on teaching employee skills to return to work and/or stay at work (i.e., to optimize reintegration). In particular, the training allowed the participants to ask job and healthrelated questions, as well as guiding communication with relevant stakeholders (e.g., supervisors, colleagues, occupational physicians, medical practitioners, labour experts, job coaches). Specifically, the intervention aimed to ''empower'' employees to access regular, curative, and occupational health care services during their illness, recovery and work reintegration. The four core elements of the intervention were: (i) a knowledge based web-site of current information focusing on work ability in relation to health; (ii) an assessment of the work ability of respondents (more information on the Work Ability Index-test can be derived from the Technical Appendix), followed by a subsequent telephone interview providing respondents with feedback on their results; (iii) personal and technical expert advice via phone and email; and (iv) an online interactive forum on work and health for sharing worker's experiences, questions, and answers with content experts. Using the intervention elements was voluntary. However, a subscription to the knowledge site was needed (http:// www.wijzermetwelder.nl/).
Aim I: evaluating the content of the web-based employability intervention The first aim of this pilot economic evaluation was to consider whom to include and how to target suitable participants for the web-based employability intervention in the future. The intervention aimed to provide an easily accessible method for interested people to ask questions about disability, health and work. Therefore, the intervention was offered to interested members of the included trade union who experience problems in relation to their employability, regardless of their type of problems or disabilities. A number of intervention features were studied. First, the uptake rates for each of the four core elements of the intervention were examined. Additionally, the economic evaluation was conducted for both the complete group of workers and several subgroups. A cost-effectiveness analysis (CEA) and a cost-utility analysis (CUA) were conducted by accounting for all costs, irrespective of who pays or bears them. Subgroup analyses were conducted based on the age (under and above 50 years), sex (males versus females), and working hours (more than 20 h per week). The results of the economic evaluation were expected to provide information on how to better target the intervention and support recommendations regarding compliance and uptake.
Aim II: evaluating the study design of the web-based employability intervention The second aim of this study was to improve the study design of web-based employability interventions by using the results of the pilot economic evaluation to carefully select outcomes and cost-categories that matter both to the employee and to the society as whole.
Therefore, first, a critical assessment was made of the differences in work ability and quality adjusted life years (QALY) among both groups. Work ability was measured via ''The Work Ability Index'' (WAI), by considering both the physical and mental demands of the work. The index helps to identify those that need support in these areas. [17] A point improvement on the WAI denoted a substantial improvement and hence, demonstrated a gained effect (i.e. better work ability). Differences in Quality Adjusted Life Years (QALYs) were calculated using the EuroQol-5D-5 L. [18,19] A crosswalk to the 3level value sets was used to calculate the utility weights based on the EuroQol scores using Dutch tariffs. [20,21] Utilities at baseline and after 3 months follow-up were used to compute QALYs, by means of the area-underthe-curve method. [22] More information on the WAI and the EuroQol-5D-5 L can be found in the Technical Appendix.
Second, in order to weigh the contribution of certain cost-categories (healthcare costs, patient and family costs, and costs in other sectors) to the calculation of the total costs, the utilization frequency of these cost-categories, and the attributed unit costs were evaluated. Questionnaires measuring healthcare resource use were provided to the respondents at baseline and 3 months after the intervention started. Healthcare costs (including primary, secondary, alternative care, hospitalization, and medication), costs to patient and family (including travel-and parking, and informal care costs), and productivity costs (including costs related to absenteeism and presenteeism) were estimated and indexed for the reference year 2013. [23][24][25][26] Costs of the web-based employability intervention included those related to implementing and operating the intervention.

Statistical analysis
Missing data per respondent were replaced by mean imputation using the mean of the series of the appropriate outcome. The analyses required complete cases responses for both baseline and 3 month follow-up measures. Therefore, respondents who were lost to follow-up were excluded from the analyses. Baseline characteristics of both groups were compared using ttests of variances, or by the non-parametric equivalent. That is, Kruskal-Wallis for analysis of continuous variables or 2 tests for categorical comparisons of proportions. The costs of both groups were compared by the nonparametric bootstrapping method using 95% confidence intervals in percentiles. By bootstrapping, samples of the same size as the original data were drawn with replacements from the observed data. An incremental approach was used to compare the employability intervention with regular trade union support. The incremental costs per ''improved'' respondent on the WAI scale and per QALY gained (i.e., the incremental cost-effectiveness ratios (ICER)) were graphically plotted on one of the four quadrants of the cost-effectiveness planes. For this, nonparametric bootstrap re-sampling techniques were, therefore, used. In the North West quadrant, if fewer effects were produced for additional costs, then the intervention was considered ''dominated'' by the control condition. In the South East quadrant, if the intervention generated superior effects (relative to the comparator condition) with fewer costs; the intervention was considered ''dominating'' the control condition. Detailed calculations relevant for the economic evaluation are found in the Technical Appendix.

Participants
Between June 2013 and March 2014, 70 potential participants were eligible for inclusion, provided their informed consent, and were allocated to either the webbased employability intervention (n ¼ 34) or the control (usual care) group (n ¼ 36). In total, 63 participants (29 in the intervention group; 34 in the control group) completed both the baseline and the 3 month followup questionnaires and were included in the analyses. The demographic characteristics of the participants are shown in Table 1. At baseline, more females were allocated to the intervention group. A significant difference was found for mean health scores with participants in the intervention group rating their health lower than the control group. Aside from gender and self-rated health, no significant differences between the two groups with respect to their demographic characteristics were found.

The content of the web-based employability intervention
The uptake rates for each of the four core elements (knowledge website, personal advice, feedback session, and forum) of the intervention were examined and found low. Out of the 29 participants in the intervention group, 50% (n ¼ 15) logged on to the knowledge site, approximately 70% (n ¼ 20) of the respondents participated in a telephone interview after completing the WAI-test to discuss their results, 16 requested personal and technical expert advice, and only one person used the online interactive forum. Its clear compliance and uptake are low. In order to improve the uptake, the costeffectiveness of the complete group and several subgroups could provide insights. First, the economic evaluation outcomes of the complete group of workers with health complaints are presented. Marginally lower scores on the WAI (À0.51) and higher total costs (E483.8) for the intervention group (when compared with the control group) resulted in a negative ICER (D total costs/D total effects) (see also Table 2). The majority of the incremental cost-effect pairs were located in the northwest quadrant indicating fewer effects, albeit higher costs, for the web-based employability intervention ( Figure 1). The 3-month ICER resulted in a large negative incremental cost utility ratio surrounded with large uncertainty which can be seen in the wide distribution of ICERs (Figure 1). Eighty-two percent of the bootstrapped cost utility pairs were situated in the north-west quadrant, representing less effectiveness (i.e., less quality of life gains), and more costs for the intervention group compared with the control group.
Second, the results of the subgroup analyses are presented in the lower level in Table 2. Subgroup analyses can distinct certain groups more or less sensitive for the intervention. Hence, subgroup analyses  In the CEA DE is the mean difference in work ability (WAI score), DC is the mean difference in societal costs. b In the CUA DE is the mean difference in QALYs, DC is the mean difference in societal costs. c NE refers to north-east quadrant of the cost-effectiveness plane, which indicates the intervention is more effective and more costly than the control. SE refers to the south-east quadrant of the cost-effectiveness plane, which indicates the intervention is more effective and less costly than the control. SW refers to south-west quadrant of the cost-effectiveness plane, which indicates the intervention is less effective and less costly than the control. NW refers to north-west quadrant of the cost-effectiveness plane, which indicates the intervention is less effective and more costly than the control.
can provide valuable information on targeting priorities. The CEA for the subgroup respondents aged above 50 years and the subgroup female genders resulted in a minimal increase in work ability (0.03 and 0.05 points improvement on the WAI, respectively), associated with higher costs compared with the respondents in the usual care condition. In order for the intervention to be effective, a point improvement on the WAI was required. However, the results of both subgroup analyses should be interpreted with caution as one cannot state that the web-based employability intervention was more effective although it had higher associated costs. Furthermore, based on the distribution of ICERs on the CE plane, these estimates are not robust. The subgroup of employees working more than 20 h per week resulted in lower work ability in the web-based employability intervention group compared with the control group. There were also higher total costs with a negative ICER of À457.69. It is important to note that the ICER does not fall within the confidence interval (Table 2). These results provide preliminary information on who to target the intervention on. Better targeting can consequently impact the effect of the intervention, and thereby impact the uptake and compliance. Content-wise adjustments derived by adequate targeting might improve the uptake for specific subgroups of employees and thus influence the results.

The study design of the web-based employability intervention
In order to guide a careful selection of appropriate outcomes, first, the differences in work ability and quality adjusted life years (QALY) were assessed among both groups. The mean incremental differences of À0.51 point improvement on work-ability and mean incremental differences of À0.01 QALY were unable to demonstrate an effect of the web-based intervention (Table 2). Second, the contribution of certain cost-categories to the calculation of the total costs showed that participants who received the intervention consulted their physiotherapist and medical specialist more often than participants in the control condition. They used more alternative care and had longer stays in the hospital. The number of visits to the general practitioner and the medical specialist were almost equal in both groups. The number of hours of informal care was larger in the control group when compared with the intervention group. Table 3 presents the uncorrected costs of different healthcare utilization costs after 3 months. As costs were not distributed normally (Table 1), aggregated costs per category (healthcare costs, patient and family costs, costs in other sectors, and societal costs) were bootstrapped. The highest total societal costs were seen in the intervention group (E1707) versus (E1223) in the control group. The relative contribution of patientand-family out-of-pocket costs was smallest; while productivity costs contributed for almost 50% of the total societal costs in both groups (Table 3).

Conclusion
This research demonstrates a number of benefits for undertaking a pilot economic evaluation to provide recommendations regarding the content and the study design of the web-based intervention. Based on the results of this pilot, an attempt is made to bridge the gap between the application of research and a reallife setting. Furthermore, this pilot economic evaluation is unique with respect to the study field of disability and reintegration as the findings and methods presented here could guide the assessment of future assistive ''e-health'' technologies for work-disabled employees.

The content of the web-based employability intervention
The first aim of the study was to consider the uptake rates and critically assess the intervention by conducting a pilot trial based economic evaluation to provide suggestions to achieve widespread implementation. The findings of the pilot economic evaluation do not indicate a widespread implementation of the web-based employability intervention in practice before improvements related to the content and compliance to the intervention are conducted.
First, all four intervention elements need to be improved to increase compliance and uptake rates. In particular, the content of the online interactive forum should be further investigated, by conducting, for example a process evaluation, as uptake of this component was very low. Second, the results of this pilot economic evaluation showed that the web-based employability intervention was not associated with superior clinical or economic impact when compared with regular trade union support. The fact that the main analyses indicated that the intervention is being dominated by the control condition (i.e., more costs and fewer effects as the ICER is located in the inferior quadrant of the cost-effectiveness plane) could also be explained by low compliance and low uptake, hampering the correct interpretation of the impact of the intervention. Based on the economic evaluation results, the subgroup analyses demonstrated a tendency females and participants aged 50 or older might be more sensitive for an intervention effect. Streamlining the intervention content more towards the employee groups which appear more sensitive for an intervention effect, in other words, ''better targeting'' the study population towards the intervention components might strengthen the trustworthiness of the results, and off course the uptake rates.

Improving the study design of the web-based employability intervention
The second aim was to provide insight in how to improve the study design by selecting those outcomes and costcategories that matter both to the employee and to the society. First, in this pilot economic evaluation, no significant improvements were found in effectiveness of the intervention when compared with the control condition. Based on the negligible mean incremental differences, and the inability of the outcome measures to detect an effect, it is unclear whether it is appropriate to assess the true effect of these types of interventions in terms of ''quality of life'' and ''self-efficacy''. Furthermore, only the intervention itself was expected to yield differences in outcomes. However, in practice, it might appear so that participants use information other than via the (web-based employability) intervention or that the intervention might be used in combination with other resources (e.g., social networking, other websites, etc.) which might hamper the correct interpretation of the true effectiveness of the intervention. Second, given the time and effort required to collect cost-effectiveness data, this pilot indicates that the collection of patientand-family costs might be omitted based on their relative contribution to the total costs. Productivity costs on the contrary appear to be essential in the economic evaluation of web-based employability.

Discussion
The study results should be placed in the light of other research conducted within the field of e-health Table 3. Cost prices, mean total costs (SD) and mean bootstrapped costs (95% CI) during the 3-month follow-up (n ¼ 63). and web-based interventions. A recently conducted review summarized the effectiveness (empowermentrelated outcomes) of several web-based interventions. [9] They concluded that when small positive effects on empowerment were found, the duration and the intensity of the interventions were much higher than the web-based employability intervention described here. [9] For clinical problems and the work ability assessment of employees, a long-term time span might have been more appropriate to capture all consequences of the intervention under study. The limited follow-up period (3 months) of this pilot might also mask the visibility of a true effect (e.g. increase in work ability over a longer period of time). However, another study with a longer follow-up period (6 months) was also unable to detect improvements in work functioning. [27] The authors indicated that the reasons for poor implementation of the e-health intervention could be attributed to the fact that the intervention was not fully integrated into the organization before testing its effectiveness. These are similar finding to our pilot study which had low uptake rates.
In addition, although randomized-controlled trials (RCT) are considered the best method to examine the effectiveness of an intervention or treatment, this pilot questions the feasibility of conducting an RCT in a pragmatic setting. The use of web-based employability interventions within a non-disease specific population raises the issue of generalizability of the results versus the applicability of the results in a disease specific population. As mentioned before, no significant effects were found. On the contary, because of the inclusion of many types of work-related disabilities, the generalizability of the results of the RCT is upturned. Considering the challenges of using RCT methods to study the effectiveness of the intervention, a pilot cost-effectiveness study could be accompanied by observational and qualitative studies to explore whether and how to conduct RCTs.

Practical implications and recommendations
Based on the findings, the voluntary usage of all four elements is advised to be reconsidered. Furthermore, methods to stimulate ongoing involvement with webbased employability interventions are suggested to be found and appraised. However, difficulties were experienced when recruiting employees to participate in the study. This might hamper potential efforts to increase participation-and intervention uptake rates. As a result, future interventions may need to specifically consider the current workplace culture, companyspecific policies regarding absenteeism and sick-leave, and the employee's own perceived health problems. Careful evaluations incorporating both quantitative and qualitative methods will be needed to document how the web-based employability intervention can be operationalized and potentially adapted for use in other settings. Next, the added value of the intervention should be made clear as the effects might also depend on personal characteristics of the individual user (e.g., healthy or unhealthy behaviour) and on the usage of alternative information and support resources. Expert opinions or other forms of qualitative research on the appropriate size of quantitatively important distinction between baseline and follow-up might be helpful.

Implications for future research
Before conducting a full-scale study, several key elements need to be improved. At this time, the web-based employability intervention was not associated with an economically relevant impact over regular trade union support for a heterogeneous population with work-and disability-related concerns. However, based on the knowledge gained from this pilot economic evaluation, the administration of data collection as well as deciding upon cost categories and outcome measures can be used to direct future research. A web-based employability intervention should attempt to be easily accessible for interested people to ask questions about disability, health, and work. The web-based tool might also benefit from better ''targeting'' given the heterogeneous population and from revising and enlarging its intensity.