Effect of e-learning on health sciences education: a protocol for systematic review and meta-analysis

ABSTRACT E-learning has been widely used in higher education as it provides better access to learning resources online, utilising technology to enhance learning. Despite growing evidence claiming that e-learning is as effective as traditional means of learning, the evidence is still very limited. This protocol aims to measure the impact of e-learning as compared to traditional face-to-face learning, both measured and perceived, on health sciences education – in terms of improving students’/health professionals’ satisfaction, knowledge, skills, and behaviours and patient-related outcomes. We will conduct a systematic review and meta-analysis of both randomised controlled trials and non-randomised controlled trials. Major databases will be searched for studies, and will be reported in accordance with PRISMA. A thematic analysis will be conducted for the included studies. If sufficient data are available, the random-effects model for meta-analysis will be performed. The outcome of this study will provide a basis for developing the best methods of e-learning in health sciences education.


Introduction
There are different meanings or interpretations of e-learning, but employing technology to provide online access to learning resources for the improvement of learning is the principal aspect of e-learning (Holmes & Gardner, 2006;Sandars, 2013). E-learning has been defined as an educational method that facilitates learning by the application of information technology and communication (Golband, Hosseini, Mojtahedzadeh, Mirhosseini, & Bigdeli, 2014). Recently, e-learning has been well recognised in medical education and other healthcare disciplines including dental, public health, nursing, and other allied healthcare education, which this protocol refers to as health sciences education (HSE). HSE includes both students (undergraduate and postgraduate) and health professionals (e.g. medical doctors, nurses, dentists, public health practitioners, physiotherapists and radiographers). The ultimate intent of all health professions' education

Aim and objectives
The aim of the proposed research is to establish, through the available literature, accurate estimates of the impact of e-learning on health sciences education. The specific objectives to achieve this are two-fold: Firstly, to measure the impact of e-learning as compared to traditional face-to-face learning on health sciences education; secondly, to identify the barriers and enablers of e-learning, and synthesise how these barriers and enablers affect the success of e-learning in improving knowledge, skills competence, and behaviour of students/health professionals and patient-related outcomes on HSE.

Review question
The systematic review study has been guided by the following research question: 'What has been the impact of e-learning as compared to traditional face-to-face learning, both measured and perceived, on health sciences education -in terms of (a) improving students'/health professionals' (i) satisfaction (reaction); (ii) knowledge; (iii) skills); and (iv) behaviours and patient effects, and (b) patient-related outcomes?' The following participant-intervention-control-outcome and types of study (PICOT) framework outlines will be used to plan the research ( Table 1). The process of formulating the review question and PICO framework is iterative (Booth, 2019).

Methods and designs
This research study will utilise an SR and MA method considering both randomised controlled trials and non-randomised controlled trials (prospective and retrospective observational studies) of good-quality studies. SR aims to 'evaluate and interpret all available research evidence relevant to a particular question . . . This differs from a traditional review in which previous work is described but not systematically identified, assessed for quality and synthesised' (National Health and Medical Research Council, 1999, p. 2). MA includes the statistical analysis for combining the results of a number of individual studies to produce summary results, e.g. pooled research studies (Khan, Kunz, Kleijnen, & Antes, 2011). This protocol has been developed in accordance with the recommendation from the Cochrane Collaboration (Higgins & Green, 2008). The Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols Table 1. PICOT framework for a systematic review.

Population
Learners/students (medical, nursing, public health, dental and allied health courses), health professionals (doctors, nurses and allied health professionals) and patients/health service users Intervention E-learning Comparison Traditional, face-to-face learning or in-class teaching Outcomes Learning effectiveness, measured (objective and/or subjective) in terms of (a) improving students'/health professionals' satisfaction (reaction), knowledge, skills and behaviours, and (b) patient-related outcomes Types of study to be included Primary study designs, both randomised and non-randomised trials (PRISMA-P) checklist has been used in the preparation of this protocol (Shamseer et al., 2015). (Additional file 1).

Inclusion and exclusion criteria
It has been argued that inclusion and exclusion criteria for any research should be set in line with the proposed aim and objectives, ensuring that peer-reviewed articles will not only be drawn within the context and boundaries of the research objectives, but also will be structured in a way that would enable the researcher to answer the proposed study objectives (Centre for Reviews and Dissemination, 2008). The inclusion and exclusion criteria for this study have been provided.

Inclusion criteria
Types of participant: We will include studies of learners/students (undergraduate and postgraduate; medical, nursing, public health, dental and allied health courses), practitioners/professionals (doctors, nurses and allied health professionals) of health sciences education, and patients/health service users.

Types of intervention
To assess the impact, studies will compare and measure the effects of e-learning or online learning versus traditional face-to-face (class) learning on health sciences education. Any interventions, i.e. computer technologies used to facilitate the provision of online access to learning resources for the improvement of learning would be considered, e.g. e-learning, online learning, online instruction, distance learning, distance teaching or computerassisted instruction.

Types of control/comparator
Traditional, face-to-face learning or in-class teaching (any interventions that are not distributed by the internet).

Types of outcome measure
The primary outcome of interest is improving students'/health professionals' (a) satisfaction (reaction) -learners' reported satisfaction with the course; (b) knowledge -subjective (e.g. learners' self-report) or objective (e.g. MCQs test) assessments of factual/conceptual understanding; (c) skills -subjective (e.g. learners' self-report) or objective (e.g. faculty ratings, or objective tests of clinical skills such as interpretation of clinical reports) assessments of learners' ability to demonstrate procedure tasks or techniques (endoscopy/endotracheal intubation); and (d) behaviours and patient effects -subjective (e.g. learners' selfreport) or objective (e.g. clinical chart audit) assessments of behaviours in practice (such as test ordering) or effects on patients (such as medical errors). These outcomes would also measure their competencies in practice using practice-based skills logs/checklists. In this review, healthcare students' and health professionals' skills and knowledge are related to the clinical or healthcare competence dimensions in terms of the concepts of (a) knowledgefactual or conceptual understanding ('know') and (b) skills -ability to demonstrate a procedure or technique ('know-how') (Miller, 1990). Secondary outcomes include patient-related outcomes, e.g. patients' preference for treatment or testing, patients' compliance with health professionals' recommendations, patient counselling and change in patients' social behaviour, e.g. avoiding crowds, healthy diets/physical exercise and effect on weight change, quality of life; and mortality and morbidity (survival to discharge).

Types of studies
To assess the impact of e-learning on health sciences education, the review considered all primary study designs, both randomised controlled trials and non-randomised trials (prospective and retrospective observational studies), evaluating the effectiveness of e-learning on HSE.

Exclusion criteria
The following studies will be excluded: • All studies published in secondary, non-empirical studies.
• Studies containing duplicate datasets, studies including commentaries, review documents, case studies, letters, discussion papers, posters, conference abstracts, congress reports that do not provide enough methodological details to be able to analyse the risk of bias. • Studies falling outside of e-learning on HSE, and studies not available in full text.

Search strategies
A systematic searching (SS) approach will be used in this review. SS 'involves applying a clear rationale to seek out the best evidence to address a research question' and aims to obtain research that is either a comprehensive collection, or a representative sample, of the available evidence (Stansfield, 2019, p. 52). A broad search strategy, therefore, has been designed to maximise the level of sensitivity to identify potential studies, and specificity to identify definitely relevant studies in searching (Higgins et al., 2019), and improve both the recall and precision ratios (Katcher, 2006). SS involving major databases -MEDLINE, EMBASE, Allied & Complementary Medicine, DH-DATA, PsycINFO, CINAHL, Global Health, BREI (British Education Index), AEI (Australian Education Index), Web of Science, CENTRAL, ERIC and Google Scholar -will be carried out for studies from inception up to 31 October 2020, and will be reported in accordance with PRISMA. We will not add a study design filter and there will be no language restriction.
We, however, include grey literature. Grey literatures are not formally published in sources such as books or journals but they may be in print or electronic format, and sources include government reports, technical reports, theses and conference proceedings. These unpublished data would be maintained by regulatory agencies, stored in trial registers or owned by industry or individual researchers (Kugley & Epstein, 2019;Lefebvre, Manheimer, & Glanville, 2011). There is little focus on how to appropriately and reliably select, appraise and integrate unpublished data with traditional research, e.g. systematic reviews and evidence-based studies, as there are no recommendations for the transparent, standard reporting of grey literatures and unpublished data (Kugley & Epstein, 2019).
Nevertheless, these literatures would be an important source of evidence contributing to evidence synthesis. Primary search terms are e-learning (synonyms and variants) and health sciences education (synonyms and variants) using 'free-text searching' -searching for a word or phrase appearing anywhere in the document, where the document is the citation (article title, journal name, author), not the full text of an article, and 'thesaurus (MeSH, EMTREE) searching', employing Boolean operators or logic to find the balance between recall and precision (National Institute for Health and Care Excellence, 2014). To maximise the sensitivity, key terms for one, MEDLINE, have been designed as shown in Table 2.
The 'Related Articles' feature in PubMed will be consulted. Searches will also be supplemented by reviewing the reference lists ('references of references') of selected articles to find any other relevant papers. A hand search of the included studies' references and relevant review studies will be completed to achieve literature saturation. We will also ask subject experts/information specialists from both the universities' libraries (Bedfordshire and Dundee) to verify the research strategy, ensuring its comprehensiveness. In addition, we will be using two elements to test the proposed search strategy. Firstly, using the test papers to validate the results using Hausner, Waffenschmidt, Kaiser and Simon's (2012) approach for developing objectively derived search strategies. To do this, we would divide the references into development and validation sets. As an example, we will pick at least one useful review on e-learning in health sciences and if it contained 30 references, we would use 20 of them to develop the search (e.g. see how they have been indexed) and then use 10 to validate the search strategy that has been constructed. Secondly, we will invite at least two senior systematic reviewers or subject librarians (one from each university) to peer-review the search strategy using the Peer Review of Electronic Search Strategies (PRESS) checklist to ensure it meets the basic standards (PRESS, 2016).

Selection of studies
The citations identified will be imported into Refworks software (https://www.refworks. com/). All studies emerging from the databases, snowballing and hand-searching will be search terms will be modified as needed for use in other databases screened twice: (a) screening of titles and abstracts using the screening guidelines suggested by Polanin et al. (2019) to make the process more clear and transparent, by two reviewers (KR, LJ) against inclusion criteria, and (b) review of full text of the included articles using the JBI Critical Appraisal Checklists (Tufanaru, Munn, Aromataris, Campbell, & Hopp, 2020). The intent of the two-stage approach is to gain efficiency without risking exclusion of potentially relevant, high-quality studies of online learning effects (Means et al., 2010;Suess, Pérez, Azarola, & Cerdà, 2014). We will use the standard PRISMA flow diagram to provide the study selection process (Moher, Liberati, Tetzlaff, & Altman, 2009).

Data extraction
Data from eligible studies will be extracted independently by two reviewers (KR, LJ) using the guidelines produced by the Cochrane Group (https://ph.cochrane.org/reviewauthors) to record (including publication, country of origin, funding source) study eligibility (study characteristics, participants, types of intervention, duration of intervention, types of outcome measures), and study details (aim of intervention, aim of study, methods, results). Summary tables would improve the process of transparency and reproducibility by better understanding what sorts of data were extracted from which studies, and also by recognising the contribution made by each study to the overall synthesis (Rodgers et al., 2009). In addition, such tables will demonstrate how the individual study area contributes to the reviewers' final conclusion.

Quality assessment
To assess the methodological quality of the included studies, we will utilise the Joanna Briggs Institute (JBI) quality checklists for both randomised controlled trials and nonrandomised controlled trials (prospective and retrospective observational studies) to assess the methodological qualities, including the possibility of bias in study design, conduct and analysis (Tufanaru et al., 2020). The results of these appraisals will be used to inform the synthesis and interpretation of the study results. These JBI tools have established content validity and have been piloted across all methodologies (Kerins et al., 2018;Pluye, 2015;Souto et al., 2015). The retrieved papers will be independently assessed by two reviewers (KR, LJ) using the standardised 13-item and nine-item critical appraisal checklists for randomised controlled trials and non-randomised trials, respectively. For interrater reliability, we will use the kappa statistics as this 'represents the extent to which the data collected in the study are correct representations of the variables measured' (McHugh, 2012, p. 276). In this research, we will also use the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to assess the certainty of the evidence, i.e. risk of bias across studies based on the study limitations, inconsistency of results, indirectness of evidence and imprecision of results (Guyatt et al., 2008). To facilitate comparison of appraisal processes, both reviewers will record the rationale for inclusion or exclusion, and discrepancies will be discussed and resolved by consensus.

Assessment of reporting biases
Publication bias, often called reporting bias and dissemination bias, refers to the concern that studies which report relatively large effects are more likely to be published as compared to studies reporting smaller effects (Borenstein, 2019, p. 155). Similarly, published studies that include multiple outcomes would be more likely to report the outcomes than if they showed statistically significant results (Sterne, Egger, & Moher, 2008). One approach to address the publication bias is to follow the Trim and Fill procedures, i.e. assessing asymmetry or symmetry in the Funnel plot if more than 10 eligible studies are identified. This approach would estimate the extent of bias or estimate of the adjusted effect size (Duval & Tweedie, 2000). We will use this approach while assessing the publication bias in the included studies. Similarly, some evidences report that studies with positive, statistically significant results may be more appealing for publication (Levay & Craven, 2019). Such bias, according to Rosenthal (1979), is called the file drawer effect, which can lead to overestimation of the benefits by underestimating of harms.
Therefore, it is important to 'include data from studies that found an intervention to be ineffective as it is to include data from studies that demonstrate a significant positive effect' (Kugley & Epstein, 2019, p. 98). It is equally interesting to note that in most cases, information on adverse events or negative outcomes may not be published; therefore, to ensure a reliable estimate of adverse events or outcomes, we need to seek and include data from unpublished sources (Golder, Loke, Wright, Norman, & Bland, 2016).

Data analysis
For quantitative data, where possible, we will measure a risk ratio (RR) or odds ratio (OR), absolute risk difference (ARD) for dichotomous/categorical outcome data, and mean difference (MD) or standardised mean difference (SMD) will be calculated for continuous data, with their 95% confidence intervals (CIs) from the data generated by each included study. If sufficient data are available, i.e. identical concerning the impact of e-learning or addressing the same fundamental question, to make an inference to a universe of comparable studies, results from the comparable groups of studies will be pooled into the statistical randomeffects model for meta-analysis to measure the effect size of e-learning on health sciences education or the strengths of relationships using the software Comprehensive Meta-Analysis (CMA, version 3. https://www.meta-analysis.com/pages/new_v3.php?cart=BT2P4569026). The purpose of using a random-effects model in the analysis is 'to incorporate the assumption that the different studies are estimating different, yet related, intervention effects' (Higgins et al., 2019). To assess the heterogeneity of effects, I 2 together with the observed effects (Q-value [Chi 2 statistics], with degrees of freedom) will be used to provide the true effects in the analysis. Q-value is the sum of the squared deviations of all effect sizes from the mean effect size. Generally, this value is on a standardised scale, so that a large deviation gets more weight if the estimate is precise, and less weight if the estimate is imprecise (Borenstein, Hedges, Higgins, & Rothstein, 2009). Confidence interval will be set at 95%.
In fact, I 2 does not tell us how much heterogeneity there is, but it tells what proportion of the observed variance reflects in true effect sizes rather than the sampling error. As such, it provides some context for understanding the forest plot (Borenstein, Higgins, Hedges, & Rothstein, 2017). If I 2 statistics are low (near zero), then most of the variance in the forest plot is due to sampling error. Conversely, if I 2 statistics are very high then most of the variance in the forest plot is due to variance in true effects. If we could somehow plot the variance of true effects, most of the variance would remain (Higgins et al., 2019). The Cochrane Handbook of Systematic Reviews of Interventions (Higgins et al., 2019) provides the following guidance for the interpretation of I 2 statistics: • 0% to 40%: might not be important; • 30% to 60%: may represent moderate heterogeneity*; • 50% to 90%: may represent substantial heterogeneity*; • 75% to 100%: considerable heterogeneity*.
* The importance of the observed value of I 2 depends on (1) magnitude and direction of effects, and (2) strength of evidence for heterogeneity (e.g. p-value from the Chi 2 test, or a confidence interval for I 2 : uncertainty in the value of I 2 is substantial when the number of studies is small).
Where meta-analysis is not possible, a thematic analysis, will be conducted for the included studies. For qualitative data, where meta-synthesis is possible, textual data will be pooled using the JBI Qualitative Assessment and Review Instrument (JBI-QARI) and Narrative, Opinion and Text Assessment and Review Instrument (JBI-NOTARI) (Joanna Briggs Institute, 2014).

Dealing with missing data
In the case of missing or poorly reported eligible studies data, corresponding authors of included studies will be contacted. Evidence suggests that poorly reported items would not only threaten the validity of the findings and conclusions but also does not permit researchers in making evidence-informed decisions in practice (Hoffmann, Erueti, & Glasziou, 2013;Meursinge Reynders, Ladu, & Di Girolamo, 2019).

Sub-group analysis
We will group by the types of e-learning (e.g. enhanced or adjunct learning; face-to-face learning, blended e-learning model; face-to-face and online learning, and pure online or fully-online learning), by students (postgraduate and undergraduate), and by health professionals (e.g. medical, nursing, dental, public health and allied healthcare professionals). Where statistical pooling is not possible, the findings will be presented in narrative form including tables and figures.

Risk of bias assessment
Risk of bias assessment will be examined, as it provides variability among studies (heterogeneity), which could be due to different types of participations, interventions and outcomes. As Whiting et al. (2016) discuss, the Risk of Bias in Systematic Reviews (ROBIS) is a tool for appraising reviews and often asks these questions about the research elements: (i) did the search include an appropriate range of sources for published and unpublished reports? (ii) were methods additional to database used to identify relevant reports? (iii) were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? and (iv) were restrictions on date, publication format, or language appropriate? (p.229).
As Higgins et al. (2019) argue, rigorously conducted studies in the SR would provide more truthful results, and the results from the studies of variable validity would give either false negative or false positive conclusions. Therefore, assessing the risk of bias in all studies in any review is important. In assessing risk, we will create a table with a row for every relevant type of potential bias, and then classify each study on each row as having a low, unclear, or high risk of bias. In this study, the issue of bias will be kept separate from the core analysis -meaning analysis will be performed without worrying about the quality/bias. We will then use the risk of bias table to provide the context for the analysis (Borenstein, 2019). As Borenstein (2019) suggests, 'if the analysis shows a clinically and/or substantially important effect, we will assess the entirety of the evidence by considering the risk of bias as well' (p.326).

Ethics
As this is a protocol for an SR and MA, neither patients nor the public will be directly involved, and ethics approval and consent will not be required either. (2019) argue, 'scientific literature is critical to the documentation and dissemination of academic scholarship and scientific knowledge' (p.96). In line with this, we aim to disseminate the study findings using the following strategies: first, a copy of the final systematic review and meta-analysis protocol will be made available at the libraries of both authors' universities (Dundee and Bedfordshire), so that students and health professionals will be able to gain benefits, largely on how to write a systematic review protocol. Second, we will be published in an academic peer-reviewed journal. Third, an abstract will be presented at suitable national/international medical education or e-learning conferences or workshops.

Discussion
To the best of our knowledge, this is the first SR and MA protocol to enable a relevant analysis on the actual impact of e-learning, trying to measure it as well as the strengths of relations between the observed impact factors. Some evidence suggests that e-learning facilitates the process of learning by supporting the instructional design and delivery mechanisms, including teaching strategies to change the practice. However, how these factors, in fact, would influence examining and synthesising internal, external and contextual factors has not been well researched in the past (Lewis, Cidon, Seto, Chen, & Mahan, 2014). Second, there are some frameworks and models of online education which employ different methods and tools to collect or gather data, and they mostly used pre-and post-test questionnaires or self-assessment checklists (Kirkpatrick & Kirkpatrick, 2006;Levy, 2006;Phillips, J, 2000;Stufflebeam, 2001). Similarly, another model was more focused on motivating learners and e-moderators in general education, but none of them is specifically designed or developed in line with medical education or HSE (Mayes & de Freitas, 2004).
These descriptions have illustrated two points: first, a lot of health sciences (or professional) education e-learning research and evaluation clearly lacks some theoretical perspectives. Second, most of the conceptual and theoretical work is based on neither medical education nor HSE, but several pieces of work look at education generally.
The potential limitations of this study would be: first, that if the retrieved studies were variable in sample size, quality and population, which may be open to bias, and heterogeneity data precludes a meaningful meta-analysis to measure the impact of e-learning on HSE, the findings might warrant generalisation. Second, there will be a small number of suitable articles to identify, appraise and synthesise the existing evidence on barriers and facilitators related to e-learning with different disciplines, i.e. nursing, physiotherapy, public health, allied healthcare sciences, etc. Third, this research is unfunded, and both time and resource might be limited. Finally, only articles published in English will be included in this study.
In light of the identified limitations or challenges, one of the major strengths of this study is to apply transparent methods and approaches for SR and MA so that future researchers, practitioners and academics will be able to reproduce its methodology easily. Grey literatures in this study will be added to provide an accurate, representative and comprehensive profile of the information on the given topic (Kugley & Epstein, 2019). Similarly, the outcome of this review will provide a useful checklist of potential factors to develop an e-learning approach in HSE. This might provide a basis for developing the best methods of e-learning in education so that e-learning policy in education and learning settings in the HSE context could be administered effectively, efficiently and equitably.

Contribution
Both authors worked closely to design and write the manuscript. KR led the methodological aspects of the research design and procedures. All authors have reviewed and approved the final version of the manuscript and given their permission for publication.