Blended SORT-IT for operational research capacity building: the model, its successes and challenges

ABSTRACT The Structured Operational Research Training Initiative (SORT-IT) has been shown to be very effective in strengthening capacity for conducting operational research, publishing in scientific journals and fostering policy and practice change. The ‘classic’ model includes three face-to-face modules during which, respectively, a study protocol, a data analysis plan, and a manuscript are elaborated. Meanwhile, the lectures of the SORT-IT are available online as YouTube videos. Given the availability of this online material and the experiences with online mentorship of the faculty, we piloted a first blended distance/residential SORT-IT. To inform future implementers of our experience with blended operational research courses, we summarize the model, successes, and challenges of this approach in this perspective paper. The blended SORT-IT consisted of an online phase, covering modules 1 and 2, followed by a face-to-face writing module 3. Four out of six participants successfully completed the course, and submitted a manuscript to a peer-reviewed journal within four weeks of completing module 3. A blended approach may make the SORT-IT course more accessible to future participants and may favour the adoption of the course by other institutions, such as national Ministries of Health.


Background
Operational research (OR) is increasingly used to study how interventions with a demonstrated efficacy can be applied in an effective manner in programmatic settings, especially when the disease burden is high, resources are scarce, and difficult choices need to be made. 1 The Structured Operational Research Training Initiative (SORT-IT), is a Global Partnership Initiative coordinated by the Special Programme for Research and Training in Tropical Diseases at the World Health Organization (WHO-TDR) since 2012 and implemented with partners. The SORT-IT is an output-based course, whereby public health programme staff are mentored in OR while working on their own research studies. About 90% of all participants have achieved the pre-defined milestone of completing and submitting a scientific paper to a peerreviewed Journal [1,2].
SORT-IT consists of 3 one-week residential modules that are spread over 9 to 12 months. To successfully complete the course, participants have to achieve four milestones, which include submission of: (1) an OR protocol at the end of module 1; (2) data collection tools and dummy tables after module 2; (3) a complete dataset before module 3; and (4) proof that the study was submitted to a peer-reviewed journal within four weeks of the end of module 3 [3]. The course uses the same standardized lecture material for the many courses that have been run in Africa, Asia, South-Pacific, Western-Europe, Eastern-Europe, and South America. The lectures have been recorded, and are currently available on YouTube (https://www.youtube.com/channel/ UC9ZRuVhbrxJm5xAjUHwo6Hw).
To join the classic SORT-IT course, participants and mentors should be available to join three face-to-face modules of one week each. However, potential participants with many competing responsibilities may have difficulties in blocking off three weeks in their agenda. Moreover, face-to-face modules are more costly for the organizer, due to travel and accommodation expenses, and visa issues may preclude worthy candidates from attending global/regional courses. The costs may also limit the adoption of the SORT-IT course by other institutions, such as national Ministries of Health, who probably could benefit most from OR on programme data. Therefore, a blended approach of distance and residential training, named 'blended SORT-IT', was proposed to explore whether these barriers could CONTACT Tom Decroo tomdecroo2@gmail.com Nationalestraat 155, 2000 Antwerpen, Belgium be overcome, while safeguarding the course's core principles. The SORT-IT staff agreed to provide modules 1 and 2 online. We anticipated a weekly student investment time of eight hours, with one lecture and one assignment per week. But we agreed that module 3 should remain face-to-face as close contact between participants and mentors and dedicated time was thought to be critical for developing and finalizing a scientific paper. The first blended SORT-IT course was piloted by Médecins Sans Frontières (MSF) Luxembourg between October 2015 and April 2016. Six participants associated with MSF were enrolled. As in the classic SORT-IT, mentors had complementary skills, including statistical and editorial experience. We aimed at including participants with a basic knowledge of managing a database and performing descriptive statistics, given that the blended SORT-IT included fewer lectures on these aspects. In this short communication we report how the classic SORT-IT course was adapted to fit a blended approach. In addition, we report the output, challenges encountered, and recommendations for future blended SORT-IT courses. The sources of data that are presented here include the documents that were used during the course implementation and the evaluations by the participants and mentors. Table 1 shows how the building blocks of the classic SORT-IT course were translated to a blended version. Modules 1 and 2 of the classic SORT-IT were provided during the online phase using the same structured approach. Participants and their mentors navigated through the different steps: first, they identified a research question; second, they developed the Introduction, Methods, and Ethics sections of a study protocol; and finally, they created dummy tables and conducted a draft data analysis. Input consisted of lectures shown through YouTube videos and weekly assignments. The course content of modules 1 and 2 of the blended SORT-IT was similar to the classic SORT-IT, except for the lectures on EpiData. As we expected participants to already have an existing database, lectures on the use of EpiData were not included. The face-to-face module of the blended SORT-IT was similar to the module 3 of the classic SORT-IT. After the lectures, participants meet with their mentors, and develop stepwise the protocol, whereby every new version elaborated by the participant is revised by the mentor. Different sections (Objectives, Introduction, Methods) are presented and commented upon during different plenary sessions.

The model
• Step 1: YouTube videos provide an introduction to OR and SORT-IT. As assignment, participants describe barriers they anticipate facing when engaging with their study. review. An example of an Introduction from another study protocol is shared. Participants develop the Introduction of their study protocol with support from their mentors.
• Step 4: YouTube videos explain study designs and terms often used in OR, basic statistics, and an example of a study protocol. Participants develop the Methods of their study protocols with support from their mentors.
• Step 5: YouTube videos explain research ethics, citation and referencing, and authorship rules. Participants add an Ethics section, edit the references, and add authors to the protocol, and then finalize their study protocols with support from their mentors. The protocol is then submitted for ethics review.

Module 2 Provided lectures include
• EpiData software for encoding and analysing data • Data analysis plan After the lectures, participants meet with their mentors, and develop the EpiData database, data analysis plan, and dummy tables/graphs.  Table 2 shows how milestones were slightly adapted to fit the blended SORT-IT. Figure 1 shows the timeline and content of the online and face-toface modules.
Outputs Table 3 shows how five participants who joined the face-to-face writing module scored several key aspects of the course at the end of module 3. Participants were especially satisfied with the availability and support of mentors during the online module:'My mentors were very supportive and helpful in clarifying the objectives and developing the protocol (that was an important educational process). It was a "back and forth" process in which they were both quick and precise.' Overall, participants were satisfied: 'I learnt a lot from my colleagues and mentors. The presenta-  Two weeks after the end of module 2: Submission of data documentation sheet, EpiData triplet files (qes, rec, and chk files) and dummy tables to the module director and course coordinator.
After finishing step 5: Submission of dummy tables/graphs to the mentor and course coordinator.

Milestone 3
Six weeks prior to the start of module 3: Submission of completed datasets and draft analysis to the module 2 facilitators and course coordinator.
Two weeks prior to the start of module 3: Submission of completed datasets and draft analysis to the mentor and course coordinator Milestone 4 Four weeks after the end of module 3: Submission of a paper to a peer-reviewed journal: copy of the submitted paper and electronic confirmation of receipt of the submitted paper by the journal to be sent to both the module director and the course coordinator.   Table 3. Score for six indicators, by five participants, at the end of module 3 of the blended SORT-IT. tions were very useful and straight to the point and practical.' This was reflected in the score of 80.8%. However, participants felt they could have been better prepared during the online module. The use of a central online repository could have improved the oversight during the online module. Moreover, the experience during the face-to-face writing module was rated as very intense. Table 4 summarizes the outputs from the course. One out of six participants dropped out before milestone 1, as he left for a mission and was scheduled to join another SORT-IT course. The remaining five participants managed to develop a protocol (milestone 1), dummy tables (milestone 2), submitted a dataset (milestone 3), and were able to join the writing module 3, although most submitted their datasets late. All participants in the face-to-face writing module completed a draft manuscript at the end of the module. Four participants achieved the fourth and final milestonesubmitting their manuscript to a peer-reviewed journal before the deadline. One participant did not submit a manuscript and thus failed. Her manuscript later served as a dissertation, submitted for obtaining a degree in Master of Public Health. One participant achieved milestone 4, but had to withdraw the paper after submission, because one partner of the multi-centric sites did not agree with the publication of the data. Three manuscripts were published by the end of September 2017, 17 months after completing the course. Table 5 summarizes the challenges encountered during the course, and shows recommendations. The main challenges were related to obtaining and preparing the data before arriving at module 3. Moreover, as some participants were less experienced than expected in managing and analysing data, most of their data cleaning and analysis had to be done during the face-to-face module. This put a lot of pressure on both the participants and mentors. Therefore, future organizers of blended SORT-IT courses should consider placing more emphasis on data analysis during the distance phase or extending the face-to-face writing module by an extra day. Alternatively, they could take up a short course on statistics prior to enrolment on a SORT-IT course.

Discussion
Between October 2015 and April 2016, the operational research course, SORT-IT, was adapted to a blended format consisting of an online phase covering modules 1 and 2, followed by a face-to-face writing module 3. Four out of six participants successfully completed the course, and submitted a manuscript to a peer-reviewed journal within four weeks of completing module 3 (milestone 4).
To assess the performance of SORT-IT courses, ambitious 80-80-80-80 targets are employed. The following four indicators should have a score of 80% or more: (1) the proportion of the total course marks given by the participants; (2) the proportion of enrolled participants who achieved milestone 4; (3) the proportion of submitted papers that were accepted, in press or published within 18 months of first submission; (4) the proportion of participants who completed their projects and provided followup information within 18 months on whether their projects informed policy and practice change [3].
For the evaluation of this course we assessed these indicators. The overall score from the five participants who joined the writing module was 80.8%. But only four out of six (66.7%) participants completed milestone 4 and three out of four submitted manuscripts were published (milestone 3). Hence, the 80-80-80-80 targets were not met, which contrasts with the success of the classic SORT-IT [1,2]. On the other hand, the proportion who successfully published is higher than that of some other OR courses [4]. The output could be improved if lessons learnt from this pilot experience are applied. Future Blended SORT-IT courses may consider a more rigorous selection of participants, taking into consideration data management skills. The pilot Blended SORT-IT was organized for MSF-affiliated participants only, and they were not selected out of a large group of candidates, as is the case for the classic version [3]. To maximize the chance of identifying candidates with the desired profile and thus a successful completion of the SORT-IT course, it may be desirable to offer the possibility of applying to a wider audience [3].
On the other hand, we think that the course achieved one of its important aims: capacity building in OR. The five participants who completed the course developed a scientific manuscript, step by step, addressing the research question they brought to the course. During this process they acquired skills to identify an OR question and write up a protocol and a manuscript. Previous studies have shown that after the course about half of the alumni complete a new research project and about one in three participants joins a SORT IT course as a facilitator, which shows a steadily increasing capacity for OR [5][6][7].
In conclusion, the blended SORT-IT was conceived as a complementary approach to building OR capacity. It may be suitable for participants with a basic knowledge of statistics, and a commitment to achieving the goals of the course, but who Mentors were intensively involved in data cleaning and analyses, especially during the first three days of the writing module.
Provide the course to participants with a basic experience in operational research and data analysis.
Online course platform Emails were used, but were difficult to track in the inboxes of both mentors and participants. As a result, it was not always easy to follow.
To engage participants and mentors more emails than strictly necessary were sent.
Consider Dropbox for the submission of documents, and subsequent versions of study protocols and dummy tables.

Online course material
The YouTube videos were used and were seen as a positive influence. However, the subjects of the presented examples should cover other themes than HIV & TB only. Students felt they could have benefitted from more lectures on statistics and software that can be used for data capture and data analysis.
Not applicable Consider developing YouTube videos covering a broader range of subjects.

Availability of participants and mentors
Mis-match of availability of the participants and mentors delayed development of the study protocols and submission of datasets and dummy tables. Although regular updates and suggested deadlines were communicated, often the participants were not able to meet them.
Deadlines were extended, and applied flexibly The availability of mentors could be organized differently during the online phase. Mentors could take turns, and be available for a specific task involving all participants during a short period, instead of being continuously available for a long period for one participant.
Apply deadlines more strictly. Preparedness for module 3, the writing module Most of the participants arrived at the writing module with less-well-developed datasets than usual for SORT-IT courses. This put extra pressure on participants and mentors to complete the write-up.
Given the delayed submission of datasets, the usual first two days of are unable to attend the classic SORT-IT courses due to time or availability constraints.