Adaptation and validation of the Children’s Surgical Assessment Tool for Rwandan district hospitals

ABSTRACT Background/Aims Paediatric surgical care is a critical component of child health and basic universal health coverage and therefore should be included in comprehensive evaluations of surgical capacity. This study adapted and validated the Children’s Surgical Assessment Tool (CSAT), a tool developed for district and tertiary hospitals in Nigeria to evaluate hospital infrastructure, workforce, service delivery, financing, and training capacity for paediatric surgery, for use in district hospitals in Rwanda. Methods We used a three-round modified Delphi process to adapt the CSAT to the Rwandan context. An expert panel of surgeons, anaesthesiologists, paediatricians, and health systems strengthening experts were invited to participate based on their experience with paediatric surgical or anaesthetic care at district hospitals or with health systems strengthening in the Rwandan context. We used the Content Validity Index to validate the final tool. Results The adapted tool had a final score of 0.84 on the Content Validity Index, indicating a high level of agreement among the expert panel. The final tool comprised 171 items across five domains: facility characteristics, service delivery, workforce, financing, and training/research. Conclusion The adapted CSAT is appropriate for use in district hospitals in Rwanda to evaluate the capacity for paediatric surgery. This study provides a framework for adapting and validating a comprehensive paediatric surgical assessment tool to local contexts in LMICs and used in similar settings in sub-Saharan Africa.


Introduction
Surgical care is an essential component of child health, yet it is often overlooked.Lack of access to surgical care disproportionately affects low-and middle-income countries (LMICs), where children make up to 42% of the population, compared to 25% globally [1].Recent estimates suggest that 61% of surgically treatable deaths in children under 20 years-old occur in LMICs [2].Despite this, only an estimated 8% of the children in LMICs have access to surgical care [3].Developing baseline metrics for paediatric surgical capacity is an important first step in understanding the unique operative, perioperative, and anaesthetic requirements for children that should be considered in efforts to scale up surgical services in LMICs.
To date, there are limited tools to assess paediatric surgery capacity in LMICs.The Pedi-PIPES tool (Paediatric Infrastructure, Process, Outcome and Cost Evaluation Score) is an example of one such tool.While there are strengths to this tool such as the incorporation of patient and family experience, overall it is limited in scope with a primary emphasis on quality and costeffectiveness, and lack of emphasis on training, education, or adequate focus on neonatal conditions [4].Various surgical subspecialties have also begun evaluating paediatric capacity through the development of specialty-specific paediatric Bellwether procedures, or essential procedures to be used as a proxy for access to paediatric subspecialty surgical care [5].However, these lists are specialty specific and only capture a small subset of surgical conditions that may not reflect the full burden of surgical disease in children.
More recently, the Children's Surgical Assessment Tool (CSAT) was developed to assess and guide strengthening of surgical systems based on currently available guidelines.The CSAT aims to identify gaps in facility infrastructure, service delivery, workforce, information management, and financing and has the potential to be adapted to the local context [6].Compared to tools such as the Pedi-PIPES and paediatric Bellwether lists, the CSAT is a more comprehensive tool that covers a broader range of domains [7].
None of the existing tools comprehensively measure resources, outcomes, accessibility, and training [7].The Global Assessment of Pediatric Surgery (GAPS) capacity assessment tool is currently being developed to fill this gap.However, this tool is being designed for paediatric surgical centres.It remains unclear whether this tool will be appropriate for use at district hospitals, which often serve as the primary entry point to surgical care for patients in LMICs.Given the different levels of infrastructure and training available at hospitals, the type of safe and appropriate surgical care differs between a first-level, district hospital and a tertiary referral hospital, and should be evaluated accordingly.
While the CSAT was developed in Nigeria with the potential to be used at a district hospital, the tool did not undergo validation in that setting.Furthermore, local regulations and guidelines around requirements for paediatric anaesthetic and operative care vary by country, leading to differing levels of appropriateness for some of the items listed on the original CSAT.The Global Initiative for Children's Surgery (GICS) recently published an 'Optimal Resources for Children's Surgery (OReCS)' consensus guideline for the optimal provision of paediatric surgical care, which provides recommendations for the infrastructure, training, and equipment required to provide essential paediatric surgical and anaesthetic care to children at each hospital level [8].These guidelines are the first multi-national guidelines to provide hospital-level specific recommendations for paediatric surgical care.
In Rwanda and other LMICs, district hospitals are an important target for scaling up surgical services.One analysis showed that scaling up surgical care at district hospitals in LMICs could avert up to 22% of the surgical deaths at the district hospital level [2].Baseline surgical capacity evaluations should be tailored to reflect the appropriate resources and training for a first-level hospital to ensure efforts to scale surgical capacity are done comprehensively, safely, and sustainably.Increasing access to surgical services at the district level is particularly important for children in Rwanda, with a significant burden of surgically treatable conditions found specifically in rural communities [9][10][11].The goal of this study was to use the OReCS guidelines for first-level hospitals and expert consensus to adapt and validate the CSAT for use at district hospitals in Rwanda.In addition to a starting list for district hospitals in other sub-Saharan African countries, this paper outlines a framework for local adaptation of CSAT for other contexts.

Study setting
The healthcare system in Rwanda is divided into four tiers: community health workers who provide basic healthcare services and education in communities, health centres which provide primary care, district hospitals that provide more specialised care including surgical care, and referral hospitals which offer the highest level of specialised care.There are eight national referral hospitals, including three teaching hospitals (Centre Hospitalier Universitaire Kigali (CHUK) and Rwanda Military Hospital (RMH) in the capital, Kigali, and Centre Hospitalier Universitaire Butare (CHUB) in the Southern Province) and a tertiary hospital in Kigali (King Faisal Hospital) [12].Recently three district hospitals were upgraded to referral hospitals and four district hospitals were upgraded to provincial hospitals to decrease the pressure of demand for services in the national referral hospitals.There is a national-based health insurance scheme which provides health insurance to over 95% of the population.
In 2017 Rwanda took a significant step towards Universal Health Coverage by developing a National Surgical, Anesthetic, and Obstetric Plans (NSOAPs).The Rwandan Ministry of Health conducted an evaluation of their NSOAP and found that the total surgical volume per population in Rwanda including cases performed in tertiary centres was calculated to be 786 procedures per 100,000 population, which is much less than the 5,000 procedures per 100,000 population recommendation [13].A retrospective chart review found a total of 1274 children were operated at CHUK and CHUB hospitals from 2013 to 2014 [14].The most common surgical conditions among paediatric patients were trauma and burn (36.58%), congenital anomaly (23.39%), and surgical infections (14.76%) [14].The total surgery, anaesthesia, and obstetric (SAO) provider density is 1.006 per 100,000 population, which is much below the target 20-40 SAO providers per 100,000 population, and many of these specialists are concentrated in tertiary centres rather than DHs [13].
One of the major goals of the Rwanda NSOAP has been to increase district hospital surgical capacity by prioritising increasing the Ministry of Health budget allocated to SOA care at DHs.There are 36 district hospitals and four provincial hospitals in Rwanda with variable ability to provide surgical care.The 2017 Rwanda NSOAP evaluation found that across the district hospitals there were an average of 1,703 surgical cases performed per hospital.There are just 25 surgical, anaesthetic, and obstetrics (SAO) providers across the 36 district hospitals.While Rwanda's NSOAP acknowledges paediatric surgical care as a subspecialty available at tertiary referral centres, there is no specific provision for it at the district hospital level.This presents a crucial gap that needs to be addressed, given that most children access healthcare services through DHs.The shortage of paediatric surgeons in the country (three fellowship-trained paediatric surgeons), coupled with a high volume of referrals from DHs, can lead to delays in care, resulting in potentially preventable morbidity and mortality.In Rwanda, individuals less than 21 years are twice as likely to have an untreated surgical condition [9].Yet, children in rural Rwanda are much more likely to experience delay in referrals compared to adults with traumatic injuries [15].
The assessment tool was adapted for use at the Ministry of Health district hospitals in rural Rwanda: Rwinkwavu District Hospital and Kirehe District Hospital in Eastern Province, and Butaro District Hospital in Northern Province.These are government-run facilities supported by Partners In Health (PIH), a US-based non-governmental organisation that accompanies the Ministry of Health in Rwanda in strengthening health systems.

Study overview
As a first step, the study team developed a preliminary adaptation of CSAT based on the OReCS guidelines.After the preliminary adaptation was completed, an expert panel was invited to participate in a threeround modified Delphi adaptation to develop consensus on the tool.The final tool was validated using the Content Validity Index (CVI) [16].

Expert panel recruitment
Expert panellists were invited based on experience with paediatric surgery, paediatric anaesthesia, or paediatric operative care at a District Hospital, or health systems strengthening in the local context (i.e.East Africa).Experts were identified by purposive sampling and invited to participate by the study team.To ensure local representation, over 50% of panellists were to be Rwandan experts.The expert panellists did not receive financial compensation for their participation and are included as co-authors in this paper.

Modified delphi method
During the internal preliminary adaptation conducted by the study team, all items on the original CSAT were sorted into three categories (propose to keep, propose to remove, and require additional discussion) based on the OReCS guidelines for a district hospital level facility.In the first round, this proposed adaptation was presented to each member of the expert panel.Individual semi-structured in-person or virtual interviews were conducted with each expert panelist to review the preliminary adaptation based on their personal experience.Feedback was recorded through notes taken during the interview and stored online.After all interviews were complete, feedback from the panellists was aggregated, and expert panellists were invited to attend one of the three optional virtual expert group sessions to discuss the feedback in a group setting.Final feedback from individual interviews and the expert group virtual meetings were used to deem each item as 'retained' from the original tool, 'removed,' and new items added by the expert panel feedback were deemed 'added by expert panel' (Table 2) for the second-round adaptation of the tool.
In the second round, expert panellists were asked to rate the relevance of each item on the adapted tool on a 4-point Likert scale (1 = not relevant at all, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant) through an online survey.Participants were also given the opportunity to provide additional feedback on each section.Items that received a score of one or two were deemed not relevant and items that received a score of three or four were deemed relevant.The proportion of expert panellist who deemed each item as relevant was calculated.Items that received an average score of >0.75 were kept in the tool, and items with a score <0.75 were subjected to a third round of expert review.Expert panellists were able to provide written feedback during Round 2 of the survey.
In the third round, expert panellists were asked to rate each item with a score <0.75 as necessary or not necessary.Items with >50% of the respondents deeming the item as necessary were included in the final tool.

Validation
We used a CVI to validate the tool [16].Each item on the tool was given an item validity score (I-CVI).I-CVI scores were calculated based on the proportion of participants who deemed items that were included on the final tool relevant (proportion of items ranked 3 or 4 on second-round survey, or proportion of items ranked as necessary, if included on the third round).An average validity (S-CVI/Average) was calculated by averaging all I-CVI scores.In alignment with researcher recommendations for minimum acceptable S-CVI Score, a threshold of an S-CVI score ≥ 0.8 was used to establish validity of the tool [17].

Results
Eighteen expert panellists contributed to the adaptation and validation process.Thirteen (72%) were from Rwanda.Table 1 provides breakdown of expert panellists by specialty and country of practice.

Round 1
Sixteen of the (88.9%) semi-structured interviews were conducted with expert panellists.Thirteen (72%) participated in one of the three virtual group discussions.Forty-three new items were added to the tool for further discussion and group evaluation (Table 2).Based on expert panellist feedback, the survey was adapted to differentiate the ability to provide care to two different age groups: patients under 5 years old and patients 5-15 years old given the national regulations requiring a physician anaesthesiologist to be present for any procedure requiring general anaesthesia.To do this, 10 questions on the survey were duplicated and modified to ask about only children <5 years old.The corresponding original questions were modified to ask about children 5-15 years old.

Round 2
Seventeen of the (94%) expert panellists participated in the second-round survey.Of the 181 items included in the second-round CSAT adaptation, 43 items that did not meet the threshold criteria of 0.75 consensuses for inclusion.

Round 3
Seventeen (94%) expert panellists participated in the third round.Out of the 43 items evaluated in the third round, there were 10 items that did not meet a 50% agreement for inclusion on the survey and were therefore removed from the final survey adaptation.

CVI
The I-CVI ranged from 0.53 for 'Does your hospital have a paediatric neurologist' to 0.94 (nine items: total number of paediatric inpatient admissions for children <5 in a year, total number of children's surgical admissions for children <5 in a year, total number of paediatric ED visits for children 5-15 years-old in a year, total number of functional incubators or paediatric warmers, most common reason for referral of paediatric surgical patients 5-15 years-old, how far away is the nearest referral hospital (hours, by car), does your hospital have an anaesthesiologist who can perform sedation, per cent surgery cases that were emergent or urgent (non-elective) cases for children of <5 years-old, and is the WHO Surgical Safety Checklist used in the operating rooms for paediatric patients).A full list of I-CVI scores for each item can be found in Table 2.The average S-CVI for the 171 items included on the final survey was 0.84 (standard deviation = .10).

Discussion
Scaling up basic surgical services at district hospitals has the potential to avert significant morbidity and mortality for children in LMICs.However, there is not yet a context-specific comprehensive tool to   S1).
When compared to existing paediatric surgical assessment tools such as Pedi-PIPES or paediatric Bellwether procedures, our CSAT R stands out as a more comprehensive tool.It includes items related to training and education, as well as an adequate focus on neonatal conditions.However, the original CSAT may overlook specific barriers and gaps that are unique to the context of district hospitals in East Africa.Our tool is specifically tailored to district hospitals and considers their distinct requirements and limitations.For example, we chose to further differentiate surgical care provision by age.In Rwanda, one of the main barriers to providing surgical care for children under 5 years-old is a lack of a physician anaesthesiologist.Non-physician anaesthetists are often the primary anaesthesia providers, particularly in district hospital settings.Given that national regulations require physician anaesthesiologists to provide general anaesthesia for patients under 5 years-old, our expert panel felt that it was important to evaluate barriers to surgical care for these two paediatric age groups separately.Moreover, our tool complements the Adult SAT, enabling district hospital s to conduct a comprehensive baseline capacity assessment for providing surgical care.This study represents the first adaptation and validation of the CSAT for a district hospital setting.A notable strength of our study lies in the robustness of the expert panel involved in developing the tool, with over 70% of panel members from Rwanda, ensuring local expertise and knowledge were incorporated.However, it is important to acknowledge that our tool's specific adaptation for district hospital s in Rwanda may limit its generalisability to other LMICs or different district hospital settings.Nevertheless, given the incorporation of both local and global health systems-level experts, we believe our tool can serve as a foundation for other district hospital s in East Africa.Future studies can follow a similar methodology to adapt the tool to fit their own contexts as needed.
One limitation of the study was conflicting opinions regarding what aspects were essential to assess, considering the balance between tool length and comprehensiveness.Additionally, the relevance of including certain areas in the assessment tool, such as emergency department visits, intensive care units, and pathology services should be carefully considered, focusing only on components pertinent to the district hospital setting.Evaluating both specialists and non-specialist care providers, such as non-physician anaesthetists and operating room nurses who often treat paediatric patients in district hospitals may provide valuable insights into paediatric surgical care.During the implementation of the tool, we received feedback from data collectors, suggesting the incorporation of branch logic to exclude irrelevant items in settings where certain information is not routinely collected by district hospitals.These insights gained during implementation should be considered in future iterations of the tool.
Finally, it is also important to acknowledge the potential for error resulting from expert panellists' providing feedback based on what they perceive as important for paediatric surgical care versus what they believe should be included on the tool (i.e.evaluated at a district hospital level).While efforts were made to clarify this point with each individual expert panellist, future implementations of the tool should provide clear instructions to avoid confusion and enhance the accuracy of the collected data.
In conclusion, this study successfully developed a context-specific paediatric surgery assessment tool for district hospitals in Rwanda.The tool can be expanded for use in other district hospitals, aiming to increase access to basic and essential paediatric surgery capacity appropriate for a firstlevel district hospital.This is particularly important to enhance access to care for children at the local level and reduce the backlog of cases referred to tertiary centres, enabling them to focus on more emergent and higher complexity cases.The adapted tool can serve as a starting point for other district hospital s in LMICs, particularly in neighbouring countries in East Africa.The methodology employed in this study can be mirrored in other studies, tailoring it to their specific contexts and contributing to the overall improvement of paediatric surgical care in district hospitals.
*All panel members from the United States have at least 10 years of experience living and working in Rwanda.**Several panelists represent and have worked for multiple institutions.
What is the highest ASA class of children operated at your institution?(ASAI = healthy patient, ASA II = patient with mild systemic disease, ASA III = patient with severe systemic disease, ASA IV = patient with severe systemic disease that is a constant threat to life)

Table 2 .
(Continued).Items that scored 0.75 or above in Round 2 were retained and not further discussed in Round 3.These are listed as N/A below. *