Predictive scores failing at identifying psychiatric disabilities following childhood bacterial meningitis calls for revision of current follow-up guidelines

Abstract Backgrounds Psychiatric disabilities affect one in three survivors of bacterial meningitis. Since current guidelines do not recommend psychiatric follow-up in all children, disabilities are often detected late. Identifying children with elevated risk of psychiatric disabilities using predictive scores could be one strategy for detecting psychiatric disabilities without having to conduct psychiatric evaluations in all children. Therefore, we searched for existing predictive scores and later tested five predictive scores’ ability to predict psychiatric disabilities following childhood bacterial meningitis. Methods From an existing dataset, we selected 73 children with bacterial meningitis of whom 22 later developed psychiatric disease and 15 experienced concentration or learning difficulties. Using these, we tested each predictive score’s sensitivity at their cut-off level for predicting psychiatric disease and concentration or learning difficulties using a chi-square test. Furthermore, we performed a receiver operating characteristic curve (ROC) analysis to assert the area under the curve (AUC) as a measure of overall predictive performance. Results The sensitivity of each predictive score’ ranged from 6 to 38% for psychiatric disease and from 8 to 57% for concentration or learning difficulties. In the ROC-analysis, the AUC was 0.59–0.73 and 0.53–0.72, respectively. Conclusions All predictive score failed at identifying children later developing psychiatric disabilities, excluding this as a feasible strategy for detecting psychiatric disabilities. Hence, current guidelines for bacterial meningitis need to be revised to recommend psychiatric evaluations in all children. KEY NOTES Current guidelines not recommending psychiatric evaluations in all children following bacterial meningitis may result in late detection of psychiatric disabilities. We tested predictive scores’ ability to identify children later developing psychiatric disabilities following bacterial meningitis. All predictive score failed at identifying children later developing psychiatric disabilities, excluding this as a feasible strategy. Hence, current guidelines for bacterial meningitis need to be revised to recommend psychiatric evaluations in all children.


Introduction
Psychiatric disabilities affect one in three survivors of bacterial meningitis. Unlike other disabilities, these are more difficult to identify, and delayed diagnosis is common [1]. This calls for better strategies for detecting psychiatric disabilities.
Psychiatric disabilities following childhood bacterial meningitis include increased risk of psychiatric disease, concentration or learning difficulties, as well as reduced quality of life due to lack of energy, anxiety, and social difficulties [1][2][3][4][5][6]. Combined, psychiatric disabilities are reported in up to 30-39% of survivors, making it one of the most common disabilities following childhood bacterial meningitis.
Contrary to well-known complications such as hearing impairment and neurological deficits, delayed diagnosis of psychiatric disabilities is common, with one study reporting a mean duration of 14 years until discovery [1].
Today, current guidelines for bacterial meningitis do not recommend routine follow-up appointments aimed specifically at detecting psychiatric disabilities [7][8][9][10][11]. Given the consequences of undetected psychiatric disabilities [1][2][3][4][5][6], the difficult task of discovering these needs to be addressed. Identifying children with elevated risk of psychiatric disabilities using predictive scores could be one strategy for detecting psychiatric disabilities without having to conduct psychiatric evaluations in all children. Therefore, the aim of this study was to test if existing predictive scores could predict psychiatric disabilities following childhood bacterial meningitis.

Materials and methods
In this retrospective cohort study, we used an existing dataset based on medical records and child health records to evaluate predictive scores' ability to predict psychiatric disabilities following childhood bacterial meningitis.

Dataset
We used a dataset containing 104 validated cases of bacterial meningitis occurring in 1986-2015 in the V€ asterbotten Region of Sweden, previously described in detail elsewhere [1,[12][13][14]. Cases for this dataset was originally identified using a population-based regional diagnosis register and regional laboratory records, and information for each case was obtained by manually systematically reviewing the patients' medical records, using a standardized protocol, from the following clinics within the V€ asterbotten Region: paediatrics, child health, otorhinolaryngology, neurology, neurosurgery, child and adolescent habilitation, rehabilitation, psychiatry, and child and adolescent psychiatry. The information obtained included any pre-existing diseases, clinical presentation at admission to the hospital, all events occurring during the hospital stay, as well as disabilities debuting after discharge. The latter also includes cerebral function graded using the Paediatric Cerebral Performance Category Scale (PCPC) [15] and psychiatric disease defined as having been diagnosed with a psychiatric disease according to the Diagnostic and Statistical Manual of Mental Disorders (DSM).

Selection criteria
For this study, we selected validated cases of bacterial meningitis in children aged 1 month to 17 years. We excluded cases where the child died during the hospital stay and children not residing in the V€ asterbotten Region. To reduce the risk of misinterpretation, we also excluded children with developmental disabilities due to pre-existing diseases. Finally, children having repeated episodes of bacterial meningitis were regarded as one case starting with their first episode of bacterial meningitis and any additional episodes occurring in the same child were excluded.

Outcomes
We used three primary outcomes of events occurring during the observational period all collected from information in the medical records; 1. psychiatric disease of any type diagnosed by a psychiatrist or by a child and adolescent psychiatrist according to the Diagnostic and Statistical Manual of Mental Disorders (DSM) (Supplementary Table S1). 2. concentration or learning difficulties diagnosed at a one of the following clinics: child and adolescent habilitation, rehabilitation, psychiatry, or child and adolescent psychiatry 3. special education started in conjunction with one of the following clinics: child and adolescent habilitation, rehabilitation, psychiatry, or child and adolescent psychiatry

Identification of predictive scores
We conducted a systematic search to identify predictive scores that could be used for predicting disabilities in children with bacterial meningitis. Specifically, we searched in the PubMed database for studies published until 31 December 2020, that included the term 'meningitis' together with either 'predict Ã ' or 'outcome'  Table  S2). Finally, by reviewing reference lists for these publications, another 19 publications were identified and also read in full.
Using this method, we identified 19 different scoring systems. Of these, four had to be excluded due to them being based on specific neurological examinations that had not been performed on the children in our retrospective material and three due to them being specific to patients receiving intensive care. In addition, seven predictive scores were excluded due to them not showing statistically significant discriminatory abilities in previous studies. The remaining five predictive scores were all included in our study; the Aronin Scale, the Herson-Todd Scale, the Meningitis Swedish Survival Scale (MeningiSSS), the Niklasson Scale, and the Simple Luanda Scale [14,[16][17][18]. For these, we retrospectively graded all cases using the individual criteria of each predictive score (Table 1).

Statistics
We performed all statistical analyses in IBM SPSS Version 24 (IBM Corp., Armonk, NY). To compare the predictive scores at their respective cut-off level, we used the chisquare test. In addition, we performed a receiver operating characteristics (ROC) curve analysis to calculate the area under the curve (AUC) for each predictive score as a test of their overall predictive performance. These results were then graded into the following previously validated performance categories; 'Excellent' (AUC ! 0.9), 'Good' (AUC ! 0.8), 'Fair' (AUC ! 0.7), 'Poor' (AUC ! 0.6) and 'Failed' (AUC < 0.6) [19]. When conducting these analyses, cases with missing variables were excluded from analyses of a specific predictive score if more than one criterion were missing in a score based on five criteria or less, or more than two criteria in a score based on more than five criteria.

Results
Of the 104 validated cases of bacterial meningitis in the dataset, 73 cases matched our selection criteria and were thus included in our study ( Table 2). The 31 cases being excluded constituted seven cases where the child died during the hospital stay and 13 cases where the child was not residing in the V€ asterbotten Region and  This figure shows the results of the systematic PubMed search that we conducted to identify predictive scores for our study, and the stepwise process of reviewing these publications. Number of publications for each step is shown in boxes. 1 At this step, reference lists of all 24 publications were reviewed to identify any additional relevant publications. This resulted in an additional 19 publications that were also read in full. None of these were deemed relevant. therefore not possible for us to follow up. In addition, eight cases in children with developmental deficits due to pre-existing diseases were also excluded. Finally, two children had repeated episodes of bacterial meningitis, both due to S. pneumoniae. These two children were only regarded as one case each resulting in a total of three cases, constituting their repeated episodes, being excluded.
During the mean observational period of 19 years and 5 months (Figure 2), a third of all children were diagnosed with some type of psychiatric disease, most commonly anxiety disorders or depression. Furthermore, one in five children experienced concentration or learning difficulties ( Table 2).

Prediction of disabilities using predictive scores
When testing the five predictive scores' ability at identifying children later developing psychiatric disease, only the Aronin Scale with an AUC of 0.72 was graded into the category 'Fair' in the ROC analysis. All other predictive scores were graded either into the category 'Poor' or 'Failed' ( Table 3). The Aronin Scale, together with the MeningiSSS, also had the highest sensitivity at 38%.
For identifying concentration or learning difficulties, the Aronin Scale with an AUC of 0.72 and the Herson-Todd Scale with an AUC of 0.70 were both graded into the category 'Fair' in the ROC analysis, whereas the remaining three predictive scores were all graded into the category 'Failed' (Table 3). Here, the Aronin Scaled had the highest sensitivity at 57%.
Finally, the Aronin Scale had the best result for predicting need of special education, being graded into the category 'Excellent' based on an AUC of 0.92 in the ROC analysis, and a sensitivity at 60%. In the same analysis, the MeningiSSS was graded into the category 'Fair' with an AUC of 0.75, whereas the Herson-Todd Scale and the Simple Luanda Scale were graded into the category 'Poor', and the Niklasson Scale was graded into the category 'Failed'.

Discussion
In this study, we tested if identifying children with elevated risk of psychiatric disabilities using predictive scores could be a feasible strategy for detecting psychiatric disabilities following childhood bacterial meningitis. However, none of the five existing predictive scores that we tested could correctly identify children later developing psychiatric disabilities. This, combined with reports of psychiatric disabilities being discovered first decades afterwards [1] raises concerns. Psychiatric disabilities are nowadays considered to be one of the most important long-term consequences of bacterial meningitis [1][2][3][4][5][6]. As several studies have shown, a delayed diagnosis of psychiatric disease and concentration or learning difficulties can lead to insufficient treatment, ineffective assistance at school, unnecessary suffering, and be a heavy social burden [20,21]. Therefore, early diagnosis is very important to reduce the burden of psychiatric disability [20][21][22][23].
Detecting psychiatric disabilities without specific psychiatric evaluations is difficult and may result in delayed diagnosis [1]. However, these investigations are timeconsuming, costly and may impose a stigma for the individual child and therefore not to be taken lightly. Contrary to guidelines for children suffering from traumatic brain injury or malignancies that recommend neuropsychiatric investigations within 2 years of diagnosis and repeated follow-up appointments until adulthood [24,25], current guidelines for bacterial meningitis do not recommend neither [7][8][9][10][11]. Instead, it is up to each individual doctor to decide if the child is in need of psychiatric evaluation [7][8][9][10][11].
For some conditions, such as pulmonary embolism and certain malignancies, predictive scores have been successfully used to direct treatment and follow-up strategies, minimising suffering in the individual patient as well as conserving resources [26,27]. Based on this, implementing predictive scores to direct long-term follow-up after bacterial meningitis would be desirable. Mainly, identifying children with elevated risk of psychiatric disabilities using predictive scores could be one strategy for detecting psychiatric disabilities without having to conduct psychiatric evaluations in all children.
From previous studies, we identified five existing predictive scores used for risk assessment in cases of bacterial meningitis. These have all shown promising results when used for either guiding treatment strategies, or predicting death or other short-term adverse outcomes [14,[16][17][18]. When we tested their ability to predict psychiatric disease, sensitivity ranged from 6-38% with moderate specificity resulting in none reaching a better grading in the ROC-analysis than the category 'Fair'. Clinical decision rules graded into this category is generally not considered adequate for implementation into clinical practice [19]. Overall, performance was only slightly higher at the task of predicting concentration or learning difficulties, but no predictive score was graded  8 11 This table shows the general features for the 73 patients included in our study, treatment strategies during the hospital stay, short-term outcome and occurrence of disabilities noted during the observation period. All 73 patients were included in all analyses. In the table, the number of affected patients (No.) are presented in the first column, followed by the corresponding occurrence presented in the second column as percentages or as means with standard deviation (SD). a This category included three cases of group B streptococci and one case each of group A streptococci and Escherichia coli. b This category included six cases of aseptic hygromas, and one cases of ischaemic injury. All cases of intracerebral structural injury were diagnosed via brain computed tomography except in one case of aseptic hygroma diagnosed via repeated brain ultrasounds. c This category included four cases of reactive arthritis, four cases of coagulation problems, three cases of respiratory problems, two cases of herpes stomatitis or other peripheral viral infections, two cases of SIADH, one case of kidney failure, one case of osteitis, one cases of skin necrosis, one case of vision impairment, one case of septic arthritis, and one case of hydrocele. d Cerebral function at discharge was retrospectively graded using the Paediatric Cerebral Performance Category Scale (PCPC) [15] where a normal function was defined as PCPC category 1 and disabilities were defined as PCPC categories 2-5. In five patients, these disabilities were previously known and were due to pre-existing diseases.
higher that the category 'Fair' at this task either. Finally, one predictive score, the Aronin Scale, with a sensitivity of 60% for need of special education was graded into the category 'Excellent' at this task. However, as there were only six patients in our study that needed special education, this result may change considerably if validated in a larger patient cohort. Unfortunately, as seen in our study, no predictive score could produce reliable results at the task of identifying children later developing psychiatric disabilities excluding this as a feasible strategy for detecting psychiatric disabilities following childhood bacterial meningitis. This, combined with the previous knowledge of suffering due to undetected psychiatric disabilities, are strong indicators that current guidelines for bacterial meningitis need to be revised to recommend psychiatric evaluations in all children during the follow-up period.

Strengths and weaknesses
The long observational period, enabling disabilities debuting later to also be detected, is the most  This table shows the ability of the five predictive scores at identifying children with bacterial meningitis later developing psychiatric disease, concentration or learning difficulties, or any type of psychiatric disability. First, the number of patients included for each predictive score (No.) are shown, followed by the area under the curve (AUC) obtained from the receiver operating curve characteristics analysis together with the performance category grading (Grade) based on this result [19]. Lastly, each predictive score's sensitivity (Sens.), specificity (Spec.), positive predictive value (PPV) and negative predictive value (NPV) when tested at their cut-off levels using a chi-square test are shown, including the p-value of this analysis (p Cut-Off ). a This category comprised of children with either psychiatric disease or with concentration or learning difficulties.
important strength of this study. In addition, the identification of predictive scores using a systematic search strategy enabling detection of several predictive scores is also a major advantage. This study has its limitations. Mainly, the retrospective study design risks missing minor disabilities, since lack of standardised protocols for clinical examinations means that all information on disabilities relies on them being brought to a healthcare providers' attention. However, since this mostly relates to minor disabilities, we are confident that the long observational period outweighs this disadvantage and that we can be confident in our results. The retrospective study design also means that other factors possibly increasing the risk of psychiatric disabilities besides the episode of bacterial meningitis may have been missed, despite the review of medical records. Finally, the transferability of our results merit discussion. Our study was conducted in a high-income country using similar treatment strategies [7][8][9][10][11] and having similar in-hospital morbidity and mortality results as in other high-income countries [1]. However, the fact that psychiatric conditions are regarded different in different countries, resulting in varying reported occurrence depending on country [1], is a factor that will have to be considered when applying our results in another setting. To reduce the impact of this variation, we have used the diagnoses using DSM which are more robust and vary less between countries. Given all this, we still consider our results to be transferable to other highincome settings to a high extent.

Conclusions
All predictive score failed at identifying children later developing psychiatric disabilities, excluding this as a feasible strategy for detecting psychiatric disabilities. Hence, current guidelines for bacterial meningitis need to be revised to recommend psychiatric evaluations in all children.