Reliability of FUNDES-Child-SE - measuring participation and independence of children and youths with disabilities

Abstract Background There is a need for an instrument to measure participation and independence in children with disabilities. FUNDES-Child-SE has its origin in the participation questionnaire Child and Adolescent Scale of Participation. Aims Test the psychometric properties of internal consistency and test-retest reliability. Material and methods This cross-sectional study included caregivers of 163 children with disability aged 6–18 years, 59 of whom were also included in the test-retest study. Descriptive statistics were used to evaluate the proportions of valid ratings. Internal consistency and test-retest reliability were tested through Cronbach’s alpha and the intra-class correlation coefficient. Results The amount of not relevant/not applicable ratings was substantial but varied between items and subdomains. Internal consistency was acceptable (0.8–0.95), and the test-retest was marginal to excellent (0.73–0.95). Conclusions The reliability together with the content validity support the use of the FUNDES-Child-SE to measure participation and independence in children with disabilities. However, results should be interpreted with caution due to the small sample size and possible selection bias. Modifications to reduce the not relevant/not applicable responses should be investigated together with the instrument’s responsiveness. Significance FUNDES-Child-SE can be used to facilitate a discussion of participation and independence and to plan interventions in a habilitation setting.


Introduction
Participation in daily activities is known to be important for a child's physical and mental development towards independence and is an important goal in Swedish habilitation services for children and youths [1,2].Therefore, reliable instruments to measure participation and independence are needed.
Several instruments for the measurement of participation in children with disabilities have been developed and used in research [3].Commonly, instruments of participation such as the Child and Adolescent Scale of Participation (CASP), are built on the World Health Organization's International Classification of Functioning, Disability and Health (ICF) definition of participation as 'involvement in life situations' [4,5].The framework Family of Participation-Related Constructs (fPRC) is intended to further define the construct and describes participation as consisting of both attendance and involvement in an activity.Attendance is a prerequisite for participation, whereas involvement may include engagement, which is an internal state involving focus or effort [6].Despite involvement being central to participation, few instruments of participation include the aspect of involvement or engagement [3].
How independently a child can participate in activities can influence both the child's attendance and involvement in the activities [6].Independence is also affected by environmental factors such as the activity setting [7].Planning interventions aimed at optimising the child's participation requires reliable information about attendance, engagement, and independence in different activities and settings [2].
When developing new instruments, it is preferable to make adaptations of existing measures of the same concept rather than create new ones [8].FUNDES-Child-SE is a translated, cross-culturally adapted [9], and further developed version of the Taiwanese instrument, the Functioning Scale of the Disability Evaluation System-Child (FUNDES-Child 7.0).FUNDES-Child 7.0 includes general information, impairments, participation, and environmental factors.The participation section (Part II) of FUNDES-Child 7.0 is based on the Chinese version of the Child and Adolescent Scale of Participation (CASP-C).The development process and adaptations from CASP to FUNDES-Child-SE have previously been described by Axelsson et al. [10].FUNDES-Child-SE only includes part II of FUNDES-Child 7.0 and a dimension of engagement as well as an open-ended question about barriers to independence have been added to each item.FUNDES-Child 7.0 is answered by caregivers through interviews with trained staff, and FUNDES-Child-SE has been responded to by caregivers through a web or paper survey.
Before an instrument can be used for research or in clinical practice, it must be tested for psychometric properties such as reliability, and several aspects of validity.Reliability is of utmost importance in establishing the clinical and scientific utility of an instrument [11].The attainment of valid data serves as a fundamental requirement for ensuring the reliability of the instrument.Reliability, as defined, refers to the instrument's capacity to effectively discriminate among subjects [12] or the extent to which the measurement remains unaffected by measurement error [13].Having a low proportion of missing data and a few items rated as not applicable indicates that the instrument can generate valid data [14].The instrument must also show the stability of results in repeated measures when no change is expected [15].
The internal consistency of CASP has previously been reported by Cronbach's alpha 0.96 [16] and 0.98 [17] and test-retest intra-class correlation (ICC) of 0.86-0.96[16][17][18][19].Cronbach's alpha of CASP-C has been reported as 0.96 [20] and the test-retest ICC of FUNDES-Child 7.0 is 0.85-0.99[21].Considering the new context, cultural adaptations of items, changed rating process, and the added dimension of engagement in FUNDES-Child-SE, there is a need to test the instrument's reliability.

Aim
The aim was to evaluate the reliability of FUNDES-Child-SE.The research questions were: i.How high is the internal consistency of items within the subdomains and dimensions in FUNDES-Child-SE?ii.How high is the test-retest reliability of FUNDES-Child-SE?

Design
This study of FUNDES-Child-SE's psychometrical properties used a cross-sectional study design as a part of the longitudinal research project on participation and mental health in children and youths with disability (CHILD-PMH).The study was approved by the Swedish Ethical Review Authority (no.2019-05028 and no.2017/496-31) and the procedure complied with the Helsinki Declaration of Medical Research [22].Reporting follows the Guidelines for Reporting Reliability and Agreement Studies (GRRAS) [12].

Participants
The participants were caregivers of children with disabilities in six regions in Sweden.The children were from their sixth year of age to the end of their 18 th year of age and were registered at a habilitation centre, meaning they lived with one or more disabilities.
For inclusion in the retest study, the retest had to have been answered within 5 to 36 days after the first survey.

Instrument
Demographic data were collected on the children's age, gender, intellectual and physical disability as well as the participants' gender, education, and whether they needed an interpreter when communicating with authorities and health care services.The 10 Question Screen (TQSI) [23] contains ten binary questions about various physical and mental problems.Four questions from the TQSI were asked to identify the child's physical and intellectual disability; To participate in the home and society (five items).Each item is rated in each of the three dimensions, Frequency of attendance, Engagement, and Independence using a four-rating scale or the response alternative not relevant/not applicable.The definition of the dimensions in FUNDES-Child-SE as well as their numeric ratings are presented in Table 1.The content validity of the cross-cultural adaptation from FUNDES-Child to FUNDES-Child-SE was tested by cognitive interviews with children, youth, and caregivers [10].As a result of these interviews, no items were removed or added.Examples of activities representing the items have been partly modified in FUNDES-Child-SE, e.g.'singing in the choir' has been added and 'parades' has been removed as a result of the content validity testing.Further, 'Not applicable' was changed to 'Not relevant/not applicable' , based on the interviews.Those who had agreed to participate were contacted and given the option of receiving a web survey or a paper survey in Swedish, English, Somali, or Arabic.Surveys were distributed from December 2020 to April 2021.For the web survey, a reminder was sent three times within two weeks.For the paper survey, a reminder was sent within two to six weeks.

Procedure
The retest survey was sent after an interval of 7 to 21 days after the participant had started answering the first web survey or had returned the paper survey.It was possible to pause the answering of the first web survey for up to three weeks after starting it.The interval between the test and retest was intended to minimise the risk of remembering the previous answers and that an actual change had occurred [24].Retest surveys answered within 5-36 days after the first survey were included in the analyses.The same reminder procedure as for the first survey was used for the retest survey, and the participant used the same type of survey (web or paper) in both the test and retest.
The proportion of valid data in FUNDES-Child-SE was analysed with descriptive statistics using the R package summarytools [14].The proportion of missing data and not relevant/not applicable ratings per item was examined, and items with ≥10% missing data or not relevant/not applicable ratings were reported.
As the rating not relevant/not applicable does not belong on the rating scale, the R package dplyr was used to transform not relevant/not applicable to rating 3 (Never do it, Not at all engaged) in the dimensions of Frequency of attendance and Engagement.It was assumed that these response options have the same meaning, i.e. if something is not relevant or applicable it is not done or engaged with.In the dimension of Independence, the rating not relevant/ not applicable was treated as missing data, as it cannot be assumed to have the same meaning as any of the rating scale alternatives.The R package mice and the mice function were used to perform imputation by multiple imputations of chained equations [30].After treatment of not relevant/not applicable ratings, surveys with ≥50% missing data within the three dimensions Frequency of attendance, Engagement, and Independence separately were excluded [31].The R package psych was used to analyse the internal consistency within items in each subdomain and dimension, using the alpha function for Cronbach's alpha.Cronbach's alpha was also examined using the test-retest sample to compare imputed values to non-imputed, as this is recommended after imputation [32].An alpha coefficient of 0.75-0.95was considered acceptable, and an alpha >0.95 indicated redundancy [13].
The R package psych was also used to analyse test-retest reliability on the index of each subdomain and dimension.The intra-class correlation coefficient (ICC) and the 95% confidence intervals were examined using the ICC function, based on a single-rating, absolute-agreement, two-way mixed-effects model [11].The index of subdomains and dimensions was calculated by the sum score of all applicable items (excluding missing data and not relevant/not applicable) in the subdomain and dimension, divided by 3 (the maximum possible score per item) and multiplied by 100 to get an index with a possible range of 0-100.An ICC between the test and retest of less than 0.5 was considered poor, 0.5-0.75marginal, 0.75-0.9good, and >0.9 excellent reliability [11].

Result
Of the 341 who agreed to participate, 200 returned the survey.Of these, 163 remained after exclusion criteria had been applied (child's age reported outside the 6-18 years range [n = 8], 100% missing data in one or more dimensions [n = 29]).For retest, 85 surveys were returned.Of these, 59 remained after exclusion criteria had been applied (response date not within 5-36 days [n = 20], 100% missing data in one or more dimensions [n = 6]).The test sample and retest sample had similar descriptive characteristics, see Table 2 for details.
In the subdomains To participate at school and To participate in the home and society, all items had ≥10% not relevant/not applicable ratings.The subdomain To participate in the home and society had the highest amount of not relevant/not applicable ratings (12.3-42.3%).Two items, Using pedagogical materials and equipment that are also available for other students or that are adapted for you/your child and Communicating with other students and adults at the school had missing ratings of ≥ 10% in all dimensions.The items Using transportation in order to move around in society and Work and responsibilities had <60% valid ratings in all dimensions.In Table 3, items with a proportion of ≥10% missing or not relevant/not applicable ratings are presented, together with the proportion of the items' valid ratings (a rating of 0-3).

Internal consistency
The number of surveys included in the analysis of internal consistency was 154 (Frequency of attendance), 152 (Engagement), and 139 (Independence).The proportion of missing data was 2.6% in Frequency of attendance, 3.7% in Engagement, and 4.1% in Independence.After the transformation, the proportion of missing data in Independence was 15.6%.
Items had acceptable internal consistency within the subdomains (alpha 0.8-0.92)and within the dimensions (alpha 0.94-0.96)(Table 4).The lowest alpha correlation coefficients (0.8-0.87) were found in the subdomain To participate in the home and society and the highest (0.91-0.92) in the subdomain To participate at school.Internal consistency was stable (differences of 0.00-0.07)when comparing the retest sample to the full sample (Table 4).

Test-retest reliability
Intra-class correlation coefficients (ICCs) of test-retest ranged from 0.73-0.9within the subdomains and 0.85-0.95 in the dimensions.ICCs of test-retest are shown in Table 4.

Discussion
This study aimed to evaluate the reliability of FUNDES-Child-SE.The main findings are that the FUNDES-Child-SE has shown a high (>10%) proportion of missing and not relevant/not applicable ratings for items in two of four subdomains, an acceptable internal consistency (alpha 0.8-0.96),and a good to excellent test-retest reliability (ICC 0.73-0.95).
The high proportion of missing and not applicable ratings, especially in the subdomains To participate at school and To participate in the home and society, could have several explanations.Unlike the instructions for CASP [33] and FUNDES-Child 7.0, FUNDES-Child-SE did not explain the intended use of the response alternative not applicable in the instructions.Instead, not relevant had been added to the response alternative, i.e.Not relevant/not applicable.Some items could be irrelevant due to the child's age, such as young children not being expected to do shopping or manage money.The impact of age on not relevant/not applicable items was confirmed in post hoc analyses (see Supplementary Tables 1-3).Also, items could be irrelevant due to e.g.lack of public transportation when living in the countryside.The low proportion of valid ratings for items in the subdimension To participate in the home and society (47.2-80.4%)and especially item 20, Work and responsibilities (50.3-55.8%),was also found in CASP [16].In CASP, 35% of the younger children had applicable ratings for that item, and the other four items in the subdimension were applicable to half their sample of 60 children aged 3-22 years with various disabilities.Item 20 was removed from the analysis of internal consistency in CASP-C due to the item being considered not applicable to most of the participants, although no data on not applicable ratings are presented [20].Additionally, as the dimension Frequency of attendance had a lower proportion of missing and not relevant/not applicable ratings compared to Engagement and Independence.This could indicate that respondents consider a rating of Engagement and Independence as being dependent on a rating of 0-2 in the dimension Frequency of attendance (0= The same or more than what is expected for the age, 1 = A bit less than what is expected for the age, and 2 = Much less than what is expected for the age).If so, a rating of Never do it or a rating of not relevant/ not applicable in the dimension Frequency of attendance excludes rating Engagement and Independence in that activity area since it is never done.Consequently, this could result in a missing or not relevant/not applicable rating in Engagement and Independence.Alternatively, the number of missing and not relevant/ not applicable ratings could be due to a high cognitive burden on the respondents as FUNDES-Child-SE in the present study was answered along with other instruments.An interview method, as used in FUNDES-Child 7.0, could perhaps decrease the amount of missing data and not relevant/not applicable through elaboration with the interviewer.The cause of missing and not relevant/not applicable ratings will require further investigation outside the scope of this study.
The internal consistency of FUNDES-Child-SE indicates a risk of redundancy within the dimensions (alpha 0.94-0.97),but not in the subdomains (0.8-0.92).Bedell et al. replaced not applicable ratings with the 60 participants' mean scores and CASP then showed internal consistency for all 20 items of 0.95 [16].Internal consistency of CASP-C had an alpha of 0.88-0.9 in the subdomains and 0.96 in the 19 items remaining after Work and responsibilities were excluded from the analysis [20].However, it is important to put the indication of redundancy in relation to the items' clinical significance.Used as an instrument to identify activity areas or settings of interest for interventions in clinical practice, all items can be relevant despite a potential redundancy [10].In research, data from subdomains or other factors can be analysed, depending on the research questions.The structural validity of FUNDES-Child 7.0 has been tested [21] and the structural validity of FUNDES-Child-SE should be tested in future studies.
Most of the subdomains have a lower 95% confidence interval above the threshold for good reliability.The dimension with the lowest ICC is Engagement, especially engagement in the subdomain To participate at school.Engagement could be difficult to rate, as it is judged through observation of the child.To rate engagement in a setting where caregivers are not frequently present, could therefore be even more difficult.This dimension and subdomain also had a high proportion of missing and not relevant/not applicable ratings (14.7-16%).Taking this, the wide scope of the items to be judged, and the test-retest period into consideration, the ICC of 0.73 and lower confidence interval of 0.61 are considered to represent an acceptable agreement between the test and retest.

Methodological limitations
This study has limitations.The high attrition rate and consequently small sample size, which does not include 9-11-year-olds, a high proportion of missing data and not relevant/not applicable ratings, as well as imputation of data are obvious threats to the validity of the results.The sample size was large enough to study test-retest reliability and internal consistency [34].The decision to conduct transformations and imputations was considered the most suitable option considering the sample size.As shown in the analysis using the retest sample, in which transformation and imputation were not performed, the internal consistency does not appear to be affected by transformations and imputations.The ICC of test-retest could have been affected by the time between test and retest in which circumstances e.g.move from school time to holiday time, could have influenced the ratings.Finally, although the translations are done by professionals, the Arabic, Somali and English versions should be validated to ensure the Swedish cultural meaning of the items has been translated and interpreted correctly.In this study, the sample using non-Swedish versions was too small to conduct separate analyses.Still, it should not have had a substantial effect on the results as only 5-9% used a non-Swedish version.

Conclusions
The results of this study indicate that reliability together with the content validity support the use of the FUNDES-Child-SE to measure participation and independence in children with disabilities.FUNDES-Child-SE can be used to facilitate a discussion and to plan interventions to improve participation and independence in children with disabilities, as recommended by Anaby et al. [2] and caregivers [10].However, before the instrument is used to evaluate interventions, it is important to also test FUNDES-Child-SE's responsiveness [13] and modifications to reduce the amount of not relevant/not applicable responses should be investigated.In a clinical setting, a dialogue on the ratings can possibly decrease the amount of missing and not relevant/not applicable.Before suggesting data management or possible revisions in the questionnaire, the factor structure will be tested to further evaluate the instrument and understand the concept of participation.Finally, the results of this study should be interpreted with caution due to the high attrition rate and possible selection bias.
Invitations were sent during the year 2020 and in January 2021, through the habilitation centre where children with disabilities were registered.The four-year longitudinal research project CHILD-PMH invited 2891 children born in 2007-2009 and 2013-2015 and their caregivers, from five regions in central Sweden to participate.In the sixth region, invitations to participate in the cross-sectional study were sent to the caregivers of 410 children born in 2003-2006.

Table 1 .
dimensions and rating scales in fundeS-child-Se.

Table 2 .
demographic data of children and participants.

Table 3 .
the 13 items with >10% missing data or not relevant/not appliable in one or more dimensions and the proportion of these items' valid ratings, per dimension (n = 163).