Development and psychometric characteristics of the SCI-QOL Ability to Participate and Satisfaction with Social Roles and Activities item banks and short forms

Objective To develop a spinal cord injury (SCI)-focused version of PROMIS and Neuro-QOL social domain item banks; evaluate the psychometric properties of items developed for adults with SCI; and report information to facilitate clinical and research use. Design We used a mixed-methods design to develop and evaluate Ability to Participate in Social Roles and Activities and Satisfaction with Social Roles and Activities items. Focus groups helped define the constructs; cognitive interviews helped revise items; and confirmatory factor analysis and item response theory methods helped calibrate item banks and evaluate differential item functioning related to demographic and injury characteristics. Setting Five SCI Model System sites and one Veterans Administration medical center. Participants The calibration sample consisted of 641 individuals; a reliability sample consisted of 245 individuals residing in the community. Results A subset of 27 Ability to Participate and 35 Satisfaction items demonstrated good measurement properties and negligible differential item functioning related to demographic and injury characteristics. The SCI-specific measures correlate strongly with the PROMIS and Neuro-QOL versions. Ten item short forms correlate >0.96 with the full banks. Variable-length CATs with a minimum of 4 items, variable-length CATs with a minimum of 8 items, fixed-length CATs of 10 items, and the 10-item short forms demonstrate construct coverage and measurement error that is comparable to the full item bank. Conclusion The Ability to Participate and Satisfaction with Social Roles and Activities CATs and short forms demonstrate excellent psychometric properties and are suitable for clinical and research applications.


Introduction
Participation in social roles and activities, a long term outcome following rehabilitation, is valued highly by persons living with the consequences of spinal cord injury (SCI). Health insurance reform legislation emphasizes the need to measure the long term outcomes of rehabilitation services using standardized measures; such measures are critical to new delivery and payment reform models, including Accountable Care Organizations, bundling demonstrations, the Continuing Care Hospital concept, and the Independence at Home Demonstration program. Needed are long term outcome measures to improve the quality of healthcare and reduce cost without sacrificing care. Thus, this report describes the development of standardized measures of long term rehabilitation outcomes focused on participation in social roles and activities and satisfaction with participation in social roles and activities.

Participation measurement
The World Health Organization defines participation limitations within the context of the International Classification of Disability and Health (ICF). Participation is defined by many rehabilitation researchers as chapters 6 to 9 of ICF's Activity and Participation domain, those pertaining to Domestic Life, Interpersonal Interactions and Relationships, Major Life Areas, and Community, Social and Civic Life, 1 and the definition we adopt in this report. Participation measurement is complicated by the World Health Organization's decision to combine activities and participation in a single classification making it impossible to distinguish performance at the individual level (activity) and performance at the societal level ( participation). Needed are reliable and valid measures of participation that allow the separation of objective participation performance and subjective satisfaction with participation.

Research on participation measurement
Magasi and Post 2 reviewed the content of contemporary participation measures. They identified 8 participation measures developed using classical test theory methods that demonstrated moderate to good validity and reliability, though reporting of measurement information was often incomplete. The measures most often assessed the ICF domains of mobility; domestic life; social interactions; major life domains; and community, social, and civic life. Dijkers 3 noted that participation is defined frequently as the primary valued outcome of rehabilitation, but that vague and incomplete definitions and inadequate measurement of participation severely limit clinicians' and investigators' ability to measure participation outcomes. He described several major issues related to participation measurement and offers conceptual and methodological guidance. Whiteneck 1 observed that the large and growing number of participation measures limits rehabilitation research and practice. He described desirable types and characteristics of participation measures and made recommendations for future participation research. He noted that participation measures vary in regards to specificity, conceptual models, development approach, psychometric properties, dimensionality, objective vs. subjective perspectives, and use of norms He encouraged development of better participation measures built on consensus regarding conceptual models.
Over the last ten years, the National Institutes of Health (NIH) has supported the development of instruments to provide common data elements for clinical research. 4,5 The Patient Reported Outcomes Measurement Information System (PROMIS) 6 is a family of instruments that measure patient-reported health status. 4,5 The National Institutes of Neurological Disorders and Stroke has taken the lead in developing core data sets for neurologic populations, including epilepsy, stroke, and Parkinson's disease among other conditions through the Quality of Life in Neurological Disorders (Neuro-QOL) project. 7 Neuro-QOL provides a conceptual framework and a common language for clinicians and investigators.
The PROMIS Social Health Workgroup conducted a systematic review of social health definitions, content, and item wording, and implemented a qualitative item review process that included identification of items, development of new items, focus group exploration of domain coverage and cognitive interviews. [8][9][10] Validation testing in diverse samples (n = 2,208 English; n = 644 Spanish) resulted in psychometricallysound and culturally-appropriate measures of social function, including Ability to Participate in Social Roles and Activities (SRA), and Satisfaction with SRA. 10 Work also began on social relationships, focusing on social support and social isolation. [8][9][10][11] The extent to which these concepts and measures apply to persons with SCI has not been evaluated.

Study objectives
The objectives of this study were to (1) develop item banks of social functioning that are relevant to persons with SCI; (2) link the item banks to either the PROMIS or Neuro-QOL (or both) through common items and transform the score to the PROMIS or Neuro-QOL metric; (3) evaluate the psychometric properties of item pools developed for adults with SCI; and (4) report information that facilitates clinical and research use.

Item set development
Based on focus group feedback, we identified initially 3 realms that are relevant to social participation: family and friends, leisure, and work. We began with 129 items generated during individual interviews 12,13 and developed 214 new items based on the focus group transcripts 14 to form the basis of item banks. Given the work completed by the PROMIS 10 and Neuro-QOL collaborators, we decided to adopt PROMIS' domains of Ability to Participate and Satisfaction with SRA. We found these subdomains to be more useful than our initial 3 realms given the assumption of unidimensionality required for item response theory (IRT) analyses and computer adaptive testing (CAT) programming. We binned the items generated from individual interviews and focus groups with 82 Neuro-QOL social domain items into subdomains of Ability to Participate in SRA and Satisfaction with SRA. Two investigators independently assigned items to subdomains; when they disagreed, they discussed and reconciled differences. Because a goal of the SCI-QOL project was to evaluate the Neuro-QOL's social domain items in an SCI population and supplement them with SCI-relevant items, we prioritized the 38 Neuro-QOL Ability to Participate items and the 44 Satisfaction items. Due to the Neuro-QOL's linkage with PROMIS, this set included 23 PROMIS Satisfaction items. The project team reviewed the remaining items and determined that all but 4 Ability and 4 Satisfaction items were redundant with the Neuro-QOL content. The team winnowed the item pool to 50 Ability and 49 Satisfaction items Following PROMIS instrument development standards, 15 we scheduled cognitive debriefing interviews 16 with individuals with SCI (n ≥ 5 per item). Because the Neuro-QOL items had already undergone this extensive review, only the 8 new SCI-QOL social items underwent this high level of review. No items required modification based on cognitive interviews.
Next, we completed a translatability review to ensure that the SCI-QOL items would be amenable to translation into other languages. An experienced team conducted a thorough review of the items to evaluate issues with vocabulary or grammar that could affect the meaning of items translated to Spanish. We modified one Ability to Participate and one Satisfaction item based on translatability feedback. We rephrased 'I am able to navigate a crowded social situation' as two new items: 'I am able to interact with people in social situations' and 'I am physically able to move through a crowded room.' We removed 'amount of' from 'I am satisfied with the amount of control I have over my daily activities.' Finally, we evaluated the reading level of new items using the Lexile Framework 17 to assure that they did not exceed a fifth grade level.

Field testing
We administered the 50 Ability to Participate and 49 Satisfaction items to a sample of individuals with SCI to provide data for IRT calibration. As outlined by Tulsky and colleagues (overview paper, this issue) 13 , 5 SCI Model System sites (Craig Hospital, Kessler Rehabilitation Hospital, Rehabilitation Institute of Chicago, University of Michigan, and University of Washington) and the James J. Peters Veterans Hospital participated in data collection. Interviewers read each question aloud from a computer screen and entered responses into a secure data platform. Institutional Review Boards at each site approved this study. We developed a procedure manual and used it to provide interviewers with training to certify their competence.

Calibration sample
A total of 641 individuals with traumatic SCI completed the two item sets. Inclusion criteria were SCI due to traumatic event, age 18 or older at the time of study participation, and ability to read and understand English. There were no further exclusion criteria. Level and severity of injury were confirmed through medical record review.

Reliability sample
An independent sample of 245 individuals from 4 SCI Model Systems centers (University of Michigan, Kessler Institute for Rehabilitation, Rehabilitation Institute of Chicago, Craig Hospital) completed the item banks twice as part of a larger project in which we tested SCI-QOL CATs and short forms at multiple intervals. 12 Each site's Institutional Review Board reviewed and approved the study protocol. Eligibility criteria were similar to the calibration study: traumatic SCI, 18 years or older, and ability to read, speak, and understand English fluently. We stratified the sample by level and completeness of injury as well as time since injury (≤2 years, 2>years). Participants were community-dwelling and sustained SCI more than 4 months before the assessment. Our goal was to have participants complete the second assessment 7-14 days after the first assessment.

Data analysis
We evaluated dimensionality of the Ability to Participate and Satisfaction item sets using confirmatory factor analysis with MPLUS version 6. Criteria for unidimensionality were a comparative fit index (CFI) and a Tucker-Lewis Index (TLI) >0.9, a root mean square error of approximation (RMSEA) <0.08 for good fit and <0.05 for excellent fit. 18

2-parameter item response theory analyses
We used a graded response IRT model 19 using MULTILOG software and examined the S-X 2 model fit statistics using the IRTFIT macro program. 20 We removed items that demonstrated local item dependence (residual correlation >|0.2|), significant (P < 0.05), misfit (S-X 2 test), 21 or differential item functioning (DIF) 22 due to sex, age (<50 vs. ≥50), education (some college or less vs. college degree or higher), level of injury ( paraplegia vs. tetraplegia), injury severity (complete vs. incomplete), and time post injury (<1 year vs.≥1 year). We transformed SCI-QOL item parameters to Neuro-QOL's general population metric following the procedure reported by Tulsky and colleagues. 12

Calibration sample characteristics
Tulsky and colleagues describe the demographic characteristics of the focus group participants. 13 Table 1 shows that the sample of 641 respondents was middle-aged (mean 42.9 years old) with a range from 18 to 91 years. Reflecting the epidemiology of SCI, men outnumbered women 3 to 1; the largest group was single and never married. The sample reflects the US population in terms of ethnicity; racial minority groups were somewhat over-represented. Educational attainment included high  Table 2 shows descriptive statistics for the retained Ability to Participate items Table 3 shows the same information for the Satisfaction items

Confirmatory factory analysis
Ability to participate We deleted 23 items due to a significant χ 2 value indicating misfit (9), bimodal distributions (3), residual correlations greater than |0.2| indicating local item dependence (11), or r 2 values less than 0.3 indicating low item-total correlation (1). Some of the Neuro-QOL items pertaining to work demonstrated a bimodal distribution of responses, suggesting that people who are employed complete the items in one manner and people who are unemployed complete the items differently. The items displayed misfit, local dependence, and differential item functioning (DIF), all of which could bias the results and performance of the scale in this population. Therefore, we removed several employment related items Satisfaction: We deleted 14 items due to bimodal distributions (6), local item dependence (11), and misfit (2). Table 4 shows the calibration statistics, including slopes and response category thresholds, for the 27 retained Ability to Participate items; all of them are Neuro-QOL items. These calibration parameters have been optimized for an SCI population but, as described below, have been transformed to the Neuro-QOL metric so they are interpretable as Neuro-QOL scores. Table 5 shows the calibration statistics, including slopes and response category thresholds, for the 35 retained Satisfaction items; 34 of them are Neuro-QOL items and 15 of these are PROMIS items. Similar to the Ability to Participate domain, these calibration parameters are optimized for an SCI population and transformed to the Neuro-QOL metric.

Differential item functioning
Seven Ability to Participate items demonstrated statistically significant DIF as did 8 Satisfaction items. When we examined the DIF effect sizes, the practical effects were negligible; thus, we decided to retain them.

Transformation to Neuro-QOL metric
We computed SCI-specific calibrations using the SCI-QOL calibration sample, then we linked these scores to the Neuro-QOL calibrations that had been developed using a general population reference group using the items that are common to SCI-QOL and Neuro-QOL. We utilized Stocking-Lord equating methods 23 to  calculate transformation slope and intercept parameters. We applied these parameters to create linear transformations so that the SCI-QOL measure maps to the Neuro-QOL metric and scores are reported in terms of general population norms and are equivalent to Neuro-QOL scores. Table 6 shows how the mean scores for each were transformed. SCI-QOL scores are about 5 Tscore points lower than Neuro-QOL scores and have less variance

Short form item selection
We selected items to comprise a fixed-length short form version of each item bank for situations in which CAT administration is not practical. First, we reviewed item locations and discrimination. Then, we assigned items into quintiles based on location and selected 1 to 2 items within each quintile with the highest slope. We considered clinical relevance, wording, and similarity to other candidate items. Our goal was to maximize the diversity of short form item content. Following PROMIS naming conventions, the short forms are titled Ability to Participate in SRA SF10a and Satisfaction with SRA SF10a, respectively. Figure 3 illustrates the reliability of the Ability to Participate SF10a vs. a variable length CAT (set to the default minimum of 4 items, maximum of 12 items, and maximum standard error 0.3), a 10-item fixedlength CAT, and the full item bank. Reliability exceeds 0.80 between T-scores of 20 and 60. Figure 4 illustrates reliability values for the various modes of administration of the Satisfaction bank. Reliability exceeds 0.80 between T-scores of 26 and 60. Table 7 shows the breadth of coverage for the Ability to Participate SF10a compared with 4-and 8-item minimum variable-length CATs, a 10-item fixed length CAT, and the full item bank. The correlation between    the 10-item Ability to Participate SF10a with the full item bank exceeds 0.96. Table 8 shows the same information for the Satisfaction SF10a. T-score means were essentially identical as was the range of scores and standard errors. The SF10a has somewhat greater ceiling effects than the other versions. Table 9 provides means, ranges, and standard errors for the Ability to Participate variable length CATs, 10item CAT, 10-item short form, and full bank. Means and standard deviations, ranges, percent of the sample at the ceiling and floor, and standard errors are essentially identical. Table 10 provides the same information for Satisfaction with SRA. Again, means and standard deviations, ranges, percent of the sample at the ceiling and floor, and standard errors are essentially identical across different versions. Tables 11 and 12 provide the raw to scaled score conversions for the 10-item short forms for Ability to Participate and Satisfaction with SRA, respectively.

Discussion
The objectives of this study were to (1) develop item banks measuring social domains of functioning in an SCI population; (2) link the measure(s) to PROMIS and Neuro-QOL; (3) evaluate the psychometric properties of item pool developed for adults with SCI; and (4) report information that facilitates clinical and research use. The 27-item Ability to Participate and a 35-item Satisfaction with Participation item bank fulfill these objectives.
The SCI-QOL Ability to Participate and Satisfaction with SRA banks are optimized versions of the Neuro-QOL/PROMIS v1.0 social item banks for use by individuals with SCI. Items calibrations were developed using a large, heterogeneous, and representative sample. When administered as a CAT, items will be selected based on their functioning in an SCI sample. Similarly, item content that is tailored to an SCI population could be added and items that do not function well in an SCI population could be identified and removed. For example, several Neuro-QOL/PROMIS items related to employment had bimodal distributions or demonstrated poor model fit, local dependence, or   differential item functioning, likely due to high rates of un-and under-employment in this sample. Given the importance of employment issues, the research team was presented with a dilemma of retaining content that could bias the overall measurement scale or remove poorly functioning items We decided to remove most problematic items at the expense of reducing content coverage. We recommend that future research develop item banks focused exclusively on employment issues. We excluded all but one (RSATIS_56, 'I am satisfied with the amount of physical contact I have with others') of the newly developed items from the final item banks. With the exception of the employment items as described above, the PROMIS and Neuro-QOL Ability to Participate and Satisfaction with SRA items describe most of the participationrelated issues that are relevant for individuals with SCI. Another significant advancement of this project is the development of calibrations optimized for individuals with SCI and items that are relevant and appropriate for this population. No other participation measure is customized to persons with SCI and used IRT methods in their development. Transformation to the Neuro-QOL metric enables direct comparison of SCI-QOL with Neuro-QOL social bank scores from other disability samples. Notably, the Ability to Participate and Satisfaction with SRA banks demonstrate lower mean scores (∼45 vs. ∼50) and restricted range (SD ∼6 vs. SD ∼10) when transformed to reflect the general population norms. This result suggests that people with SCI are limited in both their ability to participate and satisfaction with participation, and that the range of engagement in social roles and activities is more limited in individuals in SCI than in the general population. Social participation is an important issue for clinicians to target for clinical services, and for investigators to develop effective interventions.
The psychometric properties of the SCI-QOL Ability to Participate and Satisfaction CATs and short forms are excellent. Full item banks and short forms are  available as PDF files from the authors; CATs may be administered through the NIH Assessment Center, which provides users with options for customizing stopping rules such as the minimum and maximum number of items to administer, and maximum standard error.

Study limitations
This convenience sample was drawn from only 6 hospitals; it may not reflect the diversity of people with SCI living in the United States. Future studies should evaluate sensitivity to change, demonstrate known groups validity, and provide information on interpretability and meaningfulness.

Clinical applications
Screening during outpatient visits allows clinicians to identify individuals with low levels of Ability to Participate or Satisfaction with Social Roles and Activities. Discussions with patients about the meaning and consequences of the scores could help guide referrals and treatment plans.

Research opportunities
These 2 SCI-QOL social domain variables can be used in epidemiological studies to monitor population health and the effects of interventions designed to enhance participation, among other applications.

Conclusions
The Ability to Participate and Satisfaction with Social Roles and Activities items provide state-of-the-art measures that can be administered using CAT or short forms. The measures are compatible with the PROMIS and Neuro-QOL family of measures. They