Feasibility and Acceptability of the PROMIS Measures in Children and Adolescents in Active Cancer Treatment and Survivorship

Abstract

Background

Patient-reported outcomes (PROs) related to symptoms, function, and quality of life during and following cancer treatment can guide care for pediatric cancer patients. To advance the science of PROs, the National Institutes of Health funded the Patient Reported Outcomes Measurement Information System (PROMIS).

Objective

To assess feasibility and acceptability of the PROMIS pediatric measures, as defined by enrollment and attrition rates as well as missingness by measure, item, participant, and assessment time point.

Methods

8-to-18 year olds participated in two studies: PROMIS I, a cross-sectional study of children in active cancer treatment or survivorship, and PROMIS II, a longitudinal study with 3 assessment time points for children receiving curative treatment.

Results

PROMIS I (n=200) and PROMIS II (n=94) had enrollment rates of 92.5% and 89.7%, respectively. For PROMIS I, measure missingness was acceptable (8% missed any measures) and was not related to other study variables. For PROMIS II, measure missingness was minimal (0.8%), and item-level missingness was relatively low. In general, items that were skipped asked about experiences that participants had not encountered in the past 7 days.

Conclusions

In both studies, the PROMIS instruments demonstrated good feasibility and acceptability among pediatric cancer patients. Overall, we had high enrollment, low attrition, and acceptable rates of measure and item missingness.

Implications for Practice

Our results demonstrate that PROMIS measures are acceptable to 8-to-18 year-olds in different points of cancer care and feasible for use in pediatric cancer inpatient and outpatient settings.

Background

The Patient-Reported Outcomes Measurement Information System (PROMIS) is a National Institutes of Health (NIH) Roadmap Initiative, created to establish a resource of measures that effectively and efficiently documents health outcomes according to patient reports.¹ The NIH has prioritized these measures to speed up the translation of research to direct care and to accurately document the full impact of treatment on patients. The PROMIS collaborators created a conceptual framework regarding the symptoms, function, and other health outcomes for which measures would be needed (the pediatric framework can be found at http://www.nihpromis.org/measures/domainframework2). Domains in the pediatric framework include pain interference, pain intensity, upper extremity function, mobility, peer relationships, fatigue, anger, anxiety, depressive symptoms, and asthma impact. Qualitative (interview and focus group) and quantitative (item response theory) methods were used to create and test the PROMIS pediatric measures in children ages 8 to 17 years of age. In the large scale survey used to calibrate the pediatric measures, participants were recruited from general pediatrics clinic and subspecialty waiting rooms as well as after-school programs; as a result, they represented a range of health experiences, including both children who were healthy and those with chronic illnesses such as asthma and cancer.^2-7

Initial testing results with a sample of children and adolescents who have cancer indicate that the PROMIS pediatric measures do detect score differences in the domains as hypothesized, i.e., children and adolescents in treatment for cancer have worse Patient Reported Outcomes (PRO) scores than do children and adolescents who have successfully completed cancer treatment.⁸ Additional support for the use of PROs such as the PROMIS pediatric measures in clinical trials has recently been issued from the Food and Drug Administration (FDA) in the form of its Guidance for Industry paper.⁹ That guidance contains standards for types of PROs that can yield data for use in drug and device applications acceptable to the FDA.⁹ However, the published studies about the PROMIS pediatric measures have not yet reported evidence of the feasibility and acceptability of the measures to children and adolescents diagnosed with complex chronic conditions. The purpose of this report is to assess the feasibility and acceptability of the PROMIS pediatric measures in children with cancer, as defined by enrollment and attrition rates and rates of missingness by item, measure, participant, and assessment time period in two studies of children and adolescents in cancer treatment or survivorship. Feasibility and Acceptability

Feasibility and acceptability are critical aspects of patient-reported measures.^10-11 Feasibility is assessed at the level of the measure’s use with a specific target group (defined typically by characteristics or health status of the target group). It is assessed in terms of how eligible and enrolled participants react to the study itself and is measured by actual participation rates (enrollment), attrition rates, and evidence of adverse events or harm secondary to being in the study. Acceptability is assessed at the level of the measure’s items in the same target group. It is assessed by participants’ reactions to the items’ understandability and appeal and is measured descriptively by the rates of item missingness (e.g., how often an item is skipped by participants) and qualitatively by comments offered by respondents about the item/s. Feasibility and acceptability together convey appropriateness of health measures for the target group in the clinical situation where the measurement is occurring. The two provide different but complementary information about the measures. A detailed assessment of both is particularly informative when using measures in a novel way or with a new target group. Reporting the rate of missingness by measure alone could risk masking an important item pattern – i.e., a low overall rate of measure missingness but a high rate of missing responses for a specific item.

Methods

PROMIS Studies

This paper reports on participants from two separate studies that have been conducted in children and adolescents with cancer. In the first, PROMIS I, children and adolescents with cancer on or off therapy were asked to complete the PROMIS assessments at a single time period. The original aim of PROMIS I was to expand testing of PROMIS items to a diverse group of pediatric disease populations. In addition to administering PROMIS to a pediatric cancer population, this study included children and adolescents with rheumatic disease, rehabilitation need, sickle cell disease, obesity, and chronic kidney disease. The initial results in the PROMIS I cancer sample were that PROMIS pediatric measures detected hypothesized score differences between children and adolescents in active treatment for cancer and those in survivorship care.⁸

In the second study, PROMIS II, participants undergoing cycles of myelosupressive chemotherapy were asked to complete the PROMIS assessments at 3 time points – at the start of a cycle of therapy, mid-way through the cycle, and after count recovery following the cycle of chemotherapy. The specifics of the recruitment and data collection are reported below. The main objective of PROMIS II was to investigate the responsiveness of the PROMIS measures to expected changes in the health of children undergoing cancer chemotherapy.

The objective of this paper is to assess the feasibility and acceptability of the PROMIS pediatric measures in children with cancer, as defined by enrollment and attrition rates and rates of missingness by item, measure, participant, and assessment time period for both PROMIS I and PROMIS II studies. We hypothesized that enrollment and attrition rates would be acceptable and comparable to other research conducted with this population. Similarly, we hypothesized that rates of item and measure missingness would be adequate and provide evidence that PROMIS measures are acceptable to these populations.

Participants

In PROMIS I, children and adolescents between the ages of 8 and 17 years were eligible for study enrollment if they were currently receiving cure-directed cancer treatment (defined as disease-directed therapy within the past 45 days) or had completed cancer treatment and were disease-free and in follow-up care (survivorship group). In PROMIS II, patients 8-to-18 years were eligible if they were diagnosed with cancer and scheduled to receive anti-neoplastic chemotherapy. For both studies, eligible children had to be able to read and speak English, possess functional computer skills (defined as able to see and interact with a computer screen, keyboard, and mouse), and be willing to give written assent/permission for study participation. In PROMIS I, parents also needed to be able to read and speak English. In PROMIS II, consent forms were translated into Spanish, and interpreters were provided for non-English speaking parents.

Patients were excluded for either study if they had a concurrent medical or psychiatric condition which precluded study participation, cognitive or other impairment (e.g., visual) that interfered with completing a self-administered computer-based questionnaire, or if they were receiving end-of-life care (defined as supportive treatment following a decision against resuscitation or favoring terminal care with possible hospice involvement). For PROMIS II, additional exclusion criteria were enrollment on a Phase I clinical trial or treatment with a biologic agent only.

Settings

Five member sites of the Children’s Oncology Group participated in PROMIS I: Children’s National Medical Center in Washington, D.C.; Nebraska Medical Center in Omaha, Nebraska; Children’s Hospital Los Angeles in California; Palmetto Health Children’s Hospital in Columbia, South Carolina; and Emory University in Atlanta, Georgia. The number of newly diagnosed pediatric oncology patients annually at each site ranges from 35 to 478, and the number of survivors monitored in survivorship programs annually at these sites ranges from 100 to 600.

For PROMIS II, participants were recruited from two settings: Children’s National Medical Center, a tertiary medical center located in Washington, D.C., and the Center for Cancer and Blood Disorders of Northern Virginia. Children’s National main campus oncology program cares for about 250 newly diagnosed children and adolescents per year, and the Virginia clinic has about 70 new oncology patients annually.

Procedures

Training study team members

For both PROMIS studies, study personnel at each site received web-based training in study procedures and the use of the online data collection platform; the study operations manual was used as a reference for conducting the study including quality control strategies. Members of the study team were also trained by the principal investigator on procedures for explaining the study to eligible patients and their guardians as well as obtaining consent and assent. Ongoing education of site personnel occurred during weekly conference calls and quarterly refresher training sessions.

Enrollment and Data Collection

For both PROMIS studies, each assenting child and adolescent was assigned a unique identification number by a computer-based system. No other identifiers were collected using the computerized assessment method. Only de-identified data were used in the analysis. Target sample sizes for both studies were determined prior to enrollment, using power calculations based on the studies’ original research aims. Participants completed the online survey using a laptop in the clinic waiting room or a private clinic exam room or using a hospital computer in the patient’s private inpatient room. Parents completed the demographic and treatment items on the computer, and then patients completed the PROMIS pediatric measures.

Patients were recruited to PROMIS I during clinic visits or inpatient hospitalizations between May 2009 and January 2010. The majority of PROMIS I participants completed the measures in 12 to 22 minutes. Each child or adolescent received a $10 gift certificate to help compensate for their time. For PROMIS II, children and adolescents were enrolled and completed the survey first at a time of stability (T1, from 3 days before to the day following the start of a new course of chemotherapy), again 7-to-16 days following the start of a new course of chemotherapy (T2, the nadir when symptoms and adverse effects of chemotherapy are predictably present and distressing), and then concluded the study following the course of chemotherapy, generally 3-to-4 weeks from T1 (T3, when stability is predictably achieved again). Participants received gift certificates for $10 at T1 and T2, and for $15 at T3. At T1, parents completed additional demographic and treatment items. At T2, participants who were not scheduled for a clinic visit could complete the survey from their homes, either on a home computer with internet connection or on paper. Participants who were completing T2 at home were contacted by email or by phone to ask them to do the survey and to give them the log-in information. Data collection was initiated in September 2010 and concluded for this analysis in February 2013. For PROMIS II, each assessment (T1, T2, and T3) took approximately 30-40 minutes.

PROMIS Outcome Measures

PROMIS focuses on the measurement of generic health domains that are important across a variety of illnesses. As described in the pediatric framework (http://www.nihpromis.org/measures/domainframework2), the broad domains being measured are physical, mental, and social health. For both cancer studies, eight PROMIS measures were electronically administered: upper extremity function, mobility, pain interference, and fatigue as part of physical health; depressive symptoms, anxiety, and anger (mental health); and peer relationships (social health). The definitions of these measures are located at http://www.nihpromis.org/measures/domainframework2. These measures ask participants to respond based on the past 7 days and in terms of a 5-point response option ranging from ‘never’ to ‘almost always’ in most measures and from ‘with no trouble’ to ‘not able to do’ for physical functioning measures. Higher scores indicate more of the measured symptom being experienced, which signifies worse health for depressive symptoms, anxiety, anger, fatigue, and pain interference and better health for mobility, upper extremity functioning, and peer relationships.

For PROMIS I, a sampling plan was created to reduce respondent burden such that each participant was administered 96 or 84 PROMIS items from both the whole bank (so that all items were assessed in at least half of the study sample) or short form measures (see Table 1). The two study arms were randomly assigned by the computer for each participant and the actual sequence of the measures was also randomly determined.

Table 1.

Sampling Plan: Number of Items per PROMIS Pediatric Measure

Study	Arm	Mobility	Upper Extremity	Pain Interference	Fatigue	Depressive Symptoms	Anxiety	Peer Relationships	Anger	Total Number of Items
PROMIS I	1	8	8	20	36	8	8	8	0	96
	2	8	8	8	10	15	20	15	11	95
PROMIS IIa	NA	8	8	8	10	8	8	8	6	64

^aPROMIS II participants who completed the survey on the computer were first administered items using the computer adaptive test (CAT) format. After the CAT items were administered, the survey administered any short form items that had not been selected by the CAT. As a result, participants were administered different numbers of items. For the purpose of the current study, we analyzed missingness in the short form items only, as these items were administered to all participants. The numbers of short form items for each measure are listed in the table.

For PROMIS II, measures were administered using the computer adaptive test (CAT) format. CAT is when the computer algorithm chooses the items that are most discriminating for the respondent based on the response to previous items. This approach allows for a precise measurement with the lowest respondent burden. After the CAT items were administered, the survey administered any short form items that had not been selected by the CAT. Participants who completed the survey on paper were only administered the short form items.

The PROMIS measures were previously tested in a diverse group of children and adolescents; characteristics of the measures are available at www.assessmentcenter.net or in associated publications.^3-7 The previous assessments of these measures confirmed their unidimensionality and the extent to which each item was associated with the measured variable. The measures were found to distinguish levels of the measured symptom or function by participant, i.e., high or low anxiety. Prior publications have documented the reliability of the PROMIS measures for various levels of the measured domain. The PROMIS pediatric measures tested consistently achieve or exceed a reliability of 0.85 over 2 to 4 standard deviations of the domain under measurement for the short forms.^{2-4, 6} Using longer item sets than the short forms increases the reliability of measurement.

Scores on the PROMIS instruments have been established with a mean of 50 and standard deviation of 10 in the original calibration sample, a mixture of healthy children and those with chronic illnesses. Since the calibration sample was not representative of a specific group (e.g., specific disease, healthy, or general population), the score of 50 does not have specific meaning with respect to the degree of health.⁵

Other Measures

In both studies, parents completed 16 demographic items online and 12 items about their child’s cancer, treatment, the presence and intensity of 3 symptoms (altered appetite, altered sleep, and fatigue), each measured by a single item created for this study, and 1 clinical indicator (low blood counts) experienced by their child during the previous 7 days. PROMIS II child and adolescent participants also completed the Child or Adolescent Fatigue Scale^12-13 and Symptom Distress Scale.¹⁴ Missingess was not examined for these scales, but the measures are mentioned to describe the entire assessment completed by the patients. These measures were most commonly completed on the computer at all 3 study visits.

For PROMIS II, clinical indicators were collected at the time of all 3 study visits and included hemoglobin (hgb) and white blood count (WBC), which are directly related to the chemotherapy and to PROs.¹⁵ These data are routinely measured in a standard manner at each visit for a child or adolescent receiving chemotherapy and were extracted from their electronic health care records.

ANALYSIS

For both studies, feasibility and acceptability of the PROMIS measures were assessed by examining descriptive statistics for enrollment and attrition rates and rates of missingness at the levels of measures and participants. Measure and item missingness were also examined by characteristics of the sample, (e.g., sex, age) and assessment time point for PROMIS II. Measures were considered missing if no items from the measure were answered. Mean rates of missing items were compared using independent samples t-tests and ANOVA with Tukey post-hoc tests. A missing value analysis using Little’s MCAR test examined patterns of missingness to get information about whether the data were missing completely at random at the measure level for PROMIS I. Data were analyzed using Predictive Analytics Software (SPSS), version 18.0.

RESULTS

Study Participants

For PROMIS I, a total of 203 pediatric oncology patients enrolled in the study. However, 3 participants did not complete any PROMIS items for reasons that were not collected (1.5% attrition rate); therefore, the data for these three participants were removed from consideration in the study. Of the 200 participants in the final study sample, a slight majority were male (n=111, 55.5%), in survivorship care (n=107, 53.5%), and between the ages of 13 to 17 years (n=109, 54.5%). A majority were diagnosed with a leukemia or lymphoma (n=120, 60.0%). A majority were white (n=108, 54.0%), with representative numbers of black or African-American (n=41, 20.5%) and Hispanic (n=40, 20.0%) participants. For those in active treatment, the average number of days since their last treatment was 13.02 (SD = 14.34). Those in survivorship care had last received cancer treatment almost 4 years prior to the assessment (M = 1443.43 days, SD = 1291.80).

In PROMIS I, the majority of guardians who completed the demographic items were the children’s parents (n=189, 94.5%); the largest group of parents had some college education (n=74, 37.0%; Table 2). A minority of children (n=68, 34%) were described by their parents as having health problems in addition to cancer. The most commonly reported other health problems were asthma (n=25, 12.5%), being overweight (n=14, 7.0%), attention deficit disorder (n=12, 6.0%), mental illness disorders (n=11, 5.5%) and high blood pressure (n=11, 5.5%).

Table 2.

Demographic Characteristics of Sample

Characteristics	PROMIS I Participants (%) N = 200	PROMIS II Participants (%) N = 96
Child Sex
Male	111 (55.5)	52 (54.2)
Female	89 (44.5)	44 (45.8)
Child Age Full Sample (years)
8-12	91 (45.5)	40 (41.7)
13-18	109 (54.5)	56 (58.3)
Age (M, SD)	12.9 (2.9)	13.1 (3.0)
Child Race
White	108 (54)	48 (50.0)
Black or African American	41 (20.5)	25 (26.0)
Asian	8 (4.0)	4 (4.2)
Other	28 (14.0)	12 (12.5)
Multiple Races	12 (6.0)	7 (7.3)
Missing	3 (1.5)	0
Child Ethnicity
Non Hispanic	160 (80.0)	77 (80.2)
Hispanic	40 (20.0)	19 (19.8)
Respondent’s Relationship to
Child
Mother or stepmother	167 (83.5)	69 (71.9)
Father or stepfather	22 (11.0)	23 (24.0)
Grandparent	7 (3.5)	2 (2.1)
Guardian or Other	4 (2.0)	2 (2.1)
Adult Respondent’s
Education Level
< = 8th grade	4 (2.0)	6 (6.3)
Some high school	5 (2.5)	6 (6.3)
High school degree/ GED	42 (21.0)	19 (19.8)
Some college/ technical		22 (22.9)
degree	74 (37.0)
College degree	49 (24.5)	28 (29.2)
Advanced degree	26 (13.0)	13 (13.5)
Missing	0	2 (2.1)

For PROMIS II, a total of 96 pediatric oncology patients enrolled; a slight majority were male (n=52, 54.2%) and between the ages of 13 to 18 years (n=56, 58.3%). The racial composition of study participants was diverse, including white (n=48, 50.0%), black or African-American (n=25, 26.0%), Asian (n=4, 4.2%), those identifying with multiple races (n=7, 7.3%), and participants who selected other (n = 12, 12.5%). Nineteen (19.8%) reported being Spanish, Hispanic, or Latino. The majority of guardians who completed the demographic items were the children’s parents (n=92, 95.8%); the largest group of parents had a college or advanced degree (n=41, 42.7%). A minority of children (n= 32, 33.3%) were described by their parents as having health problems in addition to cancer. The most commonly reported other health problems were asthma (n=17, 17.7%), being overweight (n=5, 5.2%), being born prematurely (n=5, 5.2%), and requiring assistance to get around (n=5, 5.2%).

Enrollment Rates

Enrollment rates in PROMIS I (n=203) and PROMIS II (n=96) were 92.5% and 89.7%, respectively. Of the 226 participants screened for PROMIS I, 6 patients (2.65%) did not meet inclusion criteria (4 because they did not read or speak English and 2 because of difficulty manipulating a mouse and keyboard). Seventeen patients (7.5%) declined participation: 9 for lack of time, 3 agreed but then changed their minds about participating, 2 indicated they were not interested, 2 did not offer a reason, and 1 indicated he was on a day off from school and wanted to do ‘nothing’. Of the 165 participants screened for PROMIS II, 59 patients were found to be ineligible. The majority of ineligible patients were excluded from participation due to age outside of the study range of 8-to-18 years old. Eleven patients refused to participate (refusal rate of 10.4%): 4 patients indicated not being interested, 3 did not want to think or talk about their feelings, 1 thought doing the measures three times was too much, 1 had been involved in ‘too many’ studies already, and 2 parents declined to have their children approached about the study.

Measure Missingness

For PROMIS I, only 3 of 203 participants (1.5%) did not respond to any of the PROMIS pediatric items. Of the 200 participants who completed some or all of the PROMIS measures, 16 (8%) had missing scores for at least one of the PROMIS measures. Thirteen of these 16 participants (6.5%) ended the survey early, so that measures administered at the beginning of the survey were completed, but responses were missing for one or more measures administered at the end of the survey. Because the PROMIS measures were administered in random order, missing scores were spread out among all of the measures. For the remaining 3 participants (1.5%), an entire measure was skipped in the middle of the survey (1 anger, 2 pain interference). The reasons that participants ended their surveys early or skipped measures are not known.

A missing value analysis examined patterns of missingness at the measure level by the type of measure administered (e.g., full item bank or short form), measure content (e.g., anxiety, fatigue, mobility, etc.), parental education, child age, race, sex, presence of other health problems, type of cancer, and treatment status. The results of this analysis did not reject the hypothesis that these data were missing completely at random (Little’s MCAR test, X²(100, N = 200) = 105.32, p = 0.34); missingness did not appear to be related to the values of the other variables (e.g., PROMIS pediatric measure, child age, treatment status, type of measure administered (short form vs. full bank), etc.).

For PROMIS II, all participants responded to at least some of the PROMIS items for the assessments they completed. To date, 265 assessments have been completed, including 96 baseline (T1) with no participants withdrawing (0% attrition), 84 T2 assessments (with 12 from T1 not participating in the T2 study time point for an attrition rate of 12.6%), and 85 T3 assessments (with 3 from T2 not participating in T3 for an attrition rate of 3.6% for T3). Only 2 assessments (0.8%) included measures that had been completely skipped. In one case, a participant was interrupted to have his vital signs taken while he was in the process of completing the baseline survey. He did not return to finish the survey and as a result, his scores for three measures (anxiety, anger, and peer relationships) were missing. A second participant did not complete the peer relationships scale at her T3 assessment. This was due to an administration error; she completed the assessment on paper and the page with the peer relationships scale was missing. Due to the low overall rate of measure missingness in PROMIS II (0.8%), a missing value analysis (e.g., Little’s MCAR test) at the measure level was not conducted. Item Missingness

Missingness by item was not examined for PROMIS I. For that study, participants were not able to skip individual items due to the design of the data collection platform. The data collection platform was modified for PROMIS II to allow participants to skip individual items.

For PROMIS II, 265 assessments were completed by 96 children. When we excluded item missingness from the two participants who did not complete entire measures, we found that 25 of 64 items (39.1%) had no missing responses across the three assessment time points. Five items had missing response rates of 5% or higher in at least one of the assessments: It was hard for me to run when I had pain; Being tired made it hard for me to keep up with my schoolwork; I was too tired to do sports or exercise; Other kids wanted to be my friend; and I could open the rings in school binders. Six other items had missing response rates of 3% to <5% in at least one of the assessments (Table 4). The 11 items in PROMIS II that had missing rates of 3% or higher at one or more assessments were described in study field notes as blank because the child or adolescent had not had the opportunity to experience the activity (e.g., being with other kids his or her age, doing schoolwork, running, exercising, or opening the rings in school binders) or to experience the symptom (e.g., pain or fatigue) in the previous seven days.

Table 4.

PROMIS II Items with Missing Rates

Item	Measure	Missingness Rate T1	Missingness Rate T2	Missingness Rate T3	Notes about Participants Across Assessments
I could keep up when I played with other kids.	Mobility	4.2%	2.4%	0%	NA
I could do sports and exercise that other kids my age could do.	Mobility	1.0%	4.8%	2.4%	NA
I had trouble doing schoolwork when I had pain.	Pain Interference	4.2%	3.6%	3.5%	Three participants skipped this item on multiple assessments (2 at T1, T2, and T3; 1 at T1 and T3; one of these completed the survey in the summer). Three participants skipped this item on one of their assessments (1 at T1; 1 at T2, 1 at T3). Two of these 3 took the survey in the summer months.
It was hard for me to run when I had pain.	Pain Interference	5.2%	6.0%	4.7%	Four participants skipped this item at multiple assessments (3 at all 3 assessments). Three participants skipped this item at one time point only (2 at T1 and 1 at T2).
It was hard for me to walk one block when I had pain.	Pain Interference	4.2%	2.4%	3.5%	Two participants skipped this item at all 3 assessments, and 3 participants skipped at one assessment only (2 at T1 and 1 at T3).
I was good at making friends.	Peer Relationships	4.2%	3.6%	1.2%	Two participants skipped this item at T1 and T2. Five participants skipped this item at one assessment only (3 at T1, 1 at T2, and 1 at T3).
Other kids wanted to be my friend.	Peer Relationships	4.2%	7.1%	2.4%	Three participants skipped this item on multiple assessments (2 at T1 and T2, 1 at T2 and T3). Eight participants skipped this item on one assessment (4 at T1, 3 at T2, and 1 at T3).
I felt accepted by other kids my age.	Peer Relationships	3.1%	2.4%	1.2%	NA
Being tired made it hard for me to keep up with my schoolwork.	Fatigue	5.2%	6.0%	2.4%	Three participants skipped this item at multiple time points (1 missed T1, T2, and T3; 1 missed T1 and T2; 1 missed T2 and T3; 2 of these 3 took the survey in the summer). Five participants skipped this item on one assessment (3 at T1, 2 at T2, one of these took the survey in the summer).
I was too tired to do sports or exercise.	Fatigue	5.2%	6.0%	3.5%	Four participants skipped this item on multiple assessments (2 at T1, T2, and T3; 2 at T1 and T2). Three participants skipped this item on one assessment (1 T1, 1 T2, and 1 T3).
I could open the rings in school binders.	Upper Extremity	5.2%	6.0%	1.2%	Two participants skipped this item on multiple assessments (1 at T1 and T2; 1 at T2 and T3). Seven participants skipped this item on only one assessment (4 at T1; 3 at T2).

We then examined items missing rates of 3% or higher in more than one assessment to see if the same participants were missing the item at multiple assessments. In about half of these cases, the same participant missed an item at multiple assessments, and in other cases, a participant skipped the item at just one assessment. Detailed information about these items and participants is provided in Table 4.

For PROMIS II, the number of items missing was not significantly related to child sex (t = .53, p = .60), age group (8 to 12 year olds vs. 13 to 17 year olds; t = .98, p = .33), or assessment time point, F(2.258) = .88, p = .42. The number of participants (excluding those who missed an entire measure) with one or more missing items by measure ranged from 0 for anger and depression at T1 and anxiety at T3 to 11 for fatigue at T2 (Table 3). There were statistically significant differences in the rates of items missing (e.g., number of missing items divided by number of items in measure) by measure, F(7.2080) = 6.66, p < .001. Specifically, the average rate of missing items in the pain interference measure was higher than the average rate of items missing in upper extremity (p = .014) and anger, anxiety, and depression (p < .001 for these comparisons) measures. The average rate of missing peer relationships items was greater than the average rate of items missing in the anger (p = .012), anxiety (p = .014), and depressive symptoms (p = .01) measures.

Table 3.

Number of Participants with One or More Missing Items for PROMIS II

Measure	Assessment 1 N = 96 (%)	Assessment 2 N = 84 (%)	Assessment 3 N = 85 (%)
Anger	0	1 (1.2)	1 (1.2)
Anxiety	1 (1.0)	1 (1.2)	0
Depression	0	1 (1.2)	1 (1.2)
Fatigue	10 (10.4)	11 (13.1)	7 (8.2)
Pain Interference	9 (9.4)	9 (10.7)	7 (8.2)
Peer Relationships	8 (8.3)	9 (10.7)	1 (1.2)
Mobility	4 (4.2)	5 (6.0)	3 (3.5)
Upper Extremity	6 (6.3)	5 (6.0)	2 (2.4)

Missing Visits (PROMIS II)

A total of 15 participants (15.6%) missed the second or the third study time point: 8 represented passive refusals, 2 who reported being ‘too tired’ or ‘don’t feel like it’, 3 who had increased severity of illness leading to treatment changes, 1 whose family declined to continue, and 1 who died. Of the 15 participants not continuing, a majority were adolescent (n=9), male (n=8), and Caucasian (n=8) or African-American (n =5)

Discussion

To our knowledge, this is the first report about the feasibility and acceptability of the PROMIS pediatric measures to 8 to 18 year olds as measured by enrollment, attrition, and missingness at the levels of measure, participant, item, and assessment time point. This is a timely assessment given the importance of these measures to the NIH sponsors and the original purpose to use these measures in studies involving children and adolescents with acute and chronic illnesses to assess their symptoms, function, and quality of life. Our findings are derived from two studies involving nearly 300 children and adolescents between the ages of 8 and 18 years of age who were in treatment intended to cure cancer or survivorship. The primary aims of these studies were to examine the validity of the PROMIS instruments in diverse groups of chronic illness populations (PROMIS I) and to investigate the responsiveness over time of PROMIS instruments to PROs in children and adolescents undergoing chemotherapy (PROMIS II); although the original intent of these studies was not to examine the feasibility and acceptability of these instruments, we were able to use the data to investigate this.

Overall, our findings do support the feasibility and acceptability of these measures to this age and illness group. More specifically, our analysis found that missingness by measure, participant, item, and assessment time point was impressively low. The relatively low levels of missingness is important, given that racial and ethnic representation were strengths across both studies as was the diversity by sex and age.

Enrollment rates in both studies (92.5% and 89.7%) were equal to or higher than other pediatric instrumentation studies of symptoms, function, and quality of life.^16-18 Reasons for refusal (n=28) were primarily a perceived lack of time (n=13) and not interested (n=6). Importantly, 4 participants declined to participate in PROMIS II because of not wanting to focus on their feelings. This category of refusal was unique to PROMIS II and may be because the participants in that study were more likely to be early in their treatment as compared to those in PROMIS I and were thus earlier in their efforts to adapt to the cancer experience. Attrition rates (1.5% for PROMIS I; 0, 12.6% and 3.6% for sequential study time points in PROMIS II) were also less than projected and similar to or lower than those of other pediatric instrumentation studies. ¹⁷

For both studies, missingness at the level of measure (8%, 0.8%) was lower than anticipated, especially for the longitudinal design of PROMIS II. This missingness by measure is lower than that previously reported for a measure of patient-reported school functioning in children and adolescents in cancer treatment which ranged from 21.6% to 68.9% across 6 measurement points in one study¹⁰ and 21.5% in a second study. ¹¹

The missingness by item analysis indicated that 38.1% of the 64 items asked in PROMIS II had no missing responses across participants or assessment time point, a rate that is similar to those (40.7%) reported in other studies involving children with cancer.¹¹ Because few studies report missingness by item, a comparison between this rate and those in other pediatric instrumentation studies is not possible. However, our analysis of the type and frequency of missing item responses indicates important considerations for PROMIS pediatric measures. The 11 items in PROMIS II that had missing rates of 3% or higher at one or more assessments were identified in our study field notes as items which contain activities that may not be possible during the disease treatment experience of target children and adolescents (e.g., being with other kids his or her age, doing schoolwork, running, exercising, or opening the rings in school binders). This suggests that a certain level of missingness will likely always be present in the use of these measures with 8 to 18 year olds in cancer treatment or survivorship. An ill respondent may skip an item because he/she did not have a chance to do the activity.

Similarly, 3 of the 11 items with missing rates of 3% or higher were from the pain interference measure. In the current version of PROMIS, users who are not experiencing pain are unable to opt out of this measure, which may lead children and adolescents without pain to skip some of these items. Future research will explore the implications on scoring and validity of including an item where children and adolescents would indicate if they have pain, and those without pain would not be administered the pain interference items. Fortunately, when using the PROMIS instruments, measure scores can still be calculated by applying the already available Item Response Theory (IRT) algorithms even when item level data are missing.

The acceptability of these item missingness rates needs to be determined by researchers. Few pediatric oncology instrumentation studies report on measure, participant, item, or time point missingness^{16, 19-21}, although such analyses may be particularly informative regarding the feasibility and acceptability of using selected measures in groups of children and adolescents with complex chronic illnesses such as cancer. We recommend that pediatric instrumentation studies include missingness analyses at the levels of measure, participant, item, and assessment time point. For studies in which a considerable amount of missingness is present, we recommend an analysis at the level of site for multi-site studies and/or individual research team members to understand if different techniques are leading to different levels of missingness.

While there are many benefits to using a computer-based data collection system in which data are directly entered by participants (e.g., increased participant privacy and autonomy in answering the questions), we acknowledge a challenge for the research team member conducting the study visit is that unless a participant opts to discuss reasons for skipping items, the team member will not be notified in real time when this has occurred. Therefore, the researcher will not have the opportunity to discuss this with the participant. As a result, it is frequently unknown why a participant has elected to skip a question, leaving the research team without a clear explanation for why some items have more missingness than others. Our field notes were able to capture some of the reasons for missingness at the individual item level, but these notes may not accurately represent all of the reasons for the missingness.

This study provided an opportunity to take a comprehensive look at the feasibility and acceptability of the pediatric PROMIS instruments in a population of children and adolescents with cancer. While two other scales were administered in PROMIS II, this study only examined missingness for the PROMIS measures. Although a gold standard for item and measure completion has not been clearly established, the PROMIS measures performed adequately, with acceptable rates of measure missingness in the cross-sectional study (PROMIS I), minimal measure missingness in the longitudinal study (PROMIS II), and low rates of item missingness in PROMIS II. Enrollment and retention rates for these studies were also adequate. We also found that it was important to use a data collection platform that enabled participants to skip individual items. In the PROMIS I study where participants were not able to skip items, we found higher levels of measure missingness and more cases when participants ended the survey early. Although we do not have data about their reasons for ending the survey, it is possible that these participants would have preferred to just skip an individual item. When participants were able to skip items, we discovered some patterns in the individual items they elected to skip and gathered qualitative data about their decisions to skip items. Future studies using PROs should evaluate the feasibility and acceptability characteristics of their measures in order to guide researchers and clinicians in selecting the most effective instruments for the patient population.