An electronic patient-reported outcomes measurement system in paediatric orthopaedics

Abstract

Purpose

The purpose of the study was to evaluate the reliability, review differences and assess patient satisfaction of electronic patient-reported outcome measures (PROMs) compared with paper PROMs.

Methods

Participants between 12 and 19 years of age with a knee-related primary complaint were randomized into two groups. Group 1 completed paper PROMs followed by electronic, while Group 2 received the electronic followed by paper. PROMs included the Pediatric International Knee Documentation Committee (Pedi-IKDC), Hospital for Special Surgery (HSS) Pediatric Functional Activity Brief Scale (HSS Pedi-FABS), Tegner Activity Level Scale, Visual Analogue Scale (VAS), PedsQL Teen and a satisfaction survey.

Results

In all, 87 participants were enrolled with one excluded due to incomplete PROMs. Of the 86 participants, 54 were female and 32 were male with an average age of 14.3 years (12 to 18). A high degree of reliability was found when comparing the paper and electronic versions of the Pedi-IKDC (0.946; p < 0.001), HSS Pedi-FABS (0.923; p < 0.001), PedsQL Teen (0.894; p < 0.001), Tegner Activity Level Scale before injury (0.848; p < 0.001) and the Tegner Activity Level Scale after (0.930; p < 0.001). Differences were noted between the VAS scores, with paper scores being significantly higher than electronic (5.3 versus 4.6; p < 0.001). While not significant, a trend was noted in which electronic PROMs took, overall, less time than paper (10.0 mins versus 11.2 mins; p = 0.096).

Of all participants, 69.8% preferred the electronic PROMs, 67.4% felt they were faster, 93.0% stated they would complete forms at home prior to appointments and 91.8% were not concerned about the safety/privacy of electronic forms.

Conclusion

PROMs captured electronically were reliable when compared with paper. Electronic PROMs may be quicker, will not require manual scoring and are preferred by patients.

Level of Evidence

II

Introduction

Patient-reported outcome measures (PROMs) play an important role in today’s healthcare system, aiding in the improvement of patient care and treatment outcomes. In orthopaedics, PROMs have become the core of clinical research and evidence-based medicine. Additionally, hospitals and insurance companies have started to use PROMs as measures of physician performance, developing models to tie these questionnaires to reimbursement.¹ To properly assess the benefits of treatment, it is essential to provide evidence of the impact of treatment on the patient regarding health status, validated outcomes and health-related quality of life.²

The administration of PROMs can occur in the form of direct face-to-face interviews, telephone conversations, hard-copy forms (i.e. paper/pencil) and electronic questionnaires. Each form of data collection has advantages and disadvantages. Face-to-face interviews may be the costliest and most time consuming but may also have the highest response and completion rates, leaving no room for skipped or misinterpreted questions. Administration of PROMs in hard-copy forms has traditionally been used and considered effective, avoiding bias of an interview while providing adequate completion rates if monitored.³ Furthermore, the majority of PROMs have been validated in this format.^4-16 Nevertheless, the hard-copy methods of data collection are time-consuming for staff, requiring manual scoring and may introduce human error to score calculations.

The Pediatric International Knee Documentation Committee (Pedi-IKDC) has been validated for use in adolescents ages 10 to 18 years old.^8,9,13,17 Kocher et al⁹ demonstrated the reliability (intraclass correlation coefficient (ICC) = 0.91), criterion validity as compared with the Child Health Questionnaire (p < 0.01) and responsiveness of the Pedi-IKDC in paper form.^9,10 The Tegner Activity Level Scale,¹⁵ a well-accepted measure of knee activity level, and Pediatric Functional Activity Brief Scale (HSS Pedi-FABS),¹⁸ a paediatric-specific activity measure, have both been validated in paper format or verbal interviews. The Peds-QL is a paediatric quality of life measure that has been validated in both paper format and an online version.^11,16,19 Kruse et al¹⁹ evaluated the construct validity of the PedsQL and impact on mode of administration and found excellent correlation between modalities with a Pearson’s coefficient of 0.92 to 0.97.

The use of an electronic system may improve patient and physician efficiency, decrease cost, ensure completion, provide instantaneous information and minimize inconvenience to families.^20,21 Early data has suggested that electronic PROMs require some basic computer skills.²² Gwaltney et al²⁰ performed a meta-analysis of 65 studies assessing the equivalence of paper versus electronic version of PROMs. They found 93% of studies show comparability within +/- 5% and the average weighted correlation was 0.90.

Before transitioning PROM management from a paper format to an electronic software, it is important to assess the validation of the tools in paediatrics. The purpose of this study was to evaluate the reliability of an electronic software programme to administer PROMs in paediatric and adolescent sports medicine patients. The secondary objective was to determine the patient satisfaction and time to completion of electronic PROMs as compared with paper PROMs.

Patients and methods

An institutional review board-approved, prospective, randomized study of the efficiency and satisfaction of electronic PROMs in paediatrics was performed. Patients between 12 and 19 years of age with a primary complaint of a new knee injury were screened and identified prior to their first appointment for the corresponding injury. Patient’s guardians were contacted prior to their appointment to participate in the research study in order to arrive early for testing. Participants were included if they had not previously completed PROMs, electronically or hard copy, during a previous visit to the clinic. Participants who had a cognitive disorder or delay or could not read or write in English were excluded due to their inability to complete the PROMs.

Participants were randomly assigned into two groups, in a 1:1 fashion, to define the order in which they were to complete two sets of validated PROMs. These PROMs included the Pedi-IKDC, Hospital for Special Surgery (HSS) Pedi-FABS, Tegner Activity Level Scale, Visual Analogue Scale (VAS) and PedsQL Teen. Group 1 completed forms on paper followed by an electronic version, while Group 2 completed the electronic version followed by the paper. Electronic forms were captured in the Outcome Based Electronic Research Database (OBERD; Universal Research Solution, LLC; Columbia, Missouri), an electronic software system used to collect and manage PROMs. OBERD is designed to improve PROM completion and compliance while minimizing patient inconvenience.^23,24 The patient interface allows for ease of use without extensive computer knowledge, and the system has been validated in adult orthopaedic patients.^23,24 All electronic measures were delivered on an Apple iPad Air tablet computer (Apple Inc, Cupertino, California).

A ten-minute break, chosen based on previous published literature, occurred between the paper and electronic formats to prevent memorization of answers.^17,25-27 While completing paper forms, participants were asked to report their start times, end times and interruptions greater than two minutes. Time to completion for both paper and electronic forms was calculated by research personnel. This did not include staffing time to assess each measure for completion nor hand scoring of paper PROMs, estimated to be approximately five minutes per participant. Time was verified for electronic PROMs with the use of an electronic report, as these forms were automatically time-stamped with completion times. Time to completion of individual paper forms was not collected, however, a total time to completion of all paper forms was documented. Following completion of both sets of forms, participants completed an electronic satisfaction survey. The satisfaction survey gathered patient-reported data regarding preference of PROM format, completion time, clarity and privacy. Compensation, in the form of a $25 gift card, was provided to the participant after the satisfaction survey was completed.

Satisfaction and perceived efficiency were collected from the satisfaction survey (Appendix 1). Efficiency was calculated based on time to completion of paper PROMs, electronic PROMs and overall completion of both form sets. Scores from electronic PROMs and paper PROMs were used for comparison between formats and a reliability analysis was performed. A Pearson’s correlation was used to calculate the association between the measures and a paired t-test to compare means between the electronic PROMs and paper PROMs. Reliability analyses were conducted using an ICC calculation. With paper measures considered valid, reliability is shown when electronic measures yield statistically equivalent results to paper measures.

Results

A total of 87 participants consented and were prospectively enrolled into the study. One study participant was excluded because they failed to complete all of the PROMs. Of the 86 included participants, 54 were female and 32 were male. The mean age of all participants was 14.3 years (12 years to 18 years). Group 1 contained 44 participants and Group 2 contained 42 participants. Group 2 contained a slightly younger population (14.0 years (12 years to 17 years) versus 14.6 years (12 years to 18 years); p = 0.047) with a higher percentage of male participants (59.4% versus 40.6%; p = 0.13) when compared with Group 1.

Group 1 versus Group 2

Table 1 shows the overall comparison of mean PROM scores between groups. There were no significant differences in mean paper scores between Group 1 and Group 2. Furthermore, no differences were noted when comparing electronic scores between groups. The largest difference between groups was noted when comparing the mean paper (Group 1 = 5.6 versus Group 2 = 4.9) and electronic (Group 1 = 5.1 versus Group 2 = 4.1) VAS scores, although this was not statistically significant (p = 0.110; p = 0.085).

Table 1.

The comparison of patient-reported outcome measure scores of Group 1 and Group 2

	Overall		Group 1		Group 2
PROM	Mean	sd	Mean	sd	Mean	sd	p-value*
Pedi-IKDC
Paper	48.2	18.3	48.1	18.5	48.3	18.2	0.970
Electronic	48.8	17.5	48.4	17.8	49.3	17.4	0.821
HSS Pedi-FABS
Paper	17.3	10.7	17.5	11.1	17.1	10.4	0.863
Electronic	17.4	10.4	17.3	10.8	17.6	10.0	0.878
Tegner Activity Level Scale-Before
Paper	8.4	2	8.6	1.5	8.2	2.5	0.368
Electronic	8.5	1.7	8.7	1.3	8.3	2.1	0.223
Tegner Activity Level Scale-Current
Paper	3.7	2.8	3.8	2.7	3.7	2.9	0.863
Electronic	3.9	3.0	3.9	2.9	3.9	3.1	0.921
PedsQL Teen
Paper	70.6	14.1	70.1	13.3	71.2	14.9	0.707
Electronic	70.8	13.9	71.4	13.9	70.2	14.0	0.699
Visual Analogue Scale
Paper	5.3	2.3	5.6	2.3	4.9	2.1	0.110
Electronic	4.6	2.6	5.1	2.6	4.1	2.6	0.085

^*A two-sample t-test was used to compare the two groups.

Pedi-IKDC, Pediatric International Knee Documentation Committee; HSS Pedi-FABS, Hospital for Special Surgery Pediatric Functional Activity Brief Scale

Paper versus electronic

Table 2 shows the correlation and Table 3 shows the differences between paper and electronic PROMs. A high degree of reliability was noted when comparing the paper and electronic versions of the Pedi-IKDC, HSS Pedi-FABS, PedsQL Teen and the Tegner Activity Level Scale. When comparing the electronic and paper versions, the VAS scores indicated moderate to good reliability (ICC = 0.778; 95% confidence interval 0.63 to 0.861; p < 0.001). The greatest difference in scores was noted on the VAS, with paper scores being higher than electronic, with both Group 1 (p = 0.007) and Group 2 (p = 0.010) showing significant differences (Table 3). No differences were noted when comparing electronic and paper forms of the Pedi-IKDC, HSS Pedi-FABS, Tegner Activity Level Scale and PedsQL Teen.

Table 2.

The correlation of paper compared with electronic patient-reported outcome measures (PROMs) using an absolute agreement definition of intraclass correlation coefficients (ICC). Statistically significant p-values are noted in bold

		95% CI
PROM	ICC	Lower	Upper	p-value*
Pedi-IKDC	0.946	0.919	0.965	< 0.001
HSS Pedi-FABS	0.923	0.884	0.949	< 0.001
Tegner Activity Level Scale-Before	0.848	0.776	0.899	< 0.001
Tegner Activity Level Scale-Current	0.930	0.895	0.954	< 0.001
PedsQL Teen	0.894	0.842	0.930	< 0.001
Visual Analogue Scale	0.778	0.639	0.861	< 0.001

^*Intraclass correlation coefficients using an absolute agreement definition were used.

CI, confidence interval; Pedi-IKDC, Pediatric International Knee Documentation Committee; HSS Pedi-FABS, Hospital for Special Surgery-Pediatric Functional Activity Brief Scale

Table 3.

The comparison of paper patient-reported outcomes (PROMs) with electronic PROMs. Statistically significant p-values are noted in bold

	Overall		Paper		Electronic		Electronic-Paper
PROM	Mean	sd	Mean	sd	Mean	sd	Mean	sd	p-value*
Pedi-IKDC
Group 1	48.3	18.1	48.1	18.5	48.4	17.8	0.283	5.506	0.734
Group 2	48.8	17.7	48.3	18.2	49.3	17.4	0.996	6.265	0.309
HSS Pedi-FABS
Group 1	17.4	10.9	17.5	11.1	17.3	10.8	-0.295	5.079	0.701
Group 2	17.4	10.2	17.1	10.4	17.6	10.0	0.452	2.915	0.320
Tegner Activity Level Scale-Before
Group 1	8.7	1.4	8.6	1.5	8.7	1.3	0.093	0.895	0.499
Group 2	8.2	2.3	8.2	2.5	8.3	2.1	0.071	1.197	0.701
Tegner Activity Level Scale-Current
Group 1	3.8	2.8	3.8	2.7	3.9	2.9	0.068	0.818	0.583
Group 2	3.8	3.0	3.7	2.9	3.9	3.1	0.238	1.303	0.243
PedsQL Teen
Group 1	70.7	13.5	69.6	13.0	71.4	13.9	1.821	6.530	0.075
Group 2	70.7	14.4	71.2	14.9	70.2	14.0	-1.022	6.046	0.280
Visual Analogue Scale
Group 1	5.4	2.4	5.6	2.3	5.1	2.6	-0.545	1.266	0.007
Group 2	4.5	2.4	4.9	2.1	4.1	2.6	-0.738	1.781	0.010

^*A two-sample t-test was used to compare the groups.

Pedi-IKDC, Pediatric International Knee Documentation Committee; HSS Pedi-FABS, Hospital for Special Surgery Pediatric Functional Activity Brief Scale

Time and efficiency

Table 4 shows the comparison of time to complete outcomes measures between groups. Participants took a mean of 21.3 minutes to complete both sets of PROMs. As expected, both groups took significantly less time to complete the second set than the first set of measures (p < 0.001). While not significant, a trend was noted in which electronic PROMs took, overall, less time than paper (10.0 mins (5 mins to 30 mins) versus 11.2 mins (4 mins to 26 mins); p = 0.096). Additionally, when comparing the first set of measures for Group 1 and Group 2, paper PROMs took approximately 1.5 minutes longer to complete than electronic measures (13.6 mins (6 mins to 26 mins) versus 12.1 mins (6 mins to 30 mins); p = 0.164). The second set of PROMs for each group showed a similar trend, with electronic PROMs taking approximately 8.1 minutes (5 mins to 19 mins) compared with 8.8 minutes (4 mins to 14 mins) of paper measures (p = 0.152).

Table 4.

The comparison of time to complete patient reported outcome measures of the entire cohort, Group 1 and Group 2. For the comparison of Group 1 and Group 2 total times, statistically significant p-values are noted in bold

	Overall		Group 1		Group 2
Variable	Mean	sd	Mean	sd	Mean	sd	p-value*
Total time (mins)
Paper	11.2	4.3	13.6	4.4	8.8	2.3	0.000
Electronic	10.0	4.4	8.1	2.5	12.1	5.0	0.000
Paper and electronic combined	21.3	5.9	21.6	5.5	21.0	6.4	0.614
Electronic PROMs (mins)
Pedi-IKDC	2.8	2.2	2.1	0.9	3.6	2.8	0.001
HSS Pedi-FABS	2.0	1.1	1.5	0.9	2.5	1.2	0.000
Tegner Activity Level Scale	1.7	2.0	1.6	1.9	1.9	2.2	0.499
PedsQL Teen	2.4	1.1	1.8	0.6	3.0	1.2	0.000
Visual Analogue Scale	1.1	0.5	1.1	0.5	1.1	0.4	0.768

^*A two-sample t-test was used to compare groups.

Pedi-IKDC, Pediatric International Knee Documentation Committee; HSS Pedi-FABS, Hospital for Special Surgery Pediatric Functional Activity Brief Scale

Patient satisfaction responses

All participants endorsed that the information captured on the paper PROMs was the same as the electronic forms. More than two-thirds of participants (69.8%) stated a preference for the electronic format of the PROMs. Similarly, 67.4% of participants perceived the electronic format to be faster. Only 5.8% of participants reported the electronic forms to be hard or confusing to complete. Of the total participants, 93.0% stated they would complete forms at home, prior to appointments, if it were an option. Furthermore, 91.8% stated no concerns related to information safety or privacy of electronic forms. No differences were noted on the satisfaction survey between groups.

Discussion

PROMs are widely used in clinical practice and have been assessed in patients over the past four decades.²⁸ PROMs can enhance our understanding of patient experiences in a way that cannot be assessed through other means of medical technology.^28,29 The trend for increased use of PROMs in clinical care continues to highlight the role of patient experience as a key measure of health-care quality.³⁰ PROMs continue to be captured in multiple formats, but as our study results show, healthcare providers should consider use of electronic measures.

According to Campanella et al,³¹ electronic health records are often considered an ideal tool to be used to assess healthcare quality and monitor health providers’ performance because of the availability of stored computerized data.³¹ Similarly, electronic PROMs could be used to improve patient care and patient-provider communication, by allowing clinical staff easy, immediate access to monitor patient responses to standardized diagnostic, performance and satisfaction tools as well as give instantaneous feedback to the patient. Standardized PROMs are not only used for clinical research, but for performance assessment, benchmarking, quality improvement, monitoring patient progress and aiding in diagnosis.¹ One way to actively engage in thorough patient-provider communication is through the use of PROMs. The most important attribute to patient-centered care, is the active engagement of patients in discussions regarding their treatment and outcomes.³² Affording patients the opportunity to voice treatment feedback may lead to greater patient satisfaction with care provided.

In 2011, Common Sense Media published results from a national survey stating that approximately 72% of children, ages eight years and younger, live in a home with a computer with internet access.³³ Around the same time, a study from Lenhart and Campbell³⁴ showed that more than 75% of 12 to 17 year olds own cell phones, a 45% increase since 2004, with approximately 88% of teens identifying ‘texting’ as their preferred mode of communication. As of 2018, 95% of teens have access to a smartphone, with 45% reporting they are online almost constantly and an additional 44% reporting they go online several times a day.³⁵ With the technology advances observed over the last decade, and widespread availability of electronic resources to patient families, transitioning to electronic PROMs seems to be inevitable. Likewise, more than two-thirds of our population stated a preference for electronic PROMs over the paper format.

Electronic PROMs will afford some practice efficiencies while providing comparable scores with the paper formats. Approximately 93% of our study group stated they would complete electronic PROMs at home if available, reducing time spent in clinic. Forms completed at home prior to arrival will provide clinicians with the opportunity to review concerning answers prior to appointments in preparation for the patient visit. Electronic PROMs can reduce staff work hours and eliminate the need to transcribe paper PROMs into a compatible format for the electronic health record. The burden of data integration and analysis will be reduced. Additionally, in a study performed by Stone et al³⁶ in 2002, results demonstrated that although patients reported high compliance rates with paper forms, the actual compliance rates were much lower. Stone et al³⁶ also reported that compliance rates associated with electronic forms were much higher compared with paper.

Although not statistically significant, our population showed that electronic PROMs took less time to complete than paper, with more than 67% of the participants perceiving this to be true. Furthermore, attention should also be given to the elimination of staffing time spent to manually assess each PROM for completeness, score each measure, and, often times, manually enter each score into a database. Missed questions may often go unnoticed on paper forms, however, safeguards built into the electronic PROM system will eliminate the need to review forms thoroughly to identify overlooked or skipped questions. Although in a smaller population, after review of telephone, paper and electronic forms of a quality of life measure, Kruse et al¹⁹ was able to show trends that PROM delivery mode did not significantly change responses provided, similar to our results. Noted inconsistencies in reporting of pain on VAS should not be dismissed. Reported pain intensity may be influenced by assessment mode. Multiple pain assessment tools exist for paediatric providers including numeric, picture or colour scales. Additional studies may be needed to investigate this finding and to better understand the most accurate and consistent pain assessment format for this patient population.

One potential concern raised by the investigators was the issue of patient apprehension with electronic capture of health information as it relates to health information privacy. Public data breaches are often a concern of the general public, with multiple large companies documenting breaches over the last decade and throughout the conducting of the study.³⁷ While some studies do show that health information privacy is a concern of patients, this concern was not echoed by our participants.³⁸ More than 91% of the participants and families stated they had no concerns related to health information safety or privacy while completing electronic PROMs.

This study has limitations. The enrolled patients are reflective of a small subset of the clinical population, including only those patients ages 12 to 18 years, with knee pain. The inclusion of this focused patient population may present difficulties with generalizing the results to the clinical practice as a whole. Additionally, the selection of a ten-minute washout period may not have been sufficient to prevent recall bias. To allow for more time between modalities, electronic surveys taken off-site may provide the most favourable opportunity to eliminate potential recall.

Conclusion

PROMs captured electronically in a paediatric sports practice are reliable when compared with paper. Electronic PROMs may be quicker, will not require manual scoring and is the preferred method by patients and patient families compared with the paper method. Additionally, electronic PROMs will improve our ability to collect complete and reliable data while potentially reducing the burden on the clinical staff and patients and improving patient satisfaction.

Received 24 April 2019; accepted after revision 25 June 2019.

Appendix 1

Participants were asked to complete the following satisfaction survey questions:

1. Which form do you prefer to complete?

   Paper/Electronic

2. Do you believe they ask the same information?

   Yes/No

3. Which form do you think is faster to complete?

   Paper/Electronic

4. Was the electronic form hard to use or confusing?

   Yes/No

5. How long did it take you complete the paper form?

   2 minutes/5 minutes/10 minutes/20 minutes

6. How long did it take you to complete the electronic form?

   2 minutes/5 minutes/10 minutes/20 minutes

7. If available, would you complete at home on a phone, tablet, or computer prior to your appointment?

   Yes/No

8. Does the paper form make you concerned about your information being seen by others?

   Yes/No

9. Does the electronic form make you concerned about your information being seen by others?

   Yes/No

10. Which form do you think allowed your provider to learn the most about your condition and care?

    Paper/Electronic/Equal

Compliance with ethical standards

Funding statement

No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article.

Ethical statement

Ethical approval: This research involves human subject participants. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This study received institutional review board approval.

Informed consent: Informed consent was obtained from each participant.

ICMJE Conflict of interest statement

None declared.

Author Contributions

MJS: Study design, Data collection, Manuscript preparation.

CVG: Data collection, Manuscript preparation.

AJZ: Data collection, Manuscript preparation.

JSC: Data collection, Manuscript preparation.

SMM: Data collection, Manuscript preparation.

PLW: Study design, Manuscript preparation.

CHJ: Study design, Statistical analysis, Manuscript preparation.

HBE: Study design, Data collection, Manuscript preparation.