Validation of Remote Collection of Patient-Reported Outcomes Using Patients' Smartphones

Abstract

Backgroud

The purpose of this study was to examine the between-mode equivalence and the relative efficiency of the 2 available modes of patient-reported outcome (PRO) data collection: a web-enabled touch screen tablet and a smartphone in a sample of patients who underwent foot and ankle orthopedic surgery.

Methods

A total of 136 patients who visited the clinic after foot/ankle surgery participated in the study. All patients completed the PRO questionnaire set using tablets at the hospital. After 24 hours of completing the first PRO questionnaire, the patients completed the same PRO questionnaire at home using their personal smartphones. The outcomes were statistically compared, and the patients' preferences were surveyed.

Results

The intraclass correlation coefficients for comparing the results of PRO measurements between the 2 modes were 0.970 for the visual analog scale, 0.952 for the Foot Function Index, 0.959 for the foot and ankle outcome scale, and 0.957 for the patient's satisfaction. Sixty-eight participants (58.6%) responded that they were able to answer the questionnaires with more honesty at home using their smartphones. Regarding the mode, 60 participants (48.1%) responded that they have no preference between the devices.

Conclusions

The results of this study showed the equivalence of the 2 modes of PRO data collection: web-enabled touch screen tablets and smartphones. Smartphones may be the preferred mode of PRO measurement, due to their easy accessibility, increased privacy, and the patients' increased honesty in answering questionnaires.

A patient-reported outcome (PRO) is a self-reported outcome of the patient's state of health obtained without the researcher's input of analysis. PRO measurement is based on a stratified scoring system for patients' self-evaluation of their disease in light of the pre- or post-treatment symptoms, function, health-related quality of life, and satisfaction.¹⁾ The PRO stands in contrast to investigator-reported outcomes or conventional, objective evaluation tools such as laboratory tests and radiological examinations. The PRO allows for an accurate analysis of the patients' subjective perspective of the treatment results.²⁾

Although PROs have traditionally been collected through pencil-and-paper questionnaires, web-enabled tablets and computers are becoming a frequent method for data collection in recent years.³,⁴,⁵⁾ Both modes require patients to visit the healthcare facility and set aside additional time to fill out the questionnaire during their outpatient appointment. This is time-consuming and requires manpower. Furthermore, the hospital's environment and the presence of healthcare staff may psychologically influence the patient to give a biased self-evaluation. In addition, if the patient fails to visit the clinic, resulting in a follow-up loss, it would lead to data loss.

In 2016, a global median of 43% people owned a smartphone, which is defined as a cell phone that can access the internet and its applications. Among the highest reported rate of smartphone owners includes 88% of South Koreans, 77% of Australians, 74% of Israelis, 72% of Americans, and 71% of Spaniards.⁶⁾ The high rate of smartphone ownership allows smartphones to be a highly accessible and efficient mode of measuring PROs. A few applications have recently been developed to collect PROs using smartphones (e.g., Epic, Cerner, and Proscore).⁷⁾ However, different modalities in different environments could result in varied outcomes. The use of smartphones as a mode of remote collection of PRO data has not been validated.

Therefore, the purpose of this study was to examine the between-mode equivalence and the relative efficiency of the 2 available modes of PRO data collection: web-enabled touch screen tablets and smartphones in a sample of patients who underwent foot and ankle orthopedic operations.

METHODS

The study was conducted retrospectively by reviewing medical records and PROs. We conducted this study in compliance with the principles of the Declaration of Helsinki. The design and protocol of this retrospective study were approved by the Institutional Review Board of Inha University Hospital (IRB No. 2020-03-015). Written informed consents were waived since this study was conducted retrospectively.

The study included 116 patients who underwent foot or ankle surgery at the Inha University Hospital between September 2018 and August 2019. The median age was 35.8 years (range, 15–72 years). Thirty-seven patients underwent ligament repair surgery for chronic lateral ankle instability, 36 patients underwent correctional osteotomy for hallux valgus, 23 patients underwent cartilage regeneration operations for osteochondral lesions of the talus, and 20 patients underwent correctional osteotomies for ankle osteoarthritis. Patients who did not own smartphones or were not able to complete the PRO questionnaire independently and those who experienced changes in their operation site between the 2 PRO measurements were excluded. The study also excluded patients who did not complete the same PRO questionnaire using personal smartphones within 24 hours of receiving the web link as a text message.

PRO Questionnaire Set

The PRO questionnaire set used in this study was composed of the Korean language version of the visual analog scale (VAS), Foot Function Index (FFI),⁸⁾ Foot and Ankle Outcome Score (FAOS),⁹⁾ and an assessment of patient satisfaction. The questionnaires were displayed in the order of VAS, FFI, FAOS, and patient satisfaction.

The VAS is a simple and commonly used pain intensity scale where a patient places a mark on a 100-mm horizontal line that represents a continuum between “no pain” at the leftmost and “worst pain” at the rightmost portion. It is often used in the evaluation of adult patients' pain intensity, scaling acute pain, and pain management.¹⁰,¹¹⁾ A study by Ponkilainen et al.¹²⁾ has shown the validity of using the VAS to provide the best targeting and coverage for foot and ankle patients. FFI and FAOS are also validated PROs in the field of foot and ankle orthopedic surgery.¹³⁾ Patient's satisfaction was measured on a 0–10 point scale, with 0 representing extremely unsatisfied and 10 representing extremely satisfied.

Study Design

This study utilized Proscore (MDdatasolution, Seoul, Korea), an open platform that allows digital collection of PROs on both tablets and smartphones. For the first mode of PRO measurement, a touch screen tablet was used. The participants completed the first set of PRO questionnaires during their visit to the outpatient department (OPD). The research staff showed them how to use the touch screen tablet prior to starting the questionnaire set. They were required to answer the questionnaires' items without any assistance from others. If technical problems with the tablet arose, they were allowed to seek assistance from the research staff. A 24-hour time interval was incorporated into the study design before another PRO questionnaire set was sent to participants' smartphones to prevent them from answering duplicate questions based on the memory of responses they provided on the tablets.

For the second mode of PRO measurement, personal smartphones were used. A scheduled transfer system via the Proscore software was utilized, which allowed the participants to receive a personalized web link for PRO assessment via a text message in 24 hours after their last OPD visit. After verifying their identity, the link directed the participants to the PRO questionnaire set on the website (Figs. 1 and 2). In order to reduce the chance of including participants who changed their operative site, the study only included participants who completed the questionnaires within 24 hours upon receiving the text message. After the participants completed the second questionnaire, the research staff called each participant to check if their surgery site changed during the interval between the first and the second questionnaire sets. Also, they were asked which mode they had difficulty using, they preferred, and they answered with more honesty.

Fig. 1. Study algorithm. PRO: patient-reported outcome.

Fig. 2. Photos of a patient completing the electronic questionnaire using the patient's own smartphone.

The consistency between the 2 modes was verified using intraclass correlation coefficients and IBM SPSS ver. 19.0 (IBM Corp., Armonk, NY, USA). An alpha of 0.05 was used for the cutoff of statistical significance.

RESULTS

Of the 136 patients who agreed to participate in this study, 116 finished the PRO questionnaire set within 24 hours of receipt. The response rate for those who answered the PRO measurement on a smartphone was 85.2%. Data from the 116 patients who completed both modes of PROs were included in the statistical analysis. The intraclass correlation coefficients for the comparison of the results of the PRO measurements between the 2 modes were 0.970 for VAS, 0.952 for FFI, 0.959 for FAOS, and 0.957 for patient's satisfaction (Table 1).

Table 1. Assessment of Equivalence Using the ICC.

Variable	In-hospital tablet mode	Remote smartphone mode	ICC	p-value
Visual analog scale	2.5 ± 2.8	2.8 ± 2.8	0.970	< 0.001
Foot Function Index	21.1 ± 19	22.3 ± 22.1	0.952	< 0.001
Foot and Ankle Outcome Score	74.3 ± 16.6	74.8 ± 17.7	0.959	< 0.001
Patient satisfaction	9.3 ± 1.4	9.2 ± 1.7	0.957	< 0.001

Values are presented as mean ± standard deviation.

ICC: intraclass correlation coefficient.

Regarding the location where they answered the PRO, 68 participants (58.6%) responded that they were able to answer the questionnaire set with more honesty at home using their smartphone. Their reasons for increased honesty were as follows: ample time and space to complete the questionnaire at their convenience, psychological comfort, decreased distraction, increased concentration, and familiarity with their personal smartphones. Thirty-two participants (27.6%) responded that the place they answered or the modality of the questionnaire did not affect their attitude in answering questions (Table 2).

Table 2. Patient's Preference between the Two Modes of Data Collection.

Variable	In-hospital tablet mode	Remote smartphone mode	No difference	Sum
Honesty	16 (13.8)	68 (58.6)	32 (27.6)	116
Mode preference	44 (37.9)	12 (10.3)	60 (48.1)	116

Values are presented as number (%).

Regarding the patients' preference between 2 modes: 60 participants (48.1%) responded that they have no preference in using either mode of the PRO measurement, 44 participants (37.9%) responded that they prefer the tablet mode, and 12 participants (10.3%) responded that they prefer the smartphone mode (Table 2). The advantage of the tablet mode was the big screen size, and the disadvantage was unfamiliarity with touch screen sensation. The advantage of the smartphone mode was its familiarity and touch screen sensation, while the disadvantage was the relatively smaller screen size, which may cause clicking errors. There were no differences between the sexes in preference with regard to honesty (p = 0.574) or mode preference (p = 0.478). Due to the small number of patients with age greater than 60 years, we were not able to analyze the reliability in the elderly group (Table 3).

Table 3. Major Opinions Regarding Preference between the Two Modes of Data Collection.

Variable	In-hospital tablet mode	Remote smartphone mode
Advantage	Can ask questions to the hospital staff	Sufficient time and space
		Decreased distraction by other patients
		Psychological comfort
		Increased concentration
		Familiarity with their personal smartphones
Disadvantage	Insecure environment	Unable to ask questions
	Insufficient time
	Difficulty in concentration

DISCUSSION

The results of this study showed the high equivalence between the 2 modes of PRO data collection: the web-enabled touch screen tablet and smartphone. These results validate using the smartphone mode in tandem with the tablet mode for PRO data collection.

There are many advantages in using electronic devices as collection tools for PRO. The collected data can be saved, processed, and browsed immediately on the server network. This saves excessive time and energy required for the paper-and-pencil method. It also decreases data omission and subsequently prevents faulty or incomplete data.¹⁴⁾ However, there are still some limitations. Electronic devices such as a tablet can only be used when the patient visits the hospital. Using a smartphone as a remote data collection device does not necessitate the patient to visit the hospital and therefore does not require additional manpower. Therefore, using a smartphone is cost-effective both for the patient and the institution. The research staff can also save time in receiving and recording the follow-up data, allowing them to concentrate on patients in the outpatient clinic.

If the smartphone is used as a data collection method, the patient can complete the questionnaire at their own convenience. Also, the patient can do so in an environment that allows enough time and concentration without pressure from the medical staff. This may lead to more honest and accurate responses and help improve the integrity and quality of the data collected. Furthermore, the smartphone surveys will help clinicians identify any patients with a deteriorating health status, allowing clinicians to encourage such patients to visit the hospital for further evaluation. The easy accessibility of patients to a tertiary or a university hospital causes overcrowding, which inevitability leads to a busy environment. In such cases, it can be difficult to conduct a research survey, which requires additional time and manpower. Thus, remote collection of PRO data using smartphones can result in higher efficiency.

In this study, 68 participants (58.6%) answered that they were able to respond to questions more sincerely at home using their smartphones than at the hospital using tablets (Table 2). Their reasons for increased honesty were as follows: ample time and space to complete the questionnaire at their convenience, decreased distraction caused by the presence of other patients, psychological comfort, increased concentration, and familiarity with their personal smartphones. In this study, there was no significant difference in the PRO depending on the location of the implementation. However, there could be occasions where the PRO questionnaires completed at home and at hospital do not match. In such cases, the question of which result more accurately reflects the patient's state of health should be answered.

While 48.1% did not have any preference between the 2 devices, 37.9% preferred the tablet due to the large size of the screen. The difficulty in using the smartphone may arise due to smartphone's small touch screen size. This can be a problem especially for questions using a transverse scale bar. The transverse scale bar is only 5–6 cm long depending on the size of the phone, and accurately manipulating it with the fingertip can be difficult. A modification by changing the bar to a radial type can be considered to minimize discomfort.

An important issue regarding remote data collection is the low compliance rate. When data collection is conducted at the hospital, the medical staff is able to request the patient to complete a questionnaire in a controlled setting, which increases the reliability and compliance rate. In contrast, when data collection is done remotely in an uncontrolled environment, the compliance rate or the rate of finishing the questionnaire can decrease. In the current study, despite the participants' understanding and agreement to participate in this study, 14.7% of the patients failed to complete the questionnaire at home. A solution to increase the compliance rate should be developed.

A limitation of this study is the possible inclusion of insincere responses. In an uncontrolled setting, patients can be disturbed by unexpected events or factors when answering a questionnaire. This could result in an extended period in completing the questionnaire. In contrast, insincere responses would require a very short duration to finish the questionnaire. A technical algorithm to detect and exclude such insincere responses should be devised. In order to reduce response bias due to insincere participants, it can be helpful to measure the total time it takes for the participants to complete the questionnaire and to exclude those who take significantly small or large amount of time. This study did not take time into account; thus, we recommend future studies to implement measures to avoid dishonesty or conditions that may force a biased response. The outcomes of this study should be interpreted with caution due to the age distribution of the study group. The median age was 35.8 years (range, 15–72 years), which is relatively young, and this study included few elderly patients. Elderly patients may have more difficulty in using smartphones. Further investigation is required in the future regarding the compliance and the reliability of remote data collection using smartphones in the elderly. To minimize bias, it is important to set an appropriate time interval between questionnaires. According to a meta-analysis on the equivalence of electronic and paper administration of PRO measures, studies with a shorter interval between administrations were associated with greater equivalence.⁴⁾ A 24- to 48-hour time interval was incorporated into this study considering that a less than 24-hour time interval could cause bias due to the memory effect³⁾ and a greater than 48-hour time interval, due to the change in the condition of the operative site.

A remote implementation of PRO data collection using the patient's smartphone seemed to provide equivalent outcomes compared to in-hospital data collection performed using a tablet PC. Answering the PRO assessment questions remotely using smartphones allowed the patients to go through the questionnaire in a more comfortable environment, resulting in more honest responses.