Electronic and paper mode of data capture when assessing patient-reported outcomes in the National Dental Practice-Based Research Network

Abstract

Aims:

Our objectives were to describe the approach used in the National Dental Practice-Based Research Network to capture patient-reported outcomes and to compare electronic and paper modes of data capture in a specific network study.

Methods:

This was a prospective, multicenter cohort study of 1,862 patients with dentin hypersensitivity. Patient-reported outcomes were assessed based on patients’ perception of pain using Visual Analog Scales and Labeled Magnitude scales at baseline and at 1, 4, and 8 weeks post-baseline.

Results:

Eighty-five percent of study patients chose to complete follow-up assessments via an electronic mode; 15% completed them via a paper mode. There was not a significant difference in the proportions of patients who completed the 8-week assessment when comparing the electronic mode to the paper mode (92% vs 90.8%, p-value=0.31, Rao-Scott clustered Chi-square test).

Conclusions:

The electronic mode of data capture was as operational as the traditional paper mode, while also providing the advantage of eliminating data entry errors, not involving site research coordinators in measuring the patient-reported outcomes, and not incurring cost and potential delays due to mailing study forms. Electronic data capture of patient reported outcomes could be successfully implemented in the community dental practice setting.

Introduction

Longitudinal studies are often challenging to administer due to financial constraints, time demands on the investigators and high attrition rates of the study participants. Numerous factors need to be considered when planning such a study to ensure success. ¹ Participants who have already enrolled in the study may be difficult to locate for follow-up assessments and/or refuse to attend the assessments, potentially compromising the internal and external validity of the study. ² When assessing “real-world” outcome measures related to existing treatments or treatment recommendations, capturing the patient’s subjective experience is critical. Doing so via direct patient input is more efficient and less costly than obtaining information that requires the time of clinical staff, and more accurately reflects patient-reported outcomes. ³ The evaluation of appropriate treatment often depends on health states that are reported by patients. Numerous rating scales, ranging from impairment scales, pain scales and health–related quality of life instruments have been used to determine the current condition of the patient. ⁴

The objectives of this study were to describe the approach used in the National Dental Practice-Based Research Network (National Dental PBRN) to capture patient-reported outcomes and to compare electronic and paper modes of data capture in a specific network study, called the Management of Dentin Hypersensitivity Study. The study’s main clinical findings have been reported previously. ^{5, 6} This manuscript is limited to reporting the methodological findings to capture patient-reported outcomes as outlined in the objectives above.

Materials and Methods

Management of Dentin Hypersensitivity Study

This was a prospective, multicenter cohort study of patients with dentin hypersensitivity. Practitioners and patients were recruited from all six network regions. The structure, purpose and function of the network are described elsewhere. ⁷ The study population consisted of adult dental patients (19 years old or older) who had dentin hypersensitivity and the practitioners who recommended treatment for their condition. The total number of patients that formed the study population was 1862. Patients were asked to rate the dentin hypersensitivity they experienced within the past day (24 hours) using two Visual Analog Scales of pain intensity and unpleasantness and four Labeled Magnitude scales to assess the duration, intensity, tolerability and pain description changes at 1, 4 and 8 weeks after baseline. Practitioners who participated completed all necessary human subjects and conflict of interest training as required by the network. The study was reviewed and approved by the institutional review board of the home institution of the lead investigator (D.K.K.), as well as those for each of the six network regions. Each patient provided informed consent to participate in the study. The inclusion and exclusion criteria for the patients are described elsewhere. ⁵

At the baseline visit, the practitioners completed the patients’ oral examination, confirmed the diagnosis of dentin hypersensitivity, completed a dental history form, and recommended and/or prescribed or applied treatment for dentin hypersensitivity. Study patients completed a demographic form and a patient-reported pain form.

Study Follow-up Procedures

Patient-reported outcomes were collected independently of in-office visits. Study patients had two choices to complete post-baseline assessments of their perception of pain related to dentin hypersensitivity, via an electronic or a paper modality; their preferred method of data collection modality was established at the enrollment visit.

Study patients who chose to complete patient-reported outcomes online, were asked to complete follow-up assessments on a designated web-based data entry portal 1, 4, and 8 weeks after the baseline visit. Patients who did not have Internet access and/or chose to complete paper study forms returned the study documents by mailing them to the designated research coordinator. A designated dental office person provided study documents with instructions to patients who preferred to return study data collection forms via postal mail. Based on patients’ preferences, study patients were contacted via a text message system or telephone message to their designated telephone number or received an email prior to each follow-up interval and were asked to complete the appropriate assessment. Two to three contact attempts per assessment were implemented. The research coordinators, with the assistance from the network’s coordinating center, followed up with patients who did not complete the web-based or mail surveys by the due date, to encourage them to complete study procedures. The study assessments windows were set within ±6 days of the due date.

The coordinating center integrated the Management of Dentin Hypersensitivity Study Management System within the existing architecture of the National Dental PBRN Practitioner Database system, a proprietary customized web-based system that tracks thousands of network participants. Special study tracking pages were established within the Study Management System to enable coordinating center staff to monitor progress. The study electronic database system was tested and validated among study practitioners and research coordinators in each network region before the production mode was implemented and the electronic mode of the delivery was deployed to the study patients. The coordinating center performed extensive testing on the Study Management System, patient web site and related processes. Testing consisted of three distinct phases: 1) Unit testing, where programmers tested their own work in a central integrated development environment; 2) IT integration testing, where a technical specialist tested the functions of the system operating together; and 3) Formal system testing, where testing specialists tested all components of the system from the end-user perspective, working from a detailed test plan.

The Study Management System generated auto-notifications to patients to remind them to complete their pain status in a timely manner. Patients’ payments for completed assessments were also tracked in the Study Management System. Patients who chose the electronic option were provided with unique PINs that enabled them to access a customized web site where they could report their pain status. Study patients who had chosen to complete their patient-reported outcome measures via postal mail, sent their responses to the designated research coordinator. The designated research coordinators measured the patient-reported outcome measures with a 10 mm ruler and provided the Visual Analog Scale and Label Magnitude scale readings and the paper forms to the coordinating center for manual data entry. All of the data collection forms used in this study are publicly available at http://nationaldentalpbrn.org/study-results/management-of-dentin-hypersensitivity.php.

Study patients received up to $125 for their participation - $25 upon completion of the baseline office visit, $25 for the follow-up assessment at 1 week, $25 for the 4-week assessment, and $50 upon study completion. Practitioners and patients were free to withdraw from participation in the study at any time upon request.

Data analysis

Descriptive statistics were calculated for patient characteristics by mode of data collection. The Rao-Scott cluster adjusted Chi-Square test and mixed model ANOVA were used to compare categorical and continuous variables, respectively, between the two modes, accounting for clustering within dentists. Statistical significances was set at p < 0.05. All statistical analyses were completed using SAS statistical software, version 9.4.

Results

A total of 1,862 patients formed the study population. Seventy-four percent of patients were females and 26% were males. The socio-demographic characteristics of the study patients and treatment modalities recommended for dentin hypersensitivity are described elsewhere. ^8,5 Eighty-five percent of the study patients chose to complete the follow-up assessments electronically and the remaining 15% chose to complete the assessments using a paper mode via postal mail. As shown in Table 1, certain characteristics of these two patient groups were significantly different. Patients who chose the paper mode tended to be older and less educated. The South Central region had a substantially higher percentage of patients who chose the paper mode (28%) and the Western region also had a somewhat higher percentage of patients who chose paper mode (19%); other regions were in a range of 9-13%.

Table 1.

Characteristics of patients, by mode of data collection

Variable	Assessment mode- Electronic	Assessment mode- Paper	P-values*
	N (%)	N (%)
Gender
Male	418 (86)	67 (14)	.36*
Female	1161 (84)	216 (16)
Education
Less than high school	5 (31)	11 (69)	<.0001*
High school	183 (73)	69 (27)
Some college or Associate degree	607 (84)	112 (16)
Bachelors’ degree	452 (89)	58 (11)
Graduate degree	323 (92)	30 (8)
Decline to answer	9 (75)	3 (15)
Region
Western	188 (81)	44 (19)	.0002*
Midwest	286 (87)	43 (13)
Southwest	273 (90)	33 (10)
South Central	235 (72)	93 (28)
South Atlantic	292 (91)	29 (9)
Northeast	305 (88)	41 (12)
Age
Mean (SD)	43.6 (13.06)	51.0 (14.83)	<.0001**

^*Rao-Scott cluster adjusted Chi-Square test

^**Mixed model ANOVA

The overall retention rate for the study patients was 92% (1,717 of 1,862) at the 8-week follow-up assessment. The final patient study status is presented in Table 2. Only 10 (.5%) patients were no longer willing to participate in the study and 4% of the study patients were lost to follow-up by the 8-week assessment.

Table 2.

The final study status of participants, by mode of data collection

Final study status	Electronic mode*	Paper mode*	Frequency (n)	Percentage (%)
8-week assessment completed	1,460 of 1,579 at baseline (93%)	257 of 283 at baseline (91%)	1,717 of 1,862 at baseline	92
Patient lost to follow-up	58	9	67	4
Patient has moved and cannot be contacted	0	2	2	.1
Patient no longer able to participate	1	2	3	.2
Patient no longer willing to participate	9	1	10	.5
Other: Including window for follow-up is closed	51	12	63	3

^*There was no significant difference in the proportion of patients who completed the 8-week assessment, when comparing those who completed the study using the electronic mode to those who completed the study using the paper mode, p-value=0.31, based on the Rao-Scott clustered Chi-square test.

There was not a significant difference in the proportions of patients who completed the 8-week assessment, between those who completed the study by an electronic mode (92.5%, 1460/1579) to those who completed the study via a paper mode (90.8%, 257/283; p-value=0.31, Rao-Scott clustered Chi-square test). Among the study subjects who did not complete the 8-week follow-up assessment, 82% (119/1579) had selected the electronic mode at baseline and 18% (26/283) had selected the paper mode.

A total of 7,662 reminders were utilized in the study. The majority of study patients (82%) were contacted via an email to complete the upcoming pain assessment; 9% of study patients were contacted via a text message and the remaining 9% via a telephone call. The number of reminders, based on the assessment week was almost equally distributed, with 35% of the reminders occurring in the 1^st week, 33% of the reminders occurring in the 4^th week and 32% of the reminders occurring in the 8^th week of the study. The mean number of the reminders per study patient was 4.1 with a SD of 1.2. A total of 5,151 first and 2,404 second reminders were sent during the study. Sixty reminders were sent after completion (post facto). The post facto reminder was created due to delays with getting the “Patient Information” form data in the database on time that caused reminders to be missed. The post facto reminder was sent automatically if the “Patient Information” form data had been entered in the database after the timeframe for the first automated reminder to be sent, which was 5 days after enrollment. Thirty-nine ad-hoc reminders were sent; they occurred when the email address bounced back.

There were 59 occurrences of missing data or invalid data entry on paper forms that were mailed to the research coordinators by the patients who chose to complete the follow-up assessment on paper. The research coordinators reconciled all errors.

Discussion

According to the Food and Drug Administration, a patient-reported outcome is “any report of the status of a patient’s health condition that comes directly from the patient without interpretation of the patient’s response by a clinician or anyone else”. ⁹ The patient’s experience has become increasingly important in evaluations of the effectiveness of dental products as it complements the dentist evaluations and other measures completed in a more traditional manner. Using electronic data capture for patient-reported outcome measures was convenient for both patients and study personnel but required human interactions beyond the automated system to maximize data quantity and quality. We tested the electronic data capture system for user acceptance before the study launch. We received positive feedback during the testing procedures. The electronic data capture system was easy to access and use. Based on our experience in this study, the electronic data capture system offered many advantages over the paper mode administration. It was user friendly, eliminated additional steps of mailing completed forms to the research coordinators, eliminated assessing the Visual Analog Scales and Label Magnitude pain outcome measures by the study research coordinators, and avoided the need for data entry by staff. In our sample, 85% of the study patients completed the patient-reported outcome measures using an electronic mode of data capture and 15% of the study patients chose a paper mode. Patient retention was high in the study, irrespective of the mode of data capture. We judge that a retention rate of 92% at the fourth, final assessment of patient-related outcomes is a desirable end result, especially given that the follow-up data were collected directly from the study patients without their dentist’s involvement. The online pain perception assessments were simple and easy to complete, they eliminated missing data or incorrect responses. Study patients (n=283) who completed the pain perception assessments on paper, were susceptible to missing data or invalid data; there were 59 occurrences of missing or incorrect data that were reconciled by research coordinators.

Of note, patients who completed patient-reported outcomes via paper mode were somewhat older and less educated when comparing to the patients who completed them via electronic modality. By assuring that potential study subjects were given a choice to complete pain assessments on paper or online, we reduced the potential for selection bias and fostered enrollment of a representative population of interest. As shown by the choices made by the patients enrolled in the study, offering only electronic data capture may have excluded a fraction of the population of interest. Since the immense explosion of computer and mobile devices technology, over-time electronic data capture will reach the general population of patients and the pen-and-paper approach may become obsolete; however, at this juncture some patients still prefer pen-and-paper approach. According to Coons and colleagues, the adaptation of electronic data capture to patient-reported outcomes that were originally developed for paper mode administration avoids secondary data entry errors, allows easier implementation of skip patterns and more accurate and complete data. ¹⁰ Bischoff-Ferrari and colleagues described using electronic data capture and compared it to the traditional pen-and-paper modality in patients with osteoarthritis; they concluded that electronic data capture was a reliable assessment tool when compared to the pen-and-paper modality. Overall, more than 50% of subjects preferred the computer format while 9% preferred the paper format and 38% were undecided. ¹¹ Patients with rheumatic diseases have expressed a need for frequent measurements of relevant outcomes, including patient-reported outcomes that are extensively collected with pen-and-paper questionnaires. Christie and colleagues described collection of patient-reported outcomes in patients with inflammatory rheumatic disease via text-messages on mobile telephones when compared to a pen-and-paper modality. ¹² Patient-reported outcomes assessed using numeric rating scales and reported with text-messages on mobile telephones or with pen-and-paper questionnaires provided comparable scores. Furthermore, the text-messages modality provided high response rates in a multicenter setting. ¹² Hawley and colleagues compared paper records and electronic health records for completeness. ¹³ They found that using an electronic health record significantly improved the collection of best-practice variables in the maternity clinic setting. The electronic health records were readily available to the relevant clinical staff and were easily retrieved when compared to the paper records. ¹³ The electronic mode of capturing patient-reported outcomes offers new opportunities to potential study patients and to study teams. According to Schick-Makaroff and colleagues, the benefits of capturing patient-reported outcomes electronically are: immediate access to retrievable data, immediate scoring and presentation of data, prevention of errors in data-entry, improved data quality, less missing information, high patient satisfaction with electronic platform and storage of longitudinal data. ¹⁴ The electronic data capture system can be cost-effective when compared to traditional data collection methods. ¹⁵

Recently, Trachtenberg and colleagues assessed the feasibility of collecting electronic pain data from thalassemia patients, via an automated telephone system. ¹⁶ Bushnell and colleagues measured the validity of the electronic versions of various measures that were used among asthmatic pediatric patients, including Visual Analog Scales, in comparison to the existing paper versions of these measures. ¹⁷ Overall, intraclass correlation coefficients were excellent when comparing a paper mode to an electronic mode. In 2011, Carvalho et al, developed and validated a system of electronic data capture for oral health surveys at a national level in Belgium. ¹⁸ They found a significant reduction of errors using electronic data capture when compared to paper. Both electronic and handwritten data capture were accurate; however, general practitioners performed better using electronic data capture.

The patient-reported outcomes in this study were completed at a satisfactory rate. Multiple factors contributed to the successful implementation of the study, including patient reminders and modest incentives, despite the fact that the patient-reported outcomes were completed outside of the dental offices. Study patients who did not complete their assessments within the study window via an electronic mode were not able to complete them outside of the study window as it was closed (potentially eliminating study protocol violations). Our main study limitation was a potential for selection bias, given that the practitioners and patients were selected from the National Dental PBRN. To our knowledge, this is the first manuscript describing the use of an electronic mode of data capture when reporting patient-reported outcomes in the community practice dental setting.

Conclusion

The electronic mode of data capture was as effective as the traditional paper mode when rating patients’ perception of pain related to dentin hypersensitivity, while also providing the advantage of eliminating data entry errors, not involving site research coordinators in measuring the patient-reported outcomes, and not incurring cost and potential delays due to mailing study forms. Patients who completed patient-reported outcomes via paper mode were somewhat older and less educated when comparing to the patients who completed them electronically. There was not a significant difference in the proportions of patients who completed the 8-week assessment when comparing the electronic mode to the paper mode. Electronic data capture of patient-reported outcomes can be successfully implemented in the community dental practice setting.