Validity and Efficiency of a Smartphone-Based Electronic Data Collection Tool for Operative Data in Rotator Cuff Repair

Abstract

Purpose

This study tested validity and efficiency of Orthopaedic Minimal Data Set (OrthoMiDaS) Episode of Care (OME).

Methods

100 isolated rotator cuff repair cases in the OME database were analyzed. Surgeons completed a traditional operative note and OME report. A blinded reviewer extracted data from operative notes and implant logs in electronic medical records by manual chart review. OME and EMR data were compared with data counts and agreement between 40 variables of rotator cuff pathology and repair procedures. Data counts were assessed using raw percentages and McNemar’s test (with continuity correction). Agreement of categorical variables was analyzed using Cohen’s Kappa (κ; unweighted) and of numerical variables using the concordance correlation coefficient. Efficiency was assessed by median time to complete.

Results

OME database had significantly higher data counts for 25% (10 of 40) of variables. A high level of proportional and statistical agreement was demonstrated between the data. Among 35 categorical variables, proportional agreement was perfect for 17%, almost perfect (0.81 ≤ κ ≤ 1.00) for 37%, substantial (0.61 ≤ κ ≤ 0.80) for 20%, moderate (0.41 ≤ κ ≤ 0.60) for 14%, fair (0.21 ≤ κ ≤ 0.40) for 6%, and slight (0.0 ≤ κ ≤ 0.20) for 6%. Of 5 numerical variables, agreement was almost perfect (CCC > 0.99) for 20%, and poor (CCC < 0.90) for 80%. Median OME completion time was 161.5 seconds (IQR 116 – 224.5).

Conclusion

OME is an efficient, valid tool for collecting comprehensive, standardized data on rotator cuff repair.

Level of Evidence

Basic Science Study; Development or Validation of Outcomes Data Collection Tool

As health care in the United States continues to transition from volume-based to value-based compensation models, it is increasingly important to provide services that maximize value, which is defined as quality divided by cost.^17–19 Consequently, research will continue to focus on identifying which interventions provide the best outcomes (high quality), and at what cost, so that value can be accurately assessed. For surgical procedures, defining value necessitates identifying specific variables that are associated with quality outcomes (e.g., preoperative health status, operative technique, intraoperative pathology, etc.), as well as the cost associated with the procedure, including implant costs and other costs associated with the episode of care.

Rotator cuff repair (RCR) is one of the commonest orthopedic procedures performed and multiple studies have demonstrated a significant increase in the rate of RCR over the last few decades.^{6; 12; 25} While most patients report clinical improvement with the procedure,^{3; 11; 14} the factors responsible for poor outcomes in the proportion of patients that remain dissatisfied with their outcome are not well defined.¹⁴ This includes the surgical technique of rotator cuff repair itself, as there is currently no “gold standard” repair method. There is also no consensus on the most effective and least costly way to repair a torn rotator cuff, ³ and costs vary depending on the surgical techniques employed. Thus, defining the true value for RCR merits continued research regarding outcomes, predictors of outcomes, and economic analyses.

Historically, collecting such data has relied almost exclusively on manually reviewing operative notes and patient charts and abstracting the desired data. ^{8; 9} Like the rest of the electronic medical record (EMR), operative notes regularly underreport quantitative information and lack standardization.^{18, 22} Further, manual abstraction of data is associated with error rates between 8–23% depending on clinical site, area, surgical specialty, and medical provider.¹⁵ Reducing such errors by increasing quality assurance measures and a more thorough data review, increases time and cost.¹⁵ In contrast, large, structured datasets can serve to catalog variables that potentially impact outcomes and/or costs and allow for multivariable analysis to determine the primary predictors of interest, particularly when these variables are uniformly defined and collected. However, many existing structured datasets, such as Medicare, Nationwide Inpatient Sample (NIS), and ACS National Surgical Quality Improvement Program (ACS NSQIP), are not disease-specific, consistent or standardized, thus creating incomplete datasets and limiting the ability to conduct high quality studies. To our knowledge, no large database currently exists which has been designed to capture high-quality, prospective, standardized data surrounding rotator cuff repair techniques.^{3, 20}

To address this need, the Cleveland Clinic has developed the Orthopaedic Minimal Data Set (OrthoMiDaS) Episode of Care (OME) system, a prospectively designed electronic data collection tool to enable cost-effective, scientifically valid, and scalable data collection of patient- and surgeon-reported data surrounding a surgical episode for elective knee, hip, and shoulder surgery. The purpose of this study was to demonstrate the validity and efficiency of the OME system for collecting comprehensive and standardized data on rotator cuff repair (RCR). The authors hypothesized that, compared to a dataset constructed by manually abstracting data from traditional narrative operative notes, the OME system would demonstrate higher completion rates for clinically relevant variables with at least substantial agreement with the operative note, while not burdening surgeons with lengthy completion times.

Methods

OME System Design

The OME system was designed by a multidisciplinary team consisting of software developers, orthopedic surgeons and orthopedic administrators. It is a software-as-a-service system that builds upon the Research Electronic Data Capture (REDCap) platform to enable cost-effective, scientifically valid, and scalable data collection of patient- and surgeon-reported data surrounding a surgical episode, wherein data collection is integrated within the existing work flow.^10;24 Patients undergoing elective knee, hip, and shoulder surgery systematically enter demographic data and patient-reported outcome measures (PROMs) immediately prior to surgical intervention and at one year postoperatively, while their surgeons receive an email link for each operation on day of surgery. The data are entered prospectively with surgical procedural details using a smartphone, laptop, or desktop computer within 48 hours of the surgery. Procedural details include past surgical history, examination under anesthesia findings, and commonly-cited operative note parameters and key predictors of operative outcomes for the type of surgery being performed (such as RCR) as identified from the literature or by expert opinion of the clinicians involved in the system’s design.²³ OME was launched at Cleveland Clinic in February 2015, and by February 2018 OME had successfully captured baseline data (PROMs and surgeon-entered variables) on 97% of 21,500 eligible elective orthopedic knee, hip, and shoulder surgical procedures. OME’s design and implementation were approved by Cleveland Clinic’s Institutional Review Board and the system was vetted by the Information Security Department.

Patient Selection

This study included Cleveland Clinic patients representing the first one hundred cases of isolated rotator cuff repair (arthroscopic or open) that were collected into the OME database starting in February 2015. Patients with commonly associated procedures (subacromial decompression, biceps tenotomy or tenodesis, distal clavicle excision) were included, while those undergoing another major repair or reconstruction, including labral repair, shoulder arthroplasty, and fracture fixation were excluded.

OME Data Collection

Specifically for RCR, OME collects data on patient demographics, rotator cuff pathology and repair procedures, and additional pathology and surgical procedures performed (Table 1). Built-in branching logic is used to accelerate data entry, showing only fields applicable to each pathology and procedure, which prevents surgeons from having to answer irrelevant or unnecessary items. When RCR is selected as a surgery within OME, the surgeon views fields with choices for rotator cuff tendons repaired and tear size, the type of rotator cuff tear (i.e., partial-thickness versus full-thickness), and type of repair performed (i.e., single row versus double row repair techniques). Based on the repair type, branching logic presents the surgeon with the next group of applicable fields, including specific manufacturers and types of implants/devices used. Incomplete forms cannot be submitted within OME, which ensures that a standardized and complete dataset is obtained for every patient surgery. Total time required to complete the surgical form is also collected to measure caregiver burden. For RCR, a total of 40 variables related to rotator cuff tendon pathology and repair procedures were extracted for comparison with data from the EMR (Table 1, Supplementary Table).

Table 1:

EMR (operative note/implant log) and OME data counts

Variable	EMR	OME	P-Value§
Operative limb	100	100	> 0.99
Left shoulder past surgical history	4	100	< 0.001
Right shoulder past surgical history	8	100	< 0.001
Subscapularis
- Status	65	100	< 0.001
- Tear type	19	25	0.041
- Tear size	3	25	< 0.001
- Repaired	19	25	0.041
- Reason not repaired	2	7	0.074
- Repair type	17	18	> 0.99
- Repair approach	17	18	> 0.99
- Repair rows	17	18	> 0.99
- Fixation	17	18	> 0.99
- Number of anchors	17	17	> 0.99
- Implant manufacturer	17	17	> 0.99
- Implant make & model	17	17	> 0.99
- Augmentation	0	18	< 0.001
Superior-posterior
- Status	100	100	> 0.99
- Tear type	98	100	0.48
- Supraspinatus involved	100	100	> 0.99
- Infraspinatus involved	100	100	> 0.99
- Teres minor involved	100	100	> 0.99
- Tear size	54	100	< 0.001
- Repaired	100	100	> 0.99
- Reason not repaired	3	3	> 0.99
- Repair type	97	97	> 0.99
- Repair approach	96	97	> 0.99
- Repair rows	90	97	0.023
- Double row repair type	47	46	> 0.99
- Fixation	97	97	> 0.99
- Number of superior-posterior anchors	43	46	0.371
- Implant manufacturer	43	46	0.371
- Implant make & model	43	46	0.371
- Number of medial row anchors	47	46	> 0.99
- Medial row implant manufacturer	46	46	> 0.99
- Medial row implant make & model	46	46	> 0.99
- Number of lateral row anchors	47	46	> 0.99
- Lateral row implant manufacturer	46	46	> 0.99
- Lateral row implant make & model	46	46	> 0.99
- Number of superior-posterior sutures	3	5	0.48
- Augmentation	5	97	< 0.001

EMR Data Collection

The surgeons’ narrative operative notes and the implant logs were obtained from Epic EMR system (Epic Systems, Verona, WI, USA) for this patient cohort and queried for the same 40 variables extracted from OME by an examiner blinded to the OME data. When information about implanted anchor(s) was not specifically stated in the operative note (e.g. number, manufacturer, type), the information was obtained from the implant log. When the specific type of partial rotator cuff tear (articular or bursal-sided) was not explicitly stated in the operative notes, tear type was determined from the description of the arthroscopy procedure. When information on a variable was not present in the operative note, it was considered as “absent” in reporting data counts, but as an “implied negative” for agreement analysis. (i.e., surgeons intentionally omitting non-applicable information with the implicit understanding that the item was negative or not present). Specifically, an “implied negative” was assumed when the operative note was silent on the past surgical history of the left/right shoulder or graft augmentation of the rotator cuff repair because it was felt that surgeons would more likely than not to mention these points in the operative notes if they were applicable.

Statistical Analysis

Before the EMR and OME datasets were statistically compared, discrepancies between the two datasets were identified and all un-matched data were rechecked and verified. Subsequently, the EMR and OME data were analyzed for data counts and agreement. Data counts were analyzed by comparing raw percentages as well as with McNemar’s test (with continuity correction). Agreement of categorical variables was analyzed using Cohen’s Kappa (k; unweighted) ¹³ and of numerical variables using the concordance correlation coefficient (CCC).⁷ A 95% confidence interval was calculated for all values of agreement. In addition to these formal agreement metrics, the raw proportion of records for each variable showing agreement were also used to assess raw proportional agreement. Data was analyzed with R software (R version 3.3.3 (2017–03-06), Vienna, Austria).

Results

The first one hundred cases of isolated rotator cuff repairs (arthroscopic or open) were entered into the Cleveland Clinic’s OME database by 10 surgeons between February 2015 and May 2015. The median time to complete OME surgery data entry for these 100 cases was 161.5 seconds (IQR 116 – 224.5 seconds).

Data Counts

EMR and OME data counts of the 40 variables assessed are listed in Table 1. OME demonstrated significantly higher counts in 10 (25%) of the variables assessed (p < 0.05). Notably, superior-posterior rotator cuff tendon tear size was mentioned in 54 of 100 cases in EMR but in all 100 cases in OME. Similarly, subscapularis tendon tear size was mentioned in 3 of 100 cases in EMR and in 25 of 100 cases in OME. Further, repair of the subscapularis tendon was mentioned in 19 of 100 cases in EMR, while in 25 of 100 cases in OME. In 30 variables, there were no significant differences in data counts between EMR and OME.

A survey of the cases where data were recorded less frequently in the EMR than OME did not reveal any bias/pattern in the absent data (e.g., it was not that only “minor” or “less severe” occurrences were not mentioned in the EMR). For example, of the 46 cases where data on superior-posterior rotator cuff tendon tear size was absent in the EMR, the corresponding OME data from those patients revealed that 5 small, 11 medium-sized, 13 large, and 12 massive tears were not mentioned in the EMR.

Data Agreement

Agreement proportions and associated Cohen’s unweighted Kappa values for each of the 35 categorical variables collected are listed in Table 2. Proportional agreement >90% was observed in 77% (N=27) of the categorical variables. Agreement was perfect for 17% (N=6), almost perfect (0.81 ≤ κ ≤ 1.00) for 37% (N=13), substantial (0.61 ≤ κ ≤ 0.80) for 20% (N=7), moderate (0.41 ≤ κ ≤ 0.60) for 14% (N=5), fair (0.21 ≤ κ ≤ 0.40) for 6% (N=2), and slight (0.0 ≤ κ ≤ 0.20) for 6% (N=2) of variables.

Table 2.

Agreement between the EMR (operative note/implant log) and OME among categorical variables

Measure	Records Used	Proportional agreement	κ	95% Confidence Interval
Operative limb	100	0.98	0.960 AP	(0.905, 1.000)
Left shoulder past surgical history	100	0.94	0.545 M	(0.193, 0.898)
Right shoulder past surgical history	100	0.93	0.596 M	(0.307, 0.885)
Subscapularis
- Status	65	1.00	1.000 AP	(1.000, 1.000)
- Tear type	19	0.74	0.379 F	(−0.088, 0.846)
- Tear size	3	0.67	0.400 F	(−0.560, 1.000)
- Repaired	19	1.00	1.000 AP	(1.000, 1.000)
- Reason not repaired	2	1.00	1.000 AP	(1.000, 1.000)
- Repair type	17	1.00	N/A°	N/A
- Repair approach	17	1.00	N/A°	N/A
- Repair rows	17	1.00	N/A°	N/A
- Fixation	17	0.94	0.000 SL	(−1.000, 1.000)
- Implant manufacturer	16	1.00	1.000 AP	(1.000, 1.000)
- Implant make & model	16	0.62	0.518 M	(0.212, 0.823)
- Augmentation	18	1.00	N/A°	N/A
Superior-posterior
- Status	100	1.00	N/A°	N/A
- Tear type	98	0.93	0.809 SU	(0.672, 0.945)
- Supraspinatus involved	100	0.88	0.760 SU	(0.633, 0.888)
- Infraspinatus involved	100	1.00	1.000 AP	(1.000, 1.000)
- Teres minor involved	100	1.00	1.000 AP	(1.000, 1.000)
- Tear size	54	0.78	0.703 SU	(0.554, 0.851)
- Repaired	100	1.00	1.000AP	(1.000, 1.000)
- Reason not repaired	1	1.00	N/A°	N/A
- Repair type	97	0.96	0.693 SU	(0.398, 0.988)
- Repair approach	96	1.00	1.000 AP	(1.000, 1.000)
- Repair rows	90	0.97	0.933 AP	(0.859, 1.000)
- Double row repair type	45	0.49	0.161 SL	(−0.079, 0.400)
- Fixation	97	1.00	1.000 AP	(1.000, 1.000)
- Implant manufacturer	42	0.98	0.927 AP	(0.787, 1.000)
- Implant make & model	42	0.71	0.661 SU	(0.499, 0.823)
- Medial row implant manufacturer	44	1.00	1.000 AP	(1.000, 1.000)
- Medial row implant make & model	44	0.75	0.668 SU	(0.498, 0.838)
- Lateral row implant manufacturer	44	0.98	0.896 AP	(0.695, 1.000)
- Lateral row implant make & model	44	0.98	0.896 AP	(0.695, 1.000)
- Augmentation	97	0.99	0.663 SU	(0.006, 1.000)

^°Agreement statistics cannot be calculated in situations in which the operative note and OME are in complete agreement with only one variable appearing in each data source

^APIndicates almost perfect agreement based on the κ statistic

^SUIndicates substantial agreement based on the κ statistic

^MIndicates moderate agreement based on the κ statistic

^FIndicates fair agreement based on the κ statistic

^SLIndicates slight agreement based on the κ statistic

Implied negatives were assumed in the EMR for three variables: past surgical history on left/right shoulder and graft augmentation of the subscapularis/superior-posterior tendon repair.

The concordance correlation coefficients (CCC) for each of the five numerical variables compared between the operative note and OME are listed in Table 3. Agreement was almost perfect (CCC > 0.99) for 20% (N=1), and poor (CCC < 0.90) for 80% (N=4) of the numerical variables.

Table 3.

Agreement between EMR (operative note/implant log) and OME among numerical variables

Measure	Records Used	Proportional agreement	CCC	95% Confidence Interval
Number of subscapularis anchors	16	0.88	0.448 P	(−0.267, 1.000)
Number of superior-posterior anchors	42	0.93	0.881 P	(0.789, 0.934)
Number of superior-posterior medial row anchors	45	0.96	0.889 P	(0.807, 0.937)
Number of superior-posterior lateral row anchors	45	0.82	0.755 P	(0.595, 0.857)
Number of superior-posterior sutures	3	1.00	1.000 AP	N/A

Concordance correlation coefficient (CCC)

^APIndicates almost perfect agreement based on the CCC

^PIndicates poor agreement based on the CCC

Discussion

The purpose of this study was to demonstrate the validity and efficiency of OME for collecting comprehensive and standardized data on rotator cuff repair. Our reporting shows that OME allowed collection of preoperative and intraoperative data relevant to rotator cuff repair in a comprehensive and time-efficient manner. Data on patient demographics, rotator cuff pathology and repair procedures, and additional pathology and surgical procedures performed, were captured by the surgeon on average in less than three minutes per case.

Compared to the EMR (operative note/implant log), OME demonstrated greater counts of data capture for 25% of surgical variables relevant to RCR. Additionally, absent data in operative notes appeared to be generalized and systematic across multiple categories. Notably, superior-posterior rotator cuff tendon tear size was mentioned in 54 of 100 cases in EMR but in all 100 cases in OME. Similarly, subscapularis tendon tear size was mentioned in 3 of 100 cases in EMR and in 25 of 100 cases in OME. The absence of a key variable like tear size for a large proportion of a rotator cuff repair cohort in the EMR demonstrates why retrospective research and quality assessment from the EMR is particularly challenging and reveals the advantage of utilizing a prospectively designed and standardized database like OME. Templates have been previously used to generate operative notes, but these lack the streamlining features of branching logic and do not always require key variables to be entered into the dataset.¹⁶ The branching logic implemented in OME allows for efficient yet relevant and complete surgical documentation by streamlining the operative note process and capturing data on a prospectively designed set of clinically relevant variables. The discrete data collection format of OME because of its underlying REDCap platform also allows for efficient data extraction and retrieval, in contrast to manual data abstraction from operative notes that is associated with high error rates, and increased time and cost.¹⁵

The validity of data captured in OME was demonstrated by a high level of proportional and statistical agreement between data that existed in both datasets. This was true particularly for the categorical variables, where 77% of the variables demonstrated >90% proportional agreement, and 74% demonstrated at least substantial statistical agreement (Cohen’s unweighted κ ≥ 0.61) between the two data sources. With regard to numerical variables, OME data demonstrated poor statistical agreement with EMR data, as 80% of the variables assessed had CCC below 0.90. Since all of the numerical variables exhibited >80% proportional agreement, the corresponding low CCC indicates that when discrepancies did occur, they tended to be large.

Discrepancies between EMR and OME could arise for multiple reasons. For example, many of the discrepancies can be explained by lack of standardized content and verbiage in the surgeons’ dictated operative notes whereas standardized responses were mandated in the prospectively-designed OME database. For example, some surgeons dictated detailed descriptions of lesions and treatment using identical vocabulary and quantitative measurements as contained in OME. Other surgeons used vernacular, such as, “tear of approximately 2/3 – 3/4 of the width of supraspinatus,” or, “a very large rotator cuff tear.” Such descriptions did not align with OME documentation, which requires the surgeon to enter the data using standardized categories and defined variables, and lists tear size in centimeters corresponding to small, medium, large, or massive tears.¹

The study has several limitations. First, there is a potential for implicit bias in data collection. Surgeons are required to enter a standardized dataset in OME whereas there are no requirements as to what must be included in the operative notes. Second, while the OME database is relatively comprehensive, it is not exhaustive and will be limited by its pre-defined classifications for certain variables. Third, the study was based on data captured during the early implementation phase of OME, and may have been influenced by initial unfamiliarity of the participating surgeons. Fourth, for agreement analysis we assumed missing data in the operative notes as “implied negatives” in case of three variables; however, it is possible that some other variables that were missing could also have been “implied negative” by the surgeon, but the retrospective nature of the study does not allow us to determine the accuracy of our assumption. At present, the operative note is the most widely used method of recording surgical data, however, it is neither prospectively designed nor standardized and so there is no real “gold standard” of operative documentation. Consequently, it is difficult to fully compare OME data with the actual surgical record, and, therefore, it is challenging to fully gauge the true accuracy of the OME database.

The OME system allows a detailed, efficient, and comprehensive method of surgical documentation and data capture for RCR and various other orthopedic procedures, including ACL and meniscus repair, shoulder and hip surgery including total joint arthroplasty.^2;4;21 As of February, 2018, OME is used by 57 surgeons at 10 sites within the Cleveland Clinic Health System to document surgical details and PROMs on 97% of 21,500 elective knee, shoulder, and hip surgeries. Validation studies for OME in total knee arthroplasty,²¹ total hip arthroplasty,⁴ ACL and meniscus repair have demonstrated excellent performance of this new system in point-of-care implant documentation. Having developed and now validated the OME system for data collection, one of our future objectives is to use the OME system to create a standardized operative report and integrate with EMR systems to replace the existing methods of creating operative reports. Implementing the streamlining features of branching logic and discrete data collection format of OME in an operative note template is expected to be challenging and will require significant programming and cost.

The widespread collection of standardized prospective data captured in an electronic format in OME will allow investigation of the relationships between patient, disease, and surgical factors, and patient reported outcomes.^2;5;26 The underlying structure of OME as a secure REDCap database also provides the ability to utilize this unique data collection tool across institutions in the future for prospective multicenter cohort studies or clinical trials. The OME system is currently being used in multicenter research collaborations, and the technology has also been licensed to a company for development of commercial applications. Improvement in orthopedic outcomes research will become increasingly important as health care economics dictates justification of elective repair based on efficacy and outcomes.^{2, 11} Future areas of research may include OME data capture validation studies for other orthopedic pathologies, and using OME data to investigate the relationship between patient, disease, and surgical factors, and PROMs, for various orthopedic pathologies.

Conclusions

The prospectively designed, electronic data entry system (OME) is a valid and efficient tool for collecting comprehensive and standardized data on rotator cuff repair.