Validation of the Data Quality of a Tumor Board Registry Through Assessment of Clinicopathologic Survival Outcomes in Melanoma Patients

Introduction

Melanoma is a cancer of melanocytes and though it only accounts for ∼1% of all skin cancer diagnoses, it is a leading cause of skin cancer-related mortality. Invasive melanoma, or disease that extends beyond the most superficial layer of the skin, is the 5^th and 6^th most common malignancy in the United States in men and women, respectively.¹ 5-year survival for thin, early stage melanoma is ∼98%, but drops to <25% for patients diagnosed with advanced-stage disease. The incidence of melanoma has been rising steadily (3-7%/year) over the last ∼30 years, but mortality has stabilized and even declined in some age groups in recent years, in part due to significant advances in available treatments for late-stage disease, including targeted immunotherapies.² Over the last ∼9 years alone, twelve new immunotherapy drugs or clinical combination therapies have been approved by the Food and Drug Administration for the treatment of melanoma.^∗ Early assessments of efficacy and safety of new therapeutics is from clinical trial data, which are often less representative of the general population due to strict inclusion criteria. Outcomes studies using national registries, which allow us to expand our understanding of patient characteristics (e.g. demographics), treatment selection, and outcomes (e.g. recurrence and death), are often delayed by years. Local registries, however, can be powerful sources of real-world, near real-time data, the evaluation of which is critical to understand the immediate effect of these novel treatments on patient outcomes.

Registries (databases comprised of systematically collected disease-specific patient data) have been growing in number since the early 2000s, but have garnered significant interest recently, both academically and commercially, as translational applications have become more tangible and accessible to clinical providers (through digital health services such as Verana Health and Flatiron Health). Registries vary in the level of detail included, which influences the types of questions researchers can ask, ranging from impact on individual patient care, cancer surveillance, improving understanding of treatment and survival trends, or serving as mechanisms to monitor quality in patient care. It is clear, however, that the quality of the data in any registry is critical to the quality and accuracy of a study outcome³, and an assessment of registry quality and clinicopathologic validation must be made prior to use as a epidemiologic (research) resource. Many authors have defined what “quality” in registries means, described the types of errors most commonly seen in registries and databases, and some have recommended standardized strategies to optimize data collection and reduce error.³

Despite recent advances in data abstraction, processing, and analysis, such as natural language processing (NLP), the emergence of data lakes, and statistical learning, large, automated, searchable, user-friendly, clinician-accessible databases out of EHRs are still a promise of the future. At the moment, high-quality, detailed, controlled, and regularly-updating tumor board registries, though laborious, are already in motion. These registries are purposed to clinically assist physicians in understanding of disease and treatment courses, and have the potential to bring the learning health system to fruition.

At our large, urban academic medical center and cancer center in New England, complex patient cases are discussed at the institution’s multidisciplinary Melanoma Tumor Board (MTB), comprised of medical oncologists, plastic surgeons, surgical oncologists, dermatologists, pathologists, radiologists, and basic scientists. About 10 years ago, the MTB began building a patient registry (The Melanoma Registry; TMR), including all patients with a diagnosis of melanoma seen at the institution at any point in their care timeline. Since its inception, the purpose of this registry was to augment the ability of members to make prognostic predictions and inform treatment selection for uncommon presentations through the inclusion and interrogation of granular data elements, such as genetic/genomic information and more robust health and demographic data. This registry is a manually curated, near real-time resource, which includes comprehensive treatment regimens and a growing patient pool managed by contemporary treatment strategies. TMR contains 7414 patients with melanoma (1958-present), with 265 distinct data elements. Approximately 750 new melanoma patients are seen at the institution each year, with a variable registry entry lag time, ranging from 1 week to about 3 months. Priority of data entry is placed on cases with higher stage disease and rare tumor sites. Though each individual institutional registry has differences, TMR seems to share many features of other published melanoma registries.⁴

In this study, we first assess data quality and address data errors in TMR. To validate the registry, we examined a known clinicopathologic outcome: overall survival stratified by AJCC 8^th edition staging criteria, and find that the registry is valid when compared to prior national and regional analyses.^4,5

Methods

Registry Curation

Eligibility criteria for TMR includes patients with a primary melanoma diagnosed up until January 2020. 3500 of these patients were entered retrospectively through manual extraction of data from paper charts from 1970-2010 (though original biopsies dated back to the 1950s). In 2011, the institution implemented an electronic health record (EHR) and subsequent melanoma patients were enrolled prospectively. At this time, data was extracted directly from the EHR and augmented as necessary through searches in obituaries.

To date, a single trained data manager has captured the clinical and pathological information from patients’ medical charts (265 clinical and pathologic data points). The data elements are entered and stored in a Patient Registry Management System (PRMS) using OnCore (Forte Research Systems Inc, Madison WI, version 15.4.10). After enrollment, patients are prospectively followed, but prior to this study, the follow-up timeline was not regularly scheduled. The tumor board leadership team is currently developing a standard process for future use. For this data quality and validation study, patients were only excluded if they did not have a reported stage (n = 956 (13%)). These patients were excluded because stratification by stage was a key facet of the clinicopathologic validation outcome.

Each patient is assigned a unique identifier. Demographic information, personal medical history, social and family history, pathology and radiology reports, surgical reports (including sentinel lymph node biopsy and complete lymph node dissection data), stage, treatment types and courses, recurrence data and subsequent work-up and treatment, last visit date, death data (if, when, cause) are entered as individual data elements into the PRMS.

Registry Data Quality

For the purpose of this study, our assessment of data quality is guided by a definition of quality as,

“the totality of features and characteristics of a data set, that bear on its ability to satisfy the needs that result from the intended use of the data [...] data are of the required quality if they satisfy the requirements stated in a particular specification and the specification reflects the implied needs of the user”³.

Data errors have been characterized in many different ways. For this study, however, we will discuss errors during “set up and organization of the registry”, errors stemming from “data collection”, and errors due to lack of “quality improvement” using strategies guided by the aforementioned definition from Arts et al.

All patient information and data elements were downloaded from OnCore into excel flat files. The excel files were then loaded into R (version 3.6.1). Data elements were assessed for percentage of missing elements. Error assessment was completed by comparing overlapping, though not redundant, data elements, to ensure the data elements had equivalent information. Data discrepancies identified during data manipulation and data element characterization were subsequently addressed, as detailed in the Results section. Redundant data elements were collapsed into single data elements, when possible, which reduced percentages of missing data within certain data elements. When data errors were identified due to data entry, they were communicated to the data manager, who confirmed and corrected the errors in the OnCore PRMS.

Clinicopathologic Validation

To assess the validity of the registry, a clinicopathologic case was selected which was explicitly a function of patient-level features.⁵ Overall survival (OS) was calculated as time from the initial diagnosis (date of biopsy) to time of death. Kaplan-Meier curves were generated to estimate the OS for each staging classification and the R survfit model was used to generate and report the results. Log-rank tests were generated using the R survdiff model. Five-year OS 95% confidence intervals were estimated for stage IIC and IIIA disease. A two-tailed p-value, generated from the log-rank test, <0.05 was used to establish statistical significance.

Results

Data Structure

Each patient in the registry has a unique identifier. In TMR, data elements are entered in data entry forms that are linked directly to the patient/patient identifier or nested as a part of a biopsy (or recurrence) encounter, which is then linked to the patient. While the biopsy-central data storage structure is an ideal clinical representation of the data, it poses problems when using the registry as a informatics tool. When querying data elements that are sub-categorized by procedural events, rather than storing all data in a patient-centered format, validation challenges arise due to systematic, structural errors, as discussed in Arts et al.. Additionally, prior to this study, data review and audit occurred occasionally, but the process was not standardized. There can be up to a few months delay for data entry, though this is variable, with prioritization of later-stage tumors. The registry is cycled 1-2 times per year to update patient death information, but again, this had not been standardized. Notably, the registry was reviewed prior to our data analysis, to ensure accuracy of our output, and formal mechanisms are in development for future use.

Data errors were identified to have occurred at two phases (adapted from Arts et al.): 1. Set up and organization of the registry, which include errors due to (a) Data loss, (b) Missing data and data elements, and (c) Lack of standardization; and 2. Data collection, which include errors due to (a) Entry errors, (b) Data loss, and (c) Information loss. Table 1 presents a summary of these errors, as well as potential short and long term solutions detailed below.

1. Set up and organization of the registry

   • (a) Data loss: This error stems from loss of data through systematic changes over the lifetime of the registry or database, or in data entry. It also occurs due to data processing without sufficient knowledge of the underlying data structure. Examples of this in TMR are the implementation of dropdown boxes for the comorbidity and medication data elements. While dropdown boxes minimize typing errors from free-text entry and allow for faster input, they reduce the options of comorbidities or medications that can be entered, though the data manager can always add to the dropdown boxes. Additionally, it is often too time consuming or not possible (in the case of old paper charts) to retrospectively update old patient entries to the new format. Another example of data loss occurs when patients have more than one primary tumor biopsied on the same day. Both of these tumors/biopsy encounters are entered with the same patient identifier, for example “PT1”, if this is the patient’s first melanoma diagnosis. As a result, due to retrieval restrictions, some data elements will concatenate as (X; X') in a single column. Ultimately, this is a result of misalignment between the data entry and data storage primary keys, as the data is stored at minimum in the third normal form, evidenced during retrieval transactions joining biopsy and patient tables.

     Short-term corrections: We filtered out patients with more than one “PT1” for this analysis. We split each individual comorbidity by “;” in either the “Comorbidities” column or the “Comorbidities Dropbox” column, removed duplications and then merged the two columns into a single data element. Long-term solutions: We recommend optimization of data entry and PRMS structure upfront to minimize changes needed over time. This includes involvement of clinical informatics specialists at registry creation time, alongside other technical specialists such as biostatisticians. Additionally, moving forward, all encounters should have different names, such as “PT1-1” and “PT1-2”, to prevent the use of a composite key upon retrieval. This illustrates again, an issue encountered in TMR because data is entered in a clinical-forward, rather than informatics-forward way.

   • (b) Missing data and missing data elements: This can be split into two categories—i. absent data elements and ii. redundant data elements.

     1. Absent data elements can occur when the information is available in the EHR, but excluded by database creators/managers because it is considered unimportant for their purposes, or it is deprioritized as it increases total entry time per patient. An example of this in TMR is inclusion of markers of socioeconomic status (e.g. insurance status). Data elements may also be missing because the information is not consistently available in the EHR (not collected, not recorded in patient encounters) and this information could only be acquired by speaking with the patient again. Examples of this are a patient’s “Fitzpatrick skin type”, which is a much more accurate marker than “Race” or “Ethnicity” for a patient’s risk of development of melanoma, but requires a specific set of questions be asked of the patient that is rarely done outside of dermatology practices. Finally, absent data elements occur when information is not reliably updated in the EHR. The most important example of this is death data (i.e. whether a patient is alive or has expired, the date of death, and cause of death). Death does not trigger a separate EHR entry, so this information may not be in the EHR at all, or may be hidden in a progress note. The EHR is searched periodically to check for death in late-stage patients, and the data manager will occasionally cycle through obituaries, though this can be a laborious process.
     2. Redundant data elements can also lead to missing data in subsequent analyses if they are not completely interchangeable, leading to a form of syntactic ambiguity. An example in TMR is the assessment of “skin tone”. There are 6 different data elements that can contribute to assessment of skin type, including “Race”, “Ethnicity”, “Skin Tone”, “Sun Reaction”, “Eye Color”, and “Hair Color”. Each patient has some of these elements included in incomplete combinations, which leads to missing data and information loss when trying to integrate them into one data element, which otherwise is absent.

        Short-term corrections: We examined, through manual review, specific data elements which warranted attention (such as death), performing a more detailed chart review to correct errors.

        Long-term solutions: Regarding absent data elements, we recommend implementing a standardized data review process as described in Bajaj et al. We would also recommend a second data manager or analyst to audit the data and perform regularly scheduled checks looking for data elements with known errors or those revealed through auditing. One could also consider addition of a structured death element in the EHR. Death data is critical for accurate survival assessment and mechanisms to improve this series of data elements should be prioritized. Finally, collaborating with different clinical specialties prior to building the registry or database may help ensure more of the necessary data elements for validation are included in the registry and do not have to be added, integrated, or imputed later. This can also be applied to redundant data elements, where we can select, for example, the most useful “skin type” indicators and encourage increased collection and documentation of those specific data element(s) in patient encounters.

   • (c) Lack of standardization: This is most commonly due to variability of the recording preferences of the physician or provider that writes the notes and is a form of semantic ambiguity. In TMR, this is most visible in the data elements related to response to therapy. For example, some physicians record response using RECIST guidelines (Complete Response, Partial Response, Stable Disease, Progressive Disease), while others report “Excellent response”, “Mixed response”, or “No response”. In order to combine these into a unified response element for data analysis, we would have to interpret the original meaning, which has the potential to introduce error (especially when performed by a non-clinician). Another example is differences in the questions patients are asked. While some physicians regularly ask questions about family history, sun exposure, history of sun burns, others do not, which again leads to missing data during data entry. A similar lack of standardization is seen in pathology reports of biopsy and surgical specimens. Most reports comment on depth of invasion, presence of ulceration, and melanoma subtype because those features have been reported to have prognostic significance. However, there are many other features that may end up having significance in the future, as we learn more and guidelines change (as they did for mitotic index with the adoption of the AJCC 8^th edition in 2018).⁶ Some pathology reports will include the absence of potentially useful features (some of which have known prognostic significance in other cancer types)⁷, such as lymphovascular invasion, presence of attached nevus, perineural invasion, among others, while many exclude information about these features altogether. When extracted, these features are correctly reported as “not applicable” rather than “No”, which will obscure our ability to understand the impact of each variable. Additionally, if any of these features become critical in the future, and they are not recorded now, our ability to create retrospective cohorts will be limited.

     Short-term corrections: We made and documented the use of assumptions when standardizing data for analysis.

     Long-term solutions: Initially, we recommend identifying a focused group of high-yield data elements, a minimum viable dataset for validation, and discuss importance and utility with the clinical service lines participating in the registry to encourage improved and expanded information collection moving forward. Additionally, improvements to the registry need to be coupled with consistent training to resolve syntactic and semantic ambiguity at the point of data entry.

2. Data collection

   • (a) Entry errors: In TMR, the primary cause of entry errors is manual data entry. A goal of this registry is rapidity of data entry, which further compounds risk of entry error. Examples include not clicking “No” with the assumption that “not applicable” (which auto fills if nothing is clicked) means “No”, which is not the case. Another example includes not clicking “Yes” for “has the patient had medical therapy”, but then that same patient has a medical therapy start date.

     Short-term corrections: We compared all related data elements, for example yes/no interventions and the presence/absence of start dates or results. Programmatically we can then back fill the yes/no in the table. We went through data elements recorded by free-text and corrected errors in spelling and text case (upper vs. lower) so that data was syntactically consistent during analysis.

     Long-term solutions: Utilize the ability of enterprise registry software such as OnCore or REDCap to link directly to the EHR. If structured data elements were automatically pulled into the PRMS, this would reduce error due to manual entry, increase rapidity of entry for certain data elements, allow the data manager to spend more focused time on the unstructured data elements, and would auto-update, helping to keep the registry contemporary with reduced effort and increased accuracy.

   • (b) Information loss: This type of error occurs due to binning of data, a pre-processing technique to reduce the effects of minor observation errors. An example of binning is reducing the comprehensive lists of comorbidities into a comorbidity score (e.g. Charlson Comorbidity Index). The score is a much easier data element to manipulate, but this reduction of granularity of output leads to loss of information that may be important.

     Short-term corrections: We suggest reducing granularity of data strategically, and document when this occurs so we can better understand the effect it may have on the data.

     Long-term solutions: As the registry grows, less common data element subcategories may grow to useable numbers. Focused studies could be performed on granular elements, such as comorbidities, to minimize information loss, or improve the prognostic capacity of data.

Table 1.

Summary Table of Data Errors and their Short and Long Term Solutions

Data Errors	Short Term Solutions	Long Term Solutions
Set up and organization of registry
Data Loss	Filter patients early by redundant identifiers Clean redundant elements by splitting data, removing duplications, and re-merging into a single data element	Optimize data entry and PRMS structure upfront to minimize changes needed over time Use completely unique identifiers for every encounter
Missing data and missing data elements	Manual review of specific data elements which warranted attention (such as death), performing a more detailed chart review to correct errors	Implement a standardized data review process Utilize a second data manager or analyst to audit the data and perform regularly scheduled checks Consider addition of a structured death element in the EHR Collaborate with different clinical specialties prior to building the registry or database Select the most useful indicators and encourage increased collection and documentation of those specific data ele- ment(s) in patient encounters
Lack of standardization	Make and document the use of assumptions when standardizing data for analysis	Identify a focused group of high-yield data elements and discuss importance and utility with the clinical service lines participating in the registry to encourage improved and expanded information collection Prioritize consistent training to resolve syntactic and semantic ambiguity at the point of data entry
Data collection
Entry errors	Compare all related data elements, then programmatically back fill the missing elements using known related elements Correct free-text spelling and text case discrepancies	Link directly to the EHR, to pull structured data elements
Information loss	Reduce granularity of data strategically, and document when this occurs	Focused studies could be performed on granular elements, to minimize information loss, or improve the prognostic capacity of data, as database grows

There are many advantages to the current structure and entry process of TMR. First, all of the data has been entered by a single data manager, which standardizes the interpretation and entry. Quality control improvements can facilitate error correction during data collection. For example, after series of data entry, we recommend performing quality checks including review of default values (”not applicable” rather than yes/no) and review of overlapping and non-redundant date elements to cross-check data entry. Additionally, the flexibility of a single data manager allowed for the addition of any new data elements that were identified as useful (e.g. insurance status) to the PRMS. Adding new data elements, however, requires retrospective chart review, of which feasibility is a factor. For example, while TMR database manager can no longer update the first 3500 patients (whose data is from paper charts and no longer accessible), he can update all those with data in the EHR. Another limitation of TMR is possible introduction of correction bias. Specifically, the data manager has been willing to correct errors as they have been discovered, but this has the potential reduce error through an imbalanced mechanism, and should be addressed when discussing study results.

After addressing entry and data manipulation errors as above, through the use of the informatics methods described, we can conclude that the registry achieves quality standards for subsequent studies, as discussed by Arts et al. We then sought to validate the cleaned registry by comparing survival outcomes in our PRMS to studies published recently in the literature.^4–6

Clinicopathologic Validation

7389 patients were included for validation, including 194 stage IIC and 144 stage IIIA melanoma patients. Patients ranged from an original biopsy/diagnosis date of January 1958 to November 2019. As expected, patients with early-stage disease (melanoma in situ, stage IA, stage IB) have the best 5-year survival, and those with late-stage disease (stage IIID, stage IV) have the worst outcomes (Figure 1a) (p-value <0.001). Patients with stage II and III disease have outcomes between these two extremes (Figure 1a). As reported^4–6, patients in our cohort with stage IIC melanomas have significantly worse 5-year OS than patients with stage IIIA disease (Figure 1a, Figure 1b) (p-value <0.001).

Figure 1:

Kaplan-Meier curves of 5-year survival in TMR cohort

Conclusion

Tumor board registries can be high quality, up-to-date resources that have the ability to impact complex patient care at the individual level. When appropriately evaluated for data quality and validated through epidemiologic methods against a clinicopathologic standard, registries also have the ability to improve our understanding of disease states and contemporary treatment paradigms on a larger scale. Registry data collection comes with many known trade-offs including granularity of data elements, speed of data entry, and labor intensity required for collection. Regardless, informatics methods offer a means to assess data quality and to inform validity of analyses done using a registry for clinical or research purposes. As Arts et al explains, there will always be errors, but we have to work to minimize them, standardize them, and understand the impact they may have on our studies. Missing data and data errors matter because we cannot assess the potential impact of variables that we are not measuring on outcomes. We cannot use registries with the same fidelity in clinical and epidemiologic settings without accurate and precise demographic data elements. Furthermore, we cannot accurately assess outcomes if we are missing outcome measures.

Through the practice of informatics, we were able to evaluate the data quality of TMR, a clinically useful, detailed, longitudinal registry. We were then able to validate the registry’s data quality through evaluation of a known clinico-pathologic outcome. Nationally, the goals of tumor boards and tumor board registries are expected to be similar to those of TMR, and we suggest that the errors we have identified are generalizable. We also believe the recommendations posed for long-term improvements can be applied to current registries, and help inform the strategic development of new clinical registries.

Finally, while the paradoxical survival of stage IIC versus stage IIIA melanoma patients has been observed and reported before, few studies have identified differences in patient, tumor, or treatment characteristics that may contribute to this difference in survival.^4–6 Notably, the survival discrepancy persists when patients are staged by AJCC 7^th or 8^th edition criteria^4,6, as well as pre- and post-introduction of targeted therapy and/or immunotherapy for late-stage disease management. With a valid data source in hand, we plan to study our stage IIC and stage IIIA patient cohorts to better understand and characterize this survival discrepancy, including exploring the impact of genetic/molecular differences of these tumors, which has the potential to directly impact patient treatment selection and outcomes.