Miscoding of melanoma thickness in SEER: Research and clinical implications

Melanoma-related deaths and metastases among patients with thin (≤1mm) and ultrathin (≤0.25mm) melanomas have been reported. These observations might reflect adverse biology and/or errors in administrative data. Cumulative melanoma-related death (CMD) rates for thickness groups of patients with thin melanomas were compared among five cohorts including the Surveillance, Epidemiology and End Results registry (SEER). Thickness in one SEER region was reexamined in pathology reports. The five-year CMD rate of patients with ultrathin melanomas was higher in SEER (2.8%) compared to other registries (0.6%–0.9%). The rates across the 16 SEER regions were 0.25% to 8.4%. In SEER 21% of thin melanomas were ultrathin; in other registries they comprised 5.8%–15%. A reexamination of thickness in one SEER site revealed that 114/447 ultrathin melanomas had errors; after correction, only 17/114 remained ultrathin. The majority of errors were related to decimal point placement. The 86 thin melanomas reclassified to >1.00mm included 96% of the original ultrathin-associated deaths and and 100% of the original positive lymph nodes. Significant miscoding of thickness that is concentrated in ultrathin lesions is present in SEER and results in mischaracterization of patient outcomes. When using administrative data, validation of results can identify critical data issues.

INTRODUCTION

Individually, thin (≤1mm) primary melanomas confer a good prognosis. However, patients with thin melanomas represent a high proportion of all melanoma-related deaths (Gimotty & Guerry, 2010; Hieken et al, 2015; Whiteman et al, 2015). Given the well-characterized relationship between thickness and prognosis (Balch et al, 2001), among patients with thin melanomas those with lesions ≤0.75mm have a better prognosis than those with 0.76-1.00mm (McKinnon et al, 2003). Surprisingly, a single-institution study reported that 10% of patients with lesions <0.75mm had a recurrence within 5 years (Kalady et al, 2003). Also, a SEER-based study (Bagaria et al, 2013) reported that 10% of patients who had positive lymph nodes at diagnosis (1998–2008) had primary tumors that were ≤0.5mm, and patients with ≤0.5mm lesions had a higher CMD rate than patients with lesions that were 0.51–1.00 mm (37% versus 22%, respectively). The study's authors counseled consideration of sentinel lymph node biopsies in patients with ≤0.5mm melanomas. These unanticipated findings associated with very thin lesions raised the issue of reporting errors.

Cancer registries are an important source of information about patients with a low likelihood of death, such as patients with thin melanomas who are expected to be cured with definitive loco-regional surgery. Coding errors in prognostic and predictive variables available in a registry can produce estimates that either over- or underestimate outcomes for important subgroups of patients. The presence of such errors (or “inconsistencies”) has been identified in SEER data (Criscione & Weinstock, 2010) but has not been further investigated. In this study we tested the hypothesis that miscoding of thinner melanomas in SEER data is a primary driver of adverse outcomes that are artifactual.

RESULTS

CMD Rates by Tumor Thickness and Other Attributes

The five-year CMD rate for SEER patients with thin melanomas diagnosed between 2004 and 2010 was 5.1% (95% CI: 4.9%–5.3%), and the five-year and ten-year CMD rates for patients diagnosed between 1988 and 2003 were 3% (1.3%–4.7%) and 4.8% (4.6%–5.0%), respectively. For patients with melanomas in four thin subgroups (≤ 0.25mm, 0.26-0.50mm, 0.51-0.75mm, and 0.76-1.00mm), the five-year CMD rates for the SEER (2004-2010), Sweden, Queensland, and PLG cohorts are shown in a forest plot (Figure 1). As expected, the five-year CMD rates were maximal in all cohorts for the 0.76-1.00mm subgroup. Surprisingly, the ultrathin group in SEER had a five-year CMD rate that was 3-fold and 4-fold higher than those in Sweden and Queensland, respectively, and 2.7-fold higher than that in the University of Pennsylvania's Pigmented Lesion Group (PLG) cohort. In both the ultrathin and 0.26-0.50mm subgroups, the SEER (1988-2003) ten-year CMD rates (6% and 3%, respectively) were higher than corresponding rates observed in the four non-SEER cohorts.

Figure 1.

Forest plots of five- and ten-year cumulative melanoma-related death (CMD) rates and 95% confidence intervals by thickness subgroups and cohorts for patients with thin (≤1mm) melanomas. The German Cohort reported only ten-year CMD rates. Abbreviations: SEER, Surveillance, Epidemiology and End Results; PLG, Pigmented Lesion Group.

Tumor characteristics in the four subgroups differed among the cohorts. For example, more SEER (2004-2010) patients had ultrathin melanomas (21%) compared to the other cohorts (5.8% to 15%), and fewer SEER patients were in the 0.76-1.00mm subgroup (14.2%) compared to the other cohorts (16.6% to 27.9%) (Table 1). More level IV/V lesions were reported for the ultrathin melanomas in SEER compared to PLG (10.0% versus 1.5%), and more SEER lesions were ulcerated compared to PLG lesions (Supplementary Table S1).

Table 1.

Comparison of thickness distributions among cohorts

			Thickness (mm)
			0.01–0.25		0.26–0.50		0.51–0.75		0.76–1.00
Cohort	Period	N	%	95% CI	%	95% CI	%	95% CI	%	95% CI
SEER	2004–2010	51,855	21.0	(20.7–21.4)	42.3	(41.9–42.7)	22.4	(22.4–22.1)	14.2	(13.9–14.5)
SEER	1988–2003	42,851	18.8	(18.4–19.1)	40.8	(40.4–41.3)	24.7	(24.7–24.3)	15.7	(15.4–61.1)
Sweden	1990–2008	13,026	5.8	(5.4–6.2)	39.0	(38.2–39.8)	27.4	(27.3–26.6)	27.9	(27.1–28.6)
Queensland	1982–2006	26,736	8.9	(8.5–9.2)	43.2	(42.6–43.8)	31.3	(31.3–30.7)	16.6	(16.2–17.1)
Germany	1976–2000	12,728	8.4	(7.9–8.9)	36.3	(35.5–37.1)	29.7	(29.6–28.9)	25.6	(24.9–26.4)
PLG	1996–2006	1,315	15.0	(13.1–16.9)	41.4	(38.8–44.1)	26.1	(26.1–23.7)	17.5	(15.4–19.5)
PLG	1982–1995	1,056	10.1	(8.5–11.7)	37.6	(35.1–40.1)	29.1	(29.1–26.8)	23.2	(21.0–25.4)

Abbreviations: CI, Confidence Interval; SEER, Surveillance, Epidemiology and End Results; PLG, Pigmented Lesion Group.

Review of Pathology Reports and Coding for a SEER Region (Detroit)

Due to these inconsistencies and, particularly, the finding of a high CMD rate in SEER patients with ultrathin melanomas, we reviewed thickness in the pathology notes for the 1955 patients with thin melanomas reported by the Detroit SEER site (Supplementary Table 2). Overall, 88% of patients' lesions were correctly coded for thickness; however the percentage correctly coded varied among the thin subgroups. The ultrathin subgroup had the lowest percentage of cases correctly coded (71%). In contrast, the percentages correctly coded were 95%, 92% and 89% in the 0.26-0.50mm, 0.51-0.75mm, and 0.76-1.00mm subgroups, respectively.

Among the 190 thin melanomas with a thickness error (9.7% of 1955), 68% were due to decimal point miscoding errors (Supplementary Table 2). These errors led to three magnitudes of mistake. First, 61% of all incorrect values were too small by a factor of 10 (e.g. 0.11 became 1.10). Second, 6% were too small by a factor of 100 (e.g. 0.02 became 2.00). Third, 1% of the incorrect values for thickness were too large by a factor of 10 (e.g. 0.8 became 0.08). Decimal point errors were most frequent in the ultrathin subgroup.

Consequences of Data Correction

Among 447 thin melanomas originally characterized as ultrathin in Detroit SEER data, 75% remained ultrathin after correction of thickness, 6.7% were reclassified as 0.26-0.50mm, and 10.3% were reclassified as non-thin (>1mm) (Supplementary Table 3). Among the other three thickness subgroups, 2.3% to 3.3% of melanomas were reclassified as non-thin. The thickness distributions post- and pre-correction melanomas ≤0.50mm are shown in Figure 2 (first and second histograms). The change in the left tail reflected the elimination of 86 ultrathin melanomas, some of which were corrected to non-thin. The distribution of corrected thickness (first histogram) was similar to the distribution for the PLG cohort (third histogram).

Figure 2.

Distribution of tumor thickness for patients with melanomas ≤ 0.5mm in the post-correction (n=1,869) and pre-correction (n=1,955) SEER Detroit cohort, the PLG cohort (n=742), and all SEER regions but for Detroit (n=31,558). The red vertical line indicates 0.25mm. Abbreviations: SEER, Surveillance, Epidemiology and End Results; PLG, Pigmented Lesion Group.

Most adverse prognostic factors and poor outcomes attributed to putatively thin melanomas were associated with those that were reclassified (“corrected”) as non-thin (Table 2). The thin melanomas originally observed as ulcerated were largely reclassified as non-thin. Level IV/V ultrathin melanomas decreased, since 79% of the thin melanomas reclassified to non-thin melanomas were level IV. The percentage of patients who had lymph nodes examined in the ultrathin, 0.26-0.50mm, and 0.51-0.75mm subgroups decreased to less than 8% from the originally reported percentages of 36%, 42% and 44%, respectively. After correction, none of these patients had a lymph node positive for metastatic melanoma and none died of melanoma. Among the 86 patients reclassified as non-thin, 69% had their lymph nodes examined and 21% had at least one involved lymph node.

Table 2.

Patient outcomes and tumor characteristics among 190 corrected cases by thickness subgroups¹, pre- and post-correction²

		Thickness (mm)
		0.01–0.25		0.26–0.50		0.51–0.75		0.76–1.00		0.01–1.00		>1.00
		Pre n=114 %	Post n=20 %	Pre n=31 %	Post n=42 %	Pre n=25 %	Post n=18 %	Pre n=20 %	Post n=24 %	Pre n=190 %	Post n=104 %	Post n=86 %
Ulceration	Present	13.2	0.0	25.8	2.4	36.0	16.7	20.0	0.0	18.9	3.8	37.2
	Absent	78.9	85.0	67.7	95.2	64.0	77.8	75.0	95.8	74.7	90.4	55.8
	Unknown	7.9	15.0	6.5	2.4	0.0	5.6	5.0	4.2	6.3	5.8	7.0
Clark Level	II	33.3	75.0	12.9	52.4	16.0	27.8	0.0	12.5	24.2	43.3	2.3
	III	17.5	5.0	19.4	28.6	36.0	50.0	35.0	41.7	22.1	30.8	11.6
	IV & V	37.7	5.0	64.5	14.3	48.0	22.2	65.0	37.5	46.3	19.2	79.1
	Unknown	10.5	15.0	3.2	4.8	0.0	0.0	0.0	8.3	6.8	6.7	7.0
SEER Stage	Local	89.5	100.0	74.2	100.0	72.0	100.0	85.0	95.8	84.2	99.0	66.3
	Regional	9.6	0.0	25.8	0.0	28.0	0.0	15.0	0.0	15.3	0.0	33.7
	Unknown	0.9	0.0	0.0	0.0	0.0	0.0	0.0	4.2	0.5	1.0	0.0
Vital Status	Melanoma Death	5.3	0.0	16.1	0.0	8.0	0.0	15.0	4.2	8.4	1.0	17.4
	Other Death	4.4	5.0	6.5	0.0	0.0	0.0	10.0	8.3	4.7	2.9	7.0
	Alive	90.4	95.0	77.4	100.0	92.0	100.0	75.0	87.5	86.8	96.2	75.6
Nodes Examined	Yes	36.0	5.0	41.9	7.1	44.0	5.6	60.0	54.2	40.5	17.3	68.6
	No	64.0	95.0	58.1	92.9	72.0	94.4	40.0	45.8	61.6	82.7	31.4
Positive Nodes	≥ 1	6.1	0.0	16.1	0.0	16.0	0.0	10.0	0.0	9.5	0.0	20.9
	0 or No Exam	93.9	100.0	83.9	100.0	84.0	100.0	90.0	100.0	90.5	100.0	79.1

Abbreviations: SEER, Surveillance, Epidemiology and End Results; Pre, pre-correction; Post, post-correction.

¹Excludes patients whose charts were not found.

²Percentages computed from the post-correction data are in italics.

Heterogeneity among SEER Regions

The distribution of thickness reported from the 15 SEER sites not included in the review (n=47,828 patients; Figure 2, fourth histogram) was similar to the uncorrected cases in the Detroit SEER registry (second histogram). We ranked all SEER regions by their 5-year CMD rates for the ultrathin subset and displayed the paired rates for the ultrathin and 0.76-1.00mm subsets (Figure 3). For patients with ultrathin primaries only two regions had CMD rates between 0.3% and 0.9%, comparable to rates in the non-SEER cohorts presented in Figure 1.

Figure 3.

Forest plot of five-year cumulative melanoma-related death (CMD) rates and 95% confidence intervals for 0.01mm–0.25mm (ultrathin) melanomas (red) and 0.76-1.00mm melanomas (blue) by SEER region. Abbreviations: San Francisco (SF).

DISCUSSION

We observed in SEER a higher 5-year CMD rate for patients with ultrathin melanomas compared to rates for three non-US cohorts and the PLG cohort. We considered four potential explanations. First, adverse outcomes might reflect the uncommon presence of unfavorable prognostic factors (e.g. tumor mitoses). Second, they may be a consequence of prior, synchronous or subsequent primary cancers. Since an error in the identification of the lethal tumor could result in a biased survival rate, we focused this study on patients with single primaries in the SEER registry. Third, pathology errors may lead to misclassification of tumor attributes. Lastly, errors in coding information from medical records can occur, as demonstrated here.

SEER tumor size was reported to be 90–95% accurate (Furlow 2015). In our review we found this was not the case among the thinner of thin melanomas. In our analysis of SEER Detroit we observed a high percentage of patients with putatively ultrathin lesions whose thickness was miscoded (26%) compared to patients with 0.26–1.00 melanomas (5%) and >1mm melanomas (5%) (data not shown). After correction of thickness, the percentage of ultrathin melanomas decreased, and 40% of ultrathin melanomas with errors in thickness were reclassified as >1mm. Importantly, among patients with now correctly classified ultrathin lesions, the percentage with positive lymph nodes at the time of diagnosis and the percentage having melanoma-related deaths decreased from 5.3% to 0% and 6.1% to 0%, respectively.

The majority of the errors in SEER Detroit ultrathin melanomas were due to decimal misplacement. Melanoma is one of a few cancers in which tumor size is reported in millimeters rather than centimeters. Translation of size from pathology notes to data entry screens is an important opportunity for error. Registries can improve collection of this data element by specific numeracy training related to the metric system, designing data entry screens with built-in safeguards that facilitate correct decimal point placement, and tailoring computerized data editing. Intelligent algorithms can be developed that use other data elements to identify inconsistencies.

Our investigation of non-Detroit SEER regions strongly suggests a noteworthy error rate in coding thickness, particularly in ultrathin melanomas. Comparable miscoding may exist prior to 2004. The reported 10-year CMD rate for patients in the ultrathin subgroup in the SEER (1988-2003) was unexpectedly high (6%) and significantly higher than in the four comparison cohorts (Figure 1, right panel).

The decimal placement problem that we identified in melanoma data has also been found to underlie errors in prostate-specific antigen (PSA) data for SEER patients with prostate cancer (Furlow 2015; Sun & Trinh 2015). As these errors potentially impact many SEER-based studies of prostate cancer, SEER officials have removed PSA from its 2014 system update while pursuing corrective and preventive action (http://www.medpagetoday.com/HematologyOncology/ProstateCancer/55401 and http://www.medpagetoday.com/HematologyOncology/ProstateCancer/55399, accessed 29 April 2016).

For patients with thin melanomas the consequences of miscoding thickness are not trivial for registry-based research and related practice recommendations. Errors in thickness will result in errors in the assigned AJCC stage. Further, the high rate of misplacing a decimal point and misclassifying thick lesions as ultrathin will decrease survival rate estimates and increase the apparent likelihood of a positive regional lymph node. Our investigation will reassure clinicians that the vast majority of patients with very thin lesions should not have nodal staging and are cured with surgery at the primary site. Until the SEER Program implements a plan to correct thickness data, SEER-based studies of stage distributions and outcomes, particularly of patients with putatively thin lesions, will need to be cautiously considered.

This study demonstrates how the impact of errors can be magnified when there is a low overall error rate but a large proportion of the errors occur in a specific subset of the study population. Decimal placement errors, at least in melanoma and prostate cancer data, likely generalize to at least some similar, non-SEER databases. Recently published guidelines on the reporting of prognostic studies (TRIPOD) recommend including an external validation (Collins et al, 2015; Moons et al, 2015). Validation studies will demonstrate reproducibility of results or identify actionable weaknesses and/or inconsistencies.

MATERIALS AND METHODS

Study Populations

Study-eligible patients in 16 SEER registries (SEER Public Use Data November 2010 submission) who had a single, invasive, thin cutaneous melanoma diagnosed between 1988 and 2010 without evident metastasis and no other primaries of any type were included in the analysis (See Supplementary Table S4). Prognostic variables were collected under a unified data collection system introduced in 2004 (http://seer.cancer.gov/tools/collabstaging/, accessed 29 April 2016). The comparison cohort that had a common set of prognostic factors was the PLG Cohort (Clark et al, 1989). PLG patients provided written, informed consent under an Institutional Review Board-approved protocol. A PUBMED search yielded three non-US studies, two population/registry-based and one set in a single institution, that reported five- and/or ten-year CMD rates for patients in the four subgroups of thin melanomas: Sweden (Lyth et al, 2013), Queensland (Green et al, 2012) and Germany (Leiter et al, 2004).

Pathology Review

Twelve Detroit SEER tumor registrars reviewed pathology reports for 3,799 melanomas (2004 and 2010), including 1955 study-eligible patients with one thin primary melanoma. Data were ascertained through the SEER Data Management System that includes electronic records and images of laboratory reports. An error was called when there was a difference in thickness between that reported on the original pathology note(s) and that coded.

Survival Rates

Time to melanoma-related death was the time between diagnosis and last follow up in SEER and the three non-US studies and the time between the definitive loco-regional treatment and last follow up in PLG. Time was censored for patients who were alive at last follow up, were lost to follow up, or had died of causes unrelated to melanoma. Five- and ten-year CMD rates were estimated from the Kaplan-Meier survival distributions. Variances were estimated using Greenwood's formula (Lachin 2011).

Abbreviations

SE

Standard error

CI

confidence interval

N

sample size

CMD

cumulative melanoma-related death

SEER

Surveillance, Epidemiology and End Results

AJCC

American Joint Committee on Cancer

PLG

Pigmented Lesion Group

SF

San Francisco

PSA

prostate-specific antigen

TRIPOD

Transparent Reporting of a multivariate prediction model for Individual Prognosis or Diagnosis

US

United States

NOS

not otherwise specified