A Validated Register-Based Algorithm to Identify Patients Diagnosed with Recurrence of Malignant Melanoma in Denmark

Abstract

Purpose

Information on cancer recurrence is rarely available outside clinical trials. Wide exclusion criteria used in clinical trials tend to limit the generalizability of findings to the entire population of people living beyond a cancer disease. Therefore, population-level evidence is needed. The aim of this study was to develop and validate a register-based algorithm to identify patients diagnosed with recurrence after curative treatment of malignant melanoma.

Patients and Methods

Indicators of recurrence were diagnosis and procedure codes recorded in the Danish National Patient Register and pathology results recorded in the Danish National Pathology Register. Medical records on recurrence status and recurrence date in the Danish Melanoma Database served as the gold standard to assess the accuracy of the algorithm.

Results

The study included 1747 patients diagnosed with malignant melanoma; 95 (5.4%) were diagnosed with recurrence of malignant melanoma according to the gold standard. The algorithm reached a sensitivity of 93.7% (95% confidence interval (CI) 86.8–97.6), a specificity of 99.2% (95% CI: 98.6–99.5), a positive predictive value of 86.4% (95% CI: 78.2–92.4), and negative predictive value of 99.6% (95% CI: 99.2–99.9). Lin’s concordance correlation coefficient was 0.992 (95% CI: 0.989–0.996) for the agreement between the recurrence dates generated by the algorithm and by the gold standard.

Conclusion

The algorithm can be used to identify patients diagnosed with recurrence of malignant melanoma and to establish the timing of recurrence. This can generate population-level evidence on disease-free survival and diagnostic pathways for recurrence of malignant melanoma.

Introduction

The number of people living beyond a diagnosis of malignant melanoma has increased annually by 5–7% in women and by 6–9% in men in Denmark in recent years. The number amounted to 33,176 people in 2018, corresponding to almost 10% of all people being alive after a cancer diagnosis in Denmark.¹ The growing population of cancer survivors has emerged from advances in diagnostic technologies and cancer treatments,² and the outcomes of interest after cancer treatment are changing from overall survival towards disease-free survival and cancer recurrence.³

General practice is increasingly engaged in cancer follow-up and recurrence surveillance in many healthcare systems.⁴ Nevertheless, little evidence exists to inform this shift. Information on disease-free survival and recurrence is rarely available outside clinical trials, and clinical trials often use wide exclusion criteria, which limit the generalizability of findings to the entire population of cancer survivors.⁵ Thus, population-level evidence is warranted in this field.³

Internationally, algorithms based on routinely collected data like medical claims and cancer-related register data have been developed to identify patients diagnosed with cancer recurrence, mainly from breast and colorectal cancer; the validation of these algorithms has shown mixed results.^6–12 Recent studies have shown that cancer-specific algorithms based on Danish national health registries can accurately identify patients diagnosed with recurrence of colorectal cancer,¹³ breast cancer,^14–16 bladder cancer,¹⁷ or endometrial cancer,¹⁸ with little selection bias. However, no similar approach exists to identify patients diagnosed with recurrence of malignant melanoma.

We aimed to develop and validate a register-based algorithm to identify patients with recurrence of malignant melanoma at population level in Denmark, and to estimate the accuracy of the cancer recurrence diagnosis date derived from the algorithm.

Methods

Design and Setting

We conducted a register-based study in Denmark, where tax-funded (free of charge) healthcare is available to all citizens. The data were linked at the individual level through the unique personal registration number assigned to all Danish citizens at birth or immigration.¹⁹

Data Sources

Data were obtained from five national population-based registers. The Danish Melanoma Database (DMD)²⁰ comprises clinical information on the treatment and survival of incident cases of cutaneous melanoma, including follow-up on recurrence, since 1985. The Danish Cancer Register²¹ holds data on diagnosis codes, diagnosis dates, and tumor stage on all incident cancers diagnosed in Denmark since 1943. The Danish National Patient Register²² contains data on somatic in-hospital contacts since 1977 and on emergency and outpatient specialty care since 1995, including diagnosis codes, procedure codes, and tumor stage for cancer-related contacts. The Danish National Pathology Register²³ holds information on all pathology specimens analyzed in Denmark since 1997. The registrations are classified according to the Danish Systematized Nomenclature of Medicine (SNOMED) system, which allows for identification of malignant morphology (codes M8 and M9). The fifth digit of the morphology code denotes the specimen (eg “4: malignant, direct spread to surrounding tissue”, “6: malignant, metastasis”, and “7: malignant, recurrent”). The Danish Civil Registration System²⁴ holds (daily updated) information on migration and vital status on all Danish residents.

Gold Standard

The gold standard of recurrence from malignant melanoma was defined as recurrence status and recurrence date recorded in the DMD for patients treated for malignant melanoma at Herlev and Gentofte Hospital in 2008–2014. The hospital is located in the Capital Region of Denmark, and approximately 20% of all malignant melanomas in Denmark are treated here. In 2018–2019, recurrence registrations recorded in the DMD were updated for all patients treated at Herlev and Gentofte Hospital from 2008 and onwards. This was done by identification of metastases or recurrences from malignant melanoma in the Danish National Pathology Register.

Algorithm

The algorithm was constructed similarly to the algorithms for bladder, breast, and endometrial cancer recurrence by Rasmussen et al^15,17,18 (Figure 1). The date of surgery for malignant melanoma was defined as the first biopsy date recorded in the DMD. If a re-resection date was registered within 60 days of the first biopsy date, the re-resection date was defined as the date of surgery. To prevent inclusion of patients with residual cancer after surgery, we excluded patients with malignant disease recorded in the registries within 90 days after the date of surgery. Indicators of residual disease were: 1) distant tumor stage based on the Union for International Cancer Control (UICC) classification of malignant tumours,²⁵ 2) new malignant diagnosis code based on the International Classification of Diseases, 10th revision (ICD-10): C00-C96 and D37-D48, excluding C44 (non-melanoma skin cancer) and C43 (malignant melanoma), 3) SNOMED registration of M8–9 morphology, and 4) procedure code for excision of skin or lymph nodes, amputation, radiotherapy, or chemotherapy combined with a malignant diagnosis code (ICD-10: C00-C96 and D37-D48). This 90-day cut-off period running from primary cancer treatment until recurrence surveillance is in line with other Danish^15,17 and international^7,8 cancer recurrence algorithms.

Figure 1.

Temporal overview of the algorithm to identify patients with recurrence of malignant melanoma. Reproduced from Cancer Epidemiol, 59, Rasmussen LA, Jensen H, Flytkjær Virgilsen L, Jellesmark Thorsen LB, Offersen BV, Vedsted P. A validated algorithm for register-based identification of patients with recurrence of breast cancer—Based on Danish Breast Cancer Group (DBCG) data, 129–134, Copyright (2019), with permission from Elsevier.¹⁵

After the 90-day disease-free period, the algorithm searched for indicators of cancer recurrence (Figure 1). These indicators were identified through diagnosis codes and procedure codes in the Danish National Patient Register and through test results in the Danish National Pathology Register. A patient was defined as being diagnosed with melanoma recurrence if one of the following indicators was present:

• ICD-10: C439X (melanoma recurrence).

• ICD-10: C76*-C79* or C43xM (metastasis).

• Procedure codes: BWG* (radiotherapy), except BWGA* (scheduling), BWHA* (chemotherapy), or BOHJ* (immunotherapy) combined with one of the following ICD-10 codes: C43* (malignant melanoma), or C76*-C79* or C43xM (metastasis).

• Procedure codes: KCBB30/50/99, KDAB00/10, KDHB00/05/10, KEAA10/20/30/99, KEKB*, KQAE00/10, KQBE00/10, KQCE00/10, KQDE00/10 (surgical excision of skin), KPDJ* (excision of lymph nodes), or KNHQ*, KNDQ* (amputation), combined with one of the following ICD-10 codes: C439X (malignant melanoma recurrence), or C76*-C79* or C43xM (metastasis).

• SNOMED morphology codes: M8-M9 with 4, 6, or 7 in the fifth digit, and a morphology code similar to a morphology code registered within 90 days of the date of primary melanoma surgery.

A metastasis diagnosis code was disregarded as an indicator of recurrence if a second primary cancer diagnosis was recorded in the Danish Cancer Register or Danish National Patient Register either before the metastasis diagnosis date or within 30 days after this date. This was done to avoid confusion between a metastasis from a second primary cancer and a metastasis from malignant melanoma. If a patient had more than one indicator of recurrence, the date of the first occurring indicator was entered as the date of cancer recurrence.

Study Population

Patients from the gold standard population were eligible for inclusion if they were registered in the Danish Cancer Register with a diagnosis code of malignant melanoma (ICD-10: C43), with no prior cancer diagnosis (except for non-melanoma skin cancer, ICD-10: C44), and aged ≥18 years at the time of diagnosis. Recurrence status in the DMD was not validated for 115 patients with a positive sentinel node identified at the time of the surgery for the primary cancer, and these 115 patients were excluded. Patients who emigrated or died within 90 days of their cancer surgery were excluded. Finally, patients recorded with residual malignant disease within 90 days of cancer surgery were excluded, as described in the sub-section “Algorithm”.

Analyses

The sensitivity, specificity, and positive and negative predictive values of the algorithm with 95% confidence interval (CI) were estimated from the concordant and discordant frequencies between the recurrences identified by the algorithm and the gold standard. To investigate the influence of each of the indicators, the sensitivity, specificity, and positive and negative predictive values were analyzed separately for recurrences identified by a diagnosis code, a procedure code, and a pathology test result.

The median follow-up time with interquartile interval (IQI) was estimated from the surgery date until the date of the first of one of the following events: cancer recurrence, death, emigration, last date of follow-up in the gold standard, or end of study (10 March 2018).

The agreement between the date of recurrence identified by the algorithm and the date identified by the gold standard was measured by Lin’s concordance correlation coefficient (CCC).²⁶ Values <0.90 are considered “poor”, 0.90–0.95 “moderate”, >0.95 “substantial”, and >0.99 “almost perfect”.²⁷ We estimated the proportions of recurrence dates estimated on the same date as the gold standard date and within an interval of 7, 14, 30, 60, and 90 days. Furthermore, we estimated the median number of days between the recurrence dates of the gold standard and the recurrence dates identified by the algorithm.

A sub-analysis was performed in a study population identified through the diagnosis date in the Danish Cancer Register and procedure codes for primary melanoma surgery in the Danish National Patient Register. This was done to assess the performance of an algorithm with no information from the clinical database (DMD).

Results

The final study population consisted of 1747 patients, hereof 95 (5.4%) had recurrence from malignant melanoma according to the gold standard (Figure 2). The median time from the start of the surveillance period until identification of recurrence was 18 months (IQI: 9–34 months) according to the gold standard. Study population characteristics are shown in Table 1.

Figure 2.

Flowchart of the study population.

Table 1.

Characteristics of Included Patients Stratified on Cancer Recurrence Status in the Gold Standard^a

Population Characteristics	Cancer Recurrence		No Cancer Recurrence
	n	(%)	n	(%)
N	95	(100)	1652	(100)
Sex
Female	39	(41)	872	(53)
Male	56	(59)	780	(47)
Age, median (IQI)	68 (59;76)		59 (45;70)
Tumor stage primary cancer
IA	15	(16)	982	(59)
IB	21	(22)	276	(17)
IIA	5	(5)	54	(3)
IIB	12	(13)	38	(2)
IIC	6	(6)	15	(1)
III–IVb	5	(5)	6	(0.5)
Missing	31	(33)	281	(17)
Follow-up timec, months (IQI)	31 (17;49)		55 (38;61)

Notes: ^aNumbers are n (%) if nothing else is stated. ^bStage IV is pooled with stage III to adhere to legislation on data privacy protection. ^cTime from primary melanoma surgery to the first of the following events: recurrence, emigration, death, or end of study (10 March 2018).

Abbreviation: IQI, interquartile interval.

The algorithm identified 89 of the 95 recurrences and additional 14 false positives (Table 2), which demonstrated a sensitivity of 93.7% (95% CI: 86.8–97.6) and a specificity of 99.2% (95% CI: 98.6–99.5) (Table 3). The stratified analysis revealed that the highest sensitivity was seen for indicators of diagnosis codes, whereas the highest positive predictive value was obtained for indicators of pathology codes (Table 3).

Table 2.

Concordance of Recurrence of Malignant Melanoma Identified by the Gold Standard and the Algorithm

		Recurrence by Gold Standard
		Yes	No	Total
Recurrence by algorithm	Yes	89	14	103
	No	6	1638	1644
	Total	95	1652	1747

Table 3.

Performance of the Algorithm to Identify Recurrence from Malignant Melanoma (n=1747)

Algorithm Performance	All Indicators Combined	Indicators Stratified on Codes Related to Pathology, Procedure, and Diagnosis
	% (95% CI)	Pathology Codesa % (95% CI)	Procedure Codesb % (95% CI)	Diagnosis Codesc % (95% CI)
Sensitivity	93.7 (86.8–97.6)	52.6 (42.1–63.0)	71.6 (61.4–80.4)	75.8 (65.9–84.0)
Specificity	99.2 (98.6–99.5)	99.7 (99.3–99.9)	99.4 (98.9–99.7)	99.5 (99.0–99.8)
PPV	86.4 (78.2–92.4)	90.9 (80.0–97.0)	87.2 (77.7–93.7)	90.0 (81.2–95.6)
NPV	99.6 (99.2–99.9)	97.3 (96.5–98.1)	98.4 (97.7–98.9)	97.6 (97.9–99.1)

Notes: ^aHistology similar to primary cancer histology and coded as recurrent, metastatic, or direct spread to surrounding tissue. ^bExcision of skin, excision of lymph nodes, amputation, immunotherapy, radiotherapy, and chemotherapy. ^cICD-10: C76-C79 (except C779), C43xM, C439X.

Abbreviations: CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value.

The CCC for the agreement between recurrence dates generated by the algorithm and the dates generated by the gold standard was estimated to be almost perfect, as it showed a CCC of 0.992 (95% CI: 0.989–0.996). The date was estimated within 7 days of the gold standard date in 42% and within 60 days in 84% (Table 4). The median number of days between the recurrence date identified by the algorithm and the date identified by the gold standard was 12 days (IQI: 1–35 days).

Table 4.

Accuracy of Cancer Recurrence Date as Estimated from the Algorithm Compared with the Gold Standard

Algorithm Estimated Date	% (95% CI)
Same date	23 (15–34)
Within 7 days	42 (32–54)
Within 14 days	57 (46–67)
Within 30 days	73 (64–82)
Within 60 days	84 (76–91)
Within 90 days	87 (79–93)

In the sub-analysis, we assessed the performance of an algorithm with no information from the DMD and the Danish National Patient Register served to identify the primary melanoma surgery date. We failed to identify procedure codes indicating primary melanoma surgery in 127 patients in the Danish National Patient Register, and these patients were excluded. The final study population consisted of 1620 patients, hereof 75 had recurrence according to the gold standard. The performance of the algorithm in this population was similar to its performance in the main analyses (Supplementary Tables 1–3).

Discussion

We developed and validated a register-based algorithm to identify patients diagnosed with recurrence of malignant melanoma in Denmark. The algorithm correctly identified 94% of patients with recurrence and 99% of patients without recurrence according to the gold standard. With a recurrence prevalence of 5%, the positive predictive value was 86% and the negative predictive value over 99%. In 84% of cases, the algorithm estimated the date of recurrence within 60 days of the date estimated by the gold standard.

Strengths and Limitations

The study has several strengths. The algorithm is based on Danish national registries with near-complete high-quality records ensuring data of high validity.^19,21–24 Tax-funded, free, and equal access to healthcare in Denmark limits the risk of selection bias.

Another important strength was that the gold standard included information on all patients treated for malignant melanoma at Herlev and Gentofte Hospital. The absence of exclusion criteria for patients registered in the DMD ensures high generalizability of the results to the entire Danish population diagnosed with malignant melanoma.⁵ Furthermore, treatment and follow-up protocols for malignant melanomas in Denmark are described by the Danish Melanoma Group.²⁸ These uniform national strategies further increase the generalizability of the gold standard population to the entire population of patients treated for malignant melanoma in Denmark. Finally, the access to a large gold standard population was important as we investigated a disease with a relatively low recurrence rate. A second validation of the algorithm in a different population with information on melanoma recurrence status would further strengthen the validation of the algorithm. Unfortunately, such a population was not available.

All indicators of recurrence were based on electronic registrations in hospital records. Consequently, contact to a hospital was required for the case to be identified by the algorithm. The current Danish guidelines for recurrence surveillance after treatment for malignant melanoma recommend self-monitoring and patient-initiated follow-up of patients with low risk of recurrence (melanoma in situ and stage IA), five-year clinical and hospital-based follow-up of patients with intermediate risk of recurrence (stage IB and IIA), and specialized clinical and imaging follow-up of patients with high risk of recurrence (stage IIB, IIC and III) followed by five-year follow-up in general practice. However, in case of suspicion of recurrence, all patients are seen in a specialized hospital setting, which increases the likelihood of the algorithm to identify recurrence.

In the sub-analysis, the algorithm failed to identify procedure codes for primary cancer surgery in the Danish National Patient Register in 7% of the gold standard population, and these patients were excluded. This increased the risk of underestimating the absolute number of recurrences. However, the performance of the algorithm in terms of sensitivity, specificity, and predictive values was similar to the performance in the main analyses.

The algorithm identified 89 of 95 recurrences in the gold standard and additional 14 false positives. Half of the false positives had more than one and up to four different indicators of recurrence, and the algorithm may thus confuse second primary melanomas with melanoma recurrence. Conversely, in the validation study of recurrences at Herlev and Gentofte Hospital, the DMD may have missed a recurrence if registered as a primary tumor (not metastasis or recurrence) in the Danish National Pathology Register.

A total of 115 patients were excluded, as their recurrence status was not validated in the DMD. Most of these 115 patients were diagnosed at stage III. This is reflected in the stage distribution; less than 1% are stage III patients compared to the 10% stated in the annual report 2014 of the DMD.²⁹ After a diagnosis of malignant melanoma in stages I–III, the recurrence rates are reported to be up to 15%.³⁰ When simulating the algorithm in a population with a recurrence rate of 15%, the positive predictive value increases to 95.1%, and the negative predictive value remains almost unchanged at 98.9%. Thus, the algorithm is expected to perform superior in a non-selected population compared to the performance in the present study.

Comparison with Literature

To our knowledge, no previous studies have developed register-based algorithms to identify patients diagnosed with recurrence of malignant melanoma. The algorithm performed similarly to previously developed algorithms that used the same data sources to identify patients diagnosed with recurrence of bladder, breast and colorectal cancer.^{13–15,17,18} Studies from North America have validated algorithms to identify patients diagnosed with recurrence of breast, colorectal, and lung cancer.^6–10,12 In these studies, indicators of recurrence were based on procedure codes, diagnosis codes, and hospice and oncology visits; the sensitivity ranged from 70% to 94%, and the specificity ranged from 70% to 98%. Only two studies reached a sensitivity above 90%.^6,9 The accurate performance of the algorithms described in Danish studies seems to result from the inclusion of pathology data as indicators of recurrence.^13–15,17 Restrictions to the pathology coding prevented false positives, and the pathology data reached high positive predictive values, ranging from 91% to 100% across the studies by Rasmussen et al.^15,17,18

Studies from the United States based their algorithms on specific medical claims combined with cancer-related register data and other administrative data restricted to specific populations, geographic areas, or insurance groups.^6–8,10 This limits their generalizability to entire populations, whereas algorithms based on the population-based nationwide Danish health registers allow for national applicability.

Three studies from outside Denmark assessed the accuracy of the estimated recurrence diagnosis date; they identified 14–36% of the recurrences within 30 days of the gold standard recurrence diagnosis date^6,10 and with an average prediction error of 4.5 months.¹¹ Recurrence dates were identified more accurately in the present study; 73% of recurrences were identified within 30 days of the gold standard recurrence diagnosis date. These results were in line with previous Danish studies in bladder, breast, and endometrial cancer.^15,17,18 The validity of research in disease-free survival and recurrence pathways relies highly on the accuracy of the identified recurrence diagnosis date, and potential bias caused by the estimated date should always be considered. In the present study, deviations of recurrence dates were equally distributed before and after the gold standard date (data not shown), which limits potential bias.

Existing studies and databases have used different cut-offs to distinguish between persistent primary disease and recurrence. We used a cut-off of 90 days after primary melanoma surgery, whereas others have defined a recurrence from malignant melanoma as the return of disease as early as 30 days after the primary melanoma surgery.³¹ Studies using the algorithm to identify a population with recurrence from malignant melanoma should be highly aware of the definitions of remission, persistent disease, and recurrence in the algorithm.

Implications

The algorithm facilitates future large-scale research in recurrence from malignant melanoma, which can inform cancer follow-up strategies and improve the quality of cancer survivor care. Cancer follow-up is being reorganized in many healthcare systems. General practice is planned to play a greater and more formal role in the future cancer follow-up in Denmark and in other countries with similar healthcare systems.⁴ The recent changes in the organization of follow-up after treatment for malignant melanoma in Denmark were primarily based on expert opinions. However, the proposed algorithm has the potential to evaluate the ability of cancer follow-up programs to detect recurrences in a timely manner.

In 2018, 99% of patients diagnosed with malignant melanoma in Denmark were candidates for curative intent surgery.³² A high rate of successfully treated patients increases the relevance of using disease-free survival as the endpoint. Systemic treatment for malignant melanoma is gaining ground these years, and measurement of recurrence at population level provides an opportunity to evaluate new treatment strategies. Furthermore, the population-based design of the algorithm facilitates epidemiological research on potential predictors and causation of recurrence of malignant melanoma.

Conclusion

We developed an algorithm to identify recurrence of malignant melanoma using routinely collected data in Danish health registries. The high accuracy of the algorithm to identify cases of recurrence and to estimate the date of recurrence facilitates population-based research in the field of recurrence of malignant melanoma. This can extend our knowledge on recurrence of malignant melanoma and guide us towards the ultimate goal of improving the prognosis for the patients.

Data Sharing Statement

The data supporting the findings of this study are stored and maintained electronically at Statistics Denmark. The data is only accessible via a secured virtual private network (VPN) and only by approved collaborative partners. The data is not publicly available due to the Danish data protection legislation as the data contains information that could compromise the privacy of the research participants.

Ethics

The study is registered as “The patient pathway for cancer recurrence” (study 1, id 119) in the Record of Processing Activities at the Research Unit for General Practice in Aarhus in accordance with the provisions of the General Data Protection Regulation (GDPR). According to Danish law, the study required no approval from the Committee on Health Research Ethics in the Central Denmark Region.

Disclosure

The authors report no conflicts of interest in this work.