Is the Number of National Database Research Studies in Musculoskeletal Sarcoma Increasing, and Are These Studies Reliable?

Abstract

Background

Large national databases have become a common source of information on patterns of cancer care in the United States, particularly for low-incidence diseases such as sarcoma. Although aggregating information from many hospitals can achieve statistical power, this may come at a cost when complex variables must be abstracted from the medical record. There is a current lack of understanding of the frequency of use of the Surveillance, Epidemiology, and End Results (SEER) database and the National Cancer Database (NCDB) over the last two decades in musculoskeletal sarcoma research and whether their use tends to produce papers with conflicting findings.

Questions/purposes

(1) Is the number of published studies using the SEER and NCDB databases in musculoskeletal sarcoma research increasing over time? (2) What are the author, journal, and content characteristics of these studies? (3) Do studies using the SEER and the NCDB databases for similar diagnoses and study questions report concordant or discordant key findings? (4) Are the administrative data reported by our institution to the SEER and the NCDB databases concordant with the data in our longitudinally maintained, physician-run orthopaedic oncology dataset?

Methods

To answer our first three questions, PubMed was searched from 2001 through 2020 for all studies using the SEER or the NCDB databases to evaluate sarcoma. Studies were excluded from the review if they did not use these databases or studied anatomic locations other than the extremities, nonretroperitoneal pelvis, trunk, chest wall, or spine. To answer our first question, the number of SEER and NCDB studies were counted by year. The publication rate over the 20-year span was assessed with simple linear regression modeling. The difference in the mean number of studies between 5-year intervals (2001-2005, 2006-2010, 2011-2015, 2016-2020) was also assessed with Student t-tests. To answer our second question, we recorded and summarized descriptive data regarding author, journal, and content for these studies. To answer our third question, we grouped all studies by diagnosis, and then identified studies that shared the same diagnosis and a similar major study question with at least one other study. We then categorized study questions (and their associated studies) as having concordant findings, discordant findings, or mixed findings. Proportions of studies with concordant, discordant, or mixed findings were compared. To answer our fourth question, a coding audit was performed assessing the concordance of nationally reported administrative data from our institution with data from our longitudinally maintained, physician-run orthopaedic oncology dataset in a series of patients during the past 3 years. Our orthopaedic oncology dataset is maintained on a weekly basis by the senior author who manually records data directly from the medical record and sarcoma tumor board consensus notes; this dataset served as the gold standard for data comparison. We compared date of birth, surgery date, margin status, tumor size, clinical stage, and adjuvant treatment.

Results

The number of musculoskeletal sarcoma studies using the SEER and the NCDB databases has steadily increased over time in a linear regression model (β = 2.51; p < 0.001). The mean number of studies per year more than tripled during 2016-2020 compared with 2011-2015 (39 versus 13 studies; mean difference 26 ± 11; p = 0.03). Of the 299 studies in total, 56% (168 of 299) have been published since 2018. Nineteen institutions published more than five studies, and the most studies from one institution was 13. Orthopaedic surgeons authored 35% (104 of 299) of studies, and medical oncology journals published 44% (130 of 299). Of the 94 studies (31% of total [94 of 299]) that shared a major study question with at least one other study, 35% (33 of 94) reported discordant key findings, 29% (27 of 94) reported mixed key findings, and 44% (41 of 94) reported concordant key findings. Both concordant and discordant groups included papers on prognostic factors, demographic factors, and treatment strategies. When we compared nationally reported administrative data from our institution with our orthopaedic oncology dataset, we found clinically important discrepancies in adjuvant treatment (19% [15 of 77]), tumor size (21% [16 of 77]), surgery date (23% [18 of 77]), surgical margins (38% [29 of 77]), and clinical stage (77% [59 of 77]).

Conclusion

Appropriate use of databases in musculoskeletal cancer research is essential to promote clear interpretation of findings, as almost two-thirds of studies we evaluated that asked similar study questions produced discordant or mixed key findings. Readers should be mindful of the differences in what each database seeks to convey because asking the same questions of different databases may result in different answers depending on what information each database captures. Likewise, differences in how studies determine which patients to include or exclude, how they handle missing data, and what they choose to emphasize may result in different messages getting drawn from large-database studies. Still, given the rarity and heterogeneity of sarcomas, these databases remain particularly useful in musculoskeletal cancer research for nationwide incidence estimations, risk factor/prognostic factor assessment, patient demographic and hospital-level variable assessment, patterns of care over time, and hypothesis generation for future prospective studies.

Level of Evidence

Level III, therapeutic study.

Introduction

Retrospective studies using large national datasets have become a common source of information on the trends and patterns of cancer care in the United States [9, 62]. Sarcoma, which has the lowest incidence among all cancer types, is frequently analyzed in large national databases to improve statistical power. The National Cancer Database (NCDB) and the Surveillance, Epidemiology, and End Results (SEER) are the two largest cancer registries in the United States. The SEER was initiated by the National Cancer Institute in 1973, and the NCDB began as a project between the American College of Surgeons and the American Cancer Society in 1988. Several important similarities and differences between these two large cancer registries have been summarized and reported [10, 60]. Maintaining data integrity is an essential component of both national database programs, as data reporting is highly standardized, and validation is regularly practiced in each database.

There has been an apparent increase in SEER-based and NCDB-based studies in musculoskeletal sarcoma research in recent years. A simple PubMed search of “sarcoma and SEER” or “sarcoma and NCDB” demonstrates this notable trend, although the frequency of use of these databases over the last few decades is unknown. As an orthopaedic oncology practice that manages its own longitudinal patient dataset and is part of a tertiary cancer center that reports administrative data to both the SEER and NCDB databases, our own experiences have raised questions of the accuracy of the nationally reported data. It has been reported that sarcoma research using national cancer databases poses unique challenges because of inherent limitations in data abstraction and diagnosis coding [54]. The wider implications of these potential limitations could foster skepticism regarding the validity of some studies using these databases. However, it remains uncertain whether these databases tend to produce papers with discordant and/or unreliable findings. It is known in orthopaedic research that asking the same study question to different databases can yield different results [32, 51]. Although this is not in itself problematic, it could become so in musculoskeletal sarcoma research if an increasing number of studies with discordant findings using the same database became more of a source of confusion than clarity on clinically important topics.

In this study, we asked: (1) Is the number of published studies using the SEER and NCDB databases in musculoskeletal sarcoma research increasing over time? (2) What are the author, journal, and content characteristics of these studies? (3) Do studies using the SEER and the NCDB databases for similar diagnoses and study questions report concordant or discordant key findings? (4) Are the administrative data reported by our institution to the SEER and the NCDB databases concordant with the data in our longitudinally maintained, physician-run orthopaedic oncology dataset?

Materials and Methods

Publication Trend of SEER and NCDB Studies Over Time

To answer our question regarding publication trends, we searched PubMed from 2001 through 2020 for all published studies on sarcoma using the SEER or the NCDB. Search terms included “sarcoma and SEER” and “sarcoma and NCDB” or “National Cancer Database.” This initial query identified 659 studies (Fig. 1). Studies were excluded if sarcoma was not the main disease studied, the NCDB or SEER was not the source of reported data, or the data exclusively involved anatomic locations other than the extremities, nonretroperitoneal pelvis, trunk, chest wall, or spine. The first author (JML) manually reviewed all abstracts for inclusion. After the exclusion criteria were applied, 299 musculoskeletal sarcoma studies using the SEER or the NCDB were identified: 235 using SEER and 66 using the NCDB (two studies analyzed both SEER and the NCDB and they were counted toward both databases). Studies were tabulated by publication year in total and by separate database. We performed a statistical analysis to determine publication trend over time.

Fig. 1.

This flowchart shows how we reviewed the studies from PubMed and the process for arriving at the studies we subsequently reviewed. ^aIn total, there were 301 analyses of the NCDB or SEER databases within 299 discrete studies (two studies analyzed both NCDB and SEER).

Descriptive Characteristics of SEER and NCDB Studies

Two authors (SRJ, KSH) reviewed 299 musculoskeletal sarcoma abstracts for author, journal, and content details. Discrepant interpretations were reviewed, and consensus was made by the first author (JML). Study characteristics collected included information about the author (country, institution, and author specialty), journal (journal name and journal specialty), and content (bone or soft tissue sarcoma, type of content, use of survival analysis, and use of an advanced statistical analysis). Details about the last author of each study were used for country, institution, and author specialty data. Specific institution entities (medical school, cancer center, medical center, or university) were grouped by their parent institution (for example, university) when tabulating author institution data. Author and journal specialties included orthopaedics (including orthopaedic oncology), surgical oncology (all other surgical subspecialties), medical oncology, and other (pathology, radiation oncology, and nononcologic medical specialties). We determined the content type of each study based on the key finding(s). Studies were divided into five content types, which included: summary, demographics, tumor, treatment, and surveillance. Summary studies were an overview of a specific sarcoma type, often reporting incidence and an analysis of patient, tumor, and treatment variables with survival. Demographic studies focused on patient age, race, insurance status, marital status, employment, socioeconomic status, and treatment center volume. Tumor studies focused on pathologic findings, tumor grade, stage, location, lymph node involvement, and risk factors for metastasis and prognosis. Treatment studies focused on surgery, radiation, and chemotherapy. Surveillance studies focused on the development of secondary malignancies after sarcoma or the development of a secondary sarcoma. Survival analysis was defined as the reporting of survival outcome data, and often included a Kaplan-Meier and/or Cox proportional hazards analysis. Advanced statistical analysis was defined as the use of a statistical method other than traditional univariate and multivariate (logistic regression or Cox proportional hazards) testing. If survival analysis or statistical method information was not available in the abstract, the manuscript was reviewed.

Concordance of Key Findings From SEER and NCDB Studies With Similar Study Questions

To answer our third question regarding concordance of key findings, all 299 abstracts were grouped by specific sarcoma diagnosis. Within each diagnosis, the first author (JML) assessed each abstract if it shared a similar major study question to at least one other study; there were 94 studies with 28 study questions that were identified (Fig. 2). A major study question was defined as one addressed in the key findings of the results and/or conclusion of the abstract. The 28 study questions (and their associated studies) were classified into three groups based on the concordance of the key findings from review of all full-text manuscripts. The first author (JML) and the senior author (HSS) made classification decisions together. The three groups were concordant, discordant, or mixed. Concordant classification was defined as any two or more studies with similar key findings and similar messages. Discordant classification was defined as two or more studies with different key findings and different messages. Mixed classification was defined as any two or more studies that consisted of either: (1) different key findings though similar messages or (2) different key findings in some studies compared with secondary findings in other studies and different messages. Based on these definitions, mixed studies were considered to have less discordance than discordant studies. The message of the abstract was defined as the author’s valuation of the key findings. Key findings were defined as those found within the results and/or conclusion of the abstract. Key findings were never solely based on univariate analysis unless a multivariate analysis was not performed. Secondary findings were defined as results within the body of the study but not emphasized in the abstract. For example, a study may have analyzed 10 factors when assessing disease prognosis but only presented the significant factors from the multivariable analysis in the abstract. These significant and emphasized factors in the abstract were considered key findings, whereas those not highlighted in the abstract were considered secondary findings. Eight studies were included in more than one study question. Five studies had concordant findings for one study question and discordant findings for another study question. One study had mixed findings for two study questions, one study had mixed findings for one study question and discordant findings for another study question, and one study had mixed findings for one study question and concordant findings for another study question.

Fig. 2.

This flowchart shows how we reviewed musculoskeletal sarcoma studies for similar study questions and then subsequently classified some as having concordant, discordant, or mixed key findings. ^aConcordant classification was defined as any two or more studies with similar key findings and similar messages. Discordant classification was defined any two or more studies with different key findings and different messages. Mixed classification was defined any two or more studies that consisted of either: (1) different key findings and similar messages or (2) different key findings in some studies compared with secondary findings in other studies and different messages. ^bEight studies were included in more than one study question. Five studies had concordant findings for one study question and discordant findings for another study question. One study had mixed findings for two study questions, one study had mixed findings for one study question and discordant findings for another study question, and one study had mixed findings for one study question and concordant findings for another study question.

Differences Between Local Administrative Data and Internal Orthopaedic Oncology Dataset

To answer our fourth question regarding data concordance, we performed an internal coding audit comparing administrative data from our institution’s cancer registry submitted to national databases with data from our longitudinally maintained, physician-run orthopaedic oncology dataset. Our practice is an urban, university-based, tertiary referral center for sarcoma which treats around 400 new sarcomas per year. Our orthopaedic oncology dataset is maintained on a weekly basis by the senior author (HSS) who manually records data directly from the medical record and weekly sarcoma tumor board consensus notes. This dataset served as the gold standard for data comparison. From 2018 to 2020, two to three patients per month with a musculoskeletal sarcoma (with a ratio of 2:1 soft tissue sarcoma to bone sarcoma) were sampled from our orthopaedic oncology dataset to be included in a sample of 90 patients. The six factors assessed for concordance were date of birth, resection surgery date, surgical margin status, tumor size, clinical stage, and adjuvant treatment. All 90 patients had all six factors prerecorded in our orthopaedic oncology dataset. These 90 patients were submitted to our institutional cancer registry certified coders so that they would inform us what codes were submitted to the national databases. Thirteen patients were excluded from their audit for the following reasons: uncertain, necrotic, or altered diagnosis (four patients) or coders had yet to abstract chart after more than 6 months (nine patients). Thus, administrative data from our institution’s cancer registry were provided to us for 77 patients. These six factors were then compared for concordance between the orthopaedic oncology dataset and the cancer registry, with the orthopaedic oncology dataset serving as the gold standard. For each variable, we defined a clinically relevant and practical magnitude of difference to assess discordance between the two databases. Missing data from the administrative database were also counted as discordant. For date of birth and surgery date, an error in date of greater than one month was discordant. For margin status, a difference in R classification (R0, R1, R2) was discordant. For tumor size, a difference of greater than 50% was discordant. This definition was based on the mean tumor size of a soft tissue sarcoma to be 5 to 10 cm from our orthopaedic oncology dataset. Thus, if a tumor was 50% (around 3 cm) smaller or larger than reported, it could be either up‐staged or down‐staged with clinical implication. For stage, a difference in stage grouping was discordant. For adjuvant treatment, chemotherapy and radiation were reviewed. The inaccurate absence or presence of either, regardless of timing (preoperative or postoperative), was discordant.

Ethical Approval

This study was approved by our institutional review board at Vanderbilt University Medical Center, Nashville, TN, USA (IRB# 211598).

Statistical Analysis

The publication rate over the 20-year span was assessed with simple linear regression modeling. The difference in the mean number of studies between 5-year intervals (2001-2005, 2006-2010, 2011-2015, 2016-2020) was assessed with Student t-tests. The threshold for statistical significance was α = 0.05, and the false discovery rate was controlled at 0.05 to consider multiple comparisons. Statistical analyses were performed using R Statistical Software (version 4.1.1, R Foundation for Statistical Computing).

Results

Publication Trend of SEER and NCDB Studies Over Time

The number of studies using the SEER and NCDB databases has steadily increased between 2001 and 2020 (Fig. 3). In a linear regression model (the number of studies = b0 + b1*time), the slopes associated with time variable for NCDB, SEER, and total are 0.79, 1.74, and 2.51, respectively, with corresponding p values all less than 0.001, indicating that there has been an increasing number of studies over time. From 2001 to 2008, there were fewer than five PubMed-recorded studies per year (range 0 to 4) and all were SEER-based. The first NCDB-based study was published in 2014. The mean total number of studies more than doubled during 2011-2015 compared with 2006-2010 (13 versus 5 studies; mean difference 8 ± 3; p = 0.008), and more than tripled during 2016-2020 compared with 2011-2015 (39 versus 13 studies; mean difference 26 ± 11; p = 0.03). Of the 299 studies published between 2001 and 2020, 56% (168 of 299) have been published since 2018.

Fig. 3.

This is a graphic representation of how publications have increased by year from 2001 to 2020 for the SEER and NCDB.

Descriptive Characteristics of SEER and NCDB Studies

There were 114 institutions that published at least once using data from these databases (Table 1). There were 61 institutions with one study, 34 institutions with two to four studies, 15 institutions with five to nine studies, and four institutions with at least 10 studies. The most studies from one institution were 13. Twenty-eight percent (85 of 299) were authored in 13 non-US countries, all using SEER. Orthopaedic surgeons authored 35% (104 of 299) of studies, whereas medical oncology journals published the most studies at 44% (130 of 299). There were 102 journals that published the 299 studies. The journals Cancer (8% [23 of 299]), Journal of Surgical Oncology (6% [19 of 299]), and Clinical Orthopaedics and Related Research (6% [18 of 299]) were the most frequent publishers of SEER-based and NCDB-based studies. The subject matter of the studies was 57% (169 of 299) soft tissue sarcoma and 43% (130 of 299) skeletal sarcoma. Thirty-eight percent (113 of 299) of studies had content type characterized as a summary study. NCDB-based studies most often were treatment content–focused (39% [26 of 66]), and SEER-based studies most often were summary content–focused (42% [99 of 235]). Survival analyses were performed in 86% (257 of 299) of studies. Fourteen percent (42 of 299) of studies (all but one used SEER) used an advanced statistical method, most commonly a nomogram.

Table 1.

Publication data from 2001 to 2020

Variable	NCDB (n = 66)	SEER (n = 235)	All (n = 299)b
Author
Number of countriesa	1	14	14
Number of institutionsa	28	104	114
Institution dataa	19 institutions ≥ 5 publications 4 institutions ≥ 10 publications Most publications for single institution = 13
Author specialtya
Orthopaedics	30 (20)	36 (84)	35 (104)
Surgical oncology	36 (24)	19 (45)	23 (69)
Medical oncology	11 (7)	18 (42)	16 (47)
Other	23 (15)	27 (64)	26 (79)
Journal
Number of journals	27	93	102
Journal with most publications	Journal of Surgical Oncology 18 (12)	Cancer 9 (22)	Cancer 8 (23)
Journal specialty
Orthopaedics	8 (5)	20 (47)	17 (52)
Surgical oncology	39 (26)	7 (17)	14 (43)
Medical oncology	39 (26)	45 (106)	44 (130)
Other	14 (9)	28 (65)	25 (74)
Content
Sarcoma type
Bone	30 (20)	47 (110)	43 (130)
Soft tissue	70 (46)	53 (125)	57 (169)
Content type
Summary	23 (15)	42 (99)	38 (113)
Demographics	20 (13)	17 (41)	18 (54)
Tumor	18 (12)	16 (38)	17 (50)
Treatment	39 (26)	17 (40)	21 (65)
Surveillance	0 (0)	7 (17)	6 (17)
Advanced statistics usedc	1 (1)	17 (41)	14 (42)
Survival analysis used	83 (55)	87 (204)	86 (257)

Data presented as % (n).

^aBased on the last author.

^bThere were 299 separate publications, with 235 that used SEER and 66 that used the NCDB; two publications used both.

^cNCDB (one total): Bayesian (1); SEER (41 total): nomogram (26 studies), Bayesian (four studies), competing risk regression (four studies), machine learning (three studies), regression tree analysis (two studies), and validation (two studies).

Concordance of Key Findings From SEER and NCDB Studies With Similar Study Questions

Of the 94 studies that shared a major study question with at least one other study, 35% (33 of 94) reported discordant key findings concerning 10 study questions [1,6,8,11,12,13,18,23,26,29,36,39,40,41,44,49,53,59,63,67,68,72,79,83,93,95,98,99,100,106,109,110,111] (Table 2), 29% (27 of 94) reported mixed key findings concerning six study questions [3,7,15,19,22,25,27,28,30,35,36,37,43,45,47,50,56,75,76,82,88,89,90,91,103,104,107] (Table 3), and 44% (41 of 94) reported concordant key findings concerning 12 study questions [2,4,5,14,16,17,18,20,24,31,33,38,42,46,48,52,57,58,61,71,74,75,77,78,81,84,85,86,87,91,92,96,97,101,102,105,106,108,110,111,112] (Table 4). Of the 299 studies in total, 20% (59 of 299) reported discordant or mixed findings [1,3,6,7,8,11,12,13,15,18,19,22,23,25,26,27,28,29,30,35,36,37 39,40,41,43,44,45,47,49,50,53,56,59,63 67,68,72,75,76,79,82,83,88,89,90,91 93,95,98,99,100,103,104,106,107,109,110,111] and 14% (41 of 299) reported concordant findings [2,4,5,14,16,17,18,20,24,31,33,38,42,46,48,52,57,58,61,71,74,75,77,78,81,84,85,86,87,91,92,96,97,101,102,105,106,108,110,111,112]. Examples of topics with discordant or mixed findings include the effect of radiation on survival in several sarcoma types and the influence of race on treatment and survival. Examples of topics with concordant findings include prognostic factors or risk factors for metastasis in several sarcoma types and the impact of surgical treatment on survival in chondrosarcoma and osteosarcoma.

Table 2.

Discordant findings from national database studies in musculoskeletal sarcoma

Topic	Study	Year	Database	Key result
Marital status and OS in chondrosarcoma	Gao et al. [26]	2018	SEER	Being married associated with better OS
	Nguyen et al. [67]	2019	SEER	Being married associated with no better OS and higher risk of lung metastasis at initial presentation
Center volume and OS in soft tissue sarcoma	Bagaria et al. [8]	2018	NCDB	Higher volume center not associated with improved OS
	Venigalla et al. [93]	2018	NCDB	Higher volume center associated with improved OS
	Abarca et al. [1]	2018	NCDB	Higher volume center associated with improved OS
	Lazarides et al. [49]	2019	NCDB	Higher volume center associated with improved OS
Race and OS in Ewing sarcoma	Jawad et al. [41]	2009	SEER	Black patients with no worse OS
	Worch et al. [98]	2010	SEER	Black patients with worse OS
	Duchman et al. [18]	2015	SEER	Black patients with no different OS
	Zhou et al. [110]	2019	SEER	Black patients with worse OS
	Zhang et al. [106]	2019	SEER	Black patients with worse OS
	Zhou et al. [111]	2020	SEER	Black patients with no different OS
Race and surgery in pediatric sarcoma	Joseph et al. [44]	2017	SEER	Minorities less likely to undergo surgical resection
	Jacobs et al. [39]	2017	SEER	No difference who received surgery
Staging system performance	Cates et al. [12]	2018	SEER	8th AJCC edition not better than 7th AJCC edition
	Fisher et al. [23]	2018	NCDB	8th AJCC edition better than 7th AJCC edition
OS in osteosarcoma over the decades	Mirabello et al. [59]	2009	SEER	OS not increased since mid-1980s
	Jawad et al. [40]	2011	SEER	OS increased since mid-1980s in high-grade tumors
	Perkins et al. [72]	2014	SEER	OS not increased in last two decades
	Wu et al. [99]	2018	SEER	OS increased since mid-1980s
	Chen et al. [13]	2018	SEER	OS not increased in last two decades
Radiation therapy and OS in localized extraskeletal Ewing sarcoma	Lynch et al. [53]	2018	NCDB	Radiation therapy + chemotherapy/surgery associated with better OS
	Saiz et al. [79]	2019	NCDB	Radiation therapy + chemotherapy/surgery associated with no better OS
Radiation therapy vs. surgery and OS in localized Ewing sarcoma	Ning et al. [68]	2016	SEER	Radiation therapy noninferior to surgery alone
	Wan et al. [95]	2018	SEER	Surgery alone better than receiving radiation therapy
	Zhang et al. [109]	2018	SEER	No superior local treatment strategy
Radiation therapy and OS in chondrosarcoma	Huang et al. [36]	2019	SEER	Radiation therapy associated with worse OS
	Catanzano et al. [11]	2019	NCDB	Radiation therapy not associated with worse OS, and in select high-risk patients has survival benefit
Radiation therapy/chemotherapy and OS in synovial sarcoma	Naing et al. [63]	2015	SEER	Radiation therapy improves OS in all patients, could not assess chemotherapy
	Seo et al. [83]	2020	SEER	Radiation therapy improves OS in biphasic subtype only, chemotherapy does
	Xiong et al. [100]	2020	SEER	Radiation therapy improves OS in monophasic subtype only, chemotherapy does not
	Aytekin et al. [6]	2020	SEER	Radiation therapy improves OS, chemotherapy unclear
	Gingrich et al. [29]	2020	NCDB	Radiation therapy improves OS, chemotherapy does not

OS = overall survival; AJCC = American Joint Committee on Cancer.

Table 3.

Mixed findings from national database studies in musculoskeletal sarcoma

Topic	Study	Year	Database	Key result
Prognostic factors in chondrosarcoma	Giuffrida et al. [30]	2009	SEER	Only grade and stage (all patients)
	Song et al. [88]	2018	SEER	Age, grade, stage, size, surgery, subtype (only patients with high-grade sarcomas)
	Huang et al. [36]	2019	SEER	Age, gender, size, surgery, grade, subtype, location, radiation, chemotherapy (only patients with localized disease)
	Zhang et al. [107]	2019	SEER	Age, stage, socioeconomic status, size, surgery, chemotherapy (all patients)
Prognostic factors in chondroblastic osteosarcoma	Sun et al. [90]	2018	SEER	Age, grade, stage, surgery + race
	Gao et al. [27]	2020	SEER	Age, grade, stage, surgery + size
Prognostic factors in osteosarcoma	Duchman et al. [19]	2015	SEER	Age, size, stage, location, sex (only patients with high-grade sarcomas
	Huang et al. [37]	2019	SEER	Age, size, stage, grade (only patients with localized disease)
	Traven et al. [91]	2019	SEER	Age, stage, location, race (only patients with extremity tumors)
	Qi et al. [75]	2020	SEER	Age, size, stage, location, grade, radiation (all patients)
Race and OS in soft tissue sarcoma	Martinez et al. [56]	2008	SEER	Black patients with worse OS
	Cheung et al. [15]	2014	SEER	Black patients with slightly worse OS (4%)
	Alamanda et al. [3]	2015	SEER	Black patients with worse OS
	Lazarides et al. [50]	2018	NCDB	Black patients with worse OS
	Ramey et al. [76]	2018	SEER/NCDB	Black patients with no different OS (secondary finding)
	Featherall et al. [22]	2019	NCDB	Black patients with no different OS (secondary finding)
Radiation therapy and OS in DD-chondrosarcoma	Strotman et al. [89]	2017	SEER	Radiation therapy does not influence OS (secondary finding)
	Gao et al. [25]	2019	SEER	Adjuvant radiation therapy improves OS
Radiation therapy and OS in localized soft tissue sarcoma	Koshy et al. [47]	2010	SEER	Radiation therapy improves OS in patients with high-grade sarcomas only (not low-grade)
	Schreiber et al. [82]	2012	SEER	Radiation therapy improves OS in patients with high-grade sarcomas that are > 5 cm only (not < 5 cm)
	Bagaria et al. [7]	2014	SEER	Radiation therapy improves OS in Stage III tumors only (not Stage I/II)
	Yuen et al. [103]	2015	SEER	Radiation therapy improves OS in patients older than 65 years only (not younger than 65 years)
	Hou et al. [35]	2015	NCDB	Radiation therapy improves OS in patients with high-grade sarcomas
	Kachare et al. [45]	2015	SEER	Radiation therapy improves OS in Stage III tumors
	Yuen et al. [104]	2016	SEER	Radiation therapy improves OS in patients older than 65 years only (not in those younger than 65 years)
	Gingrich et al. [28]	2017	NCDB	Radiation therapy improves OS in all patients
	Ramey et al. [76]	2018	NCDB/SEER	Radiation therapy improves OS in patients with Stage II/III patients
	Johnson et al. [43]	2018	NCDB	Radiation therapy improves OS in NCDB but not in consortium

OS = overall survival; DD-chondrosarcoma = dedifferentiated chondrosarcoma.

Table 4.

Concordant findings from national database studies in musculoskeletal sarcoma

Topic	Study	Year	Database	Key result
Prognostic factors in Ewing sarcoma	Duchman et al. [18]	2015	SEER	Older age, larger tumor size, axial tumor location, stage IV disease associated with worse OS Four nomograms developed with similar factors
	Zhou et al. [110]	2019	SEER
	Zhang et al. [106]	2019	SEER
	Gao et al. [24]	2020	SEER
	Zhou et al. [111]	2020	SEER
Prognostic factors in malignant fibrous histiocytoma of bone	Huang et al. [38]	2020	SEER	Older age and stage associated with worse OS, whereas surgery associated with better OS
	Liu et al. [52]	2020	SEER
Prognostic factors in rhabdomyosarcoma	Perez et al. [71]	2011	SEER	Younger age, smaller tumor size, surgery, radiation, and localized disease associated with better OS
	Shen et al. [84]	2014	SEER
	Yang et al. [101]	2014	SEER
	Ren et al. [78]	2018	SEER
	Amer et al. [4]	2019	SEER
	Zhu et al. [112]	2020	SEER
	Han et al. [33]	2020	SEER
Lymph node status and OS in soft tissue sarcoma	Johannesmeyer et al. [42]	2013	SEER	Lymph node positive status associated with worse OS
	Keung et al. [46]	2018	NCDB
	Miccio et al. [57]	2019	NCDB
Risk factors for metastasis in Ewing sarcoma	Ramkumar et al. [77]	2018	SEER	Older age, axial location, and larger tumor size associated with lung metastasis at initial presentation
	Ye et al. [102]	2019	SEER
	Shi et al. [85]	2020	SEER
	Wang et al. [96]	2020	SEER
Risk factors for metastasis in osteosarcoma	Miller et al. [58]	2013	SEER	Older age, axial location, and larger tumor size associated with lung metastasis at initial presentation
	Zhang et al. [105]	2019	SEER
Rural location and OS in osteosarcoma	Cheung [14]	2013	SEER	Rural location associated with worse OS
	Wendt et al. [97]	2019	SEER
Male sex and OS in dermatofibrosarcoma protuberans	Criscito et al. [17]	2016	SEER	Male sex associated with worse OS
	Kreicher et al. [48]	2016	SEER
	Trofymenko et al. [92]	2018	NCDB
	Phan et al. [74]	2020	SEER
Marital status and OS in soft tissue sarcoma	Alamanda et al. [2]	2014	SEER	Being single associated with worse OS
	Zhang et al. [108]	2019	SEER
Surgical resection and OS in spine chondrosarcoma	Arshi et al. [5]	2017	SEER	Surgical resection associated with better OS
	Song et al. [87]	2020	SEER
	Song et al. [86]	2020	SEER
Chemotherapy and OS in soft tissue sarcoma	Movva et al. [61]	2015	NCDB	Chemotherapy associated with better OS
	Chowdhary et al. [16]	2019	NCDB
	Graham et al. [31]	2020	NCDB
Limb salvage and OS in osteosarcoma	Schrager et al. [81]	2011	SEER	Limb salvage associated with better OS than amputation
	Traven et al. [91]	2019	SEER
	Qi et al. [75]	2020	SEER
	Evans et al. [20]	2020	NCDB

OS = overall survival.

Differences Between Local Administrative Data and Internal Orthopaedic Oncology Dataset

The internal coding audit comparing administrative data with data from our orthopaedic oncology dataset showed discordance among the six factors (Table 5). Proportions of discordant values were as follows: date of birth, 4% (3 of 77); adjuvant treatment, 19% (15 of 77); tumor size, 21% (16 of 77); surgery date, 23% (18 of 77); surgical margins, 38% (29 of 77); and clinical stage, 77% (59 of 77). For surgery date, all instances of discordance occurred when a nondefinitive surgery was coded; examples of this were an incisional biopsy, a whoops procedure (an unplanned sarcoma resection), a flap or skin graft, or a take-back for infection. Tumor size was considered discordant only when there was at least a 50% difference in size in centimeters reported. All instances of discrepant results were found to be due to size data abstracted to the administrative dataset from different modalities—that is, preneoadjuvant treatment radiology size, post cytoreduction radiology size, or pathology specimen size. Margins were discordant from either R misclassification in patients with unplanned incomplete excisions performed at outside facilities or when margins reviewed at tumor board were not reflected accurately in the administrative data. Staging was most often discordant due to incomplete abstraction or missing values for two main reasons: (1) final pathology reporting was delayed due to molecular testing or when the tumor was necrotic or (2) the coders were uncertain of what grade to assign.

Table 5.

Coding audit comparing administrative data with orthopaedic oncology dataset

Year	Date of birth	Surgery date	Margin status	Tumor size	Stage	Adjuvant treatment
2018	0 (0)	4 (3)	7 (5)	8 (6)	19 (15)	5 (4)
2019	1 (1)	10 (8)	19 (15)	5 (4)	32 (24)	8 (6)
2020	3 (2)	9 (7)	12 (9)	8 (6)	26 (20)	6 (5)
Total	4 (3)	23 (18)	38 (29)	21 (16)	77 (59)	19 (15)
Discordance	4%	23%	38%	21%	77%	19%

The numbers in the rows under each column represent the % (n) of 77 patients with discordance between the administrative institutional data and our orthopaedic oncology dataset for each of the six variables; discordance was defined as when administrative data were missing or incorrect based on definitions within the methods section.

Discussion

The SEER and NCDB databases have the advantage of reporting nationwide data in large cohorts amenable to statistical analysis that is unfeasible to obtain from single institutions given the rarity of sarcoma. Little is known about the frequency of use of these databases, if their use tends to produce studies with conflicting or confusing findings, and the reliability of administrative data reported to these national databases. We describe in this study that there has been a considerable increase in the use of the SEER and the NCDB databases over the last decade, with one-fifth of all studies having varied amounts of discordant findings. Except for radiation use, there was no pattern of topics or diagnoses most associated with discordant findings. An internal audit of data submitted to national databases from our institution showed more nuanced clinical variables such as margins or stage were abstracted with less accuracy compared with harder endpoints such as dates. Some things are essential to sound interpretation of these studies: Investigators must take appropriate stewardship of these databases while readers must understand the subtleties in study design (inclusion/exclusion, missing data handling, statistical approach) and grasp the inherent limitations of these databases. Although useful in providing estimations of incidence, significance of demographics and risk factors, as well as patterns of care over time, we caution against using them to discern differences among treatment strategies that depend on many complex clinical variables, which might not be abstracted from the databases as accurately as needed. Prospective, multicenter studies such as through a consortium or the emerging Musculoskeletal Tumor Registry (MsTR), where data are collated by physicians in a coordinated method, are more appropriate for studies that seek to influence treatment strategy or health policy.

Limitations

Our study has several limitations. A major limitation of this study is that it has been shown that asking the same question of different large national databases can yield different results [32, 51]. This study was not intended to refute that concept but to provide a more nuanced understanding of the study questions from the SEER and NCDB studies that may be associated with inconsistent key findings. As well, there were several useful examples of discordant studies that evaluated the same database during the same time span, which one would expect to have more consistent findings compared with an assessment of different databases. Our first three study questions were limited by the nature of a literature review. Our search may not have included all peer‐reviewed studies because we limited it to those indexed in PubMed. We did not handsearch bibliographies of the included studies or look at grey literature such as conference proceedings, non-English sources, or preprint servers for additional studies. Nevertheless, we are confident the PubMed search outlined and performed in this study for study inclusion was reliable, as study exclusions were completed by the first author (JML) who is an attending orthopaedic oncologist. As well, it was beyond the scope of this study to assess the proportion of all sarcoma studies in the literature that come from SEER and NCDB studies, although it would have provided additional perspective on the relative increase in studies over time. For study question two, we limited collecting author details to only the last author of each study to standardize data collection. Thus, the total number of countries and institutions reported in this study are inevitably lower than if all author details were included. For study question three, we attempted to discern concordant and discordant findings between studies that shared similar study questions through our interpretation of the reported methods and statistics. To reduce our tendency toward selection and/or confirmation bias while identifying discordant studies, we employed a systematic approach of first identifying any study with a similar study question to another study when grouped by disease condition (or diagnosis). We then classified studies as being discordant, concordant, or mixed based on their key findings. Our method is inherently limited by what we defined as a major study question and a key finding. We intentionally defined these as being within the abstract of the study, as we felt the main message or messages of the study would be represented there. We created the definitions of concordant, discordant, and mixed in an attempt to provide both clarity and nuance to our classification. The approach to answering study question three was also limited in that we only used two authors to perform this classification and not a third consensus author. However, both authors (JML, HSS) are attending orthopaedic oncologists and separately reviewed all 94 papers with a shared study question. Lastly, we acknowledge although these findings may not alter specific clinical practices, they may affect how we read and initiate database studies and the import we place in them.

Our fourth study question was limited because the internal coding audit was only of our single institution and included fewer than 100 patients over a 3-year period. Our institutional coders limited the audit to fewer than 100 patients due to restraints on their time; therefore, we sampled our population. We also did not look back into the medical record for these patients to confirm the six variables for each patient. We worked under the premise that our orthopaedic oncology dataset served as the gold standard of data, given its maintenance and collation of data by the senior author (HSS) from the medical record and tumor board notes on a weekly basis. Although we feel confident that our orthopaedic oncology dataset represents the truth as closely as possible, we recognize that it is also subject to error. Lastly, although similar findings regarding the discrepancies of administrative data and the medical record at other comparable institutions are assumed, they are not guaranteed. This may have implication on the external validity of our findings and the greater impact on data accuracy of these databases nationwide.

Publication Trend of SEER and NCDB Studies Over Time

Over the past 20 years, use of the SEER and NCDB in musculoskeletal sarcoma research has progressively increased, most significantly since 2018. The overall growth is partly attributable to the introduction of the NCDB in 2014; however, that does not account for the more rapid growth in SEER studies. Even though we did not evaluate reasons for growth, an obvious although understated reason for the progressive increase in studies is the pressure to publish in academic medicine. Although little has been published on publication pressure in sarcoma research [73], many have confirmed its validity in biomedical research on increasing publication rates and influence on scientist bias [21, 80]. More likely, however, is that national databases have easy-to-access and organized datasets with a limited number of variables. As well, in the case of SEER, data are publicly available after simple institutional review board exemption is obtained rather than full institutional review board approval, according to the National Institutes of Health Office of Human Subjects Research [64]. These qualities can be particularly attractive to trainees (such as residents, fellows, and students) seeking to complete a project in a short time span, although we did not specifically evaluate this relationship.

Descriptive Characteristics of SEER and NCDB Studies

The scope of SEER and NCDB studies spanned more than 100 institutions in 14 countries (although the NCDB was only used in the United States), and four institutions published more than 10 studies during the 20-year period. It was not uncommon to find multiple studies published in close proximity by year by the same authors using the same database on the same diagnosis. Although experience using these databases may refine investigators’ abilities to perform good research, it is also plausible that institutions with increased experience navigating these databases may facilitate the tendency to ask more complicated questions that depend on variables not intended to be analyzed with great nuance. Large datasets are also easily amenable to the use of advanced statistical methods, such as a nomogram for prognosis, which occurred in nearly 20% (22 of 113) of SEER-based studies since 2018. Although the use of more novel statistical methods such as a nomogram, machine learning, or Bayesian analyses is not cause for concern, it does signify the need for the readership to familiarize themselves with these approaches to allow for clear reception of results. The most common type of study was when investigators reported a data summary of a specific sarcoma type, often including an analysis of incidence, risk factors, patterns of treatment, and survival. Demographic studies focused on health disparities using the variables of race and insurance status in both databases. Attention was also placed on the demographic variables unique to each database, as NCDB-based studies focused on treatment center volume whereas SEER-based studies focused on marital status. Additionally, we found that no single medical or surgical specialty predominated authorship, although one-third of papers were authored by orthopaedic surgeons and nearly half of studies were published in medical oncology journals. The reported authorship and journal data demonstrate the widespread use of these databases among all sarcoma subspecialties and across a variety of journals.

Concordance of Key Findings From SEER and NCDB Studies With Similar Study Questions

An in-depth review of 94 studies that shared a study question with at least one other study showed almost two-thirds of these studies had varied levels of discordant findings (discordant or mixed classification) and nearly half of these studies had concordant findings. Both concordant and discordant groups included papers on prognostic factors, demographic factors (gender, race, or marital status), and treatment strategies (radiation, surgery, or chemotherapy). The impact of radiation on survival consisted of solely discordant or mixed findings and not concordant findings. There were 24 studies concerning six study questions about radiation treatment in Ewing sarcoma, extraskeletal Ewing sarcoma, synovial sarcoma, chondrosarcoma, dedifferentiated chondrosarcoma, and soft tissue sarcoma [6, 7, 11, 25, 28, 29, 35, 36, 43, 45, 47, 53, 63, 68, 76, 79, 82, 83, 89, 95, 100, 103, 104, 109]. In studies with discordant findings, the concluding messages were notably different compared with studies we classified as having mixed findings where the concluding messages had subtle differences based on more minor discordances in key findings often based on patient inclusion/exclusion. It has been shown that assessing the impact of radiation on survival using SEER can be particularly fraught with challenges [69, 94]. One study showed that 21% of patients who were reported as receiving radiation were coded as not receiving radiation in the SEER database [70], and another study using Medicare-SEER linked data showed only moderate sensitivity in identifying patients who received radiation or chemotherapy [69]. Reasons for these findings include factors such as patient and physician preferences behind treatment options that are not captured with SEER data. These findings led SEER in 2017 to no longer “recommend comparing outcomes conditioned on treatment or comparative effectiveness research using the SEER data,” although custom request of these data on radiation and chemotherapy is allowed provided their associated limitations are acknowledged [65, 94].

Inherent limitations in the data content of any large database as previously described are assumed and must be understood by investigators and readers alike. It has also been shown that asking the same study question to different databases will often yield different results based on differences in data collection methods [32, 51]. Although SEER aggregates regional data on cancer incidence with a population-based sampling approach based on geography, the NCDB collects patient data from hospitals that are accredited through the Commission on Cancer [60]. However, one would assume asking the same study question to the same database should lead to concordant findings. This was the case for 9 of 12 study questions that had concordant database findings in our review, which seemed appropriate. What requires a more nuanced understanding of large-database research by both investigators and readers is how asking the same question to the same database will get discordant results. This was true for 8 of 10 study questions that had discordant findings in our review. Our findings showed when this occurs, it is often secondary to the methods of analysis at the discretion of investigators. For example, when assessing the study question of prognostic factors in chondroblastic osteosarcoma, two studies from 2018 and 2020 both used SEER data in a Cox regression analysis yet differed in their conclusions about key prognostic risk factors. Although both studies emphasized factors such as age, grade, stage, and surgery, one emphasized race as important [90] whereas the other instead described tumor size as a key determinant [27]. Notably, they differed in their handling of missing data, and the years of SEER evaluated (from 2004 to 2014 [90] versus from 1984 to 2015 [27]), which undoubtedly altered their patient cohorts. We classified this as mixed discordance, as the message of the studies was not grossly discordant, although particular findings were different, and the question of whether size or race are indeed independent predictors of survival remains unanswered. This example is one of many in how factors of analysis have magnified importance when large, publicly available datasets are used because different investigators can come to different key findings even as a result of well-intentioned analyses. These factors of analysis may include patient selection methods, cohort sizes, the period evaluated, statistical testing used, and even the emphasis placed on certain findings, all of which are at the discretion of the authors.

Differences Between Local Administrative Data and Internal Orthopaedic Oncology Dataset

The abstraction from the medical record and creation of a dataset that is electronically sent to a national repository for compilation is a complex task that lends itself to tiers of interpretation. Our internal audit identified varied magnitudes of discordance between a surgeon’s personal log using the electronic medical record versus the registry-reported data abstracted by certified coders at our institution through the same electronic medical record. Date of birth is considered a hard variable because interpretation is factual, and it is hard to create an error. A discordance occurred in 4% of patients, which can be considered a control or baseline for input error and at less than 5%, it was considered acceptable. When more subjective variables (such as surgery date, tumor size, margin status, clinical stage, and adjuvant treatment) require abstraction, a greater error rate occurs. For instance, abstracting the surgery date of the definitive oncologic resection for reporting to national databases is not as straightforward as expected, where all instances of discordance occurred when a nondefinitive surgery was coded. Notably, our audit identified a 38% (29 of 77) discordance in reporting surgical margin status and a 77% (59 of 77) discordance in correctly identifying the stage. For margins, although some were either missing or had an incorrect R classification, another issue was discerning the intricate subtleties of whoops procedures, planned positive margins, and counter margins. For stage, reasons included delayed pathologic results awaiting molecular analysis and changing coding and staging criteria. Others have similarly reported discordance when comparing abstracted codes with review of the medical record. Lyu et al. [55] reported 62% concordance when comparing ICD-9 and ICD-10 codes with the final pathologic results in a record review of patients with sarcomas. They recognized that coding of sarcomas is uniquely different from that of other cancers because it lacks organ specificity. For example, breast angiosarcoma is often miscoded as breast cancer because of the common organ of origin or an extraskeletal Ewing sarcoma may be included in a survival analysis intended for skeletal Ewing sarcoma. Furthermore, a 1998 study analyzed the concordance between hospital databases and a personalized surgeon’s log [34]. When comparing the surgeon’s log to the administrative database and billing claims data, only 60% concordance was identified for the hard variables that included date of birth, surgery date, CPT code, and ICD-9 code. Because CPT codes and ICD codes are often used to select the diagnosis-specific cohorts for national database studies in sarcoma, the necessity of their accuracy cannot be overstated because they impact the validity of any findings that can be drawn from these studies. Importantly, this differs from other orthopaedic specialties such as total joint arthroplasty, in which abstracting clinically relevant variables for a narrow subject can yield important results impacting clinical strategies [66].

Conclusion

The use of national databases in musculoskeletal sarcoma research is rapidly increasing. As one-fifth of all studies over the last 20 years had some number of discordant findings, it is imperative that investigators and readers understand the intent, best use, and limitations of these databases to minimize or at least appropriately interpret inevitable conflicting results. We understand the factors driving the use of large databases for sarcoma research, principally the rarity and heterogeneity of sarcomas, which make performing appropriately powered studies challenging for individual institutions. The SEER and the NCDB remain particularly useful and essential for some questions in sarcoma research, including incidence estimations, risk factor/prognostic factor assessment, patient demographic and hospital-level variable assessment, and evaluating patterns of care over time. As we and others have shown, however, more nuanced clinical variables may be abstracted with less accuracy, and thus we caution against the use of large databases to distinguish the merits of different treatment strategies, particularly given the complex factors that contribute to oncologic outcomes such as survival. The Centre for Evidence-Based Medicine, located in Oxford, UK, is an example of an organization that might create a levels of evidence pyramid that may help journals and readers of registry data account for these many complicated variables that affect reliability of data and outcomes. We submit that institutions who longitudinally maintain and analyze their data in a similar fashion, such as through multicenter partnerships (consortiums) or the emerging MsTR, may mitigate some of the limitations and challenges of large databases and should be prioritized for studies of clinical relevance.