International Expert-Based Consensus Definition, Classification Criteria, and Minimum Data Elements for Osteoradionecrosis of the Jaw: An Interdisciplinary Modified Delphi Study

Abstract

Purpose:

Osteoradionecrosis of the jaw (ORNJ) is a severe iatrogenic disease characterized by bone death after radiation therapy to the head and neck. With >9 published definitions and at least 16 classification systems, the true incidence and severity of ORNJ are obscured by lack of a standard for disease definition and severity assessment, leading to inaccurate estimation of incidence, reporting ambiguity, and likely underdiagnosis worldwide. This study aimed to achieve consensus on an explicit definition and phenotype of ORNJ and related precursor states through data standardization to facilitate effective diagnosis, monitoring, and multidisciplinary management of ORNJ.

Methods and Materials:

The Orodental Radiotherapy-Associated Late-Effects Consortium comprised 69 international experts, including representatives from medical, surgical, radiation oncology, and oral/dental disciplines. Using a web-based modified Delphi technique, panelists classified descriptive cases using existing classification systems, reviewed systems for feature extraction and specification, and iteratively classified cases based on clinical/imaging feature combinations.

Results:

The Consortium ORNJ definition was developed in alignment with Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) terminology and recent International Society of Oral Oncology-Multinational Association for Supportive Care in Cancer and the American Society of Clinical Oncology (ISOO-MASCC-ASCO) guideline recommendations. Case review using existing ORNJ classification systems showed high rates of inability to classify (up to 76%). Ten consensus statements and 9 minimum data elements (MDEs) were outlined for prospective collection and classification of precursor/ORNJ disease states.

Conclusions:

This study provides an international, consensus-based definition and MDE foundation for standardized ORNJ reporting in cancer survivors treated with radiation therapy. Head and neck surgeons, radiation, surgical, medical oncologists, and dental specialists should adopt MDEs to enable scalable health information exchange and analytics. Work is underway to develop both a human- and machine-readable knowledge representation for ORNJ (ie, ontology) and multidisciplinary resources for dissemination to improve ORNJ reporting in academic and community practice settings.

Introduction

Osteoradionecrosis of the jaw (ORNJ) is an iatrogenic disease experienced by patients with cancer treated with radiation therapy (RT) to the head and neck region. ORNJ incidence in head and neck cancer survivors is estimated to range from 5% to 15% with higher rates associated with risk factors such as poor oral hygiene, pre- and post-RT dental extractions, and high maxillary and/or mandibular radiation dose/volumes.^1–6 ORNJ can manifest as early as 6 months post-RT and, if not diagnosed and successfully managed early on, can progress to morbid states of symptom burden and poor quality of life via tooth loss, compromised orofacial function, and pain.^7–9 Financial repercussions of progressive ORNJ are substantial as major medical and surgical interventions for ORNJ can cost up to $170,000 per patient.^10,11

Until 2023, there was no International Classification of Diseases diagnostic code specific to ORNJ, resulting in an inability to formally report and assess ORNJ incidence.¹² Moreover, a historical lack of consensus in conceptualizing ORNJ is evident within existing literature, which includes at least 16 ORNJ classification systems published over a span of 4 decades, some of which only include data elements on treatment required or response to therapy in their classification schema.^1,13–27 This ambiguity in what constitutes minimum data elements (MDEs) to diagnose and classify ORNJ translates to variability in estimating the true event rate and poor intelligibility of cross-scale reporting. Ultimately, the state of classification hinders reusability of ORNJ data for comparing outcomes, building data-driven models of disease risk, and facilitating large-scale multi-institutional interventional trials to mitigate ORNJ.

To address this unmet need, the Orodental Radiotherapy-Associated Late-Effects (ORAL) Consortium was formed, comprised 69 internationally recognized multidisciplinary experts for consensus formation through a modified Delphi study. The specific aims of this study includes the following: (1) assessment of contextual overlap between existing ORNJ definitions and disease severity criteria, (2) characterization of item-specific interrater and interspecialty conceptual agreement across definitional systems for ORNJ, (3) determination of consensus-derived components of extant and proposed consensus ORNJ grading systems using best informatics principles, and (4) generation of standardized clinically relevant criteria (ie, MDEs) for clinical and radiographic diagnosis, assessment, and reporting of ORNJ across interdisciplinary care.

Methods and Materials

The consensus process

The consensus process was achieved via Delphi method and is not an update to pre-existing guidelines.^28–30 A process flowchart for this study is shown in Figure 1, and an Accurate Consensus Reporting Document guideline checklist for reporting consensus methods in biomedicine can be found in E1_ACCORD.³¹ A group of international, multidisciplinary oncology and oral/dental specialists was contacted via a closed electronic invitation process to participate and included coauthors of existing osteoradionecrosis classification systems (n = 75). Experts were also allowed to nominate additional experts to be considered for the panel. Any expert who consented to the study and completed at least 1 Delphi survey (n = 69) were collectively referred to as the ORAL Consortium. Surveys were developed in REDCap and Qualtrics (see E2 series).^32,33 Each questionnaire included an introduction, primary objectives for the round, and aggregated group feedback for consensus building. Briefly, experts were asked to review and rate existing medical terminologies and staging/grading systems for ORNJ (ie, classification systems), extract and rate clinical and radiographic data elements, resolve challenging topics for diagnosing and classifying ORNJ (ie, should a time feature be considered?), and classify text-only or image-based clinical scenarios designed to represent the full spectrum of ORNJ and its precursor states, including potential “classification threshold” cases.” After each round, items meeting consensus were reported in the following rounds as “consensus reports,” and no further questions were asked on those items. Details on Delphi methodology and statistical analysis can be found in E3_Methods. No patients were involved, and this study was approved by the University of Texas MD Anderson Cancer Center IRB.

Fig. 1.

Delphi consensus process flowchart. Abbreviation: ORN = osteoradionecrosis.

Results

The International ORAL Consortium

Characteristics of the ORAL Consortium are shown in Table 1 with representatives from head and neck surgery, radiation oncology (n = 26), medical physics, medical oncology, oral and maxillofacial surgery (OMFS), oral oncology/oral medicine, and other specialties. Nearly half were women (43%), and the average age and time in practice were 47 years and 15 years (range, 0–38 years; interquartile range, 10–21 years), respectively. Only 1 expert reported no years of clinical practice as his/her position in Radiation Oncology is primarily research-focused on RT toxicity modeling. Experts estimated a 7% annual incidence of ORNJ in their practices and treated a median of 4 cases of ORNJ per year. Participation throughout the study remained high, with 64 (93%), 60 (87%), 56 (81%), and 54 (78%) experts responding to rounds 1 through 4, respectively, and 64% of the Consortium participating in all 4 rounds.

Table 1.

The ORAL consortium expert characteristics

	Overall (N = 69)
Sex
Female	30 (43%)
Male	39 (57%)
Age (y)
Mean (SD)	47 (±9.1)
Missing	7 (10.1%)
Specialty
Medical oncology	3 (4%)
OMFS	12 (17%)
Oral oncology/medicine & dentistry	16 (23%)
Radiation oncology & physics	28 (41%)
Radiology	2 (3%)
Surgical oncology	8 (12%)
Specialty group
Dental	28 (41%)
Other	5 (7%)
Rad Onc	28 (41%)
Surg Onc	8 (12%)
Practice community
Urban (>75,000 population)	63 (91%)
Suburban (10,000–75,000)	2 (3%)
Rural (<10,000)	0 (0%)
Missing	4 (5.8%)
Practice setting
Academic	66 (96%)
Nonacademic	1 (1%)
Private	2 (3%)
Time in practice (y)
Median (IQR)	16 (10–21)
Missing	4 (5.8%)
Approximate HNC patient caseload (per mo)
Mean (SD)	55 (±70)
Missing	6 (8.7%)
Have you evaluated/treated ORN?
Yes	60 (87%)
No	5 (7%)
Missing	4 (5.8%)
Average patients evaluated for ORN annually (N)
Mean (SD)	51 (±170)
Missing	10 (14.5%)
Average patients treated for ORN annually (%)
Mean (SD)	13 (±21)
Missing	9 (13.0%)
Average HNC patients with ORN (%)
Mean (SD)	7.2 (±4.8)
Missing	11 (15.9%)

Abbreviations: HNC = head and neck cancer; OMFS = oral and maxillofacial surgery; ORN = osteoradionecrosis; SD = standard deviation.

Consensus-based definition of ORNJ (high consensus, 86%)

ORNJ is defined pathognomonically by the Consortium as “a condition in which there is a loss of blood flow to bone tissue, which causes the bone to die. Findings of bone death may be clinical (i.e., exposed bone) and/or radiographic (i. e., sclerosis, pathologic fracture). It is caused by exposure to ionizing radiation and may occur at some point in time after radiation and in the absence of active disease (i.e., cancer) in the site of bone death.” In contrast to several existing scales, the consensus definition does not require an explicit time duration of exposed bone nor explicit exclusion of concurrent local inflammation or infection (ie, osteomyelitis). It also incorporates the capacity to formally diagnose ORNJ using imaging criteria; 61/63 (97%) of experts felt that it was very/somewhat important to include “radiographic findings” in a formal definition.

Achieving this consensus-based definition required substantial iterative questioning of MDEs and derivation of expert- and specialty-specific implicit conceptual frameworks through case-based questioning. In round 1, 6 distinct elements were identified; only 3 conceptually included across highly rated definitions/scales: (1) exposed or necrotic bone, (2) RT-induced disorder, and (3) absence of tumor (ie, primary or recurrence). Round 2 capitalized on MDE identification, asking experts to review MedDRA and Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) terminologies/nomenclatures for ORNJ from which 83% (43/52) and 85% (45/53) of panelists agreed that the Consortium’s definition should align with these existing terminologies, respectively. There was also high consensus (80%) that (1) a (then-current) International Classification of Diseases 10th Revision diagnostic code for ORNJ was needed and (2) the National Cancer Institute’s definition for osteonecrosis included a relevant term of “vascular insufficiency” in defining the pathophysiology of bone death.³⁴ A total of 87% (52/60) experts agreed this vascular term or “devascularization” should be included in any definition of necrosis. This resulted in the Consortium’s first consensus statement (CS):

CS 1: The Consortium’s definition for ORNJ will reflect features in existing terminologies, including:

1. Bone disorder

2. Radiation injury and/or caused by ionizing radiation

3. Loss of blow flow or vascular insufficiency AND findings of bone death/necrosis.

The time feature for diagnosing ORNJ conundrum

During round 1, a time feature was identified in 6 of 9 published definitions, and 54 (92%) of experts favored including a time component (ie, duration of exposure) in the Consortium’s definition for ORNJ. However, the minimum duration of exposed bone required for diagnosing ORNJ varies significantly in the literature from 1 to 6 months.^15,35–37 Adding a time feature posed a significant challenge when considering different provider surveillance schedules after RT (time bias). Therefore, a time feature case scenario (Fig. 2) was presented during round 2 to further refine the nuances of time and its relevance as a diagnostic feature for ORNJ. After reviewing 3 existing terminologies, which do not include a diagnostic time feature for ORNJ, 70% (41/59) of experts strongly/somewhat agreed that a diagnosis of ORNJ could be met without a time element, whereas 84% (49/58) agreed that a time element is useful for assessing response to therapy, distinct from classifying severity of disease. This analysis resulted in the designation of an MDE for date_of_assessment (ie, date of clinical or radiographic evaluation post-RT) that should be included in ORNJ-related databases for iterative surveillance assessments.

Fig. 2.

Example case scenario for proposed timing of ORNJ diagnosis based on development of exposed and necrotic bone and timing of scheduled visits (A) and the distribution of expert responses (B). Abbreviation: ORNJ = osteoradionecrosis of the jaw.

CS 2: Although valuable to report and reflect the duration of nonhealing changes observed in irradiated bone, the time or duration of exposed bone is not a mandatory diagnostic feature for ORNJ.

Does exposed bone equal necrotic bone?

This topic was reviewed to identify differences in the conceptualization of exposed bone, which may result from other causative agents, such as trauma following a dental extraction. When asked if all cases of exposed bone automatically equate to necrotic bone, 87% (52/60) disagreed.

CS 3: Not all cases of exposed bone are necessarily considered to be exposed necrotic bone.

Intact mucosa and diagnosing ORNJ

Fifty-six (93%) of experts agreed that ORNJ can be diagnosed in cases with intact mucosa. This is significant from a clinical perspective as most published ORNJ definitions include the presence of clinically exposed bone, whereas the Consortium’s definition allows for a clinical or radiographic manifestation of features associated with ORNJ. Possible examples include intact mucosa with new lytic and/or mixed sclerotic changes seen on panoramic radiograph or cortical deconstruction of previously irradiated bone noted on surveillance CT scan.

CS 4: ORNJ can be diagnosed in a patient treated with RT presenting with intact mucosa (ie, no clinical bone exposure) if there is supporting radiographic evidence of bone death/necrosis.

Identification of elements to classify ORNJ and precursor conditions

Review of existing classification systems

Round 1 provided experts with a comprehensive overview of 15 systems. After presenting each classification system, experts were asked (1) if they had ever used the classification system before, (2) to rate the effectiveness of each system for classifying ORNJ, and (3) to classify 3 clinical scenarios with each system.

Expert utilization and expert-deemed effectiveness/utility ratings of classification systems can be found in Figure E1. The most commonly used system was the Common Terminology Criteria for Adverse Events (CTCAE)²⁶ (n = 41, 70%) followed by Notani et al²⁰ (n = 18, 32%), and Marx¹⁴ (n = 18, 31%). The inability to consistently classify the text-based cases using existing systems was evident during round 1 (Fig. 3). The inability to classify ranged from 16% to 76% (case 1), 12% to 70% (case 2), and 0% to 62% (case 3). Interestingly, the CTCAE resulted in higher completion rates of classifying cases 1 and 2 with only 16% and 12% of the Consortium reporting an inability to classify the case. Four additional systems were requested by experts and reviewed during round 2.^38–40 Of these, medication-related osteonecrosis of the jaw, a classification system for medication-related osteonecrosis, was considered the most effective for diagnosing ORNJ (57%), but its use should be cautioned as it references a distinct causative agent (ie, medications, not RT). None of the reviewed classification methods met the consensus threshold for being highly effective at classifying ORNJ severity, reflecting a critical need for developing and adopting a comprehensive and updated classification system with explicit clinical and imaging features specified to clearly differentiate between disease states. The Watson et al¹ risk-based model for ORNJ (ClinRad), which was recently published and shown to outperform existing systems for classifying ORNJ severity, was presented to the Consortium during round 4 and was evaluated favorably, with 92% (48/52) of experts agreeing with the clinical-radiographic system.

Fig. 3.

Expert classification of 3 descriptive cases of potential ORNJ using 14 classification systems. Individual bars per graph represent available grade or stage classes per system and the total count of responses per class. Magenta bars correspond to the option “unable to classify” across all systems. Abbreviation: ORNJ = osteoradionecrosis of the jaw.

Guidelines regarding application of the CTCAE grading system for ORNJ

The potential recommendation to use the CTCAE system to classify ORNJ did not meet consensus (52%). Overall, 90% of experts agreed that the CTCAE system is still valuable for toxicity reporting but not comprehensive enough for classifying ORNJ, and 95% agreed that it could be used in parallel with another ORNJ classification method.

CS 5: CTCAE is a valuable toxicity grading system that should be used in parallel with, but not replace, an ORNJ classification system inclusive of explicit and severity-specific clinical and radiographic features.

Time feature and ORNJ classification

Similar time feature-related questions were asked for classifying disease severity. During round 2, 68% (40/59) agreed that a classification system for ORNJ should be developed without a mandatory inclusion of a time feature. When rephrased to state that a time feature, which is not a clinical exam or radiographic finding, could be an optional modifier but not a necessary feature for classifying ORNJ, 83% (47/57) of the Consortium agreed (Fig. E2). The Consortium also demonstrated high agreement (85%, 49/58) in considering time features relevant for assessing response to therapy but that therapy response should be separated from a classification system characterizing ORNJ.

CS 6: A time feature is not necessary for classifying ORNJ severity. However, reporting time features may be complementary for monitoring the duration of observed ORNJ alone or in response to any therapy.

Symptoms and ORNJ classification

Specific disease-induced symptoms (ie, pain) or nonexplicit symptom presence has been used for upstaging ORNJ.^15,23,24 When reviewing symptoms, 72% (41/57) of experts agreed that symptoms are ambiguous and, therefore, should not be disease state-defining. However, documenting clear descriptions of specific symptoms, including their onset, temporal profiles, and resolution, if any, is highly encouraged in parallel to explicit clinical/radiographic findings so that a functional classification system can be developed in the future. For symptom surveillance, the Consortium recommends using validated standardized assessment tools for patient-reported outcomes (PROs), such as the MD Anderson Symptom Index Head and Neck Module, the European Organisation for Research and Treatment of Cancer questionnaire series, or the University of Washington tools.^41–44 There is ongoing work to develop a head and neck module for O3, which will include recommendations on specific PRO tools to use and report in a structured format.

CS 7: Symptoms associated with ORNJ should not be used as disease state-defining features. However, longitudinal reporting of the presence or absence of concurrent symptoms using validated PRO question items is strongly encouraged.

Minimum data element extraction and classification

A data element tracker flowsheet was developed inclusive of all clinical, radiographic, therapy, and treatment response elements identified in reviewed classification systems (Fig. E3). Experts were then asked to rate the importance of each feature (not important, somewhat important, and very important). Figure E4 shows the distribution of expert responses, with only 3 data elements being considered somewhat/very important by the entire group: pathologic fracture, extent of bone involvement, and exposed bone. Other clinical (or radiographic) features achieving >70% rating of importance included orocutaneous fistula (98%), (mucosal) ulceration (85%), sinus formation (82%), and sequestra (81%). All remaining clinical/radiographic features underwent a first pass classification by experts into the following categories—Not needed for ORNJ staging, Early/Limited ORNJ, and Advanced ORNJ (Fig. E5). Five features met high consensus (81%−97%), whereas 2 were equivocal (nonexposed bone/imaging findings beyond the AB; sequestra) and 4 were considered unnecessary for ORNJ classification including bone spicules. All these initially classified features, in addition to dental-based clinical scenarios with probe-to-bone (PTB) assessments (Fig. E6), were reclassified in round 3 into tiered groups as follows: group 1 (Not a precursor/Not related to ORNJ; precursor/Stage 0; Early ORNJ), group 2 (Early ORNJ; Intermediate ORNJ), and group 3 (Intermediate ORNJ; Advanced ORNJ). Items or cases not meeting the consensus threshold during round 3 were repeated in round 4. Additionally, 10 imaged-based cases (ie, clinical photographs and/or 2D/3D radiographic images) were reviewed during the last 2 surveys to reinforce classification of MDE combinations (Fig. 4).

Fig. 4.

Case-based classifications. Panel (A) demonstrates expert distribution of ORNJ stage assignments (ie, Not precursor/Not ORNJ, Precursor/stage 0, Early ORNJ, Intermediate ORNJ, and Advanced ORNJ) for 10 cases evaluated, whereas Panel (B) illustrates expert-based opinions and distributions of likely extent of bone involvement identified for each case. Panel (C) includes boxplots of reported level of confidence in classification of each case stratified by grouped specialty, and Panel (D) presents and summarizes findings for case 8 of the study. Abbreviation: ORNJ = osteoradionecrosis of the jaw.

Not a precursor/not related to ORNJ

• PTB test negative with periosteal reaction within AB seen on imaging: A comparable division in classifying this feature was noted during round 3 (47% Not a precursor) and round 4 (50% Not a precursor). RadOncs were the drivers of classifying this feature as a precursor stage compared with surgeons and dentists who favored it as being unrelated. The Consortium recommends this feature combination to be considered as unrelated to ORNJ to differentiate it from consensus-approved precursor features. Of more importance, 2 MDEs are valuable in this description and include clinical:PTB_test_result (positive or negative) and imaging:morphology (ie, periosteal reaction, sclerosis, and lysis). The PTB test is a commonly used dental probing examination that provides diagnostic information on periodontal health^45,46; therefore, the Consortium recommends standardized documentation of this procedure whenever performed. Moreover, these MDEs are required for other severity classification feature combinations.

Precursor/stage 0

• Intact mucosa (no clinical bone exposure) with any imaging findings within AB: This text-only description met high consensus during round 4, with 85% (45/53) of experts favoring a precursor assignment over being unrelated to ORNJ. Final round classification of image-based case 3 further supported this designation with consensus being met at 67% (33/49). Identified MDEs are clinical:mucosal_status (ie, intact and ulcerated) and the dichotomous imaging:vertical_ab_abnormality (ie, within/above AB and beyond/below AB).

• Minor bone spicules (MBS): This feature posed a significant challenge for classification across all rounds using text-only or image-based (case 5) descriptions. When asked to classify MBS (text-only) during round 3, there was a slight preference for precursor stage (45%, favored by surgeons) over being unrelated to ORNJ (43%, favored by dentists; RadOncs equivocal) or representing Early ORNJ. In round 4 (when Early ORNJ was removed as an option), the precursor status was again favored at 55%. However, when presented with clinical photography indicative of MBS, there was consistent upstaging of MBS to Early ORNJ in both rounds (49% and 50% Early ORNJ) with <30% assignment to precursor or unrelated stages. To mitigate classification ambiguity when clinically exposed bone is detected, serial clinical photographs and quantitative measurements of exposed bone (MDE clinical:exposed_bone_length_in_mm) are strongly recommended. An additional MDE for minor_bone_spicules is proposed to explicitly report MBS, which should be accompanied by a quantitative measurement.

• PTB test negative with any imaging findings (EXCEPT periosteal reaction) within AB seen on imaging: The Consortium achieved consensus during the final round for this feature combination (76%, 41/54). Associated MDEs are clinical:PTB_test_result (negative), imaging: morphology, and imaging:vertical_ab_abnormality (within AB).

• Vascular damage in bone seen on magnetic resonance imaging (MRI) without exposed bone and without other imaging findings (ie, CT shows no bony abnormalities): This scenario is considered a precursor stage as it met consensus during the final round as a text-only description (60%, 32/53) and in case 7, which depicted minor mucosal ulceration (ie, no bone exposure) with reported dynamic contrast-enhanced (DCE) MRI changes (63% agree, 31/49).

• Vascular damage in bone seen on MRI without exposed bone and with other imaging findings limited to AB (ie, X-ray shows sclerosis limited to AB): The Consortium demonstrated consensus convergence with classifying this combination as a precursor stage (78%) over Early ORNJ. This is also in alignment with the classification of cases of intact mucosa with abnormal imaging findings restricted within the alveolar bone.

Early stage ORNJ

• PTB test positive with imaging findings within AB seen on imaging: Compared with intact mucosa classification, a positive PTB test was considered to be an upstaging feature by the Consortium when combined with localized periosteal reactions (91%, 49/54) or other morphological radiographic changes (89%, 48/54) limited to the alveolar bone. This classification was supported by case 4, which showed a photograph of a positive PTB test and orthopantomogram (OPG) changes within the AB (86%, 43/50, staged it as Early ORNJ). Associated MDEs are clinical:PTB_test_result (positive), imaging:morphology, and imaging:vertical_ab_abnormality (within AB).

• Exposed bone with any imaging findings within AB: During round 3, this text-only presentation was unanimously classified as an Early ORNJ when limited to options ranging within group 1. When asked to classify case 6, a photograph with clinical bone exposure spanning the width of 1 molar and an OPG image showing periodontal ligament space widening, experts converged to consensus in assigning this feature combination as Early ORNJ (round 3, 55%; round 4, 74%). Although these examples are in alignment with the ClinRad classification model for ORNJ,¹ it became evident that the extent of visualized bone exposure influences stage designation. For example, when presented with another case with similar imaging findings but extended clinical bone exposure spanning the length of 2 molars (case 8), there was reduced consensus at classifying it as Early ORNJ (64%) over Intermediate ORNJ (36%). Case 12 showing clinical bone exposure beyond 2 teeth with text-only “reported imaging findings within AB” was presented once at the end of round 4 after review of the ClinRad model. The clinical photograph influenced the upstaging of this case of extensive bone exposure to Intermediate ORNJ (76%, 39/51). To reduce misclassification risks for the same or nearly identical patients, the Consortium again strongly recommends the use of a quantitative MDE (ie, clinical:exposed_bone_length_in_mm) for facilitated data harmonization across disciplines and evolving classification systems. Using pre-defined cutoffs (ie, </> 2 cm) is discouraged as they limit future analysis of stage subclassification or reclassification as new knowledge on ORNJ emerges. A separate quantitative measurement (clinical:exposed_bone_width_in_mm) can be considered to capture 2-dimensional data on extent of bony exposure as needed for the area classification in the modified Shaw system.²¹ Additional MDEs for this presentation include clinical:mucosal_status (absent), imaging:morphology, and imaging:vertical_ab_abnormality (beyond AB).

Intermediate stage ORNJ

• Exposed bone with any imaging findings beyond AB: This feature combination was challenging to classify. When prompted with a text description in round 3, experts were split (44% intermediate, 40% advanced). A significant association between specialty group and classification was observed with most dental specialists considering it intermediate stage (69%), whereas others (72% RadOnc, 57% surgery) favored upstaging the feature combination (P = .04). The repeated question in round 4, limited to these 2 stage options, resulted in convergence toward classification as intermediate stage (63%, 34/54, consensus threshold met). Case 9, a clinical report of exposed bone with OPG images showing full thickness sclerosis of the right mandible, and case 10, a photograph of clinical bone exposure extending less than a molar width with axial CT image showing cortical bone changes, were presented as variations of this combination, with convergence toward classifying both also as Intermediate ORNJ during the final round (case 9, 58%; case 10, 86%). However, the upstaging effect of visualizing more extensive clinical bone exposure was reproducible using case 13, which was staged as Advanced ORNJ by 71% (36/51) of experts over Intermediate ORNJ (29%). A more accurate summarization of such scenarios is through the use of standardized MDEs such as clinical:exposed_bone_length_in_mm, clinical:mucosal_status (absent), imaging:morphology, and imaging:vertical_ab_abnormality (beyond AB).

Advanced stage ORNJ

The Consortium exhibited high agreement in classifying pathologic fracture (96%, 49/51), orocutaneous fistula (92%, 47/51), and oroantral or oronasal fistula (86%, 43/51) as advanced features of ORNJ. These results are in alignment with advanced features reported in the ClinRad model and other classification systems, and their presence on clinical examination should be explicitly reported in a standardized fashion such as a disorder_present MDE with unique identifiers for each disorder.

Figure 4 shows the heterogeneity in disease severity classification and extent of bone involvement classification for 10 image-based case studies. A final list of Consortium-approved MDEs for classifying ORNJ and precursor states is shown in Table 2 along with recommended coding standards for building artificial intelligence/machine learning ready data sets.

Table 2.

Minimum data element list

Minimum data element	Example value names	Example SCTID	SCTID class
Time
date_of_assessment
Clinical
minor_bone_spicules	Present	52101004	Qualifier value
	Absent	2667000	Qualifier value
exposed_bone_length_in_mm
mucosal_status	Present	52101004	Qualifier value
	Absent	2667000	Qualifier value
PTB_test_result	Positive	404684003	Qualifier value
	Negative	260385009	Qualifier value
disorder_present	Ulceration of oral mucosa	26284000	Disorder
	Orocutaneous fistula	472978005	Disorder
	Oroantral fistula	109675004	Disorder
	Oronasal fistula	370485008	Disorder
	Pathologic fracture	268029009	Disorder
Radiographic
imaging type (DICOM standard)
morphology	Bony sclerosis	37748009	Morphologic abnormality
	Osteolysis	30425001	Morphologic abnormality
	Bony erosion	788917000	Finding
	Thinning (ie, cortical bone)	29143009	Finding
	Pathologic fracture	22640007	Morphologic abnormality
vertical_ab_abnormality	Above	352730000	Qualifier value
	Below	351726001	Qualifier value

Abbreviations: ab = alveolar bone; DICOM = Digital Imaging and Communications in Medicine; SCTID = Systemized Nomenclature of Medicine-Clinical Terms Identifier.

CS 8: All cases with clinical and/or radiographic evidence of a pathologic fracture or fistula formation (ie, orocutaneous, oroantral, and oronasal) involving previously irradiated bone should be reported as Advanced ORNJ. These individual disorders are considered classification-defining MDEs, and each should be reported separately.

Specialty-specific knowledge siloes and interrater reliability

Siloes of knowledge may occur through different knowledge acquisition per specialty-based training programs and/or practice patterns. Within the ORAL Consortium, a significant difference was found in the utilization of OPGs with experts from Oral Medicine/Oncology and OMFS (ie, Oral/Dental) using them twice as often as oncologists (84% vs 43%, P = .008; Table E1). CT scans were commonly used by dental and oncology groups (75% vs 90%; P = .166), whereas MRI was used more frequently by oncologists than oral specialists (38% vs 21%; P = .334). When questioned on the effectiveness of each modality for diagnosing ORNJ (Table E2), CT scans were rated the most effective by >86% of the Consortium, whereas approximately one-third of both specialty-condensed groups were neutral on the effectiveness of OPGs for ORNJ surveillance. Although the least adopted modality, MRI had comparable effectiveness ratings to OPG, with experts favoring the following sequences: T1-weighted with contrast (36%), T2-weighted (29%), and DCE (22%).

The potential impact of familiarity with various imaging modalities on level of confidence in diagnosing imaged-based cases is seen in Figure 4. Out of 9 scenarios presented in round 3 with a radiographic image included (excluding case 5-clinical MBS), dental specialists had the highest level of confidence (LOC) for all OPG-based cases but exhibited lowered LOC interquartile ranges comparable with RadOnc and surgery for case 7 (MRI and CT) and case 10 (CT). Cases with limited OPG-based abnormalities within the alveolar bone (ie, periodontal ligament space widening; cases 3, 4, 6, and 8) evoked lower LOC among oncologists in diagnostic capabilities, whereas radical changes seen on OPG such as a pathologic fracture (case 2) were consistently classified the same for stage and extent of bone involvement by all specialists with a reported high LOC.

During round 3, the interrater reliability (IRR) for classifying image-based cases (Table E3) varied per specialty with slight agreement among RadOnc (IRR 0.13, n = 22) and fair or better agreement between surgeons (IRR 0.29, n = 6) and dental specialists (IRR 0.34, n = 14). Based on the likelihood of categorizing the same cases per specific category, all experts showed the highest specialty group-level agreement for categorizing cases of Advanced ORNJ (RadOnc IRR 0.43; Surgery IRR 0.57; and Oral/Dental IRR 0.67), whereas the least agreement, if any, was around what cases should be considered unrelated to ORNJ (RadOnc IRR 0.1; Surgery IRR −0.07; and Oral/Dental IRR 0.15), a precursor stage (Oral/Dental IRR 0.27), or an intermediate stage (RadOnc IRR 0.17; Surgery IRR 0.17). After providing group feedback from round 3, an introduction to the ClinRad risk-based model and educational imaging resources for how to interpret OPG images, the overall specialty-level IRR during the last round improved for all groups (Oral/Dental IRR 0.38, n = 18; RadOnc IRR 0.39, n = 22; and Surgery IRR 0.58, n = 5). Moderate to substantial agreement was achieved for classifying cases as precursor/stage 0, Early ORNJ, and Intermediate ORNJ but at a cost of decreasing agreement on nonfistula and nonpathologic fracture cases of Advanced ORNJ. With regard to the image-focused educational resources, 84% of experts found the resources helpful, and 92% were interested in having an updated, comprehensive, multidimensional atlas as a support tool for case classification.

Recommendation for multidisciplinary MDE adoption

High yield recommendations and supporting rationale for diagnosing and classifying ORNJ severity have been summarized in Table E4. Given substantial variation in the classification of cases among experts, particularly “threshold” cases, which may be upstaged based on nonexplicit clinical and/or radiographic imaging features (ie, quantitative measurement of clinical bone exposure), the following CSs were also developed.

CS 9: The Consortium strongly recommends the adoption of ORNJ-focused MDEs in multidisciplinary clinical practice and clinical trial design to reduce misclassification risks and to facilitate “stage migration” across classification models.

CS 10: Inclusion of serial photographs in a patient’s medical record during post-RT surveillance, especially once changes in the mucosa (ie, ulceration) or bone (ie, progressive bone exposure) are detected, is strongly recommended. Caliper or ruler-based measurements of clinical bone exposure should also be recorded for at least the longest dimension in millimeters under the MDE, clinical:exposed_bone_length_in_mm.

Discussion

In this expert-based iterative Delphi method study, we have generated an international multidisciplinary-approved definition for ORNJ along with 10 CSs and 9 distinct MDEs that should be serially documented during dental and oncology post-RT appointments for cancer survivors undergoing ORNJ surveillance. These MDEs characterize static (ie, date of assessment) and dynamic (ie, progressive radiographic changes) features that can be used for meaningful classification of ORNJ and precursor states.

During this Delphi modeling, several authors were simultaneously involved in the development and publication of the ClinRad model¹ and the International Society of Oral Oncology-Multinational Association for Supportive Care in Cancer and the American Society of Clinical Oncology (ISOO-MASCC-ASCO) joint guideline for the prevention and management of ORNJ.⁹ The Consortium’s definition for ORNJ is in alignment with the new ISOO-MASCC-ASCO guideline, which operationally characterizes ORNJ as a “radiographic lytic or mixed sclerotic lesion of bone and/or visibly exposed bone and/or bone probed through a periodontal pocket or fistula occurring within an anatomical site previously exposed to a therapeutic dose of head and neck radiation therapy.” Moreover, both definitions demonstrate a significant departure from using a time-limiting feature for diagnosing ORNJ, thereby addressing current issues with time bias.

The ClinRad system, which outperformed classification methods such as the Notani, LENT-SOMA, and Store systems, incorporates observable clinical (ie, PTB tests) and radiographic features and uses the alveolar bone as a distinguishable threshold for disease.¹ This new classification model of ORNJ has been adopted by the new ISOO-MASCC-ASCO ORNJ guidelines.⁹ The ORAL Consortium’s favorable review of the ClinRad system before the publication of the guidelines further supports its utilization as it incorporates most of the MDEs identified in this study. However, we also demonstrate several classification challenges that should be addressed by providers before implementing the ClinRad system. There still exists a conceptualization discordance for MBS, namely whether or not it is related to ORNJ.¹ The Consortium favors the classification of MBS as a precursor event to ORNJ that has a higher likelihood of resolving over time than other MDE features or feature combinations. More importantly, utilization of a quantitative measurement such as clinical:exposed_bone_length_in_mm and inclusion of clinical photographs in the patient’s medical and dental records can facilitate data harmonization among interprofessional health care providers at different centers treating the same patient. Formal standards for image acquisition and guidance regarding dental photography collection and archiving already exist and should be leveraged by providers for standardized reporting of ORNJ.⁴⁷

The concept of specialty-specific knowledge siloes is introduced in this study and is important to recognize because the diagnosis and management of ORNJ is often a multidisciplinary task. Overall, providers tend to exhibit high agreement in perceiving severe presentations of ORNJ; however, subtle variations in physical examination or radiographic feature interpretations may cascade into differing classifications of disease for the same patient examined by different specialists. One approach to mitigate this discrepancy is by standardizing the use of MDEs across specialties and generating interdisciplinary, multimodality (ie, OPG, CT, and MRI) image-focused educational materials. Biomarkers for ORNJ and its precursor states are also being investigated, including DCE-MRI parameters (K_trans and V_e,) for assessing risk, diagnosis, and progression or treatment response of ORNJ.⁴⁸ The Consortium supports the consideration of DCE-MRI changes indicative of vascular insufficiency in previously irradiated jaw bones as a precursor event to ORNJ. Subsequent clinical guidelines will be necessary for outlining optimal MRI-based ORNJ surveillance regimens and specification of MRI-specific MDEs.

Although this interdisciplinary Delphi study has several strengths such as the Consortium size (n = 69) and sustained level of engagement, there is underrepresentation of specialties (eg radiology and OMFS) that may provide additional expertise on ORNJ characterization, including identifying disease state-defining radiographic features across imaging modalities. Fleiss kappa statistics can provide substantial insight on how reliability experts classify cases (ie, interrater reliability), but it does not provide information on whether those classifications represent the true disease state (ie, validity).⁴⁹ Although several consensus-defining methods exist,^50,51 we chose a simple agreement threshold as it is commonly used, easy to interpret, and reinforced through iterative requestioning to produce metrics of reliability. Finally, our consensus-based list of MDEs is not exhaustive in representing all multidisciplinary orodental knowledge (ie, surgery-specific MDEs are lacking). The benefit of MDEs, however, is that they are atomic (ie, indivisible) in nature, and an MDE or clinical data element repository can and should be routinely reviewed and expanded as new knowledge or data need emerge. In the interim, we recommend the application of our consensus-based “minimum” data elements and encourage future studies and/or MDE requests inclusive of all specialties involved in the management of orodental diseases. Moreover, the Operational Ontology for Oncology provides a structured format for reporting general cancer therapies, and future objectives include building an extensible ORNJ ontology that is inter-operable with O3 for wide dissemination.

Implementation of all MDEs in clinical practice can be challenging depending on resource availability (ie, subspecialties, advanced imaging). However, most of the consensus-based MDEs listed herein could be captured during routine surveillance visits in any specialty clinic (ie, clinical examination findings). Although out of the scope of this study, approaches to successful adoption of MDEs is encouraged through the development of an implementation program using well-established integrated frameworks such as RE-AIM.⁵² The RE-AIM framework consists of 5 dimensions including Reach (ie, the target population such as OMFS surgeons at a specific practice), Effectiveness (ie, efficacy of mode of MDE collection), Adoption (ie, by target staff), Implementation (ie, costs and adaptions), and Maintenance/sustainment of the intervention over time. Several planning tools are available on the RE-AIM website for providers to develop comprehensive implementation programs that are customizable to varied resource environments and are inclusive of key stakeholders’ input along with formative evaluations for prompt midprogram adaptations to institutionalize desired practices.

Conclusion

The Consortium’s definition of ORNJ and associated MDEs should be adopted as standards for reporting by head and neck surgery, oncology, radiology, and dental providers in clinical practice, research, and clinical studies. Collectively, these atomic clinical and/or radiographic elements provide a standardized foundation for defining and classifying ORNJ at a granular level. For example, standardized documentation of extent of abnormal bone findings on preoperative imaging across institutions can be integrated into predictive models of outcomes with evolving surgical techniques for ORNJ. MDEs, therefore, enable scalable health information exchange and artificial intelligence/machine learning data readiness for rigorous modeling of the disease and its precursor states. ORNJ-focused MDEs are also synergistic with recently published guidelines and newer risk-based ORNJ models that recognize the importance of combining clinical and radiographic features for ORNJ characterization. Finally, additional efforts are underway to formalize an ORNJ ontology, develop radiology standards and automated imaging feature identification and reporting, and formulate and disseminate interdisciplinary educational resources to mitigate barriers to accurate ORNJ classification.

Supplementary Material

Supplementary material associated with this article can be found in the online version at doi:10.1016/j.ijrobp.2024.12.017.

Data Sharing Statement:

In accordance with the Final NIH Policy for Data Management and Sharing NOT-OD-21–013, data that support the findings of this study are openly available in an NIH-supported generalist scientific data repository (figshare) at http://doi.org/10.6084/m9.figshare.25546723 no later than the time of an associated publication; although public data are embargoed pending peer review, referees may access these data via private link at https://figshare.com/s/625834718e80c583198d during the peer review process.