Abstract
Objective:
Medication-related osteonecrosis of the jaw (MRONJ) is a rare complication of antiresorptive therapy for osteoporosis, with risk potentially influenced by “drug holidays” around dentoalveolar procedures. MRONJ risk is higher in rheumatic diseases. Epidemiologic studies are limited by poor administrative code performance for MRONJ identification and lack of structured pharmacy data capturing drug holidays. We developed and validated a natural language processing (NLP) algorithm to classify MRONJ and detect drug holidays from free-text clinical notes in electronic health records (EHR).
Methods:
Using U.S. Veterans Health Administration EHR, we identified adults ≥50 years old with ≥1 filled prescription for a bisphosphonate or denosumab for fracture prevention (10/1/1999–12/31/2022). An NLP codebook defined 11 targets with 16 attributes. Two annotators independently labeled notes to create a reference set (n=870; 200 held out for validation) and an independent validation set (n=100) for performance assessment (precision, recall, F-measure). A decision tree-based machine-learning model used NLP-extracted features to classify MRONJ. Drug holiday was an NLP target directly.
Results:
The reference set included 6,421 annotations informing a curated vocabulary of >1,350 terms. For MRONJ classification excluding unclassifiable outputs across all validation notes, precision was 1.00 (95% confidence interval [CI] 0.86–1.00), recall 0.96 (0.80–1.00), and F-Measure 0.95 (0.88–1.00). For drug holiday detection in the held-out validation set, precision was 0.90 (0.83–0.95), recall 0.98 (0.93–1.00), and F-Measure 0.94.
Conclusion:
We developed an NLP algorithm that is sufficiently accurate in classifying MRONJ and detecting drug holidays to be a promising tool for future studies, including in persons with rheumatic diseases.
Keywords: osteoporosis, medication-related osteonecrosis of the jaw, bisphosphonates, natural language processing, artificial intelligence
The occurrence of nonhealing exposed bone in the maxillofacial region as an adverse effect of a medication, now known as medication-related osteonecrosis of the jaw (MRONJ), was first described by oral and maxillofacial surgeons in patients treated with intravenous bisphosphonates (BPs).1,2 More recently, MRONJ is recognized as an uncommon side effect of BPs, other antiresorptive agents, and antiangiogenic agents.3 MRONJ may be of particular concern for patients with inflammatory arthritis.4 Persons with rheumatoid arthritis (RA) and ankylosing spondylitis have an increased risk for osteoporosis and osteoporotic fractures compared with healthy controls.5 BP therapy reduces vertebral fracture incidence in persons with rheumatic diseases6 and denosumab increases bone mineral density at the lumbar spine and hip in persons with RA.7 Moreover, some studies suggest that RA may be an independent risk factor for the development of MRONJ.8
To date, studies identifying MRONJ events have used administrative claims-based analyses or manual electronic health record (EHR) review.9–12 Because MRONJ is rare, large-scale observational studies are required for identification of MRONJ, making manual EHR review often unfeasible. Administrative claims are also limited because International Classification of Diseases, ninth (ICD-9) and tenth revision (ICD-10) codes to identify MRONJ occurences has low positive predictive value when compared to manual EHR review.9–12 Studies using a broad number of ICD codes for MRONJ, which included both specific and nonspecific codes in order to avoid missing cases, have reported positive predictive values (PPVs) ranging from 5.8–7.1% in the United States9,10 with only somewhat higher PPVs in Europe.11,12 Existing studies have also included limited populations of men and women.9,12,13
Data on sensitivity are even more limited. Among Danish patients, the sensitivity of ICD-10 codes recorded within 180 days of a dentoalveolar procedure for identifying MRONJ was 77.2% [95% confidence interval (CI): 68.4%−84.6%].14
An intentional “drug holiday” is an interruption in osteoporosis therapy intended to minimize rare adverse events such as MRONJ.15 Dentoalveolar procedures are a known risk factor for developing MRONJ.16 When drug holidays are given around a dentoalveolar procedure, they are often short in duration17 and pharmacy dispensing data is insensitive to detecting such short medication gaps. Although epidemiologic studies have evaluated BP interruption,18 to our knowledge none have distinguished intentional clinician-recommended drug holidays from medication non-adherence using pharmacy dispensing data. Distinguishing clinician-recommended drug holidays around dentoalveolar procedures using EHR data is essential to studying current practice patterns and their effectiveness in mitigating MRONJ risk.
In this manuscript, we describe the design, construction, and validation of a natural language processing (NLP) algorithm leveraging supervised decision-tree induction to categorize MRONJ, possible MRONJ and not MRONJ diagnoses and identifying intentional antiresorptive drug holidays around dentoalveolar procedures from free-text data in the EHR clincial notes. This NLP algorithm was developed in the largest integrated health system within the United States, the Veterans Health Administration (VHA). Our goal was to identify MRONJ and antiresorptive drug holidays in persons prescribed BPs or denosumab specifically for fracture prevention.
Materials and Methods
Data and study population
The NLP system was developed on a set of Text Integration Utilities (TIU) within the U.S. national Veterans Health Administration (VHA) Corporate Data Warehouse (CDW). A TIU is the system used within the VHA CDW to store the text from VHA EHR clinical notes and documentation, hereafter referred to as “clinical notes” for clarity. The VHA CDW was queried for all persons aged 50 years and older who received at least one filled prescription for an oral (alendronate, risedronate, ibandronate) or intravenous BP (zoledronic acid, ibandronate) or subcutaneous denosumab between October 1, 1999 to December 31, 2022. Exclusion criteria are listed in Supplementary Table 1.
Figure 1 illustrates the overview of the general architecture of the NLP-based system for ascertaining MRONJ and detecting drug holidays. This study was deemed exempt by the VHA Central Institutional Review Board and conducted in accordance with the Declaration of Helsinki and its later amendments. Data underlying primary scientific publications from this study may be available to the public and other VHA and non-VHA researchers with an appropriate data use agreement (DUA) under certain conditions and consistent with policies which prioritize protecting Veteran’s privacy and confidentiality to the fullest extent possible.
Figure 1.

Overview of the natural language process-based approach for determining MRONJ status from clinic notes in men and women prescribed bisphosphonates or denosumab for fracture prevention in the U.S. Veterans Health Administration
BP: bisphosphonate; MBD: metabolic bone disease; NLP: natural language processing; MRONJ: medication-related osteonecrosis of the jaw
1Metabolic bone disease includes Paget’s disease of bone, osteogenesis imperfecta, hypophosphatasia or phosphate wasting syndromes, or osteopetrosis
Clinical notes selection
Given the low prevalence of MRONJ among individuals prescribed BPs or denosumab for fracture prevention,19,20 we used surrogate-guided sampling to efficiently select clinical notes for human annotation.21,22 We restricted sampling to Veterans with ≥1 visit between their first filled BP or denosumab prescription and 12/31/2022 with an ICD-9/10 code for a potential MRONJ event in any position. We considered a deliberately inclusive code list to maximize sensitivity for potential MRONJ events (or an anatomic orofacial variant that may warrant further evaluation for MRONJ) (Supplementary Table 2).9–13 These Veterans comprised the study development cohort, and all notes associated with qualifying visits comprised the NLP total corpus. Notes for the human-annotated reference set were selected iteratively and non-randomly from the NLP total corpus to capture sufficient linguistic diversity and outcome variation to reliably identify and refine NLP targets that distinguish the outcomes of interest: MRONJ diagnosis and drug holiday.
Constructing the reference set
Two annotators with expertise in osteoporosis management (BL, REE) independently annotated the same clinical notes following an established codebook and procedure. The codebook was created and updated iteratively by the annotators. Any disagreement was resolved through iterative discussions among annotators and consensus achieved with expert input from individuals with expertise in natural language processing (DF), osteoporosis (LDC), and dental surgeons with expertise in diagnosing and managing MRONJ (TBD, PJG). The final codebook contained 11 NLP targets, three of which had attributes collectively totaling 16 unique attributes. These targets and attributes were selected for their clinical relevance for the diagnosis of MRONJ or identification of drug holidays.23
Annotators manually identified and labeled exemplar phrases for each NLP target and attribute from free-text portions of clinical notes by thoroughly examining the author style, format, and structure of the texts in order to develop the final reference set. A drug holiday was identified by annotators with a single continuous line of text in all encountered contexts (i.e. “continue Fosamax drug holiday for 3 more months” alone was sufficient documentation to feel confident a drug holiday was intended). Therefore, Drug Holiday was a single NLP target in and of itself. In contrast, classification of MRONJ by annotators often required input from multiple non-contiguous free-text sites within the clinical note (i.e. a physical exam documenting “exposed bone” together with an assessment that the patient has “stage 2 disease”). Therefore, ten different NLP targets were developed, each of which were named entities, and then combined to inform the MRONJ diagnosis classification. The NLP targets, their attributes, descriptions, and exemplars are shown in Table 1.
Table 1.
Textual targets and attributes and their descriptions, with examples
| Target | Attributes | Descriptions | Exemplars |
|---|---|---|---|
| Compatible imaging | - | Indication of imaging findings supporting an MRONJ diagnosis on X-ray or other advanced imaging | “sclerosis” “periosteal thickening” “mixed radio-opaque radio-lucent lesion” |
| Compatible sequela | - | Indication of destructive features of possible MRONJ lesion or sequela compatible with aggressive MRONJ | “pathologic fracture” “extra oral fistula” “OA communication” “communication with maxillary sinus” |
| Drug holiday | - | Indication that the Veteran had a drug holiday from an osteoporosis drug (bisphosphonate or denosumab) either before or after a dental procedure (or planned dental procedure) | “continue Fosamax drug holiday for 3 more months” “on Fosamax however has been on a 3 month drug holiday” |
| Exposed bone | - | Indication exposed bone of jaw was present | “exposed bone” “nonhealing bone” “fistula” “sinus tract” “devitalized bone” “mucosal breakdown” |
| Inflammation | - | Indication that the individual had evidence of inflammation at the site of their MRONJ lesion | “purulent discharge” “erythema” “swelling” “abscess” |
| Jaw osteonecrosis | Definite MRONJ | Indicator of whether that instance of jaw osteonecrosis represented a case of MRONJ in that documentation, with qualification of certainty (definite or possible) based on context (Supplementary Figure 1) | “osteonecrosis of the jaw” “ONJ” “osteonecrosis of the left mandible” “ON of jaw” “medication induced osteomyelitis of the jaws” |
| Possible MRONJ | |||
| Not MRONJ | |||
| Jaw pain | - | Indication that individual was experiencing the symptom of jaw pain | “pain in the UL jaw” “pain at site of MRONJ” |
| Jaw radiation | - | Indication that the individual received jaw radiation in the past | “osteoradionecrosis” “ORN” |
| MRONJ stage | 0 | Stage 0 explicitly stated in association with MRONJ | “Stage 0” (MRONJ) |
| 1 | Stage 1 explicitly stated in association with MRONJ | “Stage 1” (MRONJ) | |
| 2 | Stage 2 explicitly stated in association with MRONJ | “Stage 2” (MRONJ) | |
| 3 | Stage 3 explicitly stated in association with MRONJ | “Stage 3” (MRONJ) | |
| MRONJ therapy | Anabolic | PTH analogue used for treatment of MRONJ (i.e. abaloparatide/Forteo or teriparatide/Tymlos) | “forteo” “teriparatide” |
| Antibiotic rinse | Antibiotic rinse used for treatment of MRONJ | “chlorhexidine” “CHX” “Peridex” “hydrogen peroxide” |
|
| Debridement | Indication that a curettage or debridement was performed for treatment of MRONJ | “ronguer was used to remove exposed ligual plate” “curette was used to remove necrotic bone” “debride down to viable bleeding bone” |
|
| Hyperbaric oxygen | Indication that hyperbaric oxygen was recommended and/or performed for treatment of MRONJ | “HBO” “hyperbaric treatments” |
|
| Jaw Reconstruction | Indication that jaw reconstruction was performed for treatment of MRONJ | “plate reconstruction” “recon plate application” “reconstructive surgery” |
|
| Jaw Resection | Indication that jaw resection was performed for treatment of MRONJ | “mandibulectomy” “maxillary resection” “excised L mandibular bone” |
|
| PENTO | Indication that PENTO protocol, or any of its components individually (pentoxifylline, tocopherol), were prescribed for treatment of MRONJ | “PENTO protocol” “vitamin E” “pentoxifylline” “PENTOCLO” |
|
| Sequestrectomy | Indication that a sequestrectomy was performed for treatment of MRONJ | “removal of exposed bony sequestrum” “sequestra removal” |
|
| Systemic antibiotic | Indication that oral or intravenous systemic antibiotics were prescribed and/or administered for treatment of MRONJ | “abx” “amoxicillin” “clindamycin” |
|
| Sequestrum | - | Indication a sequestrum was present | “sequestrum” “sequestra” |
MRONJ: Medication-related osteonecrosis of the jaw; PENTO: Pentoxifylline and tocopherol
Contextual modifiers were further included as targets to enhance the validity of the NLP algorithm. Examples of contextual modifiers included historical references such as “history of,” hypothetical references such as “may occur” or “counseled on the risks of developing,” and negation such as “patient denies” or “no signs of.” The contextual status was determined using a rule-based context algorithm originally developed by the senior author (DF) within the VHA. The algorithm maintains curated vocabularies of contextual modifier phrases (e.g. negator) classified according to their scope and directionality (e.g., negate before and/or negate after). Following concept extraction, each sentence is represented as an ordered sequence of concepts and contextual modifier terms (e.g., negate before, concept, concept, negative after). A series of predefined rules is then applied to the sequence to determine the contextual status of each extracted concept based on the type, position, and scope of nearby modifier phrases. The contextual vocabularies are iteratively expanded during annotation to incorporate newly identified expressions encountered in VHA clinical documentation. The overall approach is conceptually similar to the pyConText algorithm,24 but incorporates additional sequence-based rules and vocabulary refinements developed for the VHA Computerized Patient Record System (CPRS) EHR.
Annotation for MRONJ classification
Supplementary Figure 1 shows the a priori conceptual algorithm used by annotators to classify each clinical note as MRONJ, possible MRONJ, and not MRONJ based on NLP targets and attributes. This algorithm was constructed with input from orofacial specialists with MRONJ expertise (TBD, PJG) and aligned with the American Association of Oral and Maxillofacial Surgeons’ (AAOMS) Position Paper definition of MRONJ,23,25 supporting content validity. Notably, mention of jaw osteonecrosis or a related term was required for classification as MRONJ or possible MRONJ. In-person documentation from an orofacial specialist (dentistry, oral maxillofacial surgery, or otolaryngology) was required for an MRONJ classification, while documentation from a non-orofacial clinician could support at most possible MRONJ. Relevant VHA primary outpatient stop codes were DENTAL (180), ORAL SURGERY (452), and OTOLARYNGOLOGY/ENT (403). Because the AAOMS case definition excludes patients with prior jaw radiation, notes with a positive jaw radiation target were classified as not MRONJ regardless of whether other MRONJ criteria were met.
Annotation for drug holidays
Annotators identified drug holidays by recognizing co-occurring text elements indicating temporary discontinuation of an antiresorptive medication in relation to a dentoalveolar procedure. These elements included (1) a BP or denosumab medication (2) a phrase indicating that the medication was being withheld, (3) a time reference, and (4) an associated event (e.g., i.e. a reference to the dentoalveolar procedure). For the Drug Holiday NLP feature, the entire contiguous phrase describing the four components above comprised the annotated NLP target (Table 1).
This human-annotated reference set was used to train and evaluate the NLP algorithm.
Developing the NLP algorithm
The annotated reference set was divided into 2 groups. Most notes in the reference set were used in developing the NLP rules for the extraction routines (the training set) and 200 notes were held out for validation (the pre-specified validation set). Most of the steps in developing the NLP algorithm were shared for the outcomes of MRONJ classification and detection of drug holidays, with divergence only at the end of the process.
The first step in developing the NLP algorithm from the human-annotated reference set was to take all the annotations and use them to develop a curated vocabulary. The curation process normalized the terms, including steps such as extracting term class pairs (original span “2×3 mm area of necrotic exposed bone,” becomes “exposed bone”) and cleaning extraneous textual artifacts such as punctuation and special characters. This custom vocabulary was then reviewed by subject matter experts (LDC, TBD, PJG) for validity. Throughout this process, standardized sources were consulted whenever possible to standardize terms, including the VHA formulary of medications and the National Library of Medicine’s Unified Medical Language System.26 The custom vocabulary for each target and attribute was refined until it reached a priori target thresholds of >80% on the F-Measure. The contextual annotations were combined with an existing vocabulary of the same purpose.
Figure 2 illustrates the custom NLP pipeline used in this effort. First, as the notes were accessed in the CDW, they were sent to the preprocessing module. This module removed excess white space, most special characters, punctuation and set all the text to lower case. Next, the note was sent to our custom sentence splitter. When text notes stored in the CDW are created, they are stored as a series of 80 column text strings. This means that sentences begin and end in the middle of these strings and have newline characters at the end of each line. We developed a method of reconstructing sentences using a machine-learning model trained on 20,000 examples of text. For this method, we used C5.0, developed by Ross Quinlan as an extension of the earlier C4.5 algorithm,27,28 to generate a rule-based classifier model. Text strings are broken apart where there are sentence boundaries and joined together where sentences are broken across lines. Next, the note was subjected to a module that finds sections of the notes and stores their location based on the character offsets of the section header. These were used to avoid extracting information from areas of the note such as the medication list and the problem list. Then, in the vocabulary look-up module, each line of text was searched for every entry in the vocabulary. When items from the vocabulary were found, an annotation was created and stored with the label from the vocabulary and the character offsets to identify exactly where it appeared in the note. In the sequencer module, a sequence of all the annotations found in a line (sentence) was constructed using these character offsets. These sequences were subjected to handwritten custom rules to determine the final annotated concept. The context module used a separate vocabulary designed to identify indications in the text that render their status as negated, hypothetical or uncertain. It also used a sequence of annotations from the sentence and examines the relationship between negators and the targeted concept.
Figure 2:

Natural Language Processing Pipeline
MRONJ: Medication-related osteonecrosis of the jaw
MRONJ Classification in the NLP Algorithm
For MRONJ classification, we developed a custom function that assigned each note to one of three categories: MRONJ, Possible MRONJ, or Not MRONJ. This custom function was a machine-learning model developed on the training set with all extracted NLP features except Drug Holiday (10 NLP targets and 16 attributes; Table 1) and note author category (orofacial specialist versus non-orofacial clinician) serving as candidate predictors for the dependent variable of MRONJ diagnosis category. This intentionally restricted and clinically defined candidate predictor set limited the complexity of the feature space and reduced the potential for model overfitting. Relevant predictors were selected by decision-tree induction and included in the resulting custom function for MRONJ identification. We used the decision-tree algorithm C5.0, developed by Ross Quinlan as an extension of the earlier C4.5 algorithm.27,28 No additional hyperparameter optimization was performed. Decision-tree branches were retained only if they contained at least one example in the corresponding training fold; no additional stopping criteria or pruning parameters were applied. C5.0’s default Error-Based Pruning was disabled because its use yielded identical results. Model performance was internally validated using 10×10 cross-validation applied to the training set. C5.0 winnowing was enabled before cross-validation, with tree splits selected independently within each training fold.
Detection of Drug Holiday in the NLP Algorithm
For drug holiday detection, annotated strings were decomposed into component concepts representing medication, treatment interruption, time, and an associated dentoalveolar procedure. from the reference set were examined and broken down into smaller component parts. Rules were constructed to identify sequences of these concepts in varying order to recognize the drug holiday target. Abstracting literal text into generalized patterns allows us to anticipate patterns in unseen notes.
Validation sets
Two validation sets were used to assess NLP algorithm performance. The primary evaluation used a pre-specified, independent validation set of 200 notes randomly held out from the annotated reference set before algorithm development and reserved exclusively for final evaluation. The reference set was constructed from the underlying clinical population using surrogate-guided sampling, consistent with prior NLP studies of rare outcomes.21 Consequently, the validation set was enriched for MRONJ cases and drug holiday events relative to the intended deployment population. Because precision (i.e. PPV) and the F-Measure are prevalence-dependent, estimates from this enriched validation set may overestimate real-world performance in the broader population of individuals age≥50 years old taking antiresorptives for fracture prevention.
To address this limitation, we performed a post-hoc augmentation of the validation dataset. Two annotators with expertise in osteoporosis management (LC, REE) independently annotated an additional random sample of 100 dental notes drawn from the NLP total corpus of notes (i.e. all clinical notes from Veterans with at least 1 ICD-9/10 code for a potential MRONJ event), excluding notes in the reference set. Annotation followed the same codebook and procedures used for the reference set. Although these additional notes remained modestly enriched for MRONJ because sampling was restricted to dental encounters associated with a potential MRONJ event diagnosis code, they were not further enriched using surrogate text searches. The augmented validation set combined the pre-specified, held-out validation set (n=200) with these additional, unique 100 dental notes (total n=300).
Statistical analyses
MRONJ diagnosis was evaluated at the document level using three implementation strategies: (1) classifiable outputs only, (2) a strict MRONJ definition, and (3) an inclusive MRONJ definition. Performance metrics are presented for each strategy because different epidemiologic research questions and implementation goals may warrant different approaches. In the classifiable outputs only evaluation strategy, the reference standard was dichotomized as “MRONJ” versus “not MRONJ,” with reference standard “possible MRONJ” notes excluded. NLP outputs of “possible MRONJ” were considered unclassifiable and excluded. In the strict MRONJ definition approach, the reference standard was again dichotomized as “MRONJ” versus “not MRONJ,” with reference standard “possible MRONJ” notes excluded. However, here NLP outputs of “possible MRONJ” were classified as negative rather than excluded. Finally, in the inclusive MRONJ definition approach, the reference standard was instead dichotomized as (“MRONJ” or “possible MRONJ”) versus “not MRONJ.” NLP outputs of “possible MRONJ” were classified as positive.
Performance metrics ascertained included true positives (TP), false positives (FP), false negatives (FN), precision, recall, and F-Measure for each of the primary outcome target concepts (MRONJ diagnosis and drug holiday) as well as for the NLP target concepts and attributes contributing to MRONJ diagnosis (i.e. retained predictors from decision-tree induction). Precision was calculated as TP/(TP+FP). Recall was calculated as TP/(TP+FN). Although precision and recall are mathematically equivalent to PPV and sensitivity, respectively, we used the former terminology to align with machine learning conventions and to facilitate comparison with other NLP studies. The F-Measure is the harmonic mean of precision and recall derived from the formula: 2* (precision*recall)/(precision+recall). The 95% confidence intervals (CIs) for precision and recall were calculated using the exact binomial (Clopper-Pearson) method. For the primary outcome of MRONJ diagnosis, the 95% CI for the F-Measure was estimated using the Hsu et al. method implemented in the psF1 R package.29 F-Measure CIs were not calculated for the retained predictors used in decision-tree induction or for the drug holiday outcome because these metrics were evaluated at the span level rather than the document level. Under span-level evaluation, true negatives are not explicitly defined and are effectively unbounded, precluding specification of the total sample size required by the psF1 method. Accordingly, F-Measure CI was reported only for the document-level MRONJ diagnosis outcome.
First, performance metrics were calculated for all span-level NLP features and the three document-level MRONJ diagnosis implementation strategies using the pre-specified validation set (n=200 notes). For the three document-level MRONJ diagnosis implementation strategies, the observed prevalence of MRONJ was also reported. As a post-hoc analysis, these performance metrics, including observed prevalence, were recalculated using the augmented validation set for the three MRONJ diagnosis implementation strategies.
Finally, because the augmented validation set remained substantially enriched for MRONJ compared with the intended deployment population, a prevalence-adjusted sensitivity analysis was performed to estimate the PPV expected under real-world conditions. Specificity for each MRONJ diagnosis implementation strategy was estimated from the augmented validation set, with 95% CIs calculated using the exact binomial (Clopper-Pearson) method. The sensitivity and specificity estimates obtained from the augmented validation set (i.e. prevalence-independent test characteristics) were then employed to estimate PPV for the primary outcome of MRONJ diagnosis across a range of clinically plausible MRONJ prevalence values (0.01% to 0.03%) reported in recent reviews among men and women age≥50 years old taking BPs or denosumab for fracture prevention.19,20
Analyses were conducted using the Extensible Human Oracle Suite of Tools (eHOST) statistical software, version 1.179 and R version 4.5.2 (R Foundation for Statistical Computing, Vienna, Austria).
Results
Clinical notes and reference set
The study flow diagram is shown in Figure 3. Among 429,436 Veterans ≥50 years with ≥1 filled prescription for BP or denosumab (14.7% women), 4,600 Veterans (17.3% women) had ≥1 ICD-9/10-coded visit for a potential MRONJ event with visit-associated clinical documentation between treatment initiation and Dec 31, 2022. These Veterans contributed 22,210 notes from 10,168 unique visits. Notes were commonly authored by dentists (n=6,779), dental residents (n=1,200), oral and maxillofacial surgeons (n=302), or otolaryngologists (n=390); 4,586 had author of unknown discipline.
Figure 3:

Study flow chart for selection of clinical notes for human-annotated review
BP: bisphosphonate; MBD: Metabolic bone disease; ESRD: end-stage renal diease; ICD: International Classification of Diseases; MRONJ; Medication-related osteonecrosis of the jaw; ONJ: osteonecrosis of the jaw; NLP: natural language processing
1MBD other than osteoporosis includes Paget’s disease of bone, osteogenesis imperfecta, hypophosphatasia or phosphate wasting syndromes, or osteopetrosis
2ICD-9 and ICD-10 codes included: osteonecrosis of the jaw (733.45, M87.180); idiopathic aseptic necrosis of bone, other site (M87.08); osteonecrosis due to drug, unspecified bone (M87.10); osteonecrosis due to previous trauma, other site (M87.28); other secondary osteonecrosis, other site (M87.38); other osteonecrosis, other site (M87.88); osteonecrosis, unspecified (M87.9); inflammatory condition of jaw (526.4; M27.2; K10.2); other specified disorder of jaw (526.89; M27.8); alveolitis or osteitis (526.5; M27.3; K10.3); exostosis/torus (526.81; M27.8); periapical abscess with sinus (522.7; K04.6)
3Enrichment for underrepresented NLP targets from Phase 1 included text searches for the following terms: “alendronate”, “bisphosphonate”, “BRONJ”, “exposed bone”, “holiday”, “MRONJ”, “ONJ”, “osteonecrosis”, and “zoledronic”
4Enrichment for drug holiday in Phase 3 was performed by selecting notes with 90 days prior to a Common Procedure Terminology (CPT) code for a dental extraction (D7120, D7140, D7310, or D7320) and augmented with a text search for “drug holiday”
We annotated 870 clinical notes from 281 Veterans (16.7% women) until thematic saturation was achieved and no new emerging language categories were identified. The reference set development proceeded in 3 phases (Figure 3): (1) 398 notes enriched by text search for “ONJ”; (2) additional notes enriched for underrepresented NLP targets using expanded search terms (“alendronate”, “bisphosphonate”, “BRONJ”, “exposed bone”, “holiday”, “MRONJ”, “ONJ”, “osteonecrosis”, and “zoledronic”); and (3) notes enriched for drug holidays by sampling within 90 days before dentral extraction Common Procedure Terminology (CPT) codes (D7120, D7140, D7310, or D7320) and text searing for “drug holiday.” Of the 870 reference-set notes, 233 indicated MRONJ and 68 possible MRONJ; 121 contained ≥1 drug holiday.
The reference set was randomly partitioned into a training set (n=670) and a validation set (n=200). A total of 6,421 annotations were derived, from which a curated vocabulary of more than 1,350 unique terms was developed. Retained NLP features contributing to MRONJ diagnosis in the machine-learning model following decision-tree induction among all candidate predictors included: debridement, exposed bone, jaw osteonecrosis, jaw radiation, MRONJ stage, and sequestrectomy. Figure 4 shows the rule-based MRONJ classification function derived from decision-tree induction. The note author category (i.e. orofacial specialist versus non-orofacial clinician) was not retained in the final decision-tree model.
Figure 4.

Flow diagram of the custom rule-based function derived from decision-tree induction for MRONJ classification
1Jaw osteonecrosis was considered present only when the target span was not modified by negation, hypothetical, or uncertainty contextual modifiers
2Stage designated as 1, 2, or 3
The additional 100 clinical dental notes drawn from the NLP total corpus of notes to augment the validation set were derived from 38 Veterans (25.3% women and 73.7% men).
NLP algorithm performance
Table 2 shows the performance of the NLP system on each of the primary outcome targets (MRONJ diagnosis across three implementation strategies and drug holiday) and retained data-adaptively selected predictors of MRONJ diagnosis in the pre-specified validation set of 200 notes partitioned from the initial reference set. Performance was strong for most targets, with lower F-Measures for sequestrectomy (0.82) and debridement (0.88). The F-Measure for document-level designation of MRONJ status was strong across implementation strategies: classifiable outputs only strategy (F-Measure: 0.95 [95% CI 0.88 to 1.00]), strict MRONJ definition strategy (F-Measure: 0.94 [95% CI 0.84 to 1.00]), and inclusive MRONJ definition strategy (F-Measure: 0.95 [95% CI 0.89 to 1.00]). The F-Measure for drug holiday was likewise very strong (F-Measure: 0.94).
Table 2.
Natural language processing (NLP) performance statistics on each of the primary outcome target concepts and NLP features retained in decision-tree induction for contributing to MRONJ diagnosis1 in diagnostic accuracy analyses
| Primary Outcome Target Concept | Target Concepts Contributing to MRONJ Diagnosis | Sample size (N) |
True + (n) |
False + (n) |
False − (n) |
Prevalence | Precision (95% CI) |
Recall (95% CI) |
F-Measure (95% CI) |
|---|---|---|---|---|---|---|---|---|---|
| Pre-specified Validation Set2 | |||||||||
| Drug holiday | unbounded3 | 97 | 11 | 2 | NE3 | 0.90 (0.83, 0.95) | 0.98 (0.93, 1.00) | 0.943 | |
| Debridement | unbounded3 | 19 | 4 | 1 | NE3 | 0.83 (0.61, 0.95) | 0.95 (0.75, 1.00) | 0.883 | |
| Exposed bone | unbounded3 | 95 | 1 | 4 | NE3 | 0.99 (0.94, 1.00) | 0.96 (0.90, 0.99) | 0.973 | |
| Jaw osteonecrosis4 | unbounded3 | 159 | 5 | 8 | NE3 | 0.97 (0.93, 0.99) | 0.95 (0.91, 0.98) | 0.963 | |
| Jaw radiation | unbounded3 | 49 | 3 | 3 | NE3 | 0.94 (0.84, 0.99) | 0.94 (0.84, 0.99) | 0.943 | |
| MRONJ stage4 | unbounded3 | 34 | 5 | 1 | NE3 | 0.87 (0.73, 0.96) | 0.97 (0.85, 1.00) | 0.923 | |
| Sequestrectomy | unbounded3 | 7 | 2 | 1 | NE3 | 0.78 (0.40, 0.97) | 0.88 (0.47, 1.00) | 0.823 | |
| MRONJ, classifiable outputs5 | 180 | 18 | 0 | 0 | 10% | 1.00 (0.81, 1.00) | 1.00 (0.81, 1.00) | 0.96 (0.88, 1.00) | |
| MRONJ, strict definition6 | 183 | 18 | 0 | 1 | 10.4% | 1.00 (0.81, 1.00) | 0.95 (0.74, 1.00) | 0.94 (0.84, 1.00) | |
| MRONJ, inclusive defintion7 | 200 | 35 | 2 | 1 | 18% | 0.95 (0.82, 0.99) | 0.97 (0.85, 1.00) | 0.95 (0.89, 1.00) | |
| Augmented Validation Set8 | |||||||||
| MRONJ, classifiable outputs5 | 278 | 25 | 0 | 1 | 9.4% | 1.00 (0.86, 1.00) | 0.96 (0.80, 1.00) | 0.95 (0.88, 1.00) | |
| MRONJ, strict definition6 | 283 | 25 | 0 | 3 | 9.9% | 1.00 (0.86, 1.00) | 0.89 (0.72, 0.98) | 0.92 (0.83, 0.98) | |
| MRONJ, inclusive definition7 | 300 | 43 | 3 | 2 | 15.0% | 0.93 (0.82, 0.99) | 0.96 (0.85, 0.99) | 0.94 (0.88, 0.99) | |
MRONJ: Medication-related osteonecrosis of the jaw; CI: Confidence interval; NE: not estimable
NLP features not appearing in the table were not retained as data-adaptively selected predictors in the machine-learning model using decision tree induction
The pre-specified validation set consists of the held-out validation set created by randomly partitioning the initial annotated reference set into training and validation sets (n=200 notes)
Performance was evaluated at the span level, where true negatives are not explicitly defined and are effectively unbounded, precluding specification of the total sample size and estimation of the prevalence or the F-Measure CI using the psF1 method
This NLP target includes associated attributes (Table 1). A span-level match was considered a true positive only if its associated attribute exactly matched the reference annotation for the purposes of span level performance evaluation.
MRONJ target outcome at the document level defined with classifiable outputs only evaluation strategy. The reference standard was dichotomized as “MRONJ” versus “not MRONJ,” with reference standard “possible MRONJ” notes excluded. NLP outputs of “possible MRONJ” were considered unclassifiable and excluded.
MRONJ target outcome at the document level using a strict case definition of MRONJ. The reference standard was dichotomized as “MRONJ” versus “not MRONJ,” with reference standard “possible MRONJ” notes excluded. NLP outputs of “possible MRONJ” were classified as negative.
MRONJ target outcome at the document level using an inclusive definition of MRONJ. The reference standard was dichotomized as (“MRONJ” or “possible MRONJ”) versus “not MRONJ.” NLP outputs of “possible MRONJ” were classified as positive.
The augmented validation set includes the pre-specified validation set (n=200) notes with an additional, unique 100 dental notes randomly selected from the NLP total corpus of notes (notes from Veterans with at least 1 ICD-9/10 code for a potential MRONJ event)
Table 2 also shows the performance of the NLP algorithm across the three MRONJ diagnosis implementation strategies in the augmented validation set. Precision, Recall, and F-Measure estimates were largely unchanged across implementation strategies and observed prevalence of MRONJ remained high (9.4% to 15.0%). In post hoc sensitivity analyses assuming MRONJ prevalence of 0.01%−0.3%, estimated real-world PPV was 1.00 for both the classifiable outputs only and strict MRONJ definition NLP implementation strategies, reflecting 100% specificity in the augmented validation set (Supplementary Table 3). In contrast, estimated PPV for the inclusive MRONJ definition implementation strategy ranged from 0.01–0.20 despite specificity point estimate of 99%.
Discussion
We developed and validated an NLP algorithm leveraging supervised decision-tree induction to categorize MRONJ, possible MRONJ, and not MRONJ and identify intentional drug holidays around dentoalveolar procedures within free-text clinical documentation with excellent performance metrics. MRONJ is an ideal target for NLP because its diagnosis is not reliably identified from administrative claims and manual EHR ascertainment is infeasible due to its rarity. Previous claims-based algorithms have reported PPVs (95% CIs) of 5.68% (4.68%−6.81%)13, 5.8% (2.2%−12.1%)10, 7.1% (2.7%−14.9%)9, 18% (10%−28%)11, and 21% (10%−36%).12 Our NLP algorithm may outperform these claims-based methods, even in populations with the very low prevalence MRONJ prevalence. Observed specificity was 100% for both the classifiable outputs only and strict MRONJ definition strategies, resulting in estimated PPVs of 100% regardless of prevalence. However, even small reductions in specificity within the confidence limits substantially decreased estimated PPV. Importantly, the reference and augmented validation sets were constructed using surrogate-guided sampling, enriching negative notes for diagnostically challenging, surrogate-positive cases rather than randomly sampling all notes without MRONJ. Accordingly, specificity is likely conservative, as the algorithm was evaluated against more challenging negative examples than expected in routine clinical deployment. Thus, true specificity may be closer to the observed point estimate than the lower CI bound.
Document-level classification provides flexibility for person-level deployment across epidemiologic applications. For example, we recommend classifying Veterans with ≥1 note designated as MRONJ as having MRONJ, and those with ≥1 possible MRONJ note(s) but no MRONJ-designated notes as having possible MRONJ. Alternative approaches incorporating the number or timing of MRONJ and possible MRONJ notes could be used depending on the analytic objective, including studies of diagnostic delays. Distinguishing MRONJ from possible MRONJ provides additional flexibility. Studies requiring high outcome PPV, such as estimating MRONJ incidence, could restrict outcomes to MRONJ the classifiable outcomes only or strict definitions, whereas hypothesis generating studies could use the inclusive definition.
Intentional drug holidays around dentoalveolar procedures are particularly relevant to rheumatologists as patients with RA are at increased MRONJ risk.4 Immunosuppressive therapies for inflammatory arthritis including biologics30,31 and glucocorticoids32 have also been associated with increased MRONJ risk. Furthermore, preclinical studies report that mice with experimentally-induced RA have greater severity of MRONJ.8 Intentional drug holidays around dentoalveolar procedures are well-suited to NLP because brief treatment interruptions33 and, importantly, clinician intent cannot be reliably identified from pharmacy data alone. Whether antiresorptive therapy should be interrupted around dentoalveolar procedures remains uncertain, with conflicting recommendations reflecting limited evidence regarding potential benefits and harms25 However, consensus was not reached in the most recent guideline update, as many members of the working group no longer recommend drug holidays because the potential harms of interrupting antiresorptive therapy may outweigh any reduction in MRONJ risk.23,34,35 Employing our NLP in further epidemiologic studies, including in patients with rheumatic diseases, may help characterize current practice and determine whether drug holidays modify MRONJ risk.
Our NLP algorithm has notable strengths. Our rule-based NLP system is computationally efficient, requiring minimal memory and processing resources, making it well-suited for large-scale analysis of clinical notes. Because the vocabulary and extraction rules are explicitly defined, the algorithm is readily scalable across VHA datasets with minimal computational overhead. The annotated corpus and manually curated concept spans developed in this study also provide a valuable foundation for future supervised machine learning or large language model-based NLP approaches, enabling continued refinement and expansion of automated MRONJ phenotyping. However, the terminology and documentation patterns used to develop the algorithm reflect the VHA CPRS EHR and may not fully represent documentation practices in other healthcare systems. Accordingly, although the computational framework itself is scalable, deployment beyond the VHA will depend on the extent of vocabulary and rule-set adaptation required for implementation in other EHR systems, which is currently unknown. The codebook defining the NLP targets and attributes (Table 1) is provided as Supplementary Material to facilitate adaptation and implementation of the algorithm in other EHR systems.
Our study has limitations. Generalizability of our NLP algorithm to younger Veterans, non-Veterans, or individuals taking other medications (or cancer-treating doses of BPs or denosumab) that may cause MRONJ is not known. In particular, generalizability to non-Veteran populations with a higher proportion of women than the VHA remains uncertain. MRONJ occurs more frequently in women, likely reflecting the higher prevalence of osteoporosis, and consequently, greater exposure to antiresorptive therapy in women.23 Within the VHA, however, the annotated reference set closely reflected the broader source population of Veterans ≥50 years with at least one filled prescription for BP or denosumab before surrogate-guided sampling (16.7% versus 14.7% women), supporting the applicability of the NLP algorithm to the intended clinical deployment populations within the VHA system. A validation set representative of the intended deployment population was not feasible because the extremely low anticipated MRONJ prevalence would have required manual review of an impractically large number of clinical notes to identify a sufficient number of true-positive cases. We did not ascertain details related to the drug holiday, such as its duration or whether it was provider or patient-initiated, that may have clinical relevance in select settings. Finally, drug holiday in the context of this NLP algorithm applied to intentional cessation of antiresorptives around dentoalveolar procedures and would not be applicable to drug holidays for other indications, such as prevention of atypical femoral fracture, another uncommon adverse event of antiresorptive treatment.36,37
In conclusion, within a large national cohort of U.S. Veterans, we developed an NLP algorithm that classifies MRONJ (MRONJ, possible MRONJ, or not MRONJ) and detects drug holidays from clinical notes. Deployable within the VHA, this approach enables clinically meaningful investigations that were previously not feasible using manual chart review or claims-based algorithms.
Supplementary Material
Significance and Innovations.
Addresses a critical gap in medication-related osteonecrosis of the jaw (MRONJ) ascertainment by overcoming limitations of claims-based algorithms and enabling scalable identification from free-text electronic health record documentation using natural language processing (NLP)
Introduces a novel NLP approach to detect intentional drug holidays around dentoalveolar procedures - an exposure not reliably captured in pharmacy data and previously unmeasured in large-scale epidemiologic studies
Funding statement:
This work was supported by grant R01AG079118 from the National Institute on Aging (NIA) and the Office of the Director at the National Institutes of Health. The views expressed in this work are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs, the United States government or the National Institutes of Health.
References
- 1.Ruggiero SL, Mehrotra B, Rosenberg TJ, Engroff SL. Osteonecrosis of the jaws associated with the use of bisphosphonates: a review of 63 cases. J Oral Maxillofac Surg. May 2004;62(5):527–34. doi: 10.1016/j.joms.2004.02.004 [DOI] [PubMed] [Google Scholar]
- 2.Marx RE. Pamidronate (Aredia) and zoledronate (Zometa) induced avascular necrosis of the jaws: a growing epidemic. J Oral Maxillofac Surg. Sep 2003;61(9):1115–7. doi: 10.1016/s0278-2391(03)00720-1 [DOI] [PubMed] [Google Scholar]
- 3.Bansal H. Medication-related osteonecrosis of the jaw: An update. Natl J Maxillofac Surg. Jan-Apr 2022;13(1):5–10. doi: 10.4103/njms.NJMS_236_20 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Fujieda Y, Doi M, Asaka T, et al. Incidence and risk of antiresorptive agent-related osteonecrosis of the jaw (ARONJ) after tooth extraction in patients with autoimmune disease. J Bone Miner Metab. Jul 2020;38(4):581–588. doi: 10.1007/s00774-020-01089-y [DOI] [PubMed] [Google Scholar]
- 5.Rotta D, Fassio A, Rossini M, et al. Osteoporosis in Inflammatory Arthritides: New Perspective on Pathogenesis and Treatment. Front Med (Lausanne). 2020;7:613720. doi: 10.3389/fmed.2020.613720 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Feng Z, Zeng S, Wang Y, Zheng Z, Chen Z. Bisphosphonates for the prevention and treatment of osteoporosis in patients with rheumatic diseases: a systematic review and meta-analysis. PLoS One. 2013;8(12):e80890. doi: 10.1371/journal.pone.0080890 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Mochizuki T, Yano K, Ikari K, et al. Effects of denosumab treatment on bone mineral density and joint destruction in patients with rheumatoid arthritis. J Bone Miner Metab. Jul 2018;36(4):431–438. doi: 10.1007/s00774-017-0848-1 [DOI] [PubMed] [Google Scholar]
- 8.de Molon RS, Hsu C, Bezouglaia O, et al. Rheumatoid Arthritis Exacerbates the Severity of Osteonecrosis of the Jaws (ONJ) in Mice. A Randomized, Prospective, Controlled Animal Study. J Bone Miner Res. Aug 2016;31(8):1596–607. doi: 10.1002/jbmr.2827 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wright NC, Curtis JR, Arora T, et al. The Validity of Claims-Based Algorithms to Identify Serious Hypersensitivity Reactions and Osteonecrosis of the Jaw. PLoS One. 2015;10(7):e0131601. doi: 10.1371/journal.pone.0131601 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Stevens LAS, Spangler L, Yochum L, Ding Y, Wang FT. Assessment of a Claims-Based ICD-9 and ICD-10 Medication-Related Osteonecrosis of the Jaw Algorithm in the United States. Clin Epidemiol. 2021;13:1019–1026. doi: 10.2147/CLEP.S325080 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Bergdahl J, Jarnbring F, Ehrenstein V, et al. Evaluation of an algorithm ascertaining cases of osteonecrosis of the jaw in the Swedish National Patient Register. Clin Epidemiol. 2013;5:1–7. doi: 10.2147/clep.S37664 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Gammelager H, Svaerke C, Noerholt SE, et al. Validity of an algorithm to identify osteonecrosis of the jaw in women with postmenopausal osteoporosis in the Danish National Registry of Patients. Clin Epidemiol. 2013;5:263–7. doi: 10.2147/CLEP.S45226 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Kim HY, Kim JW, Kim SJ, Lee SH, Lee HS. Uncertainty of current algorithm for bisphosphonate-related osteonecrosis of the jaw in population-based studies: a systematic review. J Bone Miner Res. Mar 2017;32(3):584–591. doi: 10.1002/jbmr.3017 [DOI] [PubMed] [Google Scholar]
- 14.Ehrenstein V, Gammelager H, Schiødt M, et al. Evaluation of an ICD-10 algorithm to detect osteonecrosis of the jaw among cancer patients in the Danish National Registry of Patients. Pharmacoepidemiol Drug Saf. Jul 2015;24(7):693–700. doi: 10.1002/pds.3786 [DOI] [PubMed] [Google Scholar]
- 15.Diab DL, Watts NB. Bisphosphonate drug holiday: who, when and how long. Ther Adv Musculoskelet Dis. Jun 2013;5(3):107–11. doi: 10.1177/1759720X13477714 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.AlRowis R, Aldawood A, AlOtaibi M, et al. Medication-Related Osteonecrosis of the Jaw (MRONJ): A Review of Pathophysiology, Risk Factors, Preventive Measures and Treatment Strategies. Saudi Dent J. Mar 2022;34(3):202–210. doi: 10.1016/j.sdentj.2022.01.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Krishnan DG. Controversies in Dentoalveolar and Preprosthetic Surgery. Oral Maxillofac Surg Clin North Am. Nov 2017;29(4):383–390. doi: 10.1016/j.coms.2017.06.001 [DOI] [PubMed] [Google Scholar]
- 18.McGrath LJ, Hernandez RK, Overman R, et al. Initiation and interruption in intravenous bisphosphonate therapy among patients with multiple myeloma in the United States. Cancer Med. Jan 2019;8(1):374–382. doi: 10.1002/cam4.1869 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Anastasilakis AD, Pepe J, Napoli N, et al. Osteonecrosis of the Jaw and Antiresorptive Agents in Benign and Malignant Diseases: A Critical Review Organized by the ECTS. J Clin Endocrinol Metab. Apr 19 2022;107(5):1441–1460. doi: 10.1210/clinem/dgab888 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Ayers C, Kansagara D, Lazur B, Fu R, Kwon A, Harrod C. Effectiveness and Safety of Treatments to Prevent Fractures in People With Low Bone Mass or Primary Osteoporosis: A Living Systematic Review and Network Meta-analysis for the American College of Physicians. Ann Intern Med. Feb 2023;176(2):182–195. doi: 10.7326/m22-0684 [DOI] [PubMed] [Google Scholar]
- 21.Lee RY, Kross EK, Torrence J, et al. Assessment of Natural Language Processing of Electronic Health Records to Measure Goals-of-Care Discussions as a Clinical Trial Outcome. JAMA Netw Open. Mar 1 2023;6(3):e231204. doi: 10.1001/jamanetworkopen.2023.1204 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Tan WK, Heagerty PJ. Surrogate-guided sampling designs for classification of rare outcomes from electronic medical records data. Biostatistics. 2022;23(2):345–361. doi: 10.1093/biostatistics/kxaa028 [DOI] [Google Scholar]
- 23.Ruggiero SL, Dodson TB, Aghaloo T, Carlson ER, Ward BB, Kademani D. American Association of Oral and Maxillofacial Surgeons’ Position Paper on Medication-Related Osteonecrosis of the Jaws-2022 Update. J Oral Maxillofac Surg. May 2022;80(5):920–943. doi: 10.1016/j.joms.2022.02.008 [DOI] [PubMed] [Google Scholar]
- 24.Harkema H, Dowling JN, Thornblade T, Chapman WW. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. J Biomed Inform. Oct 2009;42(5):839–51. doi: 10.1016/j.jbi.2009.05.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Ruggiero SL, Dodson TB, Fantasia J, et al. American Association of Oral and Maxillofacial Surgeons position paper on medication-related osteonecrosis of the jaw−−2014 update. J Oral Maxillofac Surg. Oct 2014;72(10):1938–56. doi: 10.1016/j.joms.2014.04.031 [DOI] [PubMed] [Google Scholar]
- 26.Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. Jan 1 2004;32(Database issue):D267–70. doi: 10.1093/nar/gkh061 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Quinlan JR. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc.; 1993. [Google Scholar]
- 28.Quinlan JR. See5/C5.0 Release 2.11a. RuleQuest Research. Accessed July 15, 2026, https://rulequest.com/see5-info.html [Google Scholar]
- 29.Hsu CY, Liu Q, Shyr Y. A Unified Framework for Statistical Inference and Power Analysis of Single and Comparative Fβ Scores. Stat Med. May 2026;45(10–12):e70557. doi: 10.1002/sim.70557 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Nisi M, Izzetti R, Graziani F. Medication-Related Osteonecrosis of the Jaw Associated in a Patient Treated With Etanercept. J Craniofac Surg. Oct 1 2022;33(7):e694–e696. doi: 10.1097/SCS.0000000000008588 [DOI] [PubMed] [Google Scholar]
- 31.Hauer L, Moztarzadeh O, Baghalipour N, Gencur J. Secukinumab Causing Medication-Related Osteonecrosis of the Jaw, in a Patient Diagnosed with Psoriasis and Rheumatoid Arthritis. Psoriasis (Auckl). 2024;14:115–120. doi: 10.2147/PTT.S490982 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Toriumi S, Kobayashi A, Uesawa Y. Comprehensive Study of the Risk Factors for Medication-Related Osteonecrosis of the Jaw Based on the Japanese Adverse Drug Event Report Database. Pharmaceuticals (Basel). Dec 16 2020;13(12)doi: 10.3390/ph13120467 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Yang CT, Ngan K, Kim DH, Yang J, Liu J, Lin KJ. Establishing a Validation Framework of Treatment Discontinuation in Claims Data Using Natural Language Processing and Electronic Health Records. Clin Pharmacol Ther. Jul 2025;118(1):138–145. doi: 10.1002/cpt.3650 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Hellstein JW, Adler RA, Edwards B, et al. Managing the care of patients receiving antiresorptive therapy for prevention and treatment of osteoporosis: executive summary of recommendations from the American Dental Association Council on Scientific Affairs. J Am Dent Assoc. Nov 2011;142(11):1243–51. doi: 10.14219/jada.archive.2011.0108 [DOI] [PubMed] [Google Scholar]
- 35.Khan AA, Morrison A, Hanley DA, et al. Diagnosis and management of osteonecrosis of the jaw: a systematic review and international consensus. J Bone Miner Res. Jan 2015;30(1):3–23. doi: 10.1002/jbmr.2405 [DOI] [PubMed] [Google Scholar]
- 36.Shane E, Burr D, Abrahamsen B, et al. Atypical subtrochanteric and diaphyseal femoral fractures: second report of a task force of the American Society for Bone and Mineral Research. Journal of bone and mineral research : the official journal of the American Society for Bone and Mineral Research. Jan 2014;29(1):1–23. doi: 10.1002/jbmr.1998 [DOI] [PubMed] [Google Scholar]
- 37.Ott SM. What is the optimal duration of bisphosphonate therapy? Cleve Clin J Med. Sep 2011;78(9):619–30. doi: 10.3949/ccjm.78a.11022 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
