Improving Diagnostic Recognition of Primary Hyperparathyroidism with Machine Learning

Abstract

Importance

Parathyroidectomy offers the only cure for primary hyperparathyroidism (PHPT), but today only 50% of PHPT patients are referred for surgery, in large part because the condition is widely under-recognized. PHPT diagnosis can be especially challenging with mild biochemical indices. Machine learning (ML) is a collection of methods in which computers build predictive algorithms based on labeled examples.

Objective

With the aim of facilitating diagnosis, we tested the ability of ML to distinguish PHPT from normal physiology using clinical and laboratory data.

Design

This is a retrospective cohort study using a labeled training set and 10-fold cross-validation to evaluate algorithm accuracy. Measures of accuracy included area under the ROC curve, precision (sensitivity), and positive and negative predictive value. Several different ML algorithms and ensembles of algorithms were tested using the Weka platform.

Setting

3 high-volume endocrine surgery programs

Participants

Among 11,830 patients managed surgically from March, 2001 to August 2013, 6,777 underwent parathyroidectomy for PHPT, and 5,053 control patients without PHPT underwent thyroidectomy.

Main Outcomes and Measures

Test-set accuracies for ML models were determined using 10-fold cross-validation. Age, gender, preoperative calcium, phosphate, PTH, Vitamin D, and creatinine were defined as potential predictors of PHPT. Mild PHPT was defined as PHPT with normal preoperative calcium or PTH levels.

Results

After testing a variety of ML algorithms, Bayesian network models proved most accurate, correctly classifying 95.2% of all PHPT patients (area under ROC=0.989). Omitting PTH from the model did not significantly reduce the accuracy (area under ROC = 0.985). However, in mild disease cases, the Bayesian network model correctly classified 71.1% of patients with normal calcium and 92.1% with normal PTH levels preoperatively. Bayesian networking + AdaBoost improved the accuracy to 97.2% correctly classified (area under ROC=0.994) cases, and 91.9% of PHPT patients with mild disease. This was significantly improved relative to Bayesian networking alone (p<0.0001).

Conclusions and Relevance

ML can accurately diagnose PHPT without human input, even in mild disease. Incorporation of this tool into electronic medical record systems may greatly aid in recognition of this under-diagnosed disorder.

INTRODUCTION

Primary hyperparathyroidism (PHPT) is the state of autonomous parathyroid hormone (PTH) hypersecretion by one or more abnormal parathyroid glands. Over-secretion of PTH causes hypercalcemia and accelerates bone loss, resulting in skeletal abnormalities such as osteopenia/osteoporosis, nephrolithiasis, renal insufficiency, peptic ulcer disease, acid reflux, cardiovascular disease and hypertension. Neurocognitive changes like fatigue, depression, anxiety, memory loss, and irritability are also associated with PHPT^1–3. Surgical excision of hyperfunctioning parathyroid glands remains the only curative treatment for PHPT patients, often with greater than 95% success rates^4–6.

Today however, less than half of biochemically proven PHPT patients are referred for surgery^7,8. PHPT is the most common etiology for elevated calcium in the ambulatory population, but documented hypercalcemia often exists for many years before PHPT is formally diagnosed and evaluated^8–10. Numerous potential reasons could explain why a knowledgeable provider might delay or forgo further workup for PHPT. In one study of a large health system’s EMR, only 32% of patients with hypercalcemia had a PTH measured, suggesting that under-recognition rather than conscious observation accounts for much of the under-treatment for PHPT⁸.

The increasing availability of data through electronic medical record (EMR) systems offers a valuable opportunity to improve PHPT recognition. Federal incentives have led to the rapid proliferation of EMR systems, but the use of these tools to analyze the vast amounts of data contained within EMRs still lag behind their widespread adoption^11,12. Computer-based decision support tools have improved healthcare delivery in domains including preventative care and drug interactions^13–15 to guide clinical decision making. Decision-support tools commonly exist as alerts to prompt or alter physician order entry¹⁶. Supervised machine learning (ML) methods are focused on inferring predictive models from existing datasets that consist of labeled examples,^17,18 and in clinical medicine, can classify or predict future outcomes for a given patient^19,20. ML approaches have been employed to predict risk of breast cancer from mammographic findings²¹ or to risk stratify patients for venous thromboembolism from EMR data²². In this study, we hypothesized that ML could accurately diagnose PHPT based on readily available clinical and laboratory indicators. We describe the development and evaluation of a ML method that accurately predicts PHPT using a large, multi-institutional dataset. Improving recognition of the disease and distinguishing mild disease from normal variations drove the work described here

METHODS

Study Sample and Definitions

The Institutional Review Boards of the University of Wisconsin School of Medicine and Public Health, the University of Pittsburgh School of Medicine, and the Medical College of Wisconsin approved the collection of data. We compiled a dataset from the existing, prospectively collected perioperative datasets of 11,830 patients from three high-volume academic centers. Of these patients, 6,777 patients (STUDY) with biochemical PHPT underwent an initial parathyroid operation for sporadic disease. Only patients confirmed to have PHPT at the time of operation were included in the STUDY group. The remaining 5,053 patients (CONTROL) underwent thyroidectomy without parathyroid disease as determined by preoperative laboratory indices. The dates of operations spanned from March 2001 to August 2013. All patients undergoing parathyroidectomy were diagnosed preoperatively to have biochemical evidence of PHPT and control patients did not have biochemical evidence of PHPT, as both groups had a minimum of serum calcium and PTH levels measured preoperatively.

ML is a collection of different methods, or algorithms, each with differing classification mechanisms. The typical approach for employing ML for clinical problems is to test a variety of different algorithms to find the most accurate. To accomplish this task, we used the ML software platform available from the Waikato Environment for Knowledge Analysis (Weka Version 3.6.11). The Weka software suite contains a library of algorithms that build predictive models by learning from examples provided in user-supplied datasets. Each algorithm was trained using our patient dataset, which included 7 attributes that we considered most likely to be relevant to PHPT diagnosis: age at surgery, gender, preoperative serum calcium, paired preoperative serum PTH, serum 25-hydroxy vitamin D, serum creatinine, and serum phosphate values. Predictor variables chosen for two reasons: 1) they were consistently available within the databases from the three institutions, and 2) represent initial, routinely available data within a given patient’s electronic records that could indicate PHPT without an exhaustive or complete workup for PHPT. The studied laboratory values were those obtained closest in time to the date of surgery.

As outlined in the recent surgical guidelines on management of hyperparathyroidism (21), calcium and PTH levels characteristically vary quite a bit during the biochemical diagnosis of PHPT. In normocalcemic hyperparathyroidism, the calcium can be intermittently normal. Alternatively, the PTH level can fall within the normal range, but still be inappropriately high for the calcium level. For the purposes of this study, ‘mild’ PHTP was defined to be present when either the calcium or PTH level that was recorded closest to the operative date, was within the normal range²³. Correct diagnostic classification of mild PHPT was defined to be present when a patient’s ML-predicted disease status agreed with his physicians’ preoperative diagnostic impression based on biochemical indices.

Classification Methods

We began by testing several categories of supervised machine learning classifiers: 1) rule-based classifiers, 2) logistic regression based classifiers, 3) tree based classifiers, and 4) Bayesian network classifiers. After testing over 20 different algorithms, we selected the Bayesian Network (BayesNet) for further study as it maximized accuracy and transparency. A Bayesian Network uses a graphical representation to represent a compact encoding of the joint distribution of input variables and the outcome(s) of interest^24,25. Although more complex Bayesian network structures were evaluated, the naïve Bayes models proved as accurate, and more comprehensible, than the complex networks. A naïve Bayesian network is one in which the graph structure posits that each input variable is independent of the others when conditioned on the outcome variable. The BayesNet method handles missing data through a process called imputation where missing features are inferred from other cases where those features are present. Therefore, the algorithm does not require complete datasets for either training or prediction²⁶.

We then explored means to improve the accuracy of the diagnostic model for patients with mild biochemical disease. The method selected is known as adaptive boosting, or AdaBoost, a meta-algorithm that can be coupled with other classifiers to improve performance²⁷ by placing more weight on previously misclassified instances, thereby ‘boosting’ previously weak classifiers. AdaBoost was chosen to place more weight on the misclassified, mild biochemical cases. An ensemble of BayesNet in conjunction with AdaBoost was then run on the established datasets.

Statistical Analysis

All ML classifiers were tested using 10-fold cross validation. The performance of each classifier was evaluated based on sensitivity (recall), specificity, positive predictive value (precision), area under the receiver operating characteristic (ROC) curve, and overall accuracy. Continuous variables are represented by their mean ± standard deviation. Statistical comparisons of continuous variables were made using Student’s T-test. Categorical variables were statistically compared using Pearson’s Chi-square test.

RESULTS

Baseline Patient Characteristics and Accuracy of Bayesian Network Based Model

This sample study included 6,777 biochemically confirmed PHPT patients and 5,053 control patients, of whom 76.5% and 80.1% were female, respectively. PHPT patients (mean age, 59 ± 0.2 years) were older than control patients (mean age, 49 ± 0.2 years; p<0.0001). Serum levels of calcium, PTH, phosphate, vitamin D, and creatinine were all consistent with the biochemical definition of PHPT (Table 1)²³. 8,450 cases had at least one missing feature, but 79.1% of the cases had only 1 missing feature. The most common missing features, in descending order, were phosphate, creatinine, and vitamin D.

Table 1.

Patient Demographics and laboratory data.

	PHPT Patients (n = 6777)	Control Patients (n = 5053)	p value
N	6777	5053	-
Age	59.9 ± 0.17	49.8 ± 0.21	< 0.001
Gender (% Female)	76.5%	80.1%	< 0.001
Calcium (mg/dL)	11.3 ± 0.17	9.3 ± 0.0.007	< 0.001
Phosphate	2.9 ± 0.02	2.1 ± 0.02	< 0.001
PTH (pg/mL)	178 ± 7	41 ± 0.4	< 0.001
Vitamin D (ng/mL)	29 ± 0.17	32.4± 0.17	< 0.001
Median Creatinine (mg/dL)	1.0 ± 0.01	0.9± 0.01	0.430

Continuous variables are represented by their mean (or median where indicated) ± SEM

After testing a variety of ML methods, we found that a naïve Bayesian network most accurately predicted patient diagnoses (PHPT vs. control). A graphical representation of the network structure is shown in Figure 1, with each feature contributing independently to the probabilistic classification of patients. Here, the Bayesian network is “naïve” because each feature is conditionally dependent on the outcome alone without any arcs or probabilistic relationships between the predictor features. The Bayesian networks resulted in a 95.2% test set accuracy, and an area under the ROC curve of 0.989 (Table 2).

Figure 1.

Graphical structure of our Bayesian network classifier. A Bayesian network is represented by a directed acyclic graph that indicates the probabilistic relationships among the input features and the outcome of interest. Each arrow, or arc, represents a conditionally dependent relationship. Associated with each node is a conditional probability distribution that represents the distribution of the node variable conditioned on its parents in the graph. In this case, the model is a “naïve” Bayesian network, meaning that each input feature is treated as being independent of the others when conditioned on the outcome variable (PHPT or normal).

Table 2.

Accuracy of Bayesian Network Modeling for Predicting Disease within Different Patient Subsets

	Correctly Classified Instances	Sensitivity	Specificity	Positive Predictive Value	Area under ROC
All Patients (N=11830)	95.20%	96.40%	93.70%	95.40%	0.989
Males Only (N=2958)	95.8%	96.8%	94.2%	96.4%	--
Low Vitamin D (N=2613)	95.6%	95.5%	95.8%	97.1%	--
High Vitamin D (N=2601)	93.5%	91.8%	95.3%	95.1%	--
PHPT Patients with Mild Disease (N=1738)	86.0%	--	--	--	--
- Normal Calcium (N=760)	71.1%	--	--	--	--
- Normal PTH (N=1125)	92.1%	--	--	--	--
PHPT Patients with Non-Mild Disease (N=5039)	99.9%	--	--	--	--

Mild labs are defined as a calcium level of <10.2mg/dL and/or a PTH level of <72pg/mL

Low Vitamin D is defined as <30ng/mL

Patients with mild biochemical indices for PHPT represent a diagnostic challenge because they must be distinguished from those with secondary causes of elevated serum PTH such as vitamin D deficiency. Hence, we further tested the accuracy of our model in making diagnostic predictions that were consistent with our specialists’ clinical diagnoses specifically in these subpopulations (Table 2). The accuracy of our model did not change for patients with low vitamin D (<30 ng/mL), correctly classifying 95.6%. Among patients with mild biochemical disease (N=1,738) however, we observed a reduction in the accuracy of classification to 86.0%. These findings suggest that the accuracy of the BayesNet predictive model may be more limited when it comes to categorizing patients with milder biochemical disease.

We further investigated all the misclassified PHPT cases and compared them to the correctly classified instances. When we compared PHPT patients who were correctly diagnosed (true positives) to those mistaken as normal (false negatives), the latter group was on average significantly younger (p=6.7E-9), had higher serum phosphate levels (p=1.2E-8), and had higher serum vitamin D levels (p=0.0002). Much of the misclassification occurred in mild biochemical disease; 99% of the false negatives had mild disease, while only 23% of the true positives had mild disease. False negatives had a mean serum calcium level within normal range (9.7±0.04 mg/dL) and a much lower mean serum PTH level (110.9±10.6 pg/mL) compared to that of true positives (181.0±7.1 pg/mL, p=1.6E-8).

Control patients who were correctly classified (true negatives) were on average younger than those incorrectly classified as having PHPT (false positives) (p=3.8E-5). The false positive controls exhibited significantly higher mean calcium (10.1±0.04 mg/dL), and PTH levels (74.03±4.2 pg/mL) (p=8.0E-32). 6.3% of control patients were mistaken to have PHPT.

Adaptive Boosting of BayesNet

Although the BayesNet algorithm alone was highly accurate, this ML tool becomes even more useful to experts if it can reliably distinguish cases of mild hyperparathyroidism. To reduce the rate of misclassification among the mild cases, we employed the meta-algorithm known as adaptive boosting in conjunction with BayesNet, and observed that doing so enhanced our model’s classification accuracy from 95.2% to 97.2% with an area under the ROC of 0.994 (Table 3). The total number of misclassified patients decreased from 563 (4.76%) using BayesNet alone, to 329 (2.78%) using the BayesNet+AdaBoost model (Table 3). Since many of the previously misclassified patients had mild PHPT, the use of adaptive boosting improved classification accuracy among those with mild disease.

Table 3.

Accuracy of Bayesian Network alone versus Adaptive Boosted Modeling

		Bayesian Network Alone	Bayesian Networking + Adaptive Boosting	p-value
Correctly Classified Instances		95.2%	97.2%	--
Sensitivity		96.4%	97.8%	--
Specificity		93.7%	96.4%	--
Positive Predictive Value		95.4%	97.4%	--
Area under ROC		0.989	0.994	--
All Misclassified Instances, N		563/11830	329/11830	< 0.0001*
False Positives, N		317	180	--
False Negatives, N		246	149	--
PHPT Patients w/ Mild Disease (N=1738)	True Positives (Correctly Classified), N (%)	1494 (86.0%)	1598 (91.9%)	< 0.0001*
	False Negatives (Incorrectly Classified), N (%)	244 (14.0%)	140 (8.1%)

Mild disease defined as a calcium level and/or a PTH level within the normal range

With the same patient dataset used for the original BayesNet model, we once again compared the biochemical profiles of those misclassified to those correctly classified after BayesNet+AdaBoost. The boosted model significantly reduced the number of false negatives from 246 (2.07%) to 149 (1.26%, p < 0.0001, Table 3). Importantly, AdaBoost reduced the number of false negatives with mild disease (14.0% in BayesNet alone compared to 8.1% in BayesNet+AdaBoost, p < 0.0001, Table 3). The improvement in classification of mild cases was associated with a small increase in false positives in the control group, specifically for patients with serum PTH levels above the upper limit of normal (2.7% vs. 3.6% false positives, p < 0.0001).

Alternative ML Approach

Since the observed accuracy of the BayesNet + AdaBoost was higher than in many clinical studies using machine learning methods^22,28, we sought to test its classification accuracy using an alternate approach. Rather than employing 10-fold cross-validation (randomly chosen test sets), we used each institutional data as a separate test set allowing assessment of any institutional variations in accuracy that may indicate different referral or diagnosis patterns. The model was trained in the UW dataset and then tested on the other two institutional datasets. Although there were slight differences in sensitivity and specificity, the Bayesian network had similar overall accuracy when tested in this way, and the area under the ROC curve was nearly identical to our original analysis (Table 4).

Table 4.

Trained adaptive boosted Bayesian network model used to test other institutional cohorts

Training Set = University of Wisconsin (UW) Cohort
Supplied Test Set	Correctly Classified Instances	Sensitivity	Specificity	Positive Predictive Value	Area under ROC
MCW	94.5%	97.6%	90.8%	92.6%	0.987
PITT	96.8%	99.7%	91.4%	95.4%	0.995
MCW + PITT	96.2%	99.2%	91.2%	94.8%	0.993

MCW- Medical College of Wisconsin cohort

PITT- University of Pittsburgh cohort

Omission of PTH

The set of laboratory values included as predictive features for the above BayesNet and boosted algorithm includes a serum PTH level. However, for a clinician to order a serum PTH level means that he or she is already considering the diagnosis of PHPT. To assist the physician who either lacks awareness of PHPT or is not yet considering that particular diagnosis, we also tested several ML models omitting PTH as a predictive feature. Again, BayesNet proved the most accurate algorithm, correctly classifying 95.6% of all STUDY cases. The false positive rate was 5.4%, and the area under the ROC curve was 0.985. Meta algorithms like adaptive boosting could not improve upon the accuracy of the BayesNet alone in this particular setting.

DISCUSSION

In summary, we trained a machine learning algorithm to diagnose PHPT based on biochemical and demographic data using a multi-institutional dataset. The algorithm was highly accurate, with over 95% accuracy tested by two different techniques. The boosted algorithm performed remarkably well even for patients with mild disease. Additionally, the boosted algorithm appeared to be useful in distinguishing mild hyperparathyroidism from normal, physiologic alterations in serum calcium and/or PTH. The BayesNet model without PTH remained highly accurate even without PTH as a predictive feature. Therefore, we provide here different algorithms useful in different scenarios, depending on the informational needs of the provider: the BayesNet + AdaBoost is useful for the expert trying to distinguish mild disease from normal physiology, while the BayesNet without serum PTH can assist the clinician who lacks diagnostic conviction and/or awareness of the disease. Either or both algorithms could serve as the underlying code for a clinical decision support tool that prompts physicians to refer patients for further evaluation when labs are consistent with primary hyperparathyroidism.

The clinical diagnosis of PHPT is primarily based on interpretation of laboratory indices including PTH, calcium and vitamin D^29,30. Emerging evidence indicates that many with biochemical evidence of this disease remain undiagnosed, resulting in a vast, unrecognized population of afflicted patients^7,8. Within the Kaiser healthcare system (3.5 million patients), over one half of those with biochemical evidence of PHPT did not receive a parathyroidectomy over a 13 year period, regardless of symptoms or fulfillment of international consensus criteria for treatment⁷. However, population level studies cannot indicate why patients go untreated. Several other studies had similar findings: for patients with elevated serum calcium, unfortunately serum PTH levels are frequently not obtained despite PHPT being the most common etiology of outpatient hypercalcemia^7–10. To address situations where serum PTH is never ordered, we tested ML algorithms without PTH as a predictive feature, and found BayesNet without PTH is still over 95% sensitive for detecting PHPT.

Even for patients who meet objective diagnostic criteria, several studies have found that parathyroidectomy is significantly underutilized,^8,31 and the under-treatment appears to hinge on at least two factors: first, a lack of diagnostic awareness and second, deferment of evaluation³². The informatics tools reported here can assist with recognition, and integration into the EMR can prompt timely referral for further evaluation and treatment. Importantly, this tool utilizes readily available clinical data, and provides automated diagnosis without additional cost or invasive testing.

Parathyroidectomy benefits PHPT patients by not only correcting their biochemical disease, but also, preventing the progression of PHPT-related complications, including worsening bone mineral density and fragility fractures^33–36. Compared to observation, Almqvist et al. determined that early parathyroidectomy reduced patients’ long-term risk of hip and other bone fractures³⁷. Longstanding PHPT has also been associated with poorer quality of life, psychiatric illnesses, left ventricular hypertrophy and heightened mortality risk from cardiovascular disease^38–45. Additionally, parathyroidectomy is more cost-effective than observation⁴⁶. The tool developed here promises to aid both practitioners and patients recognize and treat the disease earlier to reduce long-term morbidity.

It is important to identify mild PHPT because patients without significant biochemical disease still experience symptoms and a decline in bone density^47,48. Accumulating studies provide evidence that mild PHPT patients benefit from parathyroidectomy despite not meeting formal consensus criteria for surgical intervention^{23,33,34,42,47,49,50} as the new AAES surgical guidelines discuss in detail (21). Moreover, the incidence of mild PHPT is on the rise⁵¹. Diagnosis of mild biochemical PHPT becomes challenging for primary providers, endocrinologists, and surgeons. Aside from difficulty distinguishing the mild biochemical PHPT from normal physiology or secondary causes of hyperparathyroidism, the symptoms associated with PHPT are often non-specific. Symptoms like fatigue or musculoskeletal pain are common in PHPT and numerous other conditions^7–9,52. Therefore, physicians may focus on other etiologies of reported symptoms prior to fully evaluating an elevated serum calcium or PTH. Adaptive boosting of the Bayesian classifier improved the classification mild PHPT, and in doing so, increased the overall accuracy of the model. For clinicians with less awareness of PHPT, the model without serum PTH can prompt further workup. For instance, a “pop-up” alert might prompt the provider to order a serum PTH. Hence, this tool can aid both the expert and the non-specialist distinguish PHPT from normal variants, or at least prompt earlier biochemical evaluation by raising awareness for clinicians less familiar with PHPT.

Emerging computer decision support tools that use machine learning reliably assist physicians in a variety of other domains such as predicting venous thromboembolism, assessing breast cancer risk, or detection of myocardial infarction^21,28,53,22. However, few decision support aids for diagnosing PHPT currently exist and are limited mainly to graphical representations of where abnormal calcium and PTH values intersect in standard laboratory reporting. These graphs use arbitrary cutoffs whereas the Bayesian Network utilized in this study was trained on nearly 12,000 examples of normal and abnormal patient laboratory results.

The tool described here uses readily available clinical data to automatically alert the provider when his or her patient exhibits certain laboratory features of PHPT, thereby prompting further evaluation or referral. Such automated alerts can become incorporated into existing electronic medical record systems as “best practices alerts.” Clinical decision-making is a complex task, requiring the physician to integrate multiple pieces of diverse data types, synthesize all this information, and bridge the “inferential gap” between the data at hand and the clinical knowledge needed to make the correct diagnosis or treatment decision^54,55. Computerized decision-support tools overcome many of these challenges, and can improve the quality of care through prompts or second opinions in real-time, at the point of care. The other advantage of incorporating ML algorithms into EMRs is the ability to continuously improve with the addition of more data. Hence, our Bayesian Network to recognize PHPT could become even more accurate as it is “trained” on more patients. Clinicians rarely work with complete data sets. Since our classification method displayed excellent accuracy despite the high frequency of missing features, we expect it to maintain its accuracy in real-world, clinical settings.

Limitations

This current study is limited by its retrospective nature; we tried to address this with the large, multi-institutional sample used to train our classifier and, in addition, used three different institutions to investigate for any institutional biases. The study patients also comprised only those referred for surgery which introduces a potential selection bias because our sample could represent those with a more severe presentation that prompted the physicians to recommend surgery rather than continued observation. A third potential limitation is the determination of test-set accuracies by cross-validation within the dataset, thus making it difficult to predict how the algorithms would perform in a different population, although the model was observed to have similar overall accuracy when evaluated across multiple institutions. The STUDY group only included patients from surgical databases confidently diagnosed with PHPT. Future testing of the tool should include those patients being observed or worked up further for PHPT. Hence, the methods described here will require prospective testing in different patient populations prior to robust clinical implementation. Another potential flaw is our decision to use in just one set of biochemical indices, taken at the time point close to surgery, which was a practical decision for evaluation of the huge masses of laboratory data in this pilot study. Future analysis of this tool should include laboratory values taken at multiple time points to reflect actual clinical practice and potentially increase the accuracy for the most challenging cases. Finally, since this is a surgical cohort, most patients affected with urinary calcium leak or familial hypocalciuric hypercalcemia (FHH) were not included, so there is no way to determine here how well the algorithm would perform on these uniquely challenging cases. However, due to the geographical location of the three examined centers and the case mix of their controls, the training set did include many patients with vitamin D deficiency and hyperthyroidism, which are two common secondary causes of elevations in PTH (21).

CONCLUSIONS

ML is a reliable adjunct for detecting PHPT to aid both specialists and non-specialists alike based on a dataset of 7 readily available clinical variables. As described above, the studied algorithm performed least well on the mild PHPT cases, but the boosted algorithm successfully reduced the misclassification of these most challenging cases. However, our purpose in this initial study was to improve upon the overall recognition of the disease rather than diagnose the most challenging cases that should probably undergo further workup regardless. Similarly, even though the model without PTH was slightly less accurate, it can facilitate the intended goal of prompting further clinical workup including PTH levels. Future studies will prospectively evaluate its performance in real time with the goal of incorporation into electronic medical record systems as clinical decision support for further evaluation or referral when results indicate PHPT is likely.