Utilizing a novel unified healthcare model to compare practice patterns between telemedicine and in-person visits

Abstract

Objective

Telemedicine practice has been shown to vary from clinical guidelines. Variations in practice patterns may be caused by disruptions in the continuity of care between traditional and telemedicine providers. This study compares virtual and in-person visits in Stanford’s ClickWell Care (CWC) – where patients see the same provider for both visit modalities.

Methods

Clinical data for two years of patient encounters at CWC from January 2015–2017 (5772 visits) were obtained through Stanford STRIDE. For the 20 most common visit categories, including 17 specific diagnoses, we compared the frequency of prescriptions, labs, procedures, and images ordered, as well as rates of repeat visits.

Results

For the 17 specific diagnoses, there are no differences in labs ordered. Two diagnoses show differences in images ordered, and four differences in prescriptions. Overall, there are more labs (0.16 virtual, 0.33 in-person p < 0.0001) and images ordered (0.07 virtual, 0.16 in-person, p < 0.0001) for in-person visits – due mainly to general medical exam visits. Repeat visits were more likely after in-person visits (19% virtual, 38% in-person, p < 0.0001), 10 out of 17 specific diagnoses showed differences in visit frequency between visit modalities. Visits for both anxiety (5.3x, p < 0.0001) and depression (5.1x, p < 0.0001) were much more frequent in the virtual setting.

Conclusions

Prescriptions, labs, and images ordered were similar between in-person and virtual visits for most diagnoses. Overall however, for in-person visits we find increased orders for labs and images, primarily from general medical exams. Finally, for anxiety and depression patients show clear preferences for virtual visits.

Introduction

Telemedicine visits are an ever-increasing part of the US healthcare system. Telemedicine dates back to the late 1960’s, however, due to recent changes in technology and reimbursement, telemedicine adoption is accelerating.^1,2 This technology has tremendous potential to reduce healthcare costs and improve access to care for underserved populations.^3,4 Less than 40% of Americans are able to get non-ED after-hours care, and telemedicine has been utilized to provide access to care for such patients.^5–8 Additionally, surveys suggest that up to 72% of consumers would choose to see a doctor virtually.⁹ However, there is concern over the quality of care in telemedicine.¹⁰ A recent systematic review showed mixed results for the effectiveness of telemedicine. There is a clear need for larger studies in this evolving and dynamic field.¹¹

Among the main concerns for telemedicine are changes in practice style and poor clinical outcomes. One study noted lower rates of diagnostic testing in virtual care than in traditional settings.¹² Another recent study highlighted a large variation in quality of care across the 8 different telemedicine companies studied.¹³ Large variations in practice and adherence to guidelines are not unique to telemedicine, and have also been found in the traditional primary care setting despite recent advances in clinical decision support.^14,15 Comparing telemedicine to in-person primary care visits is difficult because patient demographics are often variable between virtual visits in an episodic setting and those in a primary care setting. Current studies have been unable to tease out whether differences in practice are due to the modality of the visit, differences in treating physicians, or differences in patient populations.

This study compares practice patterns between virtual and in-person visits through Stanford ClickWell Care (CWC).¹⁶ CWC is a novel virtual primary care clinic developed to serve Stanford’s Accountable Care Organization’s (ACO) younger, healthier, and more tech-savvy patient population.¹⁷ This virtual primary care clinic is designed to combine the convenience of telemedicine with the patient rapport developed in a traditional primary care setting. This clinic allows patients to maintain a traditional relationship with a primary care physician, undergo both in-person and virtual (phone or video) visits, and receive on-demand care at extended hours. We believe this is an ideal way to compare practice patterns across different care delivery modalities.

Materials and methods

Participants

Participants were those who completed any encounter with CWC MD provider between January 2015 and January 2017. All patients at CWC were enrolled in the study, which was approved by the Stanford University Institutional Review Board. Demographic analysis of these patients can be found in our previous study.¹⁶ Patients eligible to sign up for CWC were Stanford Employees enrolled in Stanford's ACO. Inclusion criteria were: age ≥18 years, and English language proficiency. Patients were excluded based on any hospitalization in the past year for chronic medical conditions such as uncontrolled diabetes or hypertension, coronary artery disease, stroke, or another serious medical issue as measured by a health risk assessment (HRA). The HRA (Supplemental Material 1) was created to standardize recommendations to primary care clinic based on health risk and preference. Patients with scores of 0–4 were offered CWC; scores of 5–10 were offered traditional primary care, and scores >12 were offered Stanford Coordinated Care, a clinic for patients requiring a higher level of care coordination. No patients were excluded for mental health challenges.

Data

Data were pulled through the STRIDE database, a database containing EMR records from Stanford Health Care's version of EPIC.¹⁸ Data includes the following patient demographics: age, sex, race, ethnicity, and distance to clinic. We filtered EMR visit data to include: date, time, modality, diagnosis, tests ordered, and medications prescribed. Visit encounter data was pulled for visits completed between January of 2015 and January of 2017. All participant data was de-identified.

Analysis

ICD-9 codes were grouped into clinically meaningful condition categories using the 2012 National Ambulatory Medical Care Survey (NAMCS) and Medicare Chronic Conditions Warehouse.^19,20 The top 20 most common categories were determined by number of visits to CWC. This included 17 specific diagnoses and three other categories of visits not focused on a specific diagnosis, “general medical exam,” “gynecological exam,” and “specific procedure or aftercare.”

Prescriptions, labs, procedures, and images ordered were captured for in-person and virtual visits. Virtual visits were defined as telephone or video visits between a patient and MD provider. In-person visits were defined as those that happened in the brick and mortar CWC between a patient and an MD provider.

To capture prescription patterns, the top three prescriptions by frequency prescribed were determined for each condition category. Drugs were grouped into broad classes following the AHFS Pharmacologic-Therapeutic classification.²¹ The frequency of prescriptions was defined as the percent of visits with a prescription for the drug in question, over all visits within the condition category. These frequencies were tested for significant differences between virtual and in-person modalities with a t-test. Additionally, the frequency of labs, procedures, and images ordered were taken for each condition category. These frequencies were compared between virtual and in-person visits for each condition with a t-test.

Repeat visits for condition categories were analyzed for both virtual and in-person visits. A repeat visit was determined as any encounter with an in-network provider (including labs, phone calls, or other interactions) within one or three days, after an initial virtual or in-person visit to CWC. This definition of a repeat visit is broad, including return visits ordered at the initial visit (e.g. radiology) or visits to other providers that may be unrelated to the chief complaint. One or three day time points were chosen based on discussions with clinicians and perceived windows of time they thought may be unusual following a normal visit. The repeat visit frequencies were tested for significant differences between virtual and in-person differences with a two-sided t-test.

Software

Data processing and programs are written in Python 2.7 (Python Software Foundation, Beaverton, OR).²² Data visualization, regressions analyses, model selection and statistical adjustments are made using R 3.12 (The R Project for Statistical Computing, Vienna, Austria).²³

Results

The 20 most common condition categories were determined using all visits to CWC. Gynecological exam was removed from the diagnoses list due to having no virtual visits, despite being amongst the top 20 condition categories. After filtering for the top 20 condition categories, this resulted in 3993 unique visits that were completed over two years from January 2015-January 2017 at CWC.

Table 1 shows the number and frequency of visits by visit category, as well as the average number of labs and images ordered. Virtual visits are significantly more frequent for eight condition categories (acquired hypothyroidism, acute upper respiratory infections excluding pharyngitis, anemia, anxiety disorders, depression, disorders of lipid metabolism, normal pregnancy, and spinal disorders). In-person visits are significantly more frequent for three condition categories (acute pharyngitis, otitis media and eustachian tube disorders, and the general medical examination). Visits for anxiety and depression were 5.3x and 5.1x more frequent in the virtual setting (p < 0.0001, p < 0.0001).

Table 1.

Number and frequency of visits by diagnosis, and associated average numbers of labs and images ordered

	Number of Visits			Lab Orders (avg)			Imaging (avg)
Conditions	Virtual	Clinic	p	Virtual	Clinic	p	Virtual	Clinic	p
Acquired hypothyroidism	33 (3.2%)	19 (1.3%)	0.0027	0.303	0.316	0.9493	0.061	0.105	0.5977
Acute pharyngitis	40 (3.9%)	82 (5.7%)	0.0347	0.125	0.207	0.547	0.000	0.000	NA
Acute upper respiratory infections excluding pharyngitis	132 (12.8%)	85 (5.9%)	1E-08	0.023	0.129	0.0963	0.000	0.024	0.1585
Anemia	16 (1.6%)	8 (0.6%)	0.0212	0.188	0.500	0.2004	0.063	0.000	0.3332
Anxiety disorders	61 (5.9%)	16 (1.1%)	1E-09	0.033	0.125	0.4786	0.016	0.000	0.3213
Arthropathies and related disorders	108 (10.5%)	129 (9%)	0.2105	0.148	0.124	0.7408	0.213	0.550	3E-05
Asthma	20 (1.9%)	15 (1%)	0.076	0.000	0.133	0.1643	0.050	0.133	0.4301
Benign neoplasms	16 (1.6%)	18 (1.3%)	0.5317	0.375	0.444	0.8349	0.063	0.056	0.9343
Depression	90 (8.7%)	24 (1.7%)	1E-13	0.233	0.417	0.5229	0.011	0.042	0.4848
Disorders of lipid metabolism	31 (3%)	22 (1.5%)	0.0176	0.645	0.955	0.5807	0.032	0.136	0.3298
Essential hypertension	54 (5.2%)	53 (3.7%)	0.0678	0.537	0.415	0.6836	0.056	0.075	0.6809
General medical examination	155 (15.1%)	593 (41.2%)	2E-50	0.123	0.481	2E-07	0.026	0.091	0.0003
Heart disease excluding ischemic	7 (0.7%)	9 (0.6%)	0.8691	1.571	0.333	0.4643	0.000	0.000	NA
Normal pregnancy	19 (1.8%)	8 (0.6%)	0.0054	0.105	0.250	0.6058	0.000	0.125	0.3506
Obesity	5 (0.5%)	14 (1%)	0.149	0.000	0.071	0.3356	0.000	0.000	NA
Otitis media and eustachian tube disorders	8 (0.8%)	28 (1.9%)	0.0104	0.250	0.107	0.5998	0.000	0.036	0.3262
Rheumatism, excluding back	71 (6.9%)	89 (6.2%)	0.4836	0.070	0.124	0.4868	0.197	0.337	0.0937
Specific procedures and aftercare	20 (1.9%)	21 (1.5%)	0.3659	0.000	0.095	0.1623	0.000	0.095	0.1623
Spinal disorders	141 (13.7%)	122 (8.5%)	6E-05	0.099	0.156	0.5782	0.106	0.311	0.0004

Note: Disease condition categories formed from ICD-9 code groupings found in the 2012 National Ambulatory Medical Care Survey (NAMCS) and Medicare Chronic.

Conditions Warehouse.

Bold: p < 0.05.

For the 17 specific diagnoses, the tests ordered are highly similar. There are no significant differences in labs ordered. For imaging, there are two conditions with differences, arthropathies (p < 0.0001) and spinal disorders (p = 0.0004), with in-person visits having more orders. For categories not focused on a specific diagnosis, specific procedures and aftercare showed no differences while general medical exam showed a 3.9x increase in lab orders (p < 0.0001) and a 3.5x (p < 0.0001) increase in imaging for the in-person visits. For all visits combined there are significantly more labs (0.16 virtual, 0.33, in-person, p < 0.0001) and images ordered (0.07 virtual, 0.16 in-person, p < 0.0001) for in-person visits.

For procedures analyzed, due to low numbers of procedures per diagnosis, few statistically relevant results can be drawn, thus we have omitted these findings from the study.

Table 2 shows the prescription patterns of the most frequently prescribed drugs for each condition category. If a drug class had less than 5 orders, it was removed from the analysis due to being underpowered; this left 39 total drug classes. Overall there were a few significant differences captured in prescription patterns between virtual and in-person visits. These significant differences include the following: antibiotics were more commonly prescribed for otitis media at in-person visits as the most prescribed drug class (43% vs 0% p = 0.0001). Muscle relaxants were more commonly prescribed for spinal disorders at in-person visits as the most prescribed drug class (26% vs 15% p = 0.02). Skin prep were more commonly prescribed for arthropathies and related disorders at in-person visits as the 2nd most prescribed drug class (5% vs 1% p = 0.04, skin prep drug category includes most all topical medications). CNS drugs were more commonly prescribed for anxiety disorders at virtual visits as the 3rd most prescribed drug class (0% vs 8% p = 0.02). In three out of four of these significant differences, in-person visits had more prescriptions. However, the number of significant differences is small in comparison to the number of prescriptions analyzed, and the cumulative frequency of prescriptions was similar between modalities.

Table 2.

Top three prescriptions and frequencies of prescription for visits by diagnosis between virtual and in-person visits.

	Rx1	#Rx (%visits with Rx)			Rx2	#Rx (%visits with Rx)			Rx3	#Rx (%visits with Rx)
Conditions	Prescription	Virtual	Clinic	p	Prescription	Virtual	Clinic	p	Prescription	Virtual	Clinic	p
Acquired hypothyroidism	Thyroid Prep	52% (17)	42% (8)	0.5559	Psychotherapeutics	12% (4)	5% (1)	0.447	Gastrointestinal	9% (3)	5% (1)	0.603
Acute pharyngitis	Antibiotics	20% (8)	21% (17)	0.9258	Hormones	5% (2)	2% (2)	0.513	Biologicals	5% (2)	1% (1)	0.312
Acute upper respiratory infections excluding pharyngitis	Cough Cold	23% (31)	27% (23)	0.6432	Antibiotics	23% (30)	24% (20)	0.896	Eent prep	14% (19)	24% (20)	0.164
Anxiety disorders	Psychotherapeutics	90% (55)	88% (14)	0.8837	Analgesics	0% (0)	13% (2)	0.164	Cns drugs	8% (5)	0% (0)	0.024
Arthropathies and related disorders	Antiarthritics	3% (3)	7% (9)	0.1585	Skin prep	1% (1)	5% (7)	0.043	Eent prep	5% (5)	0% (0)	0.132
Asthma	Antiasthmatics	80% (16)	100% (15)	0.4641	Hormones	40% (8)	20% (3)	0.206	Antibiotics	5% (1)	20% (3)	0.218
Depression	Psychotherapeutics	57% (51)	75% (18)	0.2043	Contraceptives	3% (3)	8% (2)	0.417	Gastrointestinal	2% (2)	8% (2)	0.315
Disorders of lipid metabolism	Cardiovascular	52% (16)	32% (7)	0.2707	Biologicals	3% (1)	9% (2)	0.412	Gastrointestinal	0% (0)	9% (2)	0.162
Essential hypertension	Cardiovascular	43% (23)	60% (32)	0.1392	Cardiac	6% (3)	15% (8)	0.108	Diuretics	11% (6)	9% (5)	0.778
General medical examination	Antiasthmatics	8% (13)	3% (17)	0.079	Skin prep	6% (9)	4% (25)	0.481	Contraceptives	4% (6)	6% (33)	0.352
Otitis media and eustachian tube disorders	Antibiotics	0% (0)	43% (12)	0.0001	Antiasthmatics	38% (3)	0% (0)	0.197	Cough cold	25% (2)	11% (3)	0.433
Rheumatism, excluding back	Antiarthritics	4% (3)	11% (10)	0.0922	Psychotherapeutics	3% (2)	7% (6)	0.24	Biologicals	1% (1)	3% (3)	0.495
Spinal disorders	Muscle Relaxants	15% (21)	26% (32)	0.0244	Antiarthritics	15% (21)	11% (13)	0.318	Analgesics	9% (12)	15% (18)	0.15

Total number of initial virtual visits for the recorded conditions (944); total number of intitial in-person visits (1277).

Removed Anemia, Benign neoplasms, Gynecological examination, Heart disease excluding ischemic, Normal pregnancy, Obesity, Specific procedures and aftercare due to statistically insignificant low Rx counts (<5)

Drug classes are grouped following the AHFS Pharmacologic-Therapeutic Classification system.

Bold: p < 0.05

Table 3 shows repeat visit rates by condition category following a virtual or in-person visit to CWC. This is done at both a 1-day and 3-day time scale. A high % signifies a higher rate of repeat visits to CWC or another provider in the Stanford ACO within a 1 or 3 day time window following a virtual or in-person visit. For the 1-day time window, 6 of the 19 condition categories (acute pharyngitis, anemia, arthropathies and related disorders, benign neoplasms, rheumatism excluding back, spinal disorders) were found to have a significantly lower rate of repeat visits for virtual visits, with an overall average of 19% for virtual visits and 38% for in-person visits (p < 0.0001). One condition category, acute pharyngitis had a higher repeat visit rate for virtual visits (40% vs 21% p = 0.037). For the 3-day time window, 5 of the 19 condition categories (arthropathies and related disorders, benign neoplasms, rheumatism excluding back, spinal disorders, and general medical exam) were found to have a significantly lower repeat visit rate in virtual visits, with an overall average of 28% for virtual and 44% for in-person (p < 0.0001). One condition category, acute pharyngitis, had a higher repeat visit rate for virtual visits (53% vs 28% p = 0.011).

Table 3.

Repeat visit rates between virtual and in-person visits at both 1 day and 3 day intervals.

	1-day			3-day
Conditions	Virtual	In-person	p	Virtual	In-person	p
Acquired hypothyroidism	9%	32%	0.074	27%	37%	0.4935
Acute pharyngitis	40%	21%	0.0369	53%	28%	0.0115
Acute upper respiratory infections excluding pharyngitis	7%	2%	0.1064	14%	9%	0.2612
Anemia	25%	75%	0.0247	44%	75%	0.1528
Anxiety disorders	13%	13%	0.9494	18%	19%	0.9496
Arthropathies and related disorders	26%	52%	3E-05	31%	57%	5E-05
Asthma	10%	40%	0.0551	35%	40%	0.7715
Benign neoplasms	0%	39%	0.0043	0%	39%	0.0043
Depression	17%	33%	0.1259	27%	42%	0.1934
Disorders of lipid metabolism	26%	32%	0.6442	35%	41%	0.6968
Essential hypertension	19%	26%	0.3328	26%	26%	0.9546
General medical examination	21%	44%	4E-09	26%	51%	3E-09
Heart disease excluding ischemic	14%	44%	0.2043	14%	44%	0.2043
Normal pregnancy	47%	63%	0.4989	58%	75%	0.4083
Obesity	20%	43%	0.3733	40%	43%	0.9219
Otitis media and eustachian tube disorders	13%	7%	0.6993	13%	14%	0.9021
Rheumatism, excluding back	21%	43%	0.0031	30%	46%	0.0319
Specific procedures and aftercare	25%	29%	0.8023	35%	38%	0.8419
Spinal disorders	21%	40%	0.0009	30%	44%	0.0217

Bold: p < 0.05

Discussion

As telemedicine adoption continues to rise,²⁴ it is of increasing importance to understand what factors contribute to effective, high quality telemedicine. One specualted contributor to lower quality of virtual care is fragmentation²⁵–patients may benefit from continuity of care when seeing an in-person traditional primary care doctor. This is the first study to our knowledge that uses a unified in-person and virtual health system to examine similarities and differences in practice patterns between the two modalities. Other studies have looked at physician practices from separate in-person or virtual care clinics, but none offer care from the same physician in both modalities.^4,12,26 Our results show largely similar practice patterns for most conditions between modalities within CWC. For all 17 specific diagnoses there are no differences in labs ordered, only two conditions differed in images ordered, and only four in prescriptions written. There are, however, a number of important differences.

In the case of a general medical exam (where visits focused on no specific diagnosis) there is a four-fold increase in labs and a more than three-fold increase in images ordered for in-person visits. The reason for this is unknown, but may reflect physician’s perception that patients choosing in-person visits are expecting a more comprehensive examination. Alternatively, patients may choose to come in in-person if they want labs or images to be taken.

Patients and providers appear to have strong preferences for a given modality dependent on the service desired. For example, general medical exams are 2.7x more frequent in-person whereas psychiatric conditions, anxiety and depression, both are 5x more frequent virtually. We believe the differences in general checkup preferences may be explained by providers who find general checkups to be more efficient in-person. The differences for virtual visits in psychiatric conditions is particularly striking. These findings suggest that providers may find mental health encounters more suitable for virtual modalities. It also suggests the existing hypothesis that patients may be more comfortable speaking with their providers about mental health issues through virtual interfaces.^27,28 Recent studies have also reported that telemedicine is particularly effective in neurology or psychiatry settings.^29–31

Another notable finding is that lab and imaging orders are relatively similar between modalities for the vast majority of conditions. For labs, the only significant difference found is for the general medical examination (where no specific diagnosis is noted) where in-person visits recorded many more tests. For the other 17 specific diagnoses there were no significant differences in lab orders. A similar result is found for imaging, where only 2 of 17 specific diagnoses had significantly different imaging frequencies.

Cumulatively, we did find an increase in frequency of labs and images ordered in-person. This is primarily due to the large increase in labs and images ordered for the in-person general medical exam, which contribute to a large proportion of visits (41% in-person,15% virtual). Because of this, the overall average for lab orders was relatively lower in virtual visits. This coincides with recent literature that has found similar decreases in diagnostic tests in telemedicine practice.^12,13,32,33

Prescription patterns between modalities are similar as well, this coincides well with recent literature that has found a similar relationship with antibiotic prescription rates.^32,34 We propose several possible explanations for the few different prescription practice patterns. For otitis media, it is standard protocol to examine the ear before prescribing antibiotics. The greater number of antibiotic prescriptions for in-person visits is a reflection of physicians following standard of care protocol. For spinal disorders, a physician is likely more comfortable prescribing muscle relaxants if they can perform more thorough physical exams. For arthropathies and related disorders, skin prep was likely prescribed at a higher frequency due to a more thorough skin exam, however this is generally not the main concern for treatment or diagnosis. When in-person, physicians may be more likely to detect these unrelated skin issues and prescribe skin prep. For anxiety, as previously discussed, patients may feel more comfortable talking about their concerns over virtual interfaces and be more likely to receive CNS drugs. Overall, there very few significant differences in prescription patterns. For the few that are present, we see that they are likely a result of the inherent differences between the modalities of care.

Finally, we find repeat visits are more likely after in-person visits. Six condition categories have statistically higher repeat visit rates for in-person visits, whereas only one condition category has higher repeat visit rates for virtual visits (1-day windows). This possibly reflects higher acuity concerns associated with in-person visits, but we have no data on reasons for choice of modality. Interestingly, these differences in repeat visit rates suggest that virtual visits are not perceived to be “incomplete” (or in need of follow-up for further investigation) when compared to in-person visits. Our post-visit utilization analysis may not capture the underlying reason for a follow-up visit, as we count any interaction within the network (phone call, labs, primary care appointment, etc.) as a visit, which explains some of the high frequency values seen. However, we believe these are still important and interesting initial findings that prompts further future research in other study designs.

Limitations

First, practice patterns were only able to be elucidated retrospectively from the data captured through the stride database. Thus, broader practice patterns were investigated rather than individual visits. In addition, with only 2 years of data in one clinic, some statistical tests were underpowered due to low N, and may have reached significance with more data. Low N also prevented us from being able to perform a demographics analysis by diagnosis, though we have done an in-depth analysis in previous papers on the general CWC population.¹⁶ Another limitation is that the differences in practice patterns we find were not able to be analyzed in relation to outcome or patient satisfaction. For example, we were not able to look at physician’s adherence to guidelines-based practice and therefore cannot speak to the appropriateness of the labs or prescriptions ordered. Because of this, it is difficult to draw conclusions about whether certain practice patterns are better or worse. Moreover, these differences in practice patterns may be a result of selection bias rather than modality, as we are unable to control for patient choice in the study. Our analysis of repeat vists may not be a true proxy for utilization of follow-up vists. Our study uses a broad definition of repeat visits, thus visits for radiology or visits to providers unrelated to the chief complaint (routine physical therapy, behavioral health, ore other weekly apointments) may confound the analysis. Finally, these results may not be generalizable due to the specific ACO population from a generally healthy young Stanford employee population, especially in regards to education and technology acceptance.

Conclusions

This is the first study, to our knowledge, to compare practice patterns for virtual and in-person appointments where patients see the same provider for each type of visit. Overall, prescription patterns, as well as lab, procedures, and images ordered were similar between in-person and virtual visits for most diagnostic groups. However, there was a higher rate of repeat visits in the in-person setting. Finally, this study highlights a difference seen in care surrounding depression. Depressed patients are more likely to choose virtual visits, and those that were seen with this modality had less repeat visits as well. Future studies may include the collection of more data over the years and following trends longitudinally. In addition, studies may perform further exploratory data analysis to identify hidden associations and trends within the data such as sociodemographic analyses. With certain large healthcare providers anticipating more telemedicine visits than in-person visits in 2020, it will be essential that this imminent technology is studied carefully to ensure the highest quality of care.

Acknowledgements

We are grateful to STRIDE for producing the data for this manuscript. STRIDE (Stanford Translational Research Integrated Database Environment) is a research and development project¹³ at Stanford University to create a standards-based informatics platform supporting clinical and translational research. STRIDE was supported by the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through grant UL1 RR025744. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Contributorship

JXW and JGN both equally contributed to the preparation, analysis, and writing of the study. LC and SD provided feedback and advice for study design and analysis. All authors reviewed the final version of the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

This work was approved by the Stanford University Institutional Review Board.

Funding

This study was supported by the Stanford Spectrum Learning Health Care Innovation Challenge Grant and the Stanford MedScholars Research Fellowship.

Guarantor

JGN.

Peer Review

Elisabeth Medeiros De Bustos, CHU Besancon has reviewed this manuscript.

ORCID iD

Jonathan X Wang https://orcid.org/0000-0001-5208-3352

Supplemental Material

Supplemental material for this article is available online.