Abstract
Background
As patient-initiated messaging rises, identifying variation in message volume and its relationship to clinician workload is essential.
Objective
To describe the association between variation in message volume over time and time spent on the electronic health record (EHR) outside of scheduled hours.
Design
Retrospective cohort study.
Participants
Primary care clinicians at Cleveland Clinic Health System.
Main Measures
We categorized clinicians according to their number of quarterly incoming medical advice messages (i.e., message volume) between January 2019 and December 2021 using group-based trajectory modeling. We assessed change in quarterly messages and outpatient visits between October–December 2019 (Q4) and October–December 2021 (Q12). The primary outcome was time outside of scheduled hours spent on the EHR. We used mixed effects logistic regression to describe the association between incoming portal messages and time spent on the EHR by clinician messaging group and at the clinician level.
Key Results
Among the 150 clinicians, 31% were in the low-volume group (206 messages per quarter per clinician), 47% were in the moderate-volume group (505 messages), and 22% were in the high-volume group (840 messages). Mean quarterly messages increased from 340 to 695 (p < 0.001) between Q4 and Q12; mean quarterly outpatient visits fell from 711 to 575 (p = 0.005). While time spent on the EHR outside of scheduled hours increased modestly for all clinicians, this did not significantly differ by message group. Across all clinicians, each additional 10 messages was associated with an average of 12 min per quarter of additional time spent on the EHR (p < 0.001).
Conclusions
Message volume increased substantially over the study period and varied by group. While messages were associated with additional time spent on the EHR outside of scheduled hours, there was no significant difference in time spent on the EHR between the high and low message volume groups.
INTRODUCTION
Online patient portals (e.g., MyChart) allow patients to securely message their clinicians asynchronously and use has proliferated over the last decade.1 While use of online portals is associated with enhanced quality of care via improved patient self-management,2 clinicians are typically not reimbursed for offering their expertise in this format. Thus, responding to messages is not routinely built into their clinical schedule,2 causing them to have to use personal or unscheduled time to respond.
Across major health systems, clinical demands are increasing as primary care clinician shortages mount.3 Concurrently, the percentage of messages clinicians respond to after-hours has increased,4,5 contributing to burnout.6 In a qualitative study of internists and family physicians, respondents reported that the requirement to respond to messages quickly while keeping up with a busy clinic schedule eroded work-life boundaries and increased anxiety.7 This burden is not felt equally, with women physicians receiving more medical advice messages8 and spending more time on the electronic health record (EHR) than their men colleagues.9 Given the convenience of secure messaging to patients,10 it is likely that message volume will continue to increase,11–13 requiring healthcare systems to find solutions to help clinicians adapt.14
Clinicians in primary care specialties spend more time on the EHR both overall and outside of clinical hours than their specialty counterparts.15 As health systems develop new policies to mitigate the burden of increasing patient portal messaging on clinician time (e.g., billing for certain patient portal messages), understanding which clinicians receive the most messages and how much additional time they spend working outside of clinical hours is important. The objective of this study was to describe how patient portal message volume varied across clinicians and over time, and to assess the association between portal message volume and time spent by clinicians on the EHR outside of scheduled hours.
METHODS
This is a retrospective cohort study of clinicians using data from the Cleveland Clinic Health System. Clinicians included physicians (Internal and Family Medicine) and advanced practice providers (APPs) who worked at least 0.8 full time equivalent (FTE) in our health system. We measured number of patient-initiated messages for medical advice between January 2019 and December 2021 from patients who had an outpatient visit during the study period. This study was approved by Cleveland Clinic’s Institutional Review Board.
Measures
When sending messages via the portal, patients click a box to indicate the purpose of the message (e.g., refill request, appointment request, or medical advice message). The independent measure was quarterly number of patient portal messages where the patient indicated they were requesting medical advice. Per institutional policy, clinicians are expected to respond within three business days. All messages are initially sent to the nurse pool where it is determined whether it should be handled by a nurse (e.g., redirecting patient to the emergency department) or should be handled by the clinician, at which point the nurse forwards the message to the clinician’s inbox. We operationalized portal message volume as total number of messages requesting medical advice received per quarter per clinician. To understand the relationship between quarterly number of medical advice messages (i.e., message volume) and in-person patient contact, we also collected each clinician’s quarterly number of outpatient visits. To understand whether medical advice message volume mirrored volume of other patient interactions, we also identified the number of telephone encounters and number of refill requests per clinician per quarter.
The dependent measure was clinician time spent on the EHR outside of scheduled working hours, which are the hours for which they are compensated by the health system. We measured this by combining results from Epic’s time outside scheduled hours and time on unscheduled days measures. Unscheduled time is defined as pre-work mornings and post-work evenings (with a 30-min buffer on each side), and working on EPIC on days without scheduled clinical time, including weekends.16
We collected clinician characteristics, including gender, years in practice, type (physician or APP), and specialty for physicians (Internal versus Family Medicine). At the clinician level, we calculated the following panel characteristics: proportion of panel that was female, proportion with Medicaid insurance, mean age, in years, and mean age-adjusted Charlson score, to account for medical complexity of each clinician’s panel.17 We included proportion of the panel that was white race, as research has found different patterns of secure message use by race.18 Both patient race and sex were determined via the EHR.
Statistical Analysis
We categorized clinicians into groups using group-based trajectory modeling, a type of general growth curve modeling.19 The purpose of this approach is to categorize individuals into groups based on similar trajectories of an outcome, in this case, number of medical advice messages received over time. This approach allows for identification of meaningful subgroups that follow unique trajectories that are not identifiable a priori using known characteristics (i.e., gender). In contrast, most studies on changes in incoming message burden have focused on differences by clinician characteristics.9,15,20 By using group-based trajectory modeling, we can observe latent classes of clinicians according to their incoming message volume, irrespective of other characteristics. To categorize physicians into groups, we used data on incoming medical advice messages across the full 12-month study period.
Variation Across Providers
We described mean number of quarterly medical advice messages received, and clinician and panel characteristics, by clinician message group. We assessed differences by group using the chi-squared statistic or ANOVA. We then used mixed effects multivariable linear regression to model differences by message group in the quarterly (1) number of messages received, (2) time spent on the EHR outside of scheduled hours, (3) number of outpatient visits, (4) number of telephone encounters, and (5) number of refill requests, controlling for clinician and panel characteristics, including time as a random effect. From each of these models, we generated marginal effects for clinician message group and their associated 95% confidence intervals.
Changes over Time
We reported change in key measures by comparing quarter 4 (Q4, October–December 2019) and quarter 12 (Q12, October–December 2021) using ANOVA. We selected these two quarters because our study period included the initial part of the COVID-19 pandemic. Hence, we selected one time point prior to the beginning of the pandemic (Q4) and the other during a time when outpatient volume had returned to approximately pre-pandemic levels (Q12). This also allowed us to account for seasonality effects, as Q4 and Q12 both occurred during October–December.
Time Spent on the EHR Outside of Scheduled Hours by Message Group
We used mixed effects multivariable linear regression to describe the adjusted association between clinician message group and time spent on the EHR outside of scheduled hours, including number of outpatient visits, number of telephone encounters, and clinician and panel characteristics as fixed effects, and time as a random effect. We then ran this model among clinicians overall, irrespective of messaging group status. To understand if there were significant differences in change in time spent on the EHR by message group, we ran a model including an interaction between message group and time.
Correlation Between Change in Message Volume and Change in Outpatient Visits
Since messages might substitute for outpatient visits, we generated a Pearson correlation coefficient to assess whether a reduction in outpatient visits was correlated with an increase in messages across the entire study period.
All analyses were conducted in R Studio. We accounted for the potential impact of the COVID-19 pandemic by including quarterly time as a random effect in all models.
RESULTS
The sample included 150 clinicians and 325,080 patients during the study period. Fifty-seven percent (n = 85) of clinicians were men, and 54% (n = 81) specialized in Internal Medicine (Table 1).
Table 1.
Sample Characteristics, Overall and by Clinician Message Volume Group
| Overall | Low | Moderate | High | p value | |
|---|---|---|---|---|---|
| N(%) | 150(100%) | 46(31%) | 71(47%) | 33(22%) | |
| Quarterly medical advice messages, mean [IQR] | 487 [301, 653] | 206 [125, 288] | 505 [437, 582] | 840 [767, 926] | < 0.001 |
| Clinician characteristics | |||||
| Gender, N(%) | |||||
| Woman | 65(43%) | 19(41%) | 31(44%) | 15(46%) | 0.932 |
| Man | 85(57%) | 27(59%) | 40(56%) | 18(55%) | |
| Specialty and clinician type, N(%) | |||||
| Internal Medicine | 81(54%) | 18(39%) | 40(56%) | 19(58%) | 0.334 |
| Family Medicine | 59(39%) | 22(48%) | 28(39%) | 13(39%) | |
| APP | 10(7%) | 6(13%) | 3(4%) | 1(3%) | |
| Years of service, mean [IQR] | 15 [8, 23] | 13 [6, 19] | 16 [9, 24] | 16 [10, 22] | 0.023 |
| Full time equivalent (FTE), mean [IQR] | 0.96 [0.9, 1] | 0.96 [1, 1] | 0.96 [0.95, 1] | 0.95 [0.9, 1] | 0.842 |
| Patient characteristics | |||||
| Average age, mean [IQR] | 58 [56, 63] | 55.6 [52, 61] | 59.1 [56, 62] | 60.3 [56, 64] | 0.028 |
| Proportion female, mean [IQR] | 0.55 [0.51, 0.73] | 0.54 [0.47, 0.65] | 0.56 [0.41, 0.73] | 0.55 [0.38, 0.75] | 0.893 |
| Proportion white, mean [IQR] | 0.79 [0.75, 0.92] | 0.68 [0.55, 0.88] | 0.81 [0.77, 0.90] | 0.89 [0.87, 0.94] | < 0.001 |
| Proportion Medicaid insurance, mean [IQR] | 0.24 [0.18, 0.29] | 0.28 [0.19, 0.33] | 0.23 [0.18, 0.29] | 0.19 [0.16, 0.20] | < 0.001 |
| Average age-adjusted Charlson score, mean [IQR] | 4.9 [4.4, 5.7] | 4.6 [3.6, 5.7] | 4.9 [4.4, 5.6] | 5.1 [4.5, 5.8] | 0.15 |
IQR, interquartile range
Variation Across Clinicians
The group-based trajectory modeling revealed three distinct groups of clinicians (Table 1). Thirty-one percent (n = 46) were in the low-volume group, 47% (n = 71) were in the moderate-volume group, and 22% (n = 33) were in the high-volume group. Clinicians in the high-volume group received an average of 840 quarterly messages, compared to 206 messages received in the low-volume group (p < 0.001). Table 1 presents bivariate associations between message group assignment and clinician and panel characteristics.
In the adjusted analyses of differences between message groups (Table 2), clinicians in the high-volume group received, on average, 847 messages per quarter (95%CI: 759–936), compared to 529 in the moderate-volume group (95%CI: 441–617) and 262 in the low-volume group (95%CI: 174–351). Similarly, there were significant differences between the three messaging groups with respect to quarterly refill requests. There were no significant differences between the messaging groups with respect to time spent on the EHR outside of scheduled hours, number of quarterly outpatient visits, or number of quarterly telephone encounters.
Table 2.
Adjusted Utilization Measures by Clinician Messaging Group, Regression Estimates and Marginal Effects*
| Regression estimates | Marginal effects | |||
|---|---|---|---|---|
| Est | 95%CI | Est | 95%CI | |
| Medical advice messages | ||||
| Low | Ref | 262.2 | 173.5–350.9 | |
| Moderate | 266.8 | 251.9–281.8 | 529.1 | 440.9–617.3 |
| High | 585.3 | 566.6–604.9 | 847.2 | 758.5–936.2 |
| Outside hours worked | ||||
| Low | Ref | 51.7 | 42.6–60.8 | |
| Moderate | 3.8 | − 1.4 to 9.0 | 55.5 | 47.0–64.0 |
| High | 5.2 | − 1.3 to 11.7 | 56.9 | 47.9–66.0 |
| Outpatient visits | ||||
| Low | Ref | 449.7 | 336.1–563.4 | |
| Moderate | 72.0 | 50.3–93.7 | 521.7 | 408.9–634.6 |
| High | 105.8 | 78.7–132.9 | 555.6 | 442.0–669.1 |
| Telephone encounters | ||||
| Low | Ref | 719.4 | 651.3–787.4 | |
| Moderate | − 32.5 | − 79.4 to 14.4 | 686.9 | 625.4–748.4 |
| High | 42.4 | − 16.3 to 101.1 | 761.8 | 695.4–829.1 |
| Refill requests | ||||
| Low | Ref | 118.2–195.0 | 642.3 | 581.2–703.4 |
| Moderate | 156.6 | 295.6–391.7 | 798.9 | 742.7–855.1 |
| High | 343.7 | 134.5–259.6 | 985.9 | 925.4–1046.6 |
*For the regression estimates, the association is statistically significant when the 95%CI does not cross zero. For the marginal effect estimates, the association is statistically significant when the 95%CIs do not overlap
Changes over Time
Quarterly mean number of messages increased from 340 to 695 (p < 0.001) between Q4 and Q12, while outpatient volume fell from 711 to 575 (p = 0.005) (Table 3). Quarterly number of telephone encounters also declined from 910 to 801 (p < 0.001), while refill requests increased modestly (p = 0.030). Time spent on the EHR outside of scheduled hours increased from 53 h per quarter to 58 h per quarter (p = 0.017).
Table 3.
Change over Time by Clinician Message Group, per Quarter
| Message group | Quarter 4 | Quarter 12 | Change | p value | |
|---|---|---|---|---|---|
| Medical advice messages | Low | 137.7 | 339.6 | 201.9 | < 0.001 |
| Moderate | 338.4 | 702.6 | 364.3 | ||
| High | 605.6 | 1101.1 | 495.5 | ||
| Total | 339.7 | 695.0 | 355.3 | < 0.001 | |
| Outside hours on the EHR | Low | 49.4 | 51.3 | 1.9 | 0.536 |
| Moderate | 51.6 | 57.3 | 5.8 | ||
| High | 60.8 | 67.5 | 6.7 | ||
| Total | 53.0 | 58.0 | 5.0 | 0.017 | |
| Outpatient encounters | Low | 582.5 | 518.3 | − 64.2 | 0.005 |
| Moderate | 743.2 | 592.7 | − 150.5 | ||
| High | 811.5 | 604.5 | − 207.0 | ||
| Total | 711.3 | 574.7 | − 136.6 | < 0.001 | |
| Telephone encounters | Low | 847.4 | 845.3 | − 2.1 | 0.002 |
| Moderate | 897.3 | 767.7 | − 129.6 | ||
| High | 1020.1 | 819.3 | − 200.8 | ||
| Total | 910.4 | 801.5 | − 108.8 | < 0.001 | |
| Refill requests | Low | 683.5 | 769.6 | 86.1 | 0.692 |
| Moderate | 866.4 | 902.6 | 36.3 | ||
| High | 1058.9 | 1093.4 | 34.5 | ||
| Total | 856.0 | 910.3 | 54.3 | 0.030 |
Change in number of quarterly messages varied by group status, with those in the low-volume group experiencing a more modest increase in messaging from Q4 to Q12 (∆202) compared to those in the high-volume group (∆495) (p < 0.001). Change in quarterly number of outpatient visits over time varied by group, with those in the low-volume group having the smallest decline in visits (∆-64) and those in the high-volume group having the greatest decline (∆-207) (p < 0.001).
Time Spent on the EHR Outside of Scheduled Hours by Messaging Group
In the multivariable mixed effects linear regression model, controlling for clinician and panel characteristics and other utilization measures (Table 4), there was no significant difference by message group in amount of time spent on the EHR outside of scheduled hours. Among clinicians overall (Table 5), each additional 10 messages received was associated with 12 min additional time spent on the EHR outside of scheduled hours per quarter (95%CI: 6–12 min).
Table 4.
Mixed Effects Linear Regression, Differences in per-Quarter Hours Spent on the EHR Outside of Scheduled Hours, by Clinician Message Group
| Est | p value | |
|---|---|---|
| Message group | ||
| Low | Ref | |
| Moderate | 4.5 | 0.099 |
| High | 5.6 | 0.097 |
| Quarterly outpatient visits | − 0.01 | 0.293 |
| Quarterly telephone encounters | 0.01 | 0.028 |
| Specialty | ||
| Internal Medicine | Ref | |
| Family Medicine | 0.45 | 0.864 |
| APP | 26.4 | < 0.001 |
| Clinician gender | ||
| Woman | Ref | |
| Man | − 8.8 | 0.051 |
| Years of service | − 0.41 | 0.003 |
| FTE | − 16.6 | 0.312 |
| Mean patient age | 0.18 | 0.689 |
| Mean percent female | − 0.07 | 0.598 |
| Mean percent white | − 0.2 | 0.001 |
| Mean percent Medicaid | − 0.5 | < 0.001 |
| Mean age-adjusted Charlson | 9.7 | 0.001 |
Table 5.
Mixed Effects Linear Regression, Differences in Per-Quarter Hours Spent on the EHR Outside of Scheduled Hours, Overall
| Est | p value | |
|---|---|---|
| Per 10 messages | 0.20 | 0.001 |
| Quarterly outpatient visits | − 0.01 | 0.236 |
| Quarterly telephone encounters | 0.01 | 0.047 |
| Specialty | ||
| Internal Medicine | Ref | |
| Family Medicine | 0.44 | 0.865 |
| APP | 28.41 | < 0.001 |
| Clinician gender | ||
| Woman | Ref | |
| Man | − 7.65 | 0.090 |
| Years of service | − 0.38 | 0.006 |
| FTE | − 14.28 | 0.382 |
| Mean patient age | 0.23 | 0.594 |
| Mean percent female | − 0.05 | 0.675 |
| Mean percent white | − 0.26 | < 0.001 |
| Mean percent Medicaid | − 0.44 | 0.001 |
| Mean age-adjusted Charlson | 9.21 | 0.001 |
Between Q4 and Q12, the time spent on the EHR outside of scheduled hours increased significantly for all messaging groups, by 2.8 h for the high-volume group (95%CI: 1.52–4.13), by 1.5 h for the moderate-volume group (95%CI: 0.55–2.50), and by 2.0 h for the low-volume group (95%CI: 1.10–2.91). There was no significant interaction between message group and time spent on the EHR outside of scheduled hours.
Correlation Between Change in Message Volume and Change in Outpatient Visits
Across all clinicians, there was no association between the change in outpatient visits and change in message volume between Q4 and Q12. For the high-volume group, there was a moderate correlation (0.47, p = 0.007): clinicians who had a greater decrease in visits between Q4 and Q12 had a smaller increase in message volume. Clinicians in the moderate-volume group had a similar trend, but the correlation was weaker (0.26, p = 0.037). There was no significant association in the low-volume group.
DISCUSSION
Our study of primary care clinicians and their patients in a major health system found that mean number of medical advice messages received per quarter varied substantially across clinicians. Compared to clinicians in the low-volume group, clinicians in the high-volume group received over three times more messages, despite their outpatient volume only being 24% higher. Even so, there was no significant difference in the amount of time clinicians spent on the EHR outside of clinical hours by group, and while time spent on the EHR outside of scheduled hours rose modestly for clinicians in all three messaging groups, this change was similar across groups. We did not find evidence of a substitution effect between messages and outpatient visits; rather we found a modest positive correlation between messages and outpatient visits, indicating that visits appeared to generate messages rather than messages replacing visits.
The increase in message volume over the study period was large while the change in time spent on the EHR outside of scheduled hours was not. Irrespective of the number of messages clinicians received, the time spent on the EHR outside of scheduled hours was similar across groups. One possibility is that clinicians who receive the most messages may be more efficient at responding to them. Clinicians who are good at handling high volumes of portal messages also may encourage their patients to communicate with them via the portal, essentially self-selecting into the high-volume group. There is extensive literature on patient preferences for messages and portal use.21–25 Given that communication is bi-directional, a better understanding of clinician preferences regarding secure messaging and how they convey this preference to patients is needed. Indeed, a recent editorial emphasized the need for clinicians to set boundaries and expectations with patients regarding appropriate portal communication in to mitigate burnout.26 Finally, it is also possible that clinicians who receive fewer portal messages simply spend their time doing other things in the EHR, like documentation. Clinicians may allocate a fixed amount of time to work in the EHR after scheduled hours, which is why we found no difference between message volume groups.
Our study is the first in an adult patient population to use group-based trajectory modeling to identify distinct trajectories of clinicians based on trends in their incoming medical advice messages. As noted in the “Methods,” this approach is useful in identifying latent classes of clinicians based on experiences rather than characteristics. While much recent scholarship on patient messaging and EHR documentation focuses on characteristics of clinicians who receive the most messages, and who then spend disproportionate time on the EHR, our approach identified a unique group: low message volume clinicians. While these clinicians have the same clinical FTE and only slightly lower outpatient volume than clinicians in the high-volume group, they received only a quarter of the messages of the high-volume group. Some of this could be explained by differences in panel characteristics, as clinicians in the low-volume group had a higher proportion of Medicaid patients and a lower proportion of white patients.27,28 Differences in panel characteristics are likely only one explanation for the difference in messaging volume. Like the high-volume group, clinicians in the low-volume group may self-select by not encouraging their patients to contact them via the portal.
Unlike a number of other studies that have found women clinicians spent more time than their men colleagues on the EHR,5,8,9,20 we did not. One explanation for our lack of a gender effect in our study is clinicians in our study had much higher clinical loads than in these other studies. The average clinical FTE for both women and men clinicians in our study was 0.96, whereas for the Rule study5, it was 0.64 (for both genders), and 0.57 and 0.52, for women and men clinicians, respectively, in the Rittenberg study.8 Perhaps gender differences are more evident when clinicians have the option of using research or administrative time for documentation or messaging, with women and men clinicians choosing to spend this time differently. Given the relatively low clinical FTEs of clinicians in other studies, they likely reflect academic practices, whereas our study is reflective of the more generalizable setting of community-based care.
To our knowledge, ours is the first study to document differences in time spent on the EHR outside of scheduled hours by APPs versus physicians. APPs in our study were primarily in the low-message volume group yet spent considerably more time than physicians on the EHR outside of scheduled hours. One possibility is that after having patient messages forwarded to them via the nurse pool, physicians then forwarded some messages onward for APPs to handle. While not the primary focus of our study, this finding points to the need for further investigation into workload implications of increasing in-basket burden across all members of the care team.
Our study had limitations. Time spent on the EHR outside of scheduled hours was not specific to answering messages and may have included other tasks like documentation. It is unknown whether clinicians responded to messages themselves or whether they were handled by the nurse pool. This is typically determined by the complexity of the request. Given there was no difference in age-adjusted patient Charlson scores across the groups, we have no reason to believe that the level of message complexity varied by group, and therefore it is unlikely that one group’s messages was disproportionately handled by the nurse pool. Patient demographics associated with message volume are likely surrogates for socioeconomic status, which we were unable to directly measure. We were unable to validate the content of the medical advice messages and it is possible some messages were for other purposes. Consistent with their scope of practice, Family Medicine physicians cared for some pediatric patients over the study period (9% of their patients). We did not exclude this small group of patients from their records for analysis. Our study includes a relatively small number of clinicians at a single health system, limiting generalizability. Finally, definitions of key variables (e.g., time spent on the EHR outside of scheduled hours) may be specific to our health system, further limiting generalizability.
In our study using group-based trajectory modeling, we were able to identify three distinct groups of clinicians based on their incoming messaging volume over time. While incoming medical advice messages increased substantially over the study period for all groups, clinicians in the high-volume group received three times more messages from patients per quarter than clinicians in the low-volume group, without a commensurate difference in outpatient volume. Time spent on the EHR outside of scheduled hours increased modestly for all clinicians, irrespective of incoming message volume. Finally, we found a modest positive association between outpatient volume and messages, wherein those with the smallest reduction in outpatient volume over time experienced greater increases in messages. While some clinicians may prefer for patients to contact them using patient portals, health systems should monitor inbox volume, particularly as outpatient visit volume returns to pre-pandemic levels.
Data Availability
The full data from this study is not publicly available, however certain data elements may be shared with outside parties following execution of a data use agreement.
Declarations
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Footnotes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Individuals’ Access and Use of Patient Portals and Smartphone Health Apps, 2020 | HealthIT.gov. https://www.healthit.gov/data/data-briefs/individuals-access-and-use-patient-portals-and-smartphone-health-apps-2020. Accessed 22 February 2023.
- 2.Kruse CS, Bolton K, Freriks G. The Effect of Patient Portals on Quality Outcomes and Its Implications to Meaningful Use: A Systematic Review. J Med Internet Res. 2015;17(2):e44. doi: 10.2196/jmir.3171. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.The Complexities of Physician Supply and Demand: Projections from 2016 to 2030. Association of American Medical Colleges.
- 4.North F, Luhman KE, Mallmann EA, et al. A Retrospective Analysis of Provider-to-Patient Secure Messages: How Much Are They Increasing, Who Is Doing the Work, and Is the Work Happening After Hours? JMIR Med Inform. 2020;8(7):e16521. doi: 10.2196/16521. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Rule A, Shafer CM, Micek MA, Baltus JJ, Sinsky CA, Arndt BG. Gender Differences in Primary Care Physicians’ Electronic Health Record Use over Time: an Observational Study. J Gen Intern Med. Published online December 6, 2022. 10.1007/s11606-022-07837-2. [DOI] [PMC free article] [PubMed]
- 6.Tran B, Lenhart A, Ross R, Dorr DA. Burnout and EHR use among academic primary care physicians with varied clinical workloads. AMIA Jt Summits Transl Sci Proc. 2019;2019:136–144. [PMC free article] [PubMed] [Google Scholar]
- 7.Lieu TA, Freed GL. Unbounded—Parent-Physician Communication in the Era of Portal Messaging. JAMA Pediatr. 2019;173(9):811. doi: 10.1001/jamapediatrics.2019.2309. [DOI] [PubMed] [Google Scholar]
- 8.Rittenberg E, Liebman JB, Rexrode KM. Primary Care Physician Gender and Electronic Health Record Workload. J Gen Intern Med. 2022;37(13):3295–3301. doi: 10.1007/s11606-021-07298-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Rotenstein LS, Fong AS, Jeffery MM, et al. Gender Differences in Time Spent on Documentation and the Electronic Health Record in a Large Ambulatory Network. JAMA Network Open. 2022;5(3):e223935. doi: 10.1001/jamanetworkopen.2022.3935. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Hoonakker PLT, Carayon P, Cartmill RS. The impact of secure messaging on workflow in primary care: Results of a multiple-case, multiple-method study. Int J Med Inform. 2017;100:63–76. doi: 10.1016/j.ijmedinf.2017.01.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Crotty BH, Tamrat Y, Mostaghimi A, Safran C, Landon BE. Patient-to-physician messaging: volume nearly tripled as more patients joined system, but per capita rate plateaued. Health Aff (Millwood). 2014;33(10):1817–1822. doi: 10.1377/hlthaff.2013.1145. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Holmgren AJ, Downing NL, Tang M, Sharp C, Longhurst C, Huckman RS. Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use. J Am Med Inform Assoc. 2022;29(3):453–460. doi: 10.1093/jamia/ocab268. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Nath B, Williams B, Jeffery MM, et al. Trends in Electronic Health Record Inbox Messaging During the COVID-19 Pandemic in an Ambulatory Practice Network in New England. JAMA Netw Open. 2021;4(10):e2131490. doi: 10.1001/jamanetworkopen.2021.31490. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Lieu TA, Altschuler A, Weiner JZ, et al. Primary Care Physicians’ Experiences With and Strategies for Managing Electronic Messages. JAMA Netw Open. 2019;2(12):e1918287. doi: 10.1001/jamanetworkopen.2019.18287. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Rotenstein LS, Holmgren AJ, Downing NL, Bates DW. Differences in Total and After-hours Electronic Health Record Time Across Ambulatory Specialties. JAMA Intern Med. 2021;181(6):863–865. doi: 10.1001/jamainternmed.2021.0256. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Arndt BG, Micek MA, Rule A, Shafer CM, Baltus JJ, Sinsky CA. Refining Vendor-Defined Measures to Accurately Quantify EHR Workload Outside Time Scheduled With Patients. Ann Fam Med. 2023;21(3):264–268. doi: 10.1370/afm.2960. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.D’Hoore W, Bouckaert A, Tilquin C. Practical considerations on the use of the Charlson comorbidity index with administrative data bases. J Clin Epidemiol. 1996;49(12):1429–1433. doi: 10.1016/s0895-4356(96)00271-5. [DOI] [PubMed] [Google Scholar]
- 18.Heisey-Grove DM, Carretta HJ. Disparities in Secure Messaging Uptake Between Patients and Physicians: Longitudinal Analysis of Two National Cross-Sectional Surveys. J Med Internet Res. 2020;22(5):e12611. doi: 10.2196/12611. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Nagin DS. Group-based trajectory modeling: an overview. Ann Nutr Metab. 2014;65(2–3):205–210. doi: 10.1159/000360229. [DOI] [PubMed] [Google Scholar]
- 20.Rotenstein L, Jay Holmgren A. COVID exacerbated the gender disparity in physician electronic health record inbox burden. J Am Med Inform Assoc. Published online July 20, 2023:ocad141. 10.1093/jamia/ocad141. [DOI] [PMC free article] [PubMed]
- 21.Palen TE, Ross C, Powers JD, Xu S. Association of Online Patient Access to Clinicians and Medical Records With Use of Clinical Services. JAMA. 2012;308(19):2012. doi: 10.1001/jama.2012.14126. [DOI] [PubMed] [Google Scholar]
- 22.Yamin CK, Emani S, Williams DH, et al. The Digital Divide in Adoption and Use of a Personal Health Record. Arch Intern Med. 2011;171(6). 10.1001/archinternmed.2011.34. [DOI] [PubMed]
- 23.Ancker JS, Hafeez B, Kaushal R. Socioeconomic disparities in adoption of personal health records over time. Am J Manag Care. 2016;22(8):539–540. [PMC free article] [PubMed] [Google Scholar]
- 24.Mackert M, Mabry-Flynn A, Champlin S, Donovan EE, Pounders K. Health Literacy and Health Information Technology Adoption: The Potential for a New Digital Divide. J Med Internet Res. 2016;18(10):e264. doi: 10.2196/jmir.6349. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Dudas RA, Pumilia JN, Crocetti M. Pediatric caregiver attitudes and technologic readiness toward electronic follow-up communication in an urban community emergency department. Telemed J E Health. 2013;19(6):493–496. doi: 10.1089/tmj.2012.0166. [DOI] [PubMed] [Google Scholar]
- 26.Stillman M. Death by Patient Portal. JAMA. 2023;330(3):223–224. doi: 10.1001/jama.2023.11629. [DOI] [PubMed] [Google Scholar]
- 27.Margolius D, Siff J, Teng K, Einstadter D, Gunzler D, Bolen S. Primary Care Physician Factors Associated with Inbox Message Volume. J Am Board Fam Med. 2020;33(3):460–462. doi: 10.3122/jabfm.2020.03.190360. [DOI] [PubMed] [Google Scholar]
- 28.Anthony DL, Campos-Castillo C, Lim PS. Who Isn’t Using Patient Portals And Why? Evidence And Implications From A National Sample Of US Adults. Health Aff (Millwood). 2018;37(12):1948–1954. doi: 10.1377/hlthaff.2018.05117. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The full data from this study is not publicly available, however certain data elements may be shared with outside parties following execution of a data use agreement.