Behavior Shifts in Patient Portal Usage During and After Policy Changes Around Test Result Delivery and Notification

Abstract

Because of the 21st Century Cures Act, many health systems now release all test results into patient portals immediately. To investigate if changes in access to test results shifted patient portal usage, we used data from the electronic health record to evaluate how patients behaved after this policy change and a subsequent policy adjustment requiring patients to opt in for notifications about new test results. We found that following institutional compliance with the Cures Act, proportions of patients who scheduled a new appointment and messaged their clinician after accessing a new test result increased, both by 4.5%. After removing automatic notifications of new results, the proportion of patients who scheduled a new appointment increased by 2.1%, and the proportion of patients who had telemedicine encounters decreased by 0.8%. Our work identified changes in patient behavior that track how policy changes map to burden for clinicians and information-seeking behavior in patients.

Introduction

The federal 21^st Century Cures Act (Cures Act) includes the Information Blocking Rule, which states that electronic healthcare information must be made available to patients for free and without delay – constituting a new status quo of immediate access¹. The idea of equipping patients with their health information began from a policy perspective in 1996, with the Health Insurance Portability and Accountability Act (HIPAA). HIPAA established a requirement that health systems make health information available to patients upon their request and when feasible. Nearly a decade later, The HITECH Act of 2009 secured provisions for patients to have access to their health information electronically². This contextualizes an integral reason for the Cures Act, which was to ensure that health systems could not skirt HITECH provision³. Though health information was previously released if a hospital system deemed it appropriate, based on patient safety and sensitivity of results, these qualifications around result release were then redefined as information blocking. Although it would be possible to achieve Cures Act compliance with a system that responded immediately to patient requests for data, many health systems achieve compliance by making all patient data immediately available in the patient portal where previously they may only have made a limited set of data immediately available. As a result, patients now may access their test results before their clinician does.⁴

Our health system, Vanderbilt University Medical Center (VUMC), entered compliance with the Information Blocking Rule on January 20^th, 2021, by making all new health information for adult patients released to the electronic health record (EHR) available in the patient portal immediately. In our prior research, this change was associated with an uptick in both the proportion of patients reviewing their test results prior to their clinician (11.5% vs. 45.6%) and who sent a secure message via the patient portal to clinicians within 72 hours of viewing their test results (12.4% to 15.4%)^5,6. Improved information availability offers benefits to patients and clinicians⁷, but patients and clinicians have varied preferences about how and when to receive their test results^8,9. The majority of patients indicate that they would prefer to have unfettered access to their health information, such as the 86% surveyed by Rauchman et al. and the 96% of patients surveyed by Steitz et al. who appreciated immediate access to their test results. However, some have elevated concerns about releasing all health information to patients without providing contextual information such as via counseling by a clinician^10,11,12. There is still a contingent of clinicians, such as the 87% surveyed out of 82 by Winget et al. who prefer to review results with patients and make medical decisions in tandem, especially if the test result is abnormal^13,14.

An added complexity is that while the Information Blocking Rule requires that health systems make electronic health information available to patients upon their request, there is no requirement that patients be notified when that information is available. Commonly, commercial patient portals send notifications to patients every time a new test result is available. This notification behavior may influence patients who intended to wait to discuss a new result with their clinician instead to go ahead and look at it before the clinician has had a chance to see it. Based on conversations with patient advisors, health system leadership, and clinician stakeholders, on April 5, 2021, VUMC turned off default notifications to patients when new test results were available in the patient portal. With this policy, all patients were required to opt in to notifications about newly delivered test results, even if they had previously enrolled in automated notifications. As part of the policy change, patients were sent an email that detailed the new policy and provided detailed instructions about how to adjust notification settings. Patients who did not opt in to notifications would continue to receive the test results via the portal but would not get notified when they arrived.

In this study, we quantified the effects of both these policy interventions on patient portal use. Measuring patient needs and information seeking in the wake of policy changes helps form a better understanding of how health systems can respond to future information sharing changes. Our addresses a gap in knowledge around how patients interact with the healthcare system with increased access to their electronic health information. To this end, the hypotheses of study include measurements of appointment scheduling, telehealth encounters, and messaging a clinician. Using the date of VUMC’s compliance with the Cures Act and the date of the subsequent notification policy change as distinct points in time where portal practices changed, we took advantage of a natural experiment to answer these questions.

To that end, we hypothesized that at VUMC and within 3 days of reviewing new test results in the patient portal following policy changes related to the Cures Act and default notification practices

1. There was a significant change in the proportion of patients who scheduled a new clinical appointment.

2. There was a significant change in the proportion of patients who had a telemedicine encounter.

3. There was a significant change in the proportion of patients who sent a message to their clinician.

Methods

Study Site, Population, and Design

VUMC is a large academic medical center located in Nashville, Tennessee, providing inpatient, emergency, primary, and referral care to patients from throughout the Southeastern United States. Overall, VUMC cares for more than 3 million patients annually and includes many specialized care centers for cancer, diabetes, transplants, children’s care, and more¹⁵. Adult patients were eligible to be a part of our study if they had an active portal account and received a test result at any point between June 1, 2020, and January 1, 2022. This study was reviewed and approved with a waiver of consent by the Vanderbilt University Institutional Review Board.

Data Sources

We retrieved the EHR log file data about each released test result during our study period, including time of result release, time that the result was viewed by the clinician, the time that the result was viewed by the patient, the type of test, and whether the result was flagged as abnormal. We also retrieved EHR data characterizing patient demographics at the time of each test result release, including sex assigned at birth, ethnicity, age, and primary language.

The unit of analysis for our study was each patient, counted uniquely per week. Patients were deemed to have accessed their tests prior to their clinician if the time stamp of test review from the log files was prior to the time stamp of the clinician review. We extracted log files of a new appointment scheduled for a patient, a patient having a telemedicine encounter, and a patient sending a message to a clinician. We considered the events related to the test result release if they occurred within 3 days. These measures were all derived using the time that a result was viewed by a patient in the portal and the time at which a patient performed one of these behaviors as endpoints. We counted considered appointments scheduled, telemedicine encounters, and messages sent to clinicians to be potentially related to the test result view if they occurred within 3 days of the view. Additionally, VUMC policy requires that clinicians respond to patient messages within 2 business days, which is encapsulated during our study window.

Statistical Methods

We conducted an interrupted time series (ITS) analysis to evaluate the changes in proportions of various patient behaviors following Cures Act compliance and the subsequent opt-in notification policy change¹⁶. The time cut points were based on the VUMC compliance with Cures Act on January 20, 2021, and the adjustment of the notification policy on April 15, 2021. We leveraged segmented logistic regression models to estimate these changes and adjusted the model with the inclusion of time if there was a meaningful interaction between time and any of the rest of the covariates. The linear model with time in weeks is:

$$\widehat{Y_{it}}={\beta }_{1}tim{e}_t+{\beta }_{2}interruption{1}_t+{\beta }_{3}interruption{2}_t+{\beta }_4\left(tim{e}_t\ast interruption{1}_t\right)+{\beta }_5\left(tim{e}_t\ast interruption{2}_t\right)+{\text{ε}}_{it}$$

Interruption 1 and interruption 2 represent binary variables that are toggled from 0 to 1 if the associated timepoint falls after the window of intervention for Cures Act compliance and the notification policy changes, respectively. Our model evaluates $\widehat{Y_{it}}$ as the proportion represented at any given time of the unique number of patients exhibiting the behavior of note (scheduling of new appointments, having a telemedicine encounter, or messaging their clinician) divided by the total number of unique patients who received a test result within that timeframe.

Test abnormality was defined, in binary format, as laboratory results that were outside of the standard reference range for each test. Language, sex, race, and ethnicity were self-identified by patients. Insurance data is determined by which type of payor institution was used to fund the visit at which the test result originated. To ascertain differences in viewing behaviors across characteristics of patients, we performed bivariate analyses using chi-squared testing.

Results

Our analytic sample included 351,111 unique patients, with 143,579 patients viewing their results prior to their clinician and 207,532 viewing their results after their clinician. Demographics and descriptive values about our study sample are shown in Table 1. Table 1 also shows that abnormal test results were more likely to be viewed first by patients, and that privately insured patients were most likely to engage in patient-before-clinician review. Test results for patients insured by Medicare were more likely to be viewed by the clinician first.

Table 1.

Unique patients described by the data present in the EHR at their most recent test receipt, segmented by patient-first or clinician-first test review.

	Total N (%)N = 351111	Patient First View N (%)N = 143579 (40.9%)	Clinician First View N (%)N = 207532 (59.1%)	P Value
Test Abnormality				<0.001
Yes	63586 (18.1%)	27654 (19.3%)	35932 (17.3%)
No	287525 (81.9%)	115925 (80.7%)	171600 (82.7%)
Language				<0.001
English	346608 (98.7%)	141957 (98.9%)	204651 (98.6%)
Spanish	1914 (0.5%)	640 (0.4%)	1274 (0.6%)
Arabic	580 (0.2%)	244 (0.2%)	336 (0.2%)
Other	2009 (0.6%)	738 (0.5%)	1271 (0.6%)
Sex				<0.001
Female	222774 (63.4%)	91667 (63.8%)	131107 (63.2%)
Male	128328 (36.5%)	51905 (36.2%)	76423 (36.8%)
Unknown	9 (0.0%)	7 (0.0%)	2 (0.0%)
Insurance				<0.001
Private	232258 (66.1%)	102927 (71.7%)	129331 (62.3%)
Medicare	44479 (12.7%)	15180 (10.6%)	29299 (14.1%)
Medicaid	17645 (5.0%)	6823 (4.8%)	10822 (5.2%)
Workers’ Comp.	737 (0.2%)	237 (0.2%)	500 (0.2%)
Other Government	36251 (10.3%)	13384 (9.3%)	22867 (11.0%)
Other	19741 (5.6%)	5028 (3.5%)	14713 (7.1%)
Race				< 0.001
American Indian or Alaska Native	1035 (0.3%)	389 (0.3%)	646 (0.3%)
Asian	3818 (1.1%)	1612 (1.1%)	2206 (1.1%)
Black or African-American	33475 (9.5%)	12362 (8.6%)	21113 (10.2%)
Other Races	6663 (1.9%)	2646 (1.8%)	4017 (1.9%)
Prefer Not to Answer	41247 (11.7%)	18857 (13.1%)	22390 (10.8%)
White	264873 (75.4%)	107713 (75.0%)	157160 (75.7%)
Ethnic Group				< 0.001
Decline to Answer	25875 (7.4%)	9704 (6.8%)	16171 (7.8%)
Hispanic or Latino	13104 (3.7%)	5344 (3.7%)	7760 (3.7%)
Unable to Provide	312132 (88.9%)	128531 (89.5%)	183601 (88.5%)

The proportion of all patients who scheduled a new appointment within 3 days of receiving their test results increased by 4.5% at the time of Cures Act compliance (p<0.001) and increased by 2.1% at the notification policy change (p=0.037) (Table 2, Figure 1a). The rates at which proportions of patients were scheduling these appointments did not meaningfully change after Cures Act compliance or the notification policy change. Among patients who reviewed their results prior to their clinician, we observed a 5.9% increase in the proportion of patients scheduling new appointments after the implementation of Cures Act compliance (p<0.001) and a 0.4% increase after the notification policy change (p=0.004) (Table 2, Figure 1b). There were no statistically significant differences in the rate of the proportion of patients scheduling new appointments over these changes in portal practices. For patients who reviewed their results after their clinician, we observed an 5.8% increase in these patients scheduling new appointments after Cures Act compliance (p<0.001), and no changes in the rates of new appointment scheduling over time (Table 2, Figure 1c).

Table 2.

Changes in the proportion of patients who scheduled new appointments within 3 days of test result receipt

All Patients	Estimate	Standard Error	t Value	p Value
(Intercept)	0.168	0.005	31.623	<0.001
time	<0.001	<0.001	-0.117	0.907
interruption1	0.045	0.010	4.431	<0.001
time-interruption1 interaction	0.001	0.001	1.068	0.289
interruption2	0.021	0.010	2.128	0.037
time-interruption2 interaction	-0.002	0.001	-1.526	0.131
Adjusted R-squared: 0.7401
Patient-First Review Group
(Intercept)	0.118	0.007	18.160	<0.001
time	0.001	<0.001	2.401	0.019
interruption1	0.059	0.012	4.799	<0.001
time-interruption1interaction	0.001	0.002	0.541	0.590
interruption2	0.036	0.012	2.978	0.004
time-interruption2 interaction	-0.002	0.002	-1.423	0.159
Adjusted R-squared: 0.7495
Clinician-First Review Group
(Intercept)	0.159	0.005	34.835	<0.001
Time	<0.001	<0.001	-0.128	0.898
interruption1	0.058	0.009	6.669	<0.001
time-interruption1interaction	0.002	0.001	1.551	0.125
interruption2	0.010	0.008	1.226	0.224
time-interruption2 interaction	-0.002	0.001	-1.838	0.070
Adjusted R-squared: 0.8413

Figure 1a, 1b, 1c.

Proportions of patients scheduling new appointments after accessing their test results prior to their clinician or after their clinician across the time periods of Cures Act compliance and the notification policy change. Plot 1a shows the proportion of all patients scheduling new appointments. Plots 1b and 1c show the proportion of patients scheduling new appointments who reviewed their results prior to their clinician and after their clinician, respectively.

We observed a 0.8% decrease in the proportion of all patients having telemedicine encounters after the notification policy change (p<0.001). However, the rate of patients having telemedicine encounters changed less than 0.001% at the time of Cures Act compliance (p=0.008) and notification policy change (p=0.001) (Table 3, Figure 2a). There were no observed significant differences in either proportion or rate of patients having telemedicine encounters among only patients who reviewed their results prior to their clinician (Table 3, Figure 2b). For patients who reviewed their test results after their clinician, we observed a 0.9% decrease in the proportion who were having telemedicine encounters after the notification policy change (p<0.001) and changes in the rate of patients experiencing telemedicine encounters after Cures Act compliance and the notification policy change (Table 3, Figure 2c). Unlike other models presented here, the model built from the patient-first review segment for telemedicine encounters produced a low adjusted R-squared value, suggesting it was not a strong fit to the data.

Table 3.

Changes in the proportion of patients who had telemedicine encounters within 3 days

	Estimate	Standard Error	t Value	p Value
All Patients
(Intercept)	0.018	0.001	25.890	<0.001
time	<0.001	<0.001	-3.529	0.001
interruption1	-0.002	0.001	-1.241	0.218
time-interruption1 interaction	<0.001	<0.001	-2.714	0.008
interruption2	-0.008	0.001	-6.035	<0.001
time-interruption2 interaction	0.001	<0.001	3.438	0.001
Adjusted R-squared: 0.7536
Patient-First Review Group
(Intercept)	0.012	0.001	8.293	<0.001
time	<0.001	<0.001	0.860	0.393
interruption1	0.002	0.003	0.814	0.418
time-interruption1interaction	<0.001	<0.001	-1.265	0.210
interruption2	-0.005	0.003	-1.863	0.066
time-interruption2 interaction	<0.001	<0.001	1.070	0.288
Adjusted R-squared: 0.1414
Clinician-First Review Group
(Intercept)	0.015	0.001	17.004	<0.001
Time	<0.001	<0.001	-1.984	0.051
interruption1	0.001	0.002	0.407	0.685
time-interruption1 interaction	-0.001	<0.001	-2.653	0.010
interruption2	-0.009	0.002	-5.478	<0.001
time-interruption2 interaction	0.001	<0.001	3.284	0.002
Adjusted R-squared: 0.5605

Figure 2a, 2b, 2c.

Proportions of patients engaging in telemedicine encounters after accessing their test results prior to their clinician or after their clinician across the time periods of Cures Act compliance and the notification policy change. Plot 2a shows the proportion of all patients who engaged in a telemedicine encounter. Plots 2b and 2c show the proportion of patients engaging in telemedicine encounters who reviewed their results prior to their clinician and after their clinician, respectively.

The proportion of all patients who messaged their clinician within 3 days of receiving their test result increased 4.5% after Cures Act compliance (p<0.001). We did not observe a change in this proportion after notification policy changes and did not observe a change in the rate of the proportion of all patients messaging their clinician within 3 days of result receipt across Cures Act compliance or the notification policy change (Table 4, Figure 3a). Similarly, among patients who reviewed their results first, we did not observe any meaningful changes in proportion or rate of proportion over the time periods of adjustment in practices (Table 4, Figure 3b). Within patients who reviewed their results after their clinician, however, we observed a 2.6% increase in the proportion who messaged their clinicians after Cures Act compliance (p<0.001) and a 2.9% decrease after notification policy changes (p<0.001) (Table 4, Figure 3c). We also observed the rates of these proportions shifting after Cures Act compliance and the notification policy change.

Table 4.

Changes in the proportion of patients who sent a message to their clinician within 3 days

	Estimate	Standard Error	t Value	p Value
All Patients
(Intercept)	0.144	0.004	39.368	<0.001
time	<0.001	<0.001	2.561	0.012
interruption1	0.045	0.007	6.530	<0.001
time-interruption1interaction	-0.001	0.001	-1.579	0.118
interruption2	-0.009	0.007	-1.364	0.176
time-interruption2 interaction	0.001	0.001	1.004	0.319
Adjusted R-squared: 0.6253
Patient-First Review Group
(Intercept)	0.247	0.007	33.958	<0.001
time	-0.001	<0.001	-1.875	0.065
interruption1	0.011	0.014	0.787	0.434
time-interruption1interaction	<0.001	0.002	0.249	0.804
interruption2	0.019	0.014	1.375	0.173
time-interruption2 interaction	<0.001	0.002	-0.258	0.797
Adjusted R-squared: 0.2871
Clinician-First Review Group
(Intercept)	0.160	0.003	45.740	<0.001
Time	<0.001	<0.001	1.456	0.149
interruption1	0.026	0.007	3.939	<0.001
time-interruption1interaction	-0.004	0.001	-3.942	<0.001
interruption2	-0.029	0.007	-4.462	<0.001
time-interruption2 interaction	0.003	0.001	3.783	<0.001
Adjusted R-squared: 0.1969

Figure 3a, 3b, 3c.

Proportions of patients messaging their clinician after accessing their test results prior to their clinician or after their clinician across the time periods of Cures Act compliance and the notification policy change. Plot 3a shows the proportion of all patients who messaged a clinician after accessing their test results. Plots 3b and 3c show the proportion of patients messaging a clinician who reviewed their results prior to their clinician and after their clinician, respectively.

Discussion and Conclusions

Overview of findings

We observed that the policy changes were associated with time-based changes to the rate of new appointment scheduling at both time points. Additionally, we found that the second policy change was associated with the rate at which patients had telemedicine encounters. Lastly, we found that the first policy change was associated with the rate at which patients would message their clinician.

The statistics presented in Table 1 suggest that there are differences between the populations that view their results before their clinician and after their clinician across all sets of patient characteristics. However, despite the fact that most of the p values are significant, very few of the differences appear large enough to be meaningful. The exceptions might be the categories of test abnormality (which show that abnormal tests were more likely to be viewed by the patient first), and insurance (which show that commercially insured patients were more likely to view results before their clinician).

Interpreting our findings

In the case of scheduling new appointments, there appeared to be a trend among all patients that showed an increase on the proportion of patients who made a new appointment. Perhaps this points to an overarching trend in patients scheduling more new appointments over time as a result of being able to access more health information, especially since both of the segmented groups of patient-first review and clinician-first review experienced a significant increase at the notification policy change. Additionally, this could capture a trend of patients using the portal more than other platforms or routes of appointment scheduling, such as calling in, since they now have more information available to them in the portal¹⁷. This suggests that patients are apt to make a new appointment more now than they were prior to these practice changes. In response, it is possible that patients will book more appointments with clinicians, and the health system should work to manage operational resources in parallel, perhaps by adding more appointment slots and hiring more clinicians to staff them. Increasing transparency and communication in the workflow alongside information delivery to patients might assuage worries that patients may have about clinicians reviewing their results. Anticipatory guidance, pre-counseling patients, and providing more educational resources may also prepare patients better as they review more health information and make more appointments.

The trends in telemedicine encounters appear to suggest that there was an overall decrease in encounters, both through time and through the practice changes, but the effect of each intervention seems relatively small. This may be an artifact of the temporary and large increase in telehealth utilization during the pandemic¹⁸. It is possible that we are observing a gradual drop-off in usage now that more patients are comfortable seeking care in person and health systems are scaling back telemedicine offerings. The proportion of patients having a telemedicine encounter is just above 2% of all patients at its maximum in our analysis, so though we may see the influence of the practice changes on the data we do not, for the most part, observe meaningful swings in the actual proportion of patients having telemedicine encounters. Patients sending a new message to a clinician, however, seems to be meaningfully increased by Cures Act compliance. The segmentation of patient-first versus clinician-first view is interesting in this case, as the uptick in patients messaging their clinician seems to be primarily in patients who review their results prior to their clinician. The difference in proportion of new messages sent to a clinician between these two viewing segments itself is notable, with around 25% of the patient-first review group messaging and only 16% of the clinician-first review group messaging. Some health systems have responded to this uptick in messaging by hiring someone to triage and address messages upfront, which is a model for inbox management that could be replicated across institutions to ensure messages receive responses without overwhelming care teams. Further investigation is necessary to determine why patients may send a message after reviewing their results, and whether this may be motivated by concern, relief, or another emotion, as well as the implications on clinician workload.

Our work fits alongside literature that estimates the changes in clinician workload associated with an uptick in telemedicine use and portal messaging inbound. Beyond research findings around COVID-19 public health emergency based increases in telemedicine implementation, our results, if telemedicine encounter volume followed suit with what we observed in the portal, potentially echo the idea that the system burden produced by this new virtual care delivery is negligible¹⁹. The increase in the proportion of patients scheduling new appointments through the portal and sending messages to a clinician resembles other healthcare system factors that increase clinician burden. Measurements of so-called “pajama time,” or the time spent by clinicians in the EHR outside of their normal working hours, have increased as a result of clinical, team-based, and scheduling factors^20,21. Our quantification of the proportion of patients using the portal to access care point to information access-mediated inflections in healthcare use that could inflate work for clinicians and lead to similar increases in pajama time.

Limitations

It is important to interpret these results about patient behaviors observed through portal data with the understanding that patients who use the portal do not represent all patients receiving care. Since patients who use the portal are more likely to be socioeconomically advantaged, this work would greatly benefit from an extension into populations that are not as privileged to ensure representation in our study^22,23. Additionally, our study was performed at a single site with already high levels of portal engagement²⁴. As we study how the patient portal mediates and improves ownership over healthcare and the way that this potentially translates into positive health outcomes, we must consider that the patients who may derive the greatest benefit from this kind of access to health data management tools are not currently able to access them.

Previous research also suggests that there are differences in patient behavior based on the severity of the result that they receive or are anticipating receiving⁵. To further strengthen the potential linkages between various observed patient behaviors and test results, the type of result that each patient is receiving could be factored in. This would allow us to consider how patient behavior changes not only alongside changing information access but also with regard to severity of test result. Since the majority of test results are routine, zooming in on the behavior of patients who receive severe and abnormal results and comparing them against those that receive routine results could help us to better characterize how patients behave in the patient portal under stress.

Another angle to measuring how patient behavior in the portal is influenced by policy would be through providing patients with pre-counseling prior to the review of their results. Not only would this effectively serve as a kind of intervention, but also it would help us to potentially understand if and how serving information to patients before they are given their medical results makes a difference on their behaviors and perhaps which patients are best helped by this measure, if at all. Conducting a formative usability study on our pre-counseling materials could equip us to translate the learnings from these studies into something that patients could leverage to their benefit when absorbing personal health information.

Conclusions

Our investigation into how patients are behaving around the receipt of their test results, using EHR and audit log data sources, help us to better understand which real-life actions follow result release. We evaluated which practice changes, over time, contributed to a change in patient behavior. We assessed the test results for 351,111 unique patients across a time range of nearly two years. We observed an increase in the proportion of patients who reviewed their results prior to their clinician, an increase in the proportion of patients who scheduled new appointments through the portal, and an increase in the proportion of patients who messaged their clinician after Cures Act compliance. After the notification policy was changed, we observed a decrease in the proportion of patients who reviewed their results prior to their clinician, an increase in the proportion of patients who scheduled new appointments, and a decrease in the proportion of patients who had telemedicine encounters. Our findings indicate that changes in information access from the two policy changes affected how many patients schedule new appointments, have telemedicine encounters, and message their clinician.

As this work continues to develop, relating these changes in patient behavior back to clinical context will be vital. This work observed changes in patient behavior as a result of portal policy changes, but future work could assess how patient anxiety contributed to this shift or how these behavioral changes impact clinician workload. Quantifying the observed shifts in patient behavior back to metrics of burden for clinicians, patients, and the healthcare system will help to ground these findings in actionable, operational steps that can be taken to improve the experience of result release. Considering how information access relates to outcomes will enhance the design of healthcare infrastructure around the patient portal.

Figures & Tables

Table 5.

Summary of results by study outcome

Study Outcomes	Policy 1: Immediate Release	Policy 2: Opt in for Notifications	Difference Between Pt/Prov 1st Groups
1. New Appointments	↑	↑	Yes
2. Telemedicine Encounters	–	↓	No
3. Messaging	↑	–	Yes

↑indicates statistically significant increase in proportion of patients, ↓ indicates statistically significant decrease in proportion of patients, – indicates no significant change, Yes/No indicates if change in proportion of patients was different in direction between patient- and clinician-first test review groups