Abstract
Background
Patient portals are becoming increasingly common, but the safety of patient messages and eVisits has not been well studied. Unlike patient-to-nurse telephonic communication, patient messages and eVisits involve an asynchronous process that could be hazardous if patients were using it for time-sensitive symptoms such as chest pain or dyspnea.
Methods
We retrospectively analyzed 7322 messages (6430 secure messages and 892 eVisits). To assess the overall risk associated with the messages, we looked for deaths within 30 days of the message and hospitalizations and emergency department (ED) visits within 7 days following the message. We also examined message content for symptoms of chest pain, breathing concerns, and other symptoms associated with high risk.
Results
Two deaths occurred within 30 days of a patient-generated message, but were not related to the message. There were six hospitalizations related to a previous secure message (0.09% of secure messages), and two hospitalizations related to a previous eVisit (0.22% of eVisits). High-risk symptoms were present in 3.5% of messages but a subject line search to identify these high-risk messages had a sensitivity of only 15% and a positive predictive value of 29%.
Conclusions
Patients use portal messages 3.5% of the time for potentially high-risk symptoms of chest pain, breathing concerns, abdominal pain, palpitations, lightheadedness, and vomiting. Death, hospitalization, or an ED visit was an infrequent outcome following a secure message or eVisit. Screening the message subject line for high-risk symptoms was not successful in identifying high-risk message content.
Keywords: patient portal, eHealth, eVisit, secure message, safety
Introduction
In a broad sense, patient portals connect patients to their health system via the internet. At its most basic level, the patient portal allows patients to have an online view of elements of their medical record. Increasingly, patient portals have become more than passive views into medical records; they now allow secure communication between healthcare providers and patients. Many major medical institutions now have patient portals, which allow patients electronic access to appointment scheduling, medication refills, and secure communication with their provider or care team.1–3 In the USA, major health systems offering patient portals include the Veterans Health Administration, Partners HealthCare, Intermountain Healthcare, University of Pittsburgh, Kaiser Permanente, Cleveland Clinic, and Mayo Clinic.4–7 Some patient portals offer not only the ability to send secure messages to their provider but also offer structured communication about symptoms, termed eVisits.8–10 The structured communication in eVisits is intended to provide sufficient symptom information for the provider to make an assessment about the need for further care.
Although the term eVisit is relatively new, symptom assessment from a distance is not. Telephone triage call centers have been providing symptom assessments for decades and have used computerized algorithms to provide structured assessments of symptoms in real time.11 Although symptom assessment by telephone has generally been found to be safe,12 little is known about the safety of patient portal secure messages or eVisits. A major concern is whether patients and health systems are using patient portal communication safely. A safety advantage of telephone triage centers is that patients are able to communicate rapidly and synchronously with medically trained personnel. For example, a national telephone triage call center in the UK (National Health Service Direct) takes over 5 million calls yearly with 24/7 availability and wait times of only a few minutes.13 In contrast, with asynchronous secure messages and eVisits, there may not be a 24/7 process to triage the communication. If patients use the patient portal strictly for non-urgent medical problems, then the turnaround time for messages and eVisits may not be a safety issue. Currently, there is little information on how safely patients are using this new form of communication.
A major knowledge gap exists in how often patients are using secure messaging and eVisits for acute, high-risk symptoms. Telephone triage literature demonstrates a significant percentage of telephone callers have symptoms that require urgent medical attention.14 If current callers shift their communication preference to portal messages and eVisits, then some safeguards may be needed. For example, it may be important to design a messaging and eVisit process so symptoms conferring medical urgency are not drifting for hours in cyberspace while awaiting a medical opinion. On the other hand, if patients are uniformly using portal communication for low-risk and non-acute symptoms, then messages and eVisits may not require an additional triage process.
In this study, we reviewed the content of over 7000 patient-generated secure messages and eVisits. Death, hospitalization, and emergency department (ED) visits were measured outcomes. We used established telephone triage literature to guide the selection of potentially dangerous symptoms and used this information to determine whether patients were using portal messaging and eVisits for symptoms requiring urgent evaluation.
Methods
Setting
The study took place in the primary care practice at Mayo Clinic in Rochester, Minnesota, which has an empanelled population of 141 000. These patients are split among three primary care specialties: family medicine (73 000), primary care internal medicine (46 000), and community pediatrics and adolescent medicine (22 000). There are 121 staff physician providers and 59 staff mid-level providers. The physician providers have academic commitments along with patient care responsibilities, and postgraduate training programs are associated with each of the primary care specialties. There are 100 000 adult patients in primary care family medicine and primary care internal medicine (the final source of the messages after the exclusions below). Of these adult primary care patients, 54% are women, 51% are Mayo employees or dependents, 19% are 65 years and over, and 30% are aged 18–34 years. The age groups 35–49 and 50–64 years account for 25% and 26%, respectively.
Patient online services (Mayo Clinic patient portal) were available starting 7 April 2010, and were available exclusively to primary care empanelled patients up to 20 August 2011. After that date, patient online services were available to other Mayo Clinic patients. This study examined only primary care patient-generated messages from the portal during the time frame of 7 April 2010 to 20 August 2011, when primary care patients had sole access to portal messaging. During that time, patients were able to send a message to their healthcare provider either as a secure message or an eVisit. There was a charge of US$39 for the eVisit and no charge for a secure message. Both secure messages and eVisits were handled first by a nurse before being resolved or forwarded to a provider, appointment secretary, or other recipient based on the content of the message.
Secure messages and eVisits were captured in their entirety and identified by patient clinic numbers and the time the messages were sent. Secure messages and eVisits had a similar data structure of separate text variables representing a subject line and message content. Only patient or proxy initiated messages and eVisits were included in the study. Portal users also had the option of requesting new appointments, making requests for corrections to their medication or allergy lists, and reporting immunizations. Those message types were not included in this study.
Instructions for eVisits and secure messages
The eVisit was preceded by the following statement on the portal web site: If you are reporting a symptom, please click Begin an eVisit. The eVisit start page was also linked to a frequently asked question page which stated:
Q. How long will it take for my provider to respond to my eVisit?
We respond to all eVisits within 24 h during standard business days (which exclude weekends and holidays). If you are experiencing a medical emergency, call 911 immediately.
Q. Should I use eVisits for emergency purposes?
NO. In an emergency, call 911 (or your emergency contact) to seek immediate medical attention.
There was no explicit warning other than not to use the eVisit for an emergency. During the eVisit process, patients were led through a structured interview for pertinent positive and negative information based on their symptoms. Secure messages had just free-text subject and body fields like an email. There was no structured set of questions as contained in the eVisit.
The web page instruction for secure messages had examples of appropriate message content. Specific examples of appropriate messages were: ’I have a follow-up question about a recent visit‘, ’My provider asked me for an update on my blood pressure‘, ’Am I due for my mammogram?’ and ’My acne medication isn't working as planned’. As noted above, symptom concerns were directed to the eVisit.
There were several ongoing promotions to increase portal registration, including a video that visually demonstrated an example of a patient sending a message and describing an eVisit.15 These efforts had little educational content about the differences between secure messages and eVisits, so other than what was contained on the secure message and eVisit web pages, there was limited instruction on how to use messages and eVisits.
Message selection for content review
Messages concerning pediatric patients (those in the pediatric practice and those under 18 years) were excluded. In addition, we reviewed all message content and excluded messages from surrogates who used their own secure message account rather than that of the patient. Surrogate senders, especially in the pediatric population, sometimes used their own portal account for messages rather than the proxy account for the patient. To avoid linking outcomes to the surrogate instead of the patient, we cleaned the dataset of those messages (pediatric and many adult surrogate messages).
We used data from the National Hospital Ambulatory Medical Survey and telephone triage literature to find symptoms associated with a high risk of hospitalization and urgent triage recommendations.16 To capture messages and eVisits concerning high-risk symptoms, we used software (Access 2010, Microsoft Corp, Redmond, Washington, USA) to perform character string searches on each subject line for keywords and parts of words associated with high-risk symptoms. Table 3 contains our search term list. As a reference, we randomly selected a 5% sample (323 secure messages, 43 eVisits) for content abstraction and used this for a manual review to calculate subject line text search sensitivity and specificity. For the high-risk symptoms, we reviewed the content and categorized them as present for over 24 h if the duration of symptoms was clearly stated to be greater than 24 h or 1 day. We also used this random sample of 366 to obtain estimates of content categories contained in secure messages and eVisits (see appendix).
Table 3.
High-risk symptoms: comparison of subject line string search to complete message abstraction
| Message symptom capture by subject line string search (N=366) | Message symptom capture by complete message abstraction (N=366) | String search terms | |
|---|---|---|---|
| Symptom | n (%) | n (%) | |
| Abdominal pain | 2 (0.5) | 4 (1.1) | Abdominal, abdomen, stomach, belly |
| Breathing concerns | 1 (0.3) | 3 (0.8) | Breath |
| Chest pain | 1 (0.3) | 1 (0.27) | Chest, heart |
| Lightheaded/palpitation | 1 (0.3) | 2 (0.55) | Palpitation, heart, lighthead, faint, pass out |
| Nausea/vomiting | 2 (0.5) | 3 (0.8) | Nausea, vomit |
| Total | 7 (1.9) | 13 (3.55) | NA |
Response times and potential response delays
We captured the time a message was sent and subtracted that from the date/time of the response message to get the response time for each secure message and eVisit. The distribution of response times was skewed, so we used the median response time to compare the difference between secure messages and eVisits. We also looked at the times and days of the week that the messages were submitted and compared them to when the clinic was open. By subtracting message sent times from clinic opening times, we calculated the number of hours that a secure message or eVisit might remain in cyberspace before being responded to. For example, messages sent on Friday evening could wait over 60 h before being addressed at 07:00 hours on Monday morning.
Capture of hospitalization, ED visit, and death outcome
We used billing data to capture the hospitalizations and ED visits within 7 days after a message or eVisit. We chose the 7-day time frame based on triage literature, which demonstrates that most ED and hospitalization outcomes following a call occur within 3 days.14 There were instances of more than one message (secure messages or eVisits) associated with one ED visit or hospitalization. In only one case was there one message associated with more than one ED visit. This lack of one-to-one correspondence between messages and the outcome variables is evident in the results.
Demographic and comorbidity comparison with telephone triage calls
As a comparison for the secure messages we also examined our triage calls during the same study time interval. We collected demographic and comorbidity information from secure messages, eVisits and triage calls. Demographic variables of age, sex, race, and Mayo employee status were obtained for message patients and for telephone triage patients. In addition, we used the Deyo derivation of the Charlson index to obtain a comorbidity comparison between portal message and telephone triage patients.17 18
Message and eVisit content review for hospitalizations and ED visit outcomes
ED visits and hospitalizations were not always related to the previous secure message or eVisit. For example, there were cases of accidents requiring ED visits that coincidentally occurred within 7 days after an unrelated secure message or eVisit. Similarly, there were hospitalizations for elective surgery occurring within 7 days of a portal message. The messages sometimes had no discernable relationship to the reason for hospitalization. To address this, we had two independent reviewers: one of the authors, and a study coordinator abstractor, who examined the combined 108 messages and eVisits that preceded ED visits and hospitalizations by 7 days. The reviewers examined the subject line and body of the message to determine whether message content could be associated with the ED or hospital dismissal diagnosis. Disagreements were resolved by arbitration to reach a consensus. The two independent reviewers agreed 90% of the time (97 of 108) for a kappa of 0.8 (95% CI 0.68 to 0.91). We report both the total count of the subsequent hospitalizations and ED visits as well as those determined by consensus related to the secure message or eVisit.
Statistical analysis
We used JMP V.9.01 software (SAS Institute Inc., Cary, North Carolina, USA) for statistical analysis. Comparisons of categorical data were performed with the Fisher's exact test and continuous data with the Wilcoxon rank sum test. OR and 95% CI from the logistic model were used to compare differences between secure messages and eVisits.
This study was approved by the Mayo Clinic Institutional Review Board.
Results
Patient demographics and message content
There was a total of 8789 secure messages and eVisits from 3024 patients during the study interval. Of those, we excluded 492 messages (5.6%) that were generated by 154 (5.1%) patients who declined research participation. We also excluded all 494 messages from the pediatric practice and an additional 481 messages from surrogates. Our analyzed messages totaled 7322 from 2668 patients. The median secure message count per patient was two (range 1–65, interquartile 25–75%, 1–3). Median eVisits per patient were one (range 1–9, interquartile 25–75%, 1–2).
Tables 1 and 2 compare the age ranges, Charlson index, race, sex, and Mayo Clinic employee status of patients using secure messages with that of telephone triage. There were significant differences in most demographic categories between patients using portal messages and those using telephone triage. In addition, comorbidities (by Charlson index) were fewer in patients using portal messages compared to those using telephone triage. Compared to our adult primary care panel, those with portal messages were younger and more likely to be employed by Mayo. Mayo employees and dependents accounted for 71% of those using portal messages but only represent 59% of our adult primary care population; those aged 65 years and over accounted for 9% of the portal message senders but represent 19% of our adult primary care patients. Message content differed between eVisits and secure messages (see appendix). Symptoms were the main content in 95% of eVisits but only in 23% of secure messages. Most symptoms in both secure messages and eVisits were over 24 h in duration before the message or were chronic, recurrent, or previously evaluated symptoms. Only 3% of secure messages and 14% of eVisits were about acute symptoms less than 24 h old.
Table 1.
Age, comorbidity index (Charlson), and count comparison of adult portal message patients and telephone triage patients
| Age group (years) | Charlson index | Age group percentage | ||||
|---|---|---|---|---|---|---|
| Portal patients (N=2668) | Telephone triage patients (N=31794) | Portal patients (N=2668) | Telephone triage patients (N=31794) | |||
| Mean (SD) | Mean (SD) | p Value | % (n) | % (n) | p Value | |
| 18–34 | 0.45 (0.92) | 0.46 (1.06) | 0.85 | 26 (695) | 26 (8342) | 0.83 |
| 35–49 | 0.7 (1.42) | 0.69 (1.41) | 0.78 | 32 (849) | 25 (7818) | <0.001 |
| 50–64 | 1.12 (1.81) | 1.25 (2.02) | 0.07 | 33 (881) | 25 (8035) | <0.001 |
| 65+ | 2.27 (2.51) | 2.79 (2.87) | 0.006 | 9 (243) | 24 (7599) | <0.001 |
| Overall | 0.92 (1.66) | 1.27 (2.14) | <0.001 | 2668 | 31794 | <0.001 |
Table 2.
Demographics comparison of adult portal message and telephone triage patients
| Portal (N=2668) | Telephone triage (N=31794) | ||
|---|---|---|---|
| Demographic | % (n) | % (n) | p Value |
| Female, % (n) | 73 (1949) | 66 (21018) | <0.001 |
| Caucasian, % (n) | 94 (2519) | 90 (28772) | <0.001 |
| Resides in local county, % (n) | 71 (1902) | 73 (23263) | 0.036 |
| Employee % (n) | 71 (1890) | 32 (10038) | <0.001 |
Provider recipients of messages
There were 241 provider recipients of messages (101 trainees and 140 staff). Staff providers received 6772 messages (93%), while the postgraduate trainees received 550 (7%). Staff providers received a median of 34 messages each; trainees received a median of four messages.
Deaths
There were only two deaths within 30 days of a message. Both happened after secure messages. One death was 7 days after a surrogate sent a secure message. The message was about thyroid replacement medication and there were no previous messages. The patient was already hospitalized on the date of the message, and the terminal illness was autopsy-confirmed organizing diffuse alveolar damage and chronic obstructive pulmonary disease with contributing conditions of cirrhosis and hepatic necrosis, severe calcific coronary atherosclerosis and Alzheimer type II gliosis. A secure message was sent 24 days before the second death and was about a rash following chemotherapy for small cell carcinoma. The rash was not mentioned in an oncology note 8 days before the patient's cancer-related death due to small cell carcinoma.
Potentially unsafe messages: high-risk symptom content and potential long response times
Table 3 examines high-risk symptoms based on abstraction of a 5% random sample of complete messages. Overall, 3.5% of the random sample of messages had high-risk symptoms (12 secure messages and one eVisit). Sensitivity of the subject line search for high-risk symptoms was poor at 15% (two of 13 in the 366 message sample). Specificity of the search was 99% with a positive predictive value of 29% and negative predictive value of 97%. Table 4 compares the potential for time delay from message to response for eVisits and secure messages.
Table 4.
Message delivery times and response time comparisons: secure messages to eVisits
| Secure message n=6430 (%) | eVisits n=892 (%) | Fisher exact test p value (Ho: OR 1) | OR eVisit to secure message | OR 95% CI | |
|---|---|---|---|---|---|
| Message delivery time | |||||
| 8–5 Monday–Friday | 4783 (74.4) | 683 (76.6) | 0.16 | 1.12 | 0.95 to 1.33 |
| 8 or more hours until office open | 503 (7.8) | 57 (6.4) | 0.14 | 0.80 | 0.60 to 1.07 |
| 12 or more hours until office open | 439 (6.8) | 48 (5.4) | 0.11 | 0.78 | 0.57 to 1.06 |
| 24 or more hours until office open | 265 (4.1) | 23 (2.60) | 0.03 | 0.62 | 0.40 to 0.95 |
| 48 or more hours until office open | 99 (1.5) | 10 (1.2) | 0.38 | 0.73 | 0.38 to 1.4 |
| Wilcoxon rank sum | |||||
| Median message response time (h) | 4 | 3 | <0.008 | NA | NA |
Outcomes: hospitalizations and ED visits
There were 20 unique patients who were hospitalized within 7 days of a portal secure message or eVisit (0.3%). Fifteen of the hospitalizations were initiated by an admission through the ED, with the remaining five being direct hospitalizations without an associated ED visit. Of the 15 hospitalizations going through the ED, none were diagnosed with a myocardial infarction or pulmonary embolus. The most common hospital discharge diagnosis was depression (four in total, two with suicidal ideations indicated on their dismissal). In only eight of the hospitalizations was the final diagnosis judged to be associated with the message content. Potentially reversible serious illnesses associated with messages included one serious infection from a cat bite requiring hospitalization for intravenous antibiotics and an asthma exacerbation requiring hospitalization.
Table 5 displays the occurrence of hospitalizations and ED visits within 7 days following initiation of a secure message or eVisit. The table also contains the consensus counts of those secure messages and eVisits, which on content review were judged to have a relation to the subsequent hospitalization or ED visit. Overall, only 47 of 108 secure messages and eVisits (44%) were judged to be related to the subsequent hospitalization or ED visit diagnosis. Of the 61 messages that were not related to an ED or hospitalization outcome, 40 had no mention of a symptom at all. These messages were about medication questions or renewals, test results or appointments. Twenty-one of the messages had a mention of a symptom but in six of those there was unrelated coincidental trauma (laceration, motor vehicle accident or fall) happening after the mention of the symptom. In 15 there was a symptom mentioned in the message that could not be related to the actual diagnosis in the hospital or ED. Another factor differentiating secure messages was the time lapse between message and ED or hospital visit. The 47 messages in which we could find a definite association between message and ED visit or hospitalization had a mean of 2.0 days between message and ED or hospitalization, while the 61 messages without an association had a mean of 3.3 days between message and ED or hospitalization (p<0.002). This increased time interval is consistent with an entirely new symptom developing after the message was sent.
Table 5.
ED visit and hospitalization outcomes associated with eVisits and secure messages.
| Outcome count associated with secure message n=6430 (%) | Outcome count associated with eVisit* n=892 (%) | Fisher's exact p value | OR (95% CI) eVisit to secure message | |
|---|---|---|---|---|
| ED visit or hospitalization count | ||||
| Unique ED visits within 7 days after message | 56 (0.87) | 9 (1.0) | 0.70 | 1.2 (0.6 to 2.4) |
| Consensus unique ED visits actually related to any previous message | 28 (0.44) | 7 (0.78) | 0.19 | 1.8 (0.8 to 4.2) |
| Unique hospitalization within 7 days after message | 17 (0.26) | 3 (0.34) | 0.73 | 1.3 (0.4 to 4.4) |
| Consensus unique hospitalizations actually related to any previous message | 6 (0.09) | 2 (0.22) | 0.25 | 2.4 (0.5 to 12.0) |
| Message count† (multiple messages per outcome possible) | ||||
| Unique messages within 7 days before ED visit | 85 (1.3) | 11 (1.2) | 0.99 | 0.9 (0.5 to 1.7) |
| Consensus unique messages determined actually related to ED diagnosis | 37 (0.6) | 8 (0.9) | 0.25 | 1.6 (0.7 to 3.4) |
| Unique messages within 7 days before hospitalization | 31 (0.48) | 3 (0.34) | 0.79 | 0.70 (0.2 to 2.3) |
| Consensus unique messages determined actually related to hospitalization diagnosis | 6 (0.09) | 2 (0. 22) | 0.25 | 2.4 (0.5 to 12.0) |
Message counts for those outcomes (multiple messages for some hospitalization or ED visit outcomes).
*For multiple previous messages, outcome was attributed to eVisit column if at least one eVisit was generated regardless of additional number of associated secure messages.
†The 108 secure messages and eVisits we reviewed are fewer than the total messages in this table because several unique messages were counted both as associated with an ED visit and a hospitalization (eg, message from a patient who went to the ED and was subsequently hospitalized).
ED, emergency department.
Of the 47 ED visits and hospitalizations that could be linked to the messages, message responses did not appear to have major safety concerns. Appointments were offered or made in response to 13 (28%), eight for the same day. For 10 (21%) the message response was to call for nurse triage, and for three there was a suggestion to go to the ED or be directly admitted to the hospital. Medication changes were the response to seven (15%). No message response appeared to provoke an ED visit or hospitalization by delay in diagnosis or treatment. Two ED visits appeared to be unnecessary, diagnosed as paronychia and non-traumatic leg pain. The message response to those patients was a same-day appointment in the office.
Discussion
To our knowledge, this is the first study to examine the safety of patient messages and eVisits by searching for content about specific high-risk symptoms and examining subsequent outcomes of death, hospitalizations, and ED visits. Our secure messages and eVisits had hospitalization rates of 0.26% and 0.34%, respectively. There were six hospitalizations judged to be related to the 6430 secure messages (0.09%); only two hospitalizations were judged to be related to the 982 eVisits (0.22%).
There are few benchmarks to assess the risk of portal messages. A possible reference group for portal messages is telephone triage, which also communicates symptom information. However, we found significant differences both in demographics and comorbidities between patients using telephone triage and those using the portal (tables 1 and 2). Telephone triage also differs significantly in message content. For example, all our triage calls had symptom content, but only 31% of the portal messages were about symptoms. Although telephone triage has substantially higher rates of hospitalization (2.5% for adults aged 18–39 years and 4.2% for those aged 40–65 years),14 differences in content and user population preclude a direct comparison of telephone triage to portal messages.
Another reference group are emails from patient to doctor. A study by Houston et al19 showed that patients used emails for potentially high-risk symptoms of chest pain and shortness of breath, as well as suicidal ideation. Our results add to the evidence that patients do use asynchronous forms of communication for mood disorders. Four of the 20 hospitalizations following messages were for depression (two with suicidal ideation). However, a careful review of the preceding portal message content revealed no content to suggest suicidal ideation in any of the four. A possible explanation may be that patients perceive email as more private and personal than communication through the portal. Our findings combined with those of Houston et al19 suggest that mood disorder symptoms may challenge current forms of asynchronous communication and that further study on mood disorders and portal messaging is needed.
A large percentage of the secure messages were for questions regarding medications or laboratory test results, confirming that most patients who used secure messages were using this technology as intended. However, patients did use secure messages for new symptoms even though they were directed to use the eVisit for that purpose. A mitigating factor for this potentially risky use of secure messages was that the content of high-risk secure messages showed many of them to be follow-ups from a previous visit.
Over 75% of the patients sent secure messages and eVisits during regular office hours when the messages could be promptly viewed and addressed. However, despite warnings about potential delays, patients still sent them at times when the message could be stored for over 24 h before the office would open and someone would look at the message. This happened as frequently with eVisits as with secure messages (table 4).
There are several potential solutions to address high-risk symptoms. First, there may need to be an expansion in the hours of service to 24/7. Service provision outside of the typical business hours may require a healthcare provider with prescribing privileges such as a mid-level provider rather than a registered nurse. The asynchronous (store and forward) eVisit/messaging tool may need to allow for the option of live chat (digital or telephonic). Second, minor enhancements to the current eVisit/message tool could communicate the expected time of response, which would assist patients in deciding if they should seek alternative care options. For example, on creation of the message, software could alert the patient of the hours expected before someone would address the message. A message on Saturday evening could generate an immediate message back to confirm that the time delay is acceptable and, if not, then there could be a message to call a telephone triage number or another appropriate care source.
Subject line string searches had both low sensitivity and low positive predictive value and were not sensitive enough to screen messages for potentially serious symptoms such as chest pain and breathing concerns. Patient-generated messages will be an important challenge for natural language processing and how it can be applied to patient-generated text.
This study has limitations. Our findings are based on early users of this technology who may be more medically knowledgeable than those using it later. In addition, a large number of the messages were from Mayo Clinic employees, many of whom may have some knowledge of the high-risk symptoms of chest pain and shortness of breath. Our study contains a high percentage of healthcare workers, limiting its generalizability. Excluding all pediatric patients and many surrogate users also limits our findings. Because of this we were not able to address message risk for pediatric patients. Our study also lacks the complete surrogate message contribution, which may be an important marker of increased message risk as is suggested by telephone triage literature.20 The source for hospitalizations and ED visits was Mayo Clinic billing data, which captures almost all local hospitalizations and ED visits, but could miss similar outcomes if the patient was hospitalized or had an ED visit outside of the Mayo system. Although we did not do a formal sensitivity analysis for this study, previous internal studies examining hospitalization and ED visits for our population resulted in a conservative estimate that 90% of our study hospitalizations and ED visits were captured by billing data.
A limitation concerning the risk of secure messages is that there was a cost differential between secure messages (no cost) and eVisits (US$39). A patient reporting a symptom may be weighing the severity of the symptom against the cost differential when deciding whether to use a secure message or eVisit. A portal that does not charge for an eVisit or has a lower cost differential between the eVisit and secure message might be expected to have fewer symptom-related secure messages. Symptom content was also higher for eVisits than secure messages (see appendix). This limits a direct comparison of eVisits to secure messages for hospitalization risk and ED visits. Another limitation is that providers and patients can move between modes of communication. Our analysis just examined specific outcomes of ED visits and hospitalizations associated with secure messages and eVisits. We did not examine additional levels of communication complexity that may be involved, such as a secure message that gets escalated to a telephone call or an office encounter. Future research will be needed to help understand patient motivation in selecting a particular form of care or communication. We were not able to accomplish that in this study. With the increasing range of synchronous and asynchronous options it will be important to understand factors that influence whether the patient makes a telephone call, sends a secure message, opts to purchase an eVisit, or seeks care at the ED.
Our finding that 3.5% of portal messages contain content about potentially high-risk symptoms needs further study. We need to determine what interventions should be used to decrease high-risk symptom messages. Portal messaging has increased the number of options patients have to address their acute symptom. Not long ago, patients with acute symptoms had few options. They called the office, an answering service, or went to the ED. Now a symptomatic patient has a choice of calling a provider's office, sending a portal message, creating an eVisit, searching the internet for symptom advice, or calling a dedicated triage line or 911. For some symptoms, a delay in assessment can have adverse consequences. Mobile platforms will make messaging even more convenient, and there may be a great temptation to use a secure message or eVisit before other forms of access. As this technology becomes more widespread, the population using it may also have a wider diversity of medical literacy. We will need to be vigilant about possible untoward effects of this powerful messaging tool as its use grows and its audience diversifies.
Acknowledgments
The authors acknowledge Eric Manley, Chris Rock, Brian Kabat, and Stephanie Quigg for their contributions to this manuscript.
Appendix
Comparison of secure message and eVisit content abstracted from a 5% random sample of messages (N=366)
| Secure message (N=323) | eVisit (N=43) | |
|---|---|---|
| Message content | % (n) | % (n) |
| Medication related (renewal, request for different medication, question about current or not current medication) | 37 (120) | 9 (4) |
| Symptom related (new symptom or recurrent symptom) | 23 (73) | 95 (41) |
| New symptom under 24 h duration or unknown duration | 3 (10) | 14 (6) |
| New symptom over 24 h duration or recurrent, chronic, or previously evaluated | 20 (63) | 81 (35) |
| Test related (test requested, result wanted, or negotiation ordered test) | 20 (64) | 5 (2) |
| Medical question, additional information or correction | 7 (23) | 2 (1) |
| Referral request | 7 (22) | 9 (4) |
| Acknowledgment or thanks | 6 (20) | 0 (0) |
| Request to fill out form | 5 (15) | 0 (0) |
| Greater than one issue | 9 (28) | 19 (8) |
Footnotes
Contributors: Study design: FN, SJC, RJS, SMT-S; data collection: FN, SSC; critical revision of manuscript: FN, SJC, RJS, SSC, ESE, SMT-S.
Funding: Funding was received from the Mayo Foundation.
Competing interests: None.
Ethics approval: The study received ethics approval from the Mayo Clinic Institutional Review Board.
Provenance and peer review: Not commissioned; externally peer reviewed.
References
- 1.Weingart SN, Rind D, Tofias Z, et al. Who uses the patient internet portal? The PatientSite experience. J Am Med Inform Assoc 2006;13:91–5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Gamble KH. Is it registering? Patient portals, part II (see Financial Department for part I). Healthc Inform 2009;26:24; 26; 28 [PubMed] [Google Scholar]
- 3.Tang PC, Ash JS, Bates DW, et al. Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J Am Med Inform Assoc 2006;13:121–6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Wald JS, Businger A, Gandhi TK, et al. Implementing practice-linked pre-visit electronic journals in primary care: patient and physician use and satisfaction. J AmMed Inform Assoc 2010;17:502–6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Nazi KM. Veterans’ voices: use of the American Customer Satisfaction Index (ACSI) Survey to identify My HealtheVet personal health record users’ characteristics, needs, and preferences. J Am Med Inform Assoc 2010;17:203–11 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Page D. The two paths to PHRs. Hosp Health Netw 2010;84:44; 46; 42 [PubMed] [Google Scholar]
- 7.Roblin DW, Houston TK, Allison JJ, et al. Disparities in use of a personal health record in a managed care organization. J Am Med Inform Assoc 2009;16:683–9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Padman R, Shevchik G, Paone S, et al. eVisit: a pilot study of a new kind of healthcare delivery. Stud Health Technol Inform 2010;160:262–6 [PubMed] [Google Scholar]
- 9.Adamson SC, Bachman JW. Pilot study of providing online care in a primary care setting. Mayo Clin Proc 2010;85:704–10 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Jung C, Padman R, Shevchik G, et al. Who are portal users vs. early E-visit adopters? A preliminary analysis. AMIA Annu Symp Proc 2011;2011:1070–9 [PMC free article] [PubMed] [Google Scholar]
- 11.O'Cathain A, Sampson FC, Munro JF, et al. Nurses’ views of using computerized decision support software in NHS Direct. J Adv Nurs 2004;45:280–6 [DOI] [PubMed] [Google Scholar]
- 12.Lattimer V, George S, Thompson F, et al. Safety and effectiveness of nurse telephone consultation in out of hours primary care: randomised controlled trial. The South Wiltshire Out of Hours Project (SWOOP) Group. BMJ 1998;317:1054–9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.NHS Direct National Health Service Trust Annual Report & Accounts 2010/11. http://www.nhsdirect.nhs.uk/en/About/CorporateInformation/OperatingStatistics/AnnualReport2010–2011 (accessed 21 Mar 2012).
- 14.North F, Varkey P. How serious are the symptoms of callers to a telephone triage call centre? J Telemed Telecare 2010;16:383–8 [DOI] [PubMed] [Google Scholar]
- 15.North F, Hanna BK, Crane SJ, et al. Patient portal doldrums: does an exam room promotional video during an office visit increase patient portal registrations and portal use? J Am Med Inform Assoc 2011;18:i24–7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.North F, Ward WJ, Varkey P, et al. Should you search the internet for information about your acute symptom? Telemed J E Health 2012;18:213–18 [DOI] [PubMed] [Google Scholar]
- 17.Charlson ME, Pompei P, Ales KL, et al. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40:373–83 [DOI] [PubMed] [Google Scholar]
- 18.Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992;45:613–19 [DOI] [PubMed] [Google Scholar]
- 19.Houston TK, Sands DZ, Jenckes MW, et al. Experiences of patients who were early adopters of electronic communication with their physician: satisfaction, benefits, and concerns. Am J Manag Care 2004;10:601–8 [PubMed] [Google Scholar]
- 20.North F, Muthu A, Varkey P. Differences between surrogate telephone triage calls in an adult population and self calls. J Telemed Telecare 2010;17:118–22 [DOI] [PubMed] [Google Scholar]