Telemedicine Physical Examination Utilizing a Consumer Device Demonstrates Poor Concordance with In-Person Physical Examination in Emergency Department Patients with Sore Throat: A Prospective Blinded Study

Abstract

Background: Telemedicine allows patients to connect with healthcare providers remotely. It has recently expanded to evaluate low-acuity illnesses such as pharyngitis by using patients' personal communication devices. The purpose of our study was to compare the telemedicine-facilitated physical examination with an in-person examination in emergency department (ED) patients with sore throat.

Materials and Methods: This was a prospective, observational, blinded diagnostic concordance study of patients being seen for sore throat in a 60,000-visit Midwestern academic ED. A telemedicine and a face-to-face examination were performed independently by two advanced practice providers (APP), blinded to the results of the other evaluator. The primary outcome was agreement on pharyngeal redness between the evaluators, with secondary outcomes of agreement and inter-rater reliability on 14 other aspects of the pharyngeal physical examination. We also conducted a survey of patients and providers to evaluate perceptions and preferences for sore throat evaluation using telemedicine.

Results: Sixty-two patients were enrolled, with a median tonsil size of 1.0. Inter-rater agreement (kappa) for tonsil size was 0.394, which was worse than our predetermined concordance threshold. Other kappa values ranged from 0 to 0.434, and telemedicine was best for detecting abnormal coloration of the palate and tender superficial cervical lymph nodes (anterior structures), but poor for detecting abnormal submandibular lymph nodes or asymmetry of the posterior pharynx (posterior structures). In survey responses, telemedicine was judged easier to use and more comfortable for providers than patients; however, neither patients nor providers preferred in-person to telemedicine evaluation.

Conclusion: Telemedicine exhibited poor agreement with the in-person physical examination on the primary outcome of tonsil size, but exhibited moderate agreement on coloration of the palate and cervical lymphadenopathy. Future work should better characterize the importance of the physical examination in treatment decisions for patients with sore throat and the use of telemedicine in avoiding in-person healthcare visits.

Introduction

Pharyngitis is one of the most common causes of physician visits, with an estimated 15 million U.S. outpatient visits in 2007, accounting for 6% of all visits to pediatricians and family practitioners.^1–3 Pharyngitis is also expensive, with pediatric group A streptococcal pharyngitis alone estimating to cost anywhere from $224-$539 million per year.⁴

Many visits to the emergency department (ED) are driven by doctors' offices being closed and patients' lacking access to other providers,^5,6 including simple ailments such as pharyngitis.⁷ ED waiting rooms continue to be full, with the average ED wait time increasing by 25% to 58 min in 2009.⁸ Poor access to care and long wait times have led to many healthcare innovations, such as acute care clinics, retail health clinics, urgent care centers, and direct-to-consumer telehealth services.⁸

Telemedicine is the delivery of medical care remotely using telecommunication technology. In direct-to-consumer telemedicine, patients can connect with providers using personal communication devices without having to travel to a healthcare provider's clinic.⁹ As direct-to-consumer telemedicine has grown, some have questioned whether the objective data available using telecommunications technology can allow providers to deliver the same quality of care as in-person visits.^10,11 Others have even suggested that the inability to perform a reliable physical examination may lead to increased antibiotic prescribing and poor quality care.¹² Telemedicine is currently being used for pharyngitis evaluation as one commonly evaluated urgent care complaint.

This study was conducted to determine whether the physical exam in patients with pharyngitis correlates with an in-person examination. Additionally, we aimed to understand patient and provider comfort with telemedicine as an examination medium for sore throat, with the intention of validating that telemedicine provides noninferior information with high concordance to the in-person evaluation.

Materials and Methods

Study design, setting, and population

This study was a cross-sectional prospective blinded diagnostic study of adults (age ≥18 years) presenting to a 60,000-visit university-based Midwestern ED complaining of sore throat between January 2016 and March 2017. Participants were enrolled using a convenience sample during day and evening shifts by a single research assistant.

All participants had a face-to-face and a telemedicine-enabled physical examination performed by separate advanced practice providers (APPs) blinded to the results of the other evaluator. Prisoners, pregnant women, non-English-speaking patients, and those unable to provide consent were excluded. Participants provided written informed consent, and the study was approved by the local Institutional Review Board. This study is reported in accordance with the Standards for Reporting Diagnostic Accuracy (STARD) statement.¹³

Study protocol

After giving informed consent, study participants had an in-person physical examination performed by an APP using a standardized care report form. This form contained an evaluation of the oropharynx (soft palate, uvula, tonsils, and posterior oropharynx); presence of exudate, swelling, or ulceration; tonsil size grade; and evaluation of lymph nodes (Fig. 1), using standard clinical criteria. After the primary clinical examination, the in-person evaluator left the room, and a research assistant established a telemedicine link with a second APP in the same ED who had not examined the participant. A two-way video telemedicine connection was established using an Apple iPad Air 2 (^® Apple, Cupertino, CA) with the patient and secure telemedicine software (VidyoMobile™, Vidyo, Inc., Hackensack, NJ). The second APP (who was physically present in another room in the same ED) conducted a physical examination with the patient's help, using no assistants in the room with the participant during the evaluation.

Fig. 1.

Standardized data collection form used for in-person and telemedicine consult visits.

Participants were given a standard flashlight (appropriate to visualize the posterior oropharynx) and tongue depressor with instructions given by the telemedicine provider on how to optimize oropharyngeal viewing (Fig. 2) and how to palpate lymph nodes for tenderness and swelling. All examinations were performed by a small group of four APPs trained in a detailed pharyngeal examination. They used a standard case report form based on the detailed pharyngeal physical examination according to Bate's Nursing Guide to Physical Examination and History Taking (13). An identical case report form was used for both examinations, and a satisfaction survey developed by the study team was administered to both participants and providers to better understand the telemedicine experience using a 5-point Likert scale. The four questions on the survey captured ease/difficulty, comparability between the two modalities, comfort, and preference of telemedicine and in-person visits. All diagnostics, therapy, and decision making that influenced a patient's care were performed entirely by the provider who was practicing in the ED in person. The telemedicine examination was done in addition to medical care for the purpose of research, and in no way did this evaluation influence clinical care.

Fig. 2.

Provider view of patient's oropharynx granted by flashlight and telemedicine interface. Used with permission, Signal Center for Health Innovation, University of Iowa Health Ventures.

Blinding

Providers of both examinations were blinded to the other examination and were not permitted to communicate until after data were recorded. Case report forms for the in-person examination were completed before the telemedicine examination was being performed, and all forms were reviewed at the time of submission to ensure no data were missing. Telemedicine examiners did not have access to the participant's medical record, but were able to ask additional questions as part of the interaction.

Outcomes

The primary outcome in this study was concordance between the evaluators in assessing pharyngeal redness. This concordance was measured by inter-rater agreement (kappa) as the primary measure and raw concordance as the secondary measure. Secondary outcomes included presence of posterior pharynx asymmetry, uvular deviation, exudate, swelling, and ulceration. The oropharynx was inspected for abnormal coloration of the soft palate, uvula, tonsils, and posterior oropharynx. Tonsil size was evaluated on a four-point scale (0 = absent tonsils, 1+ = tonsils were visible, 2+ = tonsils are visible between tonsillar pillars and uvula, 3+ = tonsils are touching the uvula, or 4+ = tonsils are touching each other). The presence of swelling and/or tenderness of the submandibular, submental, superficial cervical, and posterior cervical lymph nodes were also compared. The in-person physical exams were used as the gold standard to compare telemedicine physical exams against.

Sample size

The study was powered to detect a 20% difference in the proportion of participants with pharyngeal redness, assuming the prevalence of redness to be 70%, for which 62 patients were expected to be enrolled (power = 80%, α = 0.05).

Statistical analysis

Inter-rater agreement between exams was measured by Cohen's kappa statistic for binary outcomes, the weighted kappa statistic for tonsil size, and percent overall agreement. In addition to the weighted kappa measure and percent agreement for exact matches on tonsil size, we also evaluated agreement on tonsil size within ±1 score, to reduce the penalty for nearly concordant values. Interpretation of kappa used widely accepted interpretation guidelines (14). The surveys were analyzed by summary statistics and are reported in aggregate using medians and interquartile ranges for Likert scales. Responses to identical questions were compared between the provider and participant surveys using the Wilcoxon signed-rank test for paired observations. Statistical hypothesis tests are performed using two-tailed tests with alpha <0.05, and all analyses were conducted with SAS version 9.4 (SAS Institute, Cary, NC).

Results

During the study period, 62 patients were enrolled. The primary outcome (pharyngeal redness) was evident in 26 participants (42%) during the in-person examination, and oropharyngeal swelling was observed in 16% of study participants (Table 1). Agreement between the two exams was poor on posterior pharyngeal redness (worse than the a priori-defined 20% concordance threshold), with a kappa of 0.361.

Table 1.

Inter-Rater Reliability Measures Comparing Consults through Physical Visit and Telemedicine

			KAPPA MEASUREMENT
MEASURE	% POSITIVE ACCORDING TO GOLD STANDARD (IN-PERSON VISIT)	PERCENT AGREEMENT	KAPPA STATISTIC	95% CI
Asymmetry of posterior pharynx	4.8	93.3	−0.026	−0.064 to 0.013
Uvula deviation	4.8	95.0	NA	NA
Abnormal coloration
Palate	17.7	86.4	0.434	0.121–0.747
Uvula	19.4	74.6	0.071	−0.198 to 0.340
Tonsils	27.4	75.9	0.251	−0.012 to 0.512
Posterior oropharynx	41.9	69.0	0.361	0.119–0.604]
Oropharynx
Exudate	4.8	93.2	0.298	−0.205 to 0.801
Swelling	16.1	84.7	0.236	−0.100 to 0.572
Ulceration	3.2	96.6	0.483	−0.132 to 1.000
Lymph Nodes
Submandibular
Tender	8.1	80.6	0.044	−0.222 to 0.309
Swollen	4.8	87.1	−0.064	−0.115 to −0.014
Submental
Tender	3.2	90.2	0.211	−0.184 to −0.607
Swollen	3.2	95.1	0.375	−0.180 to 0.931
Superficial Cervical
Tender	11.3	82.0	0.431	0.170–0.692
Swollen	6.5	85.2	0.239	−0.097 to 0.574
Posterior Cervical
Tender	0	90.2	NA	NA
Swollen	0	96.7	NA	NA

CI, confidence interval; NA, not applicable.

Agreement between exams

Overall percent agreement was generally high (approximately ≥70%), which was driven by the prevalence of most measures. When assessing inter-rater reliability, several measures such as superficial cervical lymph node tenderness and abnormal palate coloration had moderate agreement (Table 1). Several parameters have no kappa calculated due to the overall low prevalence of those measures, and no detection of the measure by at least one mode of examination.

Tonsillar size had poor agreement (54%); however, when reclassifying agreement as being within 1 tonsillar size level (e.g., 3+ tonsils agreeing with tonsils recorded as 2+ or 4+), 32% of nonconcordant cases were then found to be concordant (overall concordance = 75%, Table 2).

Table 2.

Summary of Tonsil Size Measure by Group

TELEMEDICINE \ USED	NO \ TELEMEDICINE
	TONSIL SIZE	0	1	2	3	4	Total
	0	14 (25.5)	3 (5.5)	1 (1.8)	0 (0)	5 (9.1)	23
	1	6 (10.9)	3 (5.5)	2 (3.6)	0 (0)	1 (1.8)	12
	2	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0
	3	0 (0)	0 (0)	2 (3.6)	0 (0)	0 (0)	2
	4	4 (7.3)	3 (5.5)	0 (0)	0 (0)	11 (20.0)	18
	Total	24	9	5	0	17	55

Bold numbers indicate an exact match.

Survey results

Providers uniformly felt more favorably about the technology than participants (p < 0.01 on ease of use, quality of the exam, and comfort with future use) (Table 3). APPs specifically ranked the telemedicine medium favorably, with a median score of 5 for ease of use. Study participants, however, were less impressed with ease of use (Likert median 4.0 vs. 5.0, p < 0.001) and comfort with using telemedicine for an exam in the future (Likert median 3.0 vs. 4.0, p = 0.009).

Table 3.

Telemedicine Survey Results Between Patients and Advanced Practice Providers

	PATIENT		APP
Survey Question	MEDIAN	IQR	MEDIAN	IQR	p VALUE
On a scale of 1–5 with 1 being the hardest and 5 being the easiest, how easy was using the telemedicine media?	4.0	3.0–4.0	5.0	4.0–5.0	<0.001
Do you feel the telemedicine physical exam is the same as an in-person physical exam? Circle one number below from 1 to 5 with 1 being not the same at all to 5 being exactly the same.	3.0	2.0–3.0	3.5	3.0–4.0	<0.001
Would you be comfortable using telemedicine in the future for the physical exam? Circle one number below from 1 to 5 with being you would not feel comfortable at all and 5 being you would feel completely comfortable.	3.0	3.0–4.0	4.0	3.0–5.0	0.009
If given the option, would you prefer using telemedicine for performing the physical exam, or would you rather have an in-person physical exam? Circle one number below from 1 to 5 with 1 being you would prefer an in-person physical exam, 5 being you would prefer using telemedicine, and 3 being no preference.	3.0	2.0–3.0	3.0	2.0–4.0	0.527

APP, advanced practice provider; IQR, interquartile range.

Discussion

The use of direct-to-consumer telemedicine services for urgent care visits has increased rapidly in the last decade,^14,15 constituting over 1 million U.S. visits in 2014 alone.¹⁶ Many insurance companies and health systems have supported the development of these systems as a low-cost, efficient alternative to clinic visits,^17–19 and several vendors have developed large-scale networks to provide rapid and convenient evaluations.²⁰ Despite the rising popularity, some have raised questions about the quality of care being delivered by telemedicine, and some of the criticism surrounds the accuracy of the physical examination.

Although this is the first comparative report of specific elements of the physical examination using telemedicine, several prior reports have raised questions about examination elements. A 2015 study published by the RAND Corporation reported the results of a comparative analysis of antibiotic prescribing in urgent care visits using Teladoc, a national direct-to-consumer telemedicine company. In their analysis, the overall antibiotic prescribing rate for patients with pharyngitis was significantly higher in telemedicine encounters than physician office encounters (72% vs. 53%, p < 0.01), and even among those receiving antibiotics, broad-spectrum antibiotics were much more likely to be prescribed (87% vs. 39%, p < 0.01). This difference was larger than other upper respiratory illness (e.g., upper respiratory infection, sinusitis, bronchitis), and it suggests that pharyngitis may confer significant uncertainty during a telemedicine evaluation.²¹

Another evaluation of claims related to Teladoc visits evaluated adherence with accepted quality measures. In this analysis of data from the California Public Employees Retirement System, the use of streptococcus testing was much lower among telemedicine visits compared with in-person visits (3% vs. 50%, p < 0.01), likely owing to the difficulty in obtaining the test when it cannot be performed at the site of care. In the same analysis, antibiotic prescribing for bronchitis was higher as well (83% vs. 72%, p < 0.01).²²

Another study of 492 ambulatory patients in a general pediatric practice compared diagnostic agreement for a cohort of patients receiving both a telemedicine encounter and an in-person evaluation. In this study, telemedicine diagnosis agreed very closely with in-person diagnosis for complaints not involving the ear (89% vs. 90%, p = 0.65). Among patients with symptoms attributed to the ear, however, agreement was much worse (82% vs. 94%, p < 0.02), suggesting that difficulty in performing an ear exam with telemedicine was a significant barrier in diagnostic accuracy.²³

Other elements of the physical examination have been much better validated. For instance, telemedicine-enabled stroke care has also evolved very quickly, and the quality of the neurological evaluation is felt to closely resemble in-person neurologist evaluation. In fact, in a study of 41 patients in an inpatient stroke unit, inter-rater reliability on elements of the neurological examination were uniformly excellent, with weighted kappa ranging from 0.85 to 0.99.¹² Elements of care that are easily observed can likely be observed accurately with simple telemedicine cameras, but more invasive elements that either require special lighting, perspective, or tactile components may be much more limited.

These studies suggest that the physical examination likely remains an important part of the diagnostic process, but that the utility of telemedicine to interpret data from the physical examination is heterogeneous. That finding also corroborates the findings of our study. In general, agreement between examiners was highest for elements of the pharyngeal examination that were easily observed (anterior structures, lymph node palpation in easily definable regions, epithelial defects, asymmetry), and much poorer for more challenging elements of the examination (observing and lighting posterior elements, color judgment). It also highlights some technical challenges with the equipment commonly used for direct-to-consumer visits. This anticipated challenge was the reason that participants in our study were given commonly available household tools, such as flashlight, iPad, and tongue depressor. While technology exists to accurately illuminate the posterior pharynx or middle ear and transmit high-definition color-balanced images, that technology is not available in common consumer electronics.^24,25

Our study also validates prior literature that shows high patient and provider satisfaction,^26–28 although we observed provider enthusiasm eclipsing patient enthusiasm. Because participants in our study were manipulating the iPad themselves, trying to get a good image while holding the flashlight, etc., we expected that patients might feel that there were more limitations than in prior reports. Understandably, the experience of managing local communication technology is a different experience from having the technology available transparently in a clinic, but it is one that is important when using telemedicine for these types of complaints. Importantly, participants were not asked to weigh the relative value of avoiding an in-person visit with the inconvenience of the technology, and participants still felt that the telemedicine connection was valuable.

One final question remains in interpreting the results of our study: what is the role of the physical examination in diagnostic decision making? Some studies have suggested that the physical examination offers limited diagnostic utility, and that use of more advanced diagnostic testing is warranted.^29–32 With many urgent care complaints, however, additional diagnostic testing is not required; the medical history and physical examination are key to diagnostic accuracy. For pharyngitis specifically, elements of the physical examination predict the etiology of infectious symptoms, and patients with throat complaints are commonly examined in clinical practice.³³ The examination remains important, and further study should evaluate the minimum information needed to adequately evaluate a variety of chief complaints, with an assessment of whether that information can be obtained using telemedicine.

Our study has several limitations. First, we compared agreement only to an absolute gold standard of another control provider. Although we do not have a control group comparison (comparing 2 providers' evaluations on in-person examinations alone), we have limited the number of providers to four providers in the same practice with similar training. Each of those providers were both telemedicine and in-person providers, to limit systematic bias in the assignment of cases. We have selected some subjective elements of the physical examination (color, size, swelling), but even with subjectivity, certain elements of the examination exhibit good concordance. Finally, the quality of the technology used for the study protocol is not the best available telemedicine-enabled equipment. While clearly better technology exists, we aimed to replicate technology used for commonly available consumer electronics, so this study is more pragmatic in replicating real-world experiences.

In conclusion, our study has shown that a telemedicine-enabled physical examination replicating a direct-to-consumer interaction has poor concordance with in-person examinations. Elements of the physical examination that are easily observed show high concordance, but posterior structures and elements that are difficult to observe using consumer communications technology limit the utility of the pharyngitis physical examination. Future work should more rigorously evaluate the impact of telemedicine-associated limitations on diagnostic accuracy and treatment variation, as providers more carefully risk stratify patients in the era of telemedicine.

Acknowledgments

This research was funded by the National Heart, Lung, and Blood Institute at the National Institutes of Health (T35 HL007485), the University of Iowa Carver College of Medicine, and support from the Rural Telehealth Research Center, funded by the Federal Office of Rural Health Policy (FORHP), Health Resources and Services Administration (HRSA), and U.S. Department of Health and Human Services (USDHHS) (cooperative agreement U1C RH29074). The information and conclusions in this article are those of the authors and do not infer endorsement by FORHP, HRSA, or USDHHS.

Disclosure Statement

The authors have no competing financial interests.