Using Unannounced Standardized Patients to Explore Variation in Care for Patients With Depression

ABSTRACT

Background 

Physicians across specialties need to be skilled at diagnosing and treating depression, yet studies show underrecognition and inadequate treatment. Understanding the reasons requires specifying the influence of patient presentation, screening, and physician competence.

Objective 

We deployed an unannounced standardized patient (SP) case to assess clinic screening and internal medicine (IM) residents' practices in identifying, documenting, and treating depression.

Methods 

The SP represented a new patient presenting to the outpatient clinic, complaining of fatigue, with positive Patient Health Questionnaire (PHQ) items 2 and 9 and a family history of depression. The SPs assessed clinic screening and IM resident practices; appropriate treatment was assessed through chart review and defined as the resident doing at least 1 of the following: prescribing a selective serotonin reuptake inhibitor (SSRI), making a referral, or scheduling a 2-week follow-up.

Results 

Of 129 IM residents, 85 (66%) provided appropriate treatment, 79 (61%) appropriately referred, 59 (46%) prescribed an SSRI, and 49 (38%) scheduled a 2-week follow-up, while 40 (31%) did not add depression to the problem list. The IM residents who used PHQ-2 and PHQ-9 were more likely to appropriately (89%) versus inappropriately (50%) treat (P < .001). Compared with those who did not, residents who treated appropriately assessed depression symptoms more (P < .001) and had better communication (73% versus 50%, P = .02), patient centeredness (74% versus 42%, P = .03), and patient activation skills (35% versus 11%, P < .001).

Conclusions 

The use of unannounced SPs helps identify targets for training residents to provide evidence-based treatment of depression.

What was known and gap

Studies have shown underrecognition and inadequate treatment of depression in primary care settings.

What is new

Unannounced standardized patients presented with symptoms of depression to an internal medicine ambulatory clinic to assess the percentage of cases where residents appropriately diagnosed and initiated treatment.

Limitations

Single site, single specialty study limits generalizability; impact of supervising faculty on treatment decisions was not assessed.

Bottom line

Use of standardized patients helped identify areas for improvement in diagnosing and treating depression in ambulatory internal medicine.

Introduction

Diagnosing depression is often complicated by the initial clinical presentation. Approximately two-thirds of depressed individuals present to primary care with somatic complaints,^1–4 which reduce the likelihood of recognition, and therefore is the leading contributor to missed depression diagnoses.⁴ Missed diagnoses may result in a search for other explanations of the symptoms, causing unnecessary medical testing.⁵ Even with a positive screening Patient Health Questionnaire (PHQ)-2, only 5% of patients receive a follow-up PHQ-9,⁵ and only 34% have these results addressed.⁶ Even with a correct diagnosis, only half of the patients with diagnosed depression receive treatment.⁵

Internal medicine (IM) residents' accuracy at assessing depression in patients has been shown to be poor^7,8 but can be improved by a screening or inventory tool.⁸ While depression cannot be managed effectively without being identified, screening alone is not sufficient. Systems-based approaches to ensure comprehensive screening, diagnosis, and collaborative management should contribute to patients receiving appropriate care,^9,10 yet disaggregating the contribution of the individual physician from that of the care system to inadequate depression care is difficult.

Unannounced standardized patients (SPs) have been used to assess clinical skills, residents' professionalism, physician compliance with clinical guidelines, response to requests for direct-to-consumer advertised antidepressants, and efficacy of educational interventions.^11,12 Studies have explored the acceptable feasibility and validity of unannounced SPs to assess the quality of care and have shown SPs to be an effective means for obtaining detailed information on practice in context.¹³ Since 2009, our institution has used unannounced SPs to unobtrusively assess learners and the clinical microsystem.¹⁴ Currently, residents in our IM program may see between 4 and 6 unannounced SP cases (including the depression case described in this article) designed to assess the care provided to a range of common clinical primary care issues.

The study took advantage of the controlled nature of unannounced SPs to deliver a consistent portrayal of a depressed patient and to explore whether residents' ability to diagnose and manage depression is associated with clinic screening processes and trainees' patient interaction skills.

Methods

Unannounced SPs were seen by IM residents (including categorical and primary care tracks) during their continuity clinic at 2 urban primary care clinics. Residents were informed that they would see an unannounced SP as part of their own assessment. Data are reported only for residents who consented to include their data in a research registry (92%, 129 of 140 residents). Residents who consented did not differ from those who did not consent in terms of postgraduate year [PGY], sex, or age. The program's curriculum on depression included a 2-hour session during ambulatory block, practice of skills and feedback in annual objective structured clinical examinations with depression cases, and ongoing just-in-time teaching during precepting.

The SPs were men in their early 20s, and they were trained to present to the clinic as the same new patient with fatigue and insomnia. If asked or screened, they described symptoms positive for depression based on the PHQ-2 and PHQ-9, widely used depression screening tools with evidence of reliability and validity.¹⁵ Full case details are provided as online supplemental material. The SPs (actors) received 6 hours of training in assessment and character portrayal, including how to meet criteria for a positive response on PHQ-2 and PHQ-9 (scores of 14–16).

In both clinics, medical assistants are expected to screen all new patients for depression using the PHQ-2 and, if positive, conduct the full PHQ-9. Results are expected to be communicated to the physician via the electronic health record (EHR) prior to the patient encounter.

Case fidelity was maintained through ongoing review of audio recordings of the SP-physician encounter. The documented PHQ-9 scores of the SPs averaged 14.6 over 129 visits (SD = 2.3), suggesting consistent, accurate portrayal of symptoms. Based on resident feedback, the estimated detection rate for this case was 10%, with almost all detections occurring after the visit. Overall scores did not differ by whether the visit was detected.

We used 2 assessments to describe residents' management of this case: (1) a comprehensive SP checklist to capture specific practices associated with the visit and competence in clinical patient interaction skills using behaviorally anchored items (provided as online supplemental material),¹⁶ and (2) a systematic review of the EHR notes written by residents. Data were collected and managed using REDCap electronic data capture tools hosted at the New York University Langone Medical Center.¹⁷

The SP checklist assessed whether the medical assistant screened for depression using the PHQ-2 and followed up with PHQ-9, and it evaluated residents' exploration of depressive symptom criteria (sleep, energy, concentration, appetite, psychomotor retardation, guilt, suicidal thoughts). The unannounced SPs indicated the degree to which the resident explored the patient's social/family support, family medical history, unhealthy habits (alcohol, smoking, drug use), and past medical history, and reported on the extent of education they received from the resident about the connection between the presenting symptoms and depression.

Residents' clinical skills, as assessed through the checklist, included 12 items in the domains of communication skills (4 information gathering, 5 relationship development, 3 patient education and counseling); 4 items on patient centeredness (answered all questions, took a personal interest in the patient); and 4 patient activation items (encounter helped the patient understand the condition and feel confident he or she could manage it). Items used a not done, partly done, and well done scale with behavioral anchors. Domain scores were calculated as percentage of items rated well done. Internal consistency for each domain, measured using Cronbach's alpha, exceeded 0.70.

We reviewed residents' EHR notes for treatment recommendations, prescriptions, referrals, recommended time frames for follow-up visits, and whether depression was included on problem lists. Chart reviews were completed by 2 research assistants, who were blind to the study hypotheses. Initial checks on interrater reliability using a random sample of 20 charts showed strong agreement between raters (>85%) for all abstracted items. Appropriate treatment was defined as 1 or more of the following: prescribing a selective serotonin reuptake inhibitor (SSRI), making a referral, and scheduling a follow-up visit in less than 2 weeks.

Residents were grouped into those who treated the depression case appropriately and those who did not. The 2 groups were compared in terms of resident visit practices and overall clinical skills. We also explored whether treatment patterns differed by PGY, residency track (categorical or primary care), or site. We used independent sample t tests to compare group means on scores (percentage well done) and chi-square statistics to compare distributions.

This study was conducted through a resident medical education research registry¹⁸ approved by the New York University School of Medicine Institutional Review Board. Residents consented to including their education data in a research database.

Results

A total of 129 IM residents saw the unannounced SP depression case. From walking in the front door to leaving the clinic, mean visit length was 43 minutes (SD = 18 minutes, range 14–125 minutes).

Of the SP cases, 92% (119 of 129) were screened for depression by the medical assistant. Screening rates did not differ among clinic locations or over time. In the 8% (10 of 129) of cases where the SP was not screened by the medical assistant, no resident listed depression as the primary diagnosis. Overall, 29% (37 of 129) of residents listed depression/dysthymia as the primary diagnosis in the EHR problem list, and 35% (45 of 129) ranked insomnia as the primary diagnosis. Only 48% of residents (62 of 129) included depression/dysthymia on the problem list at all (64% [83 of 129] documented it as a routine examination, and 54% [70 of 129] documented it as insomnia). In addition, 70% (90 of 129) fully explored the patients' depression symptoms, and 53% (68 of 129) connected the presenting symptom (fatigue) with depression.

The Figure describes how residents managed this depression case, classifying practice into appropriate (66%, 85 of 129 residents) or inappropriate treatment (34%, 44 of 129 residents). Residents made a psychiatric or collaborative care referral in 61% (52 of 85) of visits, prescribed an SSRI in 46% (39 of 85) of visits, and scheduled a follow-up appointment in 38% (32 of 85) of visits. The Figure shows the distinct subgroups of treatment combinations.

Figure.

Description of Appropriate and Inappropriate Treatment for Standardized Depression Case

Characteristics for the appropriate and inappropriate treatment groups are shown in Tables 1 and 2. Both groups had similar rates of assessing patients' medical history and substance use, but significantly more of those who provided appropriate treatment fully explored the depressive symptoms (83%, 71 of 85) than those who did not (41%, 18 of 44; X² (1, N = 128) = 25.93; P < .001). Significantly more residents who treated appropriately had the patients' PHQ-9 (in addition to PHQ-2) data (89% [76 of 85] versus 50% [22 of 44]; chi-square [1, N = 129] = 24.67; P < .001) and included depression on the problem list (69% [59 of 85] versus 14% [6 of 44]; chi-square [1, N = 129] = 36.08; P < .001). The 2 groups did not significantly differ in terms of visit length, average return visit, PGY, attending physician, or clinic location.

Table 1.

Differences in Context, Clinic, and Resident Characteristics by Appropriate Versus Inappropriate Treatment

Table 2.

Differences Between Residents Who Provided Appropriate Treatment and Those Who Did Not: Core Clinical Skills

Residents who provided appropriate treatment had significantly higher scores in communication domains and subdomains (information gathering, relationship development, patient education, and counseling). This group also had higher scores in patient centeredness (74% [63 of 85] versus 42% [18 of 44], P = .003) and patient activation (35% [30 of 85] versus 11% [5 of 44], P < .001). Residents who provided appropriate treatment in the depression case did not consistently have better performance in other unannounced SP cases compared with those who did not.

Discussion

We found substantial variation of care for a common presentation of depression in a resident continuity clinic. Residents were more likely to provide appropriate treatment when a PHQ-9 was completed by medical assistants, supporting the proposition that physicians' clinical reasoning may be influenced by how the resident and clinical microsystem work together.

We found that the majority of visits with inappropriate treatment were associated with a positive PHQ-2, suggesting that these residents did not know, or were reluctant to follow, depression management guidelines. Equally striking was the fact that when making appropriate treatment decisions, only two-thirds of residents committed to a depression diagnosis in the medical record. While it is understandable that a resident may not want to commit to this diagnosis on the first visit, it is troubling that return visits were not consistently scheduled to enable proper monitoring of the patient.

Residents with more effective communication, patient centeredness, and patient activation skills were more likely to diagnose depression and initiate appropriate treatment. This is consistent with previous literature, which found positive associations between patient satisfaction, physician empathy, and treatment adherence for depressed patients,¹⁹ and showed that shared decision-making increases the probability that depressed patients see an improvement in symptoms.²⁰ Since patient activation is critical in improving health outcomes for depressed patients,²¹ the low scores in this domain are an area for improvement.²²

Our study has limitations. The unannounced SP methodology is limited to new patient visits and does not provide insight into what would transpire during follow-up visits. Our determination of appropriate versus inappropriate treatment was based on chart reviews, which may not fully capture residents' clinical decisions. The sample was limited to residents from 1 program and, thus, may not generalize; it is also unclear how attending supervisors influenced treatment decisions.

The unannounced SP case helped uncover deficits in residents' diagnosis and treatment of depression, while reinforcing the importance of depression screening protocols in the clinical microsystem. We plan to do more resident education and faculty development on depression management and documentation. In future studies, we will explore both why residents were not comfortable in making a diagnosis even when following appropriate treatment guidelines and their reluctance to activate depression management protocols in the face of a positive screening.

Conclusion

The use of an unannounced SP presenting with a somatic complaint and underlying depression reinforced the importance of effective clinic screening and helped target areas for improvement in residents' depression curriculum and training to ensure that residents effectively treat depression in the primary continuity care setting.