Screening for Psychiatric Illness with a Combined Screening and Diagnostic Instrument

Abstract

OBJECTIVE

To determine 1) if the PRIME-MD, a two-step screening and diagnostic instrument for psychiatric disorders, increases diagnosis and intervention when actively implemented in a busy general medicine clinic, and 2) the type of staff support required to achieve sufficient implementation to realize gains in diagnosis and treatment.

DESIGN

We introduced the PRIME-MD into a large general medicine clinic with repeated rotation of four support conditions for implementation: (1) no support, (2) nonclinical staff support (NCSS), (3) nursing staff (RN) support, and (4) a written “Prompt” condition.

SETTING AND PATIENTS

Patients (N= 2,263) attending a general medicine clinic at a Veterans Affairs Medical Center.

MEASUREMENTS AND MAIN RESULTS

Outcome measures were (1) PRIME-MD questionnaire and interview use, (2) overall psychiatric diagnosis, (3) new psychiatric diagnosis, and (4) provider intervention for psychiatric conditions. The NCSS, RN support, and prompt conditions resulted in similar rates of questionnaire use but significantly different rates of structured interview use. The NCSS condition was associated with significant increases in new diagnosis, and the RN support and Prompt condition were associated with significant increases in new diagnosis and intervention compared with no support.

CONCLUSIONS

Nursing staff support resulted in sufficient PRIME-MD implementation to achieve gains in both new diagnosis and provider intervention compared with no support. These gains occurred in a busy primary care clinic with nonselected providers and customary visit lengths. This level of support should be achievable in most clinical settings.

Psychiatric disorders are common in primary care settings. Twenty-five percent to 33% of patients treated in primary care clinics have a mental disorder,^1,2 but only a small minority of these patients are recognized and treated.^3–6 Patients with untreated psychiatric disorders often have significant morbidity and functional impairment.^7–10 Improving detection of mental illness in primary care may be an important step in addressing unmet mental health needs and decreasing associated morbidity.

Several studies have examined the effect of patient screening questionnaires on primary care provider (PCP) detection and treatment of psychiatric disorders.^{11–14,17,18} These studies have produced mixed results. Some studies have demonstrated an increase in the rate of psychiatric diagnosis with screening questionnaires,^11–18 while others have found screening questionnaires to have little or no effect on PCP diagnosis or management of psychiatric disorders.^15,16,19 No large-scale studies have clearly demonstrated improved patient outcomes as a result of screening programs.²⁰

Investigators have attributed the lack of uniform increases in diagnosis or treatment with questionnaire use to a number of factors. Some have concluded that screening instruments identify too many “false positives,”^16,21,22 or focus too narrowly on one psychiatric disorder—usually depression—while patients frequently have another psychiatric disorder or suffer from more than one disorder.^23–25 Others argue that general physicians lack the time for or interest and expertise in the treatment of psychiatric disorders.²⁶ They note that PCPs may not be familiar with the diagnostic criteria for psychiatric disorders,³ and may not know how to follow up on indications of distress on screening questionnaires.

The PRIME-MD is an innovative screening procedure for mental disorders that directly addresses the concern that PCPs fail to diagnose or treat patients because of a lack of familiarity with diagnostic criteria or because they are using a screen with a narrow focus on one mental disorder.²⁷ The PRIME-MD includes both a self-administered patient questionnaire and a provider-administered structured diagnostic interview. The 26-item patient questionnaire screens for five common categories of mental disorders in primary care: mood disorders, anxiety disorders, somatoform disorders, eating disorders, and alcohol abuse. If a patient has a positive screen on the questionnaire, the provider follows up with the structured diagnostic interview, the Clinician Evaluation Guide (CEG). The CEG assists the physician in either making or ruling out psychiatric disorders and can be completed in approximately 8 minutes.

Diagnoses made by PCPs with the CEG appear to be valid. There is good agreement between diagnoses made by PCPs using the CEG and diagnoses made by mental health specialists conducting more extensive telephone interviews, a comparison diagnostic standard (Cohen's κ = 0.71 for “any psychiatric diagnosis”).²⁷ Patients with PRIME-MD diagnoses have lower functional scores on the SF-20, more disability days, and higher rates of health care utilization.^27,28 Forty-eight percent of patients who received diagnoses on the PRIME-MD in research settings had not been previously recognized as having a psychiatric disorder.²⁷

Clearly, the PRIME-MD is a promising instrument that may prove useful in primary care settings. However, the study that examined the utility of the PRIME-MD provided levels of support sufficient to ensure that all portions of the instrument—questionnaire and the structured interview (CEG)—were completed as designed.²⁷ Volunteer physicians who were interested in psychosocial issues administered the instrument, and they received extra time and other incentives for its use.²⁷

For the PRIME-MD to be effective, it also must perform well in clinics with realistic levels of staff interest and support. Approximately 30,000 physicians and health care professionals are expected to receive the PRIME-MD by the end of 1997, yet anecdotal accounts have cited difficulties in implementing the instrument in clinical settings.²⁹ There are no data on either the numbers of physicians using the PRIME-MD in clinical practice or the manner in which the test influences patient care.²⁹

The aim of this study was to determine the effectiveness of the PRIME-MD instrument when implemented in a clinic with nonselected providers, customary visit lengths, and realistic levels of clinic support. Specifically, this study examined the type of support that would be required to achieve sufficient instrument implementation to realize gains in diagnosis and provider intervention.

METHODS

Study Site

The General Medicine Clinic at the Ann Arbor Vetrans Affairs (VA) Medical Center is staffed by resident physicians, staff physicians, physician assistants, and nurse practitioners. Residents are divided equally between the first, second, and third year of postgraduate training in internal medicine. Nurse practitioners and physician assistants in the clinic have considerable primary care experience, with an average of 7.5 years of experience (range 1.6 –13.9 years). Resident physicians see an average of six to seven patients in the morning clinic and work one to three mornings per week in the clinic. Physician assistants and nurse practitioners see approximately eight patients in the morning clinic and work two to five mornings per week. Providers average 9 to 15 minutes in face-to-face contact during a scheduled visit.

A total of 54 providers, 45 residents, 2 staff physicians, and 7 physician assistants or nurse practitioners, provided direct patient care in the clinic during the study period. The most common medical diagnoses in the clinic were hypertension, coronary artery disease, arthritic conditions, chronic obstructive pulmonary disease, and diabetes. As is traditional in VA settings, scheduled patients make no direct payments for PCP services, although private insurance was occasionally billed.

Patient Population

Patients were included in the study if they were seen in the general medicine clinic between the hours of 8 and 10:30 am during study weeks 1 through 6 and between 8 and 11 am during weeks 7 through 15; if this was their first or only visit to the clinic during the study period; and if their chart note was returned to the clinic clerk by 12:30 pm on the day of the visit.

A total of 3,491 patient visits occurred between the hours of 8 and 10:30 am during the 15-week study period; 2,735 visits (78%) had chart notes returned by 12:30 pm; 2,282 of these 2,735 visits represented first or only visits (unique patients) and were eligible for inclusion in the study.

Intervention

Training

All primary care providers in the General Medicine Clinic were offered formal training in use of the PRIME-MD instrument. The 1-hour training session consisted of a review of a videotape provided by the Pfizer Corporation, followed by practice using the instrument under the supervision of a study psychiatrist. The majority of providers underwent this 1-hour group training. Providers who missed the group training sessions were contacted by a study psychiatrist (SF or MV) for individual training in instrument use. Providers were informed that study personnel were available to answer questions about the PRIME-MD instrument at any time during the study period.

We did not verify that providers had learned to use the PRIME-MD instrument correctly. We also did not explicitly train the new first-postgraduate-year residents (n = 5) who entered the clinic after cessation of active support and were present during the last study week (week 15). Rather, we offered training that would be feasible in most community clinics and allowed staff physicians and preceptors to either transmit or not transmit use of this instrument to newcomers, along with other clinic procedures and advice on patient care.

Instrument Implementation and Support Conditions

After training in use of the instrument, the PRIME-MD was implemented in the clinic with varying types and intensity of staff and clinic support. There were three active support conditions for implementation in addition to a “no-support” condition; the active support conditions were (1) nonclinical staff support (NCSS), (2) nursing (RN) support, and (3) a provider Prompt condition. Support conditions were rotated weekly.

In the no-support condition, the PRIME-MD instrument was available but staff members were not given specific responsibility for its distribution or collection. In the NCSS condition, nonclinical research staff were given responsibility for distributing, collecting, and placing questionnaires in the chart prior to physician interview. In the RN support condition, nursing staff who already worked in the clinic were given responsibility for distributing, collecting, and placing patient questionnaires in the chart prior to the physician interview. Finally, in the prompt condition, both nonclinical staff and nurses had responsibility for distributing and collecting patient questionnaires. In addition, research staff screened the questionnaires and placed written prompts in the charts of patients with positive screens. Provider prompts were printed on brightly colored paper and advised the providers to complete the appropriate modules in the structured interviews. The prompts were worded as follows:

“PROVIDER: This patient screened POSITIVE on the PRIME-MD questionnaire. Please evaluate the patient with the enclosed Clinical Evaluation Guide on the following modules: □ MOOD, □ ANXIETY, □ ALCOHOL, □ EATING, □ SOMATOFORM.”

In the final week of the prompt condition (study week 13), a clinic nurse completed the CEG for 19 patients with positive screens on a convenience basis. Data from these patients were eliminated from study analyses.

Active support conditions were alternated each week between March 27 and June 16, 1995 (study weeks 2–13). The rotation was planned to distribute support conditions throughout the study period and ensure that each active support condition was in place during 4 weeks of the 15-week study period. Weeks were not randomly assigned. The week prior to initiating active support (study week 1), the week immediately following cessation of active support (study week 14), and a week that followed 8 weeks after cessation of active support (study week 15) constituted the no-support condition.

Data Collection

Data were collected for five dichotomous outcome variables: (1) PRIME-MD questionnaire use, (2) PRIME-MD structured interview (CEG) use, (3) any psychiatric diagnosis during the visit, (4) new psychiatric diagnosis, and (5) provider intervention for psychiatric conditions. Questionnaire completion and rates of interview (CEG) use were recorded by research assistants during each support condition. Information about clinician diagnosis and initiation of treatment was obtained through structured review of chart notes.

Providers were considered to have used the semistructured interview, the CEG, if any module in the CEG showed written notation or if CEG use was recorded in the patient note.

Providers were considered to have made a psychiatric diagnosis if any specific or nonspecific diagnostic notation was noted on the CEG, the patient questionnaire, or the progress note. For example, a provider who noted “anxiety” as an important patient problem was considered to have made a diagnosis of “anxiety, not otherwise specified.” Providers were counted as having made a new diagnosis if the progress note indicated that the physician was making a diagnosis during the study visit and there was no indication in the note that the patient had a past psychiatric history. Providers were considered to have intervened for a psychiatric condition if they (1) made a new referral to a mental health provider, (2) started a new psychotropic medication, (3) provided supportive counseling, (4) advised the patient to continue ongoing mental health treatment, or (5) continued a previously prescribed psychotropic medication.

Statistical Analyses

All analyses were conducted on 2,263 patient visits—the 2,282 patient visits that met the eligibility criteria outlined above, with data from the 19 patients with nurse-administered CEGs eliminated. The rates of positive response to the five dichotomous outcome variables of questionnaire completion, CEG completion, psychiatric diagnosis, new psychiatric diagnosis, and provider intervention were calculated for each support condition. The frequency of questionnaire and CEG completion was calculated for weeks 2 through 15 (excluding week 1) as there was no use of the PRIME-MD questionnaire or interview prior to initiation of active support for implementation in week 2. Rates of psychiatric diagnosis and provider intervention were calculated for all weeks (1–15) of the study.

Bivariate analyses of the relation between PRIME-MD use and psychiatric diagnosis or intervention and between support conditions and psychiatric diagnosis or intervention were conducted with contingency tables and χ² statistics.

A multivariable generalized estimating equation (GEE) was completed for each of the five dichotomous outcome variables—questionnaire completion, CEG completion, psychiatric diagnosis, new psychiatric diagnosis, and provider intervention. The five multivariable GEE analyses included dummy 0 –1 predictor variables for each of the support conditions and for patient age group. The analyses also accounted for correlation of observations by provider.

Patient age group was categorized as follows: (1) patients under 65 years of age, (2) patients 65 to 74 years of age, and (3) patients over 75 years of age. Week of study was included as a linear term in all five analyses to adjust for learning or extinction effects from cumulative instrument exposure. The possibility of a nonlinear relation between each of the five outcomes and study week was also investigated. A “week-squared” term was found to be significant in the model for CEG use and was incorporated into this analysis. Estimated parameters were adjusted for all predictors in the model.

All statistical analyses were completed using the SAS software, version 6.12 (SAS Inc., Cary, NC).

RESULTS

Patient Demographics

Study patients had a mean age of 63.1 years (range 24 –89 years, SD 11.5 years). Forty-six percent of patients were under 65 years of age, 42% were between 65 and 74 years of age, and 12% were over 75 years of age. Ninety-seven percent of the patients were male.

Relations of PRIME-MD Use and Patient Diagnosis or Treatment

Questionnaires were completed by 69.8% of study patients during the entire 15-week study period. Eighty-one percent of patients completing a questionnaire had a “positive screen.” Providers used the interview (CEG) in 20.8% of patients with a positive screen.

Patients who completed a questionnaire were significantly more likely to receive a psychiatric diagnosis (χ² = 7.5, p < .01), a new psychiatric diagnosis (χ² = 32.7, p < .0001), and a provider intervention (χ² = 10.2, p < .001) than patients who did not complete a questionnaire. Twenty percent of patients who completed a questionnaire were given a psychiatric diagnosis compared with 15.2% of patients who did not complete a questionnaire. Seven percent of patients who completed a questionnaire received a new psychiatric diagnosis compared with 1% of patients who did not complete a questionnaire, and 12% of patients who completed a questionnaire received a provider intervention compared with 8% of patients who did not complete a questionnaire.

Patients who were interviewed with CEG were far more likely to receive a psychiatric diagnosis (χ² = 100.187, p < .0001), a new psychiatric diagnosis (χ² = 372.7, p < .0001), and a provider action (χ² = 60.4, p < .0001) than patients who were not interviewed with the CEG.

Relations of Support Conditions and PRIME-MD Use

The relation between support conditions and PRIME-MD questionnaire and CEG use is summarized in Table 1. All active support conditions (NCSS, RN, and prompt) significantly increased questionnaire usage over no-support conditions; however, the NCSS, RN, and prompt conditions did not differentially affect the frequency of questionnaire completion. There were high completion rates (80% – 81%) in all active support conditions.

Table 1.

Relation of Support Conditions to Instrument Use and Diagnosis or Intervention

All active conditions increased CEG use compared with no support, and there were significant differences in rates of CEG use among the active support conditions. The RN condition showed a strong trend toward increased CEG use compared with NCSS (odds ratio [OR] 1.5; 95% confidence interval [CI] 1.0, 2.2;p = .055), and the prompt condition was significantly more likely to result in CEG completion than either NCSS (OR 2.5; 95% CI 1.6, 4.1;p < .0001) or RN support (OR 1.7; 95% CI 1.1, 2.8;p < .05) (Fig. 1). The CEG was used very selectively by physicians under all support conditions, even when patients had three or more positive screens on the questionnaire.

Figure Support conditions and CEG use. After adjustment for study week and patient age group, there was a trend for RN support to result in significantly more CEG use than NCSS ( p = .55). The prompt condition resulted in significantly more CEG use than either NCSS or RN support (p < .001, and p < .05, respectively).

As illustrated in Figure 2, CEG use showed a nonlinear relation to week of study. There was no CEG use in the week after training but before institution of support for implementation. Use of CEG increased dramatically with the institution of active support, then declined rapidly with the cessation of active support. The PRIME-MD was used in the minority of patients in the week that immediately followed withdrawal of active support. Eight weeks after the withdrawal of active support, the PRIME-MD questionnaire and CEG were not used at all.

Figure Percentage of CEG use by study week.

There was no association between patient age group and questionnaire/CEG use. There was also no relationship between study week and questionnaire use.

Relations of Support Conditions and Diagnosis or Intervention

Table 1 summarizes the rates of psychiatric diagnosis, new psychiatric diagnosis, and provider intervention under different support conditions and includes results of unadjusted bivariate χ² analyses. Table 2 summarizes the adjusted ORs of psychiatric diagnosis, new diagnosis, and provider action under the active support conditions compared with no support.

Table 2.

Results of GEE Analyses Comparing Active Support Conditions to No Support*

When adjusted for age category and study week, there were no significant differences between the NCSS and RN conditions and no support in the rate of “any diagnosis.” The prompt condition showed a trend toward an increase in “any diagnosis” over no support (p = .06). All active support conditions resulted in significantly more new psychiatric diagnoses than no-support conditions, and the RN and prompt conditions resulted in significantly more provider actions than no support.

The active support conditions (NCSS, RN, and prompt) did not differentially affect the rates of psychiatric diagnosis, new psychiatric diagnosis, or provider intervention. Study week was not significantly related to the rate of any psychiatric diagnosis (p = .58), new psychiatric diagnosis (p = .17), or provider intervention (p = .29) in multivariable GEE analyses.

DISCUSSION

The utility of the PRIME-MD has been examined with levels of support that allowed complete implementation of the two-step instrument.²⁷ This study examined the effectiveness of the PRIME-MD under conditions more likely to be achievable in clinical practice. We delivered predetermined levels of clinic support rather than ensuring full instrument implementation. Realistic support conditions resulted in differential use of the questionnaire and semistructured interview (CEG), and allowed us to determine the usefulness of partial implementation. We specifically examined the level of support required for sufficient implementation to realize gains in diagnosis or intervention.

Use of PRIME-MD was associated with increased rates of psychiatric diagnosis and provider action when administered by nonselected providers with realistic levels of support. A high percentage of patients completed the self-administered questionnaire. Consequently, response bias, or selective completion of questionnaires by patients with psychiatric difficulties, was not likely to have played a role in the association between questionnaire use and diagnosis or treatment. The rate of positive screens on questionnaires in this study was comparable to those observed in the study by Spitzer and colleagues in which all participating patients completed questionnaires. The association between CEG completion and diagnosis or intervention needs to be interpreted with more caution. Providers used the CEG very selectively in eligible patients, and there were most likely intervening factors in the clinical encounter that increased both CEG use and diagnosis or intervention.

This study primarily investigated the impact of staff support on the effectiveness of the PRIME-MD; however, our data support previous work showing that screening questionnaires affect diagnosis and management.^11,14,18 Rates of overall notation of psychiatric diagnosis and new diagnosis rose significantly with questionnaire completion. The impact of PRIME-MD questionnaires may have been increased by the availability (despite low use) of a follow-up structured interview.

The active staff support conditions for PRIME-MD implementation resulted in similar rates of questionnaire completion but very different rates of semistructured interview (CEG) use in the general medicine clinic. Self-administered patient questionnaires proved easy to implement, while semistructured interviews were used selectively even when accompanied by provider prompts. Increments in active staff support increased CEG use. However, increments in active staff support did not result in significant increases in rates of psychiatric diagnosis. Rates of provider action increased with each increment in staff support, but again these increases did not reach statistical significance. These data suggest that supplying extra clinic support to boost CEG use beyond 20% to 22% of positively screening patients may not result in further gains in diagnosis or provider action.

Our data also suggest that PRIME-MD use depends on continued active clinic support and will decay rapidly without such support. There was essentially no PRIME-MD use without active clinic support. Rates of new diagnosis also declined with withdrawal of active support for implementation. However, increases in overall diagnosis and provider intervention that occurred with prolonged exposure to the PRIME-MD appeared to be more robust and were still present 8 weeks after cessation of active support.

Study Limitations

Several study limitations may limit the generalizability of the conclusions discussed above. The study was conducted in one site, a general medicine clinic in a VA Medical Center. Findings from this site may not generalize to other primary care settings. The patient population was both older than many primary care populations and predominantly male. Screening has been noted to be more effective in increasing diagnosis and treatment in an older male population than in the general primary care population.¹⁵ Also, many of the providers in this study were either resident trainees or midlevel practitioners and may have been more amenable to using screening instruments than experienced PCPs practicing in the community.

Although we rotated weeks of support, we were not able to randomize providers to different support conditions, and most providers were exposed to several support conditions. This may have “contaminated” provider responses and made the support conditions look more similar in their effects on diagnosis or intervention than if we had randomized providers. In study analyses, we included week of study to control for the effect of cumulative exposure to the instrument and also accounted for correlation of observations by provider. This may have partially mitigated the effect of multiple exposure.

There may be concerns about the definition of “provider intervention” used in this study. Specifically, concerns may arise about including “continuing psychotropic medication” or “advising reinitiation or continuation of mental health treatment” as provider actions. These interventions seem passive—actions that may be unaffected by the PRIME-MD and serve only to dilute the apparent effects of support for implementation. However, in post-hoc analyses, we found that “continuing a psychotropic medication” or “advising continuation of mental health treatment” was significantly increased by support for implementation. This suggests that these interventions are responsive to PRIME-MD implementation. PCPs may be unaware of the specialized mental health treatments that their patients have received or are currently receiving. Becoming aware of previous or parallel mental health treatment and participating in treatment through medication monitoring or discussion is likely to improve care and increase collaboration. We therefore considered this to be an important PCP action and included it in our analyses.

Lastly, this study does not address whether the increases in new diagnosis and provider intervention that occurred with PRIME-MD implementation were accompanied by improved patient outcomes. As pointed out by Schwenk in an earlier editorial in JGIM, this is an essential element in the decision about whether to recommend screening at all.²² There has been considerable debate about whether increased detection and treatment of mental disorders in primary care settings will actually benefit patients,³⁰ and the U.S. Preventive Services Task Force has concluded that there is “insufficient evidence to conclude that routine depression screening is indicated in unselected patients, because it has not been shown that the early detection and treatment of depression in primary care leads to improved outcome . . . .”³¹ The Task Force does recommend routine screening for problem drinking, one of the other screens included in the PRIME-MD, noting benefits from intervention with problem drinkers. The value of screening for other psychiatric disorders in primary care is unknown and awaits further research.

Summary

Nonclinical staff support for PRIME-MD implementation was sufficient to increase rates of new diagnosis compared with no support, and nursing staff support was sufficient to obtain increases in both new diagnosis and provider intervention compared with no support. These levels of clinic support should be readily achievable in most settings.