Computerization of Mental Health Integration Complexity Scores at Intermountain Healthcare

Abstract

Intermountain Healthcare’s Mental Health Integration (MHI) Care Process Model (CPM) contains formal scoring criteria for assessing a patient’s mental health complexity as “mild,” “medium,” or “high” based on patient data. The complexity score attempts to assist Primary Care Physicians in assessing the mental health needs of their patients and what resources will need to be brought to bear. We describe an effort to computerize the scoring. Informatics and MHI personnel collaboratively and iteratively refined the criteria to make them adequately explicit and reflective of MHI objectives. When tested on retrospective data of 540 patients, the clinician agreed with the computer’s conclusion in 52.8% of the cases (285/540). We considered the analysis sufficiently successful to begin piloting the computerized score in prospective clinical care. So far in the pilot, clinicians have agreed with the computer in 70.6% of the cases (24/34).

Introduction

Frequently in today’s primary care setting, providers are confronted with diagnosing and treating mental health concerns. Gatchel and Oordt estimated that 70% of primary care visits stem from psychosocial issues.¹ And Kessler et al. found that almost 50% of mental health disorders were treated in primary care.² Unfortunately, as Collins et al. lament, “most primary care doctors are ill-equipped or lack the time to fully address the wide range of psychosocial issues that are presented by the patients.”³ Indeed, Vermani et al. cite evidence of low detection rates and poor quality of care for a variety of mental health disorders in the primary care setting.⁴

In this context, researchers at Intermountain Healthcare have sought to assist the Primary Care Physician (PCP) with the task of managing mental health issues in the primary care setting. Intermountain has organized itself around key clinical processes and developed “Clinical Programs”⁵ in nine clinical areas to address those processes. In multidisciplinary fashion, the clinical programs identify best practices and standardize the care delivered within the organization. The Primary Care Clinical Program is one such program. Within the Primary Care Clinical Program, Intermountain researchers and clinicians have developed a Mental Health Integration (MHI) program^6,7 that espouses a patient-centered, team-based approach to diagnosing and treating mental illness in the primary care setting. The central concept of the program is to eliminate “the chasm between physical health care and mental health care—rolling both into a comprehensive whole that is addressed by a high-functioning team.”⁶ The program standardizes processes and supports the PCP with expanded team support (including mental health professionals, community advocates, and care management) to address the mental health needs of their patients and families. Each member of the team is trained in specific responsibilities (communication, shared decision-making, etc.) that contribute to a collective whole health care plan. A standardized process is outlined via a Care Process Model (CPM).

A primary objective of the CPM is to determine which of three levels of care is appropriate for the patient – routine (PCP-based) care, collaborative care (which adds care manager and mental health specialist participation), or enhanced care (which further increases care manager and mental health specialist participation). The care determination is made by assessing “Mental Health Complexity.” A complexity of “mild” suggests routine care is appropriate, a complexity of “moderate” suggests collaborative care, and a complexity of “high” suggest enhanced care.

To assess complexity, the PCP asks the patient to fill out a paper-based “MHI packet” as a screening mechanism when a patient presents to the PCP with possible mental health concerns. The packet solicits 47 pieces of data from the patient on 21 facets related to mental health (some facets involve more than one piece of data) such as:

• Previous mental health treatment

• Level of chronic pain

• Family Rating Scale (assessing level of support of family relationships)

• Validated tools for diagnosing and treating patients with mental health conditions, such as the Patient Health Questionnaire (PHQ-9)

Our researchers have developed classification criteria for each facet that classify a patient as “mild,” “moderate,” or “high” complexity for the facet. (These will be referred to as “sub-scores” for the rest of the paper.) For illustration purposes, Table 1 shows the criteria for several sub-scores. Some sub-score classification criteria, such as those for the Patient Health Questionnaire, are drawn from published, validated instruments. Other sub-score criteria, such as those for chronic pain severity, are based on the experience of Intermountain clinicians and researchers. Over the past decade, Intermountain MHI researchers have refined and formalized the sub-score complexity classification criteria and have documented and published them internally.

Table 1.

An extract of sub-score criteria for several sub-scores.

Mild	Moderate	High
Previous mental health treatment: 1 episode	Previous mental health treatment: ≥ 2 episodes	Previous mental health treatment: multiple treatment failures
Chronic Pain Severity: 0–3 on a scale of 10	Chronic Pain Severity: 4–6 on a scale of 10	Chronic Pain Severity: 7–10 on a scale of 10
Family Rating Scale: Style III (balanced/secure)	Family Rating Scale: Style II (confused/chaotic)	Family Rating Scale: Style I (disconnected/avoidant)
Patient Health Questionnaire: Symptom score: <5 (out of a list of 9) Severity score: 10–14 (maximum possible=27)	Patient Health Questionnaire: Symptom score: >=5 (out of a list of 9) Severity score: 15–19 (maximum possible=27)	Patient Health Questionnaire: Symptom score: >=5 (out of a list of 9) Severity score: >=20 (maximum possible=27)

Currently, a portion of the patient responses (corresponding to 14 of the 21 facets) is transcribed by clinical staff into Intermountain’s HELP2⁸ clinical information system. This portion represents those items most frequently filled in by patients and most influential in determining complexity based on our experience. The PCP is then asked to apply the sub-score criteria to the information to obtain sub-scores. Finally, the PCP aggregates the sub-scores into an overall “high”/“moderate”/“mild” complexity score that will dictate the initial level of care for the patient. The guidelines for aggregation have heretofore been incompletely specified; examples of sub-scores and corresponding overall complexity had been published internally, but logically-complete criteria had not been defined. The PCP reviews the sub-scores and the examples and makes his or her best judgment of an overall complexity.

The MHI program has shown positive clinical and financial outcomes.^6,7 However, improving the consistency and appropriateness of complexity scores is critical to the long-term success of the program. Consequently, the MHI researchers sought the assistance of the Intermountain Medical Informatics group to automate the scoring process. Together, we reasoned that computerization would accrue three benefits:

1. More scores will be generated. Clinician scoring has not been mandatory. Thus, in the busy clinic environment, some patients do not receive a score, even though the patient has filled out a Mental Health packet. We theorized that more scores would result in more appropriately directed care.

2. Scores will be more consistent. Undoubtedly, there has been variability in clinicians’ scoring – especially in the aggregation of sub-scores into an overall complexity. Computerization would process inputs objectively and consistently, promoting standardization of care and enabling more reliable stratification of patients for research purposes.

3. Scores will be more appropriate. There is no “correct” or “gold standard” overall complexity; our clinician researchers have conceived of it in experience-based fashion to assist the PCPs in managing issues outside their expertise. We anticipate, though, that we can continually improve the complexity criteria as our researchers and the PCPs learn from each other through an iterative process of applying the criteria to patients, evaluating the care levels dictated, and refining again.

Our plan was as follows:

• Implement the classification criteria in Intermountain’s decision support system

• Execute the classification criteria on retrospective data

• Analyze results and refine the classification criteria

• Pilot the classification criteria with several primary care providers and solicit their feedback

• Make necessary refinements

Once these steps are completed, the intent is to implement the classification in Intermountain’s production system. When the patient’s packet information is entered, the computer will deduce a complexity and present it to the patient’s PCP, thus suggesting a level of care to begin with. The PCP will be able to either accept that level of care or set another level based on his or her clinical judgment.

We report here on our efforts through the piloting step.

Materials and Methods

Availability of data in the system

The first step in implementation of the algorithm was to verify the availability of the input data. We verified that the 24 pieces of data needed to conclude the 14 sub-scores were available in HELP2. Table 2 shows the 14 sub-scores included in the computerized algorithm, with the number of data items involved in concluding each sub-score. These 24 data items became the basis of the computerization.

Table 2.

The 14 sub-scores addressed by the computerized algorithm

Sub-score	Category	Number of Data Items (Data Items)
Number of Somatic Complaints	S1	1 (Number of Somatic Complaints, 0–9)
Chronic Pain Severity	S1	1 (Chronic Pain Severity, 0–10)
Sleep Problem Severity	S1	1 (Sleep Problem Severity, 0–10)
Substance Use	S1	1 (Current Substance Use, Y/N)
Overall Impairment	S2	1 (Impairment Rating Score, 1–7)
Overall Health	S2	1 (Overall Health, 1–10)
Family Relational Style	O1	3 (Family Style I Score, Family Style II Score, Family Style III Score)
Family Pattern Profile (Support Person)	O1	1 (Most Common Support)
Patient Health Questionnaire (PHQ-9)	O1	2 (PHQ-9 Symptom Count, PHQ-9 Severity Score)
Suicide Assessment	O2	3 (PHQ-9 Q9 response, Suicide State, Suicide Risk)
Anxiety/Stress Disorders (GAD-7)	O3	3 (GAD-7 Q1 Score, GAD-7 Q2–5 Score, GAD-7 Q6–7 Score)
Mood Disorder Questionnaire (MDQ)	O3	3 (MDQ Q1 Score, MDQ Q2 Response, MDQ Q3 Response
Mood Regulation	O3	2 (Symptom Score, Impairment Score)
Attention Deficit Hyperactivity Disorder (ADHD)	O3	1 (ASRS-v1.1 Part A Score)
Total number of sub-scores: 14		Total number of data items: 24

PHQ-9: Patient Health Questionnaire-9⁹

GAD-7: Generalized Anxiety Disorder-7¹⁰

MDQ: Mood Disorder Questionnaire¹¹

ASRS-v1.1: Adult ADHD Self-Report Scale (ASRS) Version 1.1¹²

Adequately expressive logic

Morris¹³ asserts, “An adequately explicit clinical method is one that contains adequate detail to generate specific instructions (patient-specific orders) without requiring judgments by the clinician.” Adequate explicitness is essential in the computerization of clinical decision support criteria.^13–16 We held several discussions in which we strengthened the explicitness of the sub-score algorithms. For example, the last row of Table 1 shows the original logic for deducing one particular sub-score – the sub-score for the PHQ-9.

Examination found that cases such as the following would have indeterminate results:

• A symptom count < 5 with a severity score < 10

• A symptom count of 5 with a severity score of 10–14

• A symptom count of 5 with a severity score < 9

As we completed the logical “decision table,” we recognized that the “symptom” component was actually superfluous to the logic. Refining the criteria resulted in the simplification shown in Table 3.

Table 3.

A simplification of the sub-score criteria for the Patient Health Questionnaire.

Mild	Moderate	High
Patient Health Questionnaire: Severity score: <15 (maximum possible=27)	Patient Health Questionnaire: Severity score: 15–19 (maximum possible=27)	Patient Health Questionnaire: Severity score: >=20 (maximum possible=27)

Discussions regarding boundary conditions, logical operations (clarification of ORs vs. ANDs), and logical completeness resulted in fully computerizable sub-score classification criteria.

The next step was to fully specify the criteria for aggregating sub-scores into an overall complexity. In the clinical judgment of our MHI researchers, subjective data items contributed differently to complexity than did objective items. They consequently categorized each sub-score as either a subjective (S) or an objective (O) measure, depending on whether the data contributing to the sub-score were subjective or objective in nature. They further divided the subjective scores into two subcategories (S1 and S2), the S1 scores being more critical (based on their clinical experience) to the overall complexity. The resulting categories are shown in Table 2. This allowed the researchers to develop the empiric rules shown in Figure 1.

Figure 1.

Logic for aggregating sub-scores into overall complexity.

Note the rules were arranged in waterfall fashion (IF-THEN-ELSEIF), the last condition being

ELSE overall complexity = “mild”

In essence, the “default” complexity in the absence of data was “mild,” i.e., the patient should receive routine care.

Computerization and execution

Two medical knowledge engineers converted the classification criteria into Java code within Foresight, Intermountain’s existing Computerized Decision Support (CDSS) system.¹⁷ Foresight is a J2EE-compliant, Intermountain-developed decision logic execution engine coupled with a sophisticated clinical data monitor. Foresight rules are written in Java and deployed in an Oracle Weblogic server.

We executed the classification criteria on 500 patients for whom providers had entered a complexity score in HELP2 between August and November of 2012. The results were exported into Oracle tables and made available to the clinical experts for review and analysis.

Preliminary Analysis

We narrowed our preliminary analysis to 273 patients who had suicide assessment (the most critical facet to complexity) or greater than 80% of the 24 items. In this population, the clinician agreed with the computer in 53.1% of cases. An MHI researcher (BR) reviewed the results of the classification criteria for twenty randomly sampled cases. Upon reviewing the patient data, she judged that the classification was not weighing two factors heavily enough: an MHI packet question regarding the patient’s level of thoughts of death and suicide, and a question regarding the patient’s access to a support person. We recognized this would result in a general increase in complexity – possibly higher than the clinicians might tend to score – but we reasoned that at this learning and improvement stage it was better to “overstate” complexity (and hence “over-involve” the mental health specialists than “under-involve” them). If, as a result, the specialists told the PCPs their involvement was unnecessary, learning would occur and the criteria could be adjusted. We adjusted the criteria to be more sensitive to these factors.

We re-ran the classification on the 2012 patients and on 500 patients whose complexity was entered between June and October of 2013. Again, we focused analysis on patients who either had suicide data entered or who had greater than 80% of the 24 data items entered. These criteria left 540 patients to analyze.

Results

Retrospective Data Analysis Results

Clinician agreement frequency on the set of 540 patients is shown in Table 4. The clinician agreed with the computer in 285 of the 540 cases (52.8%). Of the 255 cases in which the provider and the computer disagreed, the algorithm was “higher” 179 times (70.2%). A graphical view of the frequencies, comparing the frequency of cases in which the clinician agreed with the computer, cases in which the clinician was “higher,” and cases in which the computer was “higher” is shown in Figure 2.

Table 4.

Clinician and computer agreement in the retrospective data.

Computer
		Mild	Moderate	High	TOTAL
Clinician	Mild	47	89	14	150 290 100
	Moderate	36	178	76
	High	4	36	60
	TOTAL	87	303	150	540

Figure 2.

Clinician agreement with computer classification.

We (BR and PB) performed a case-by-case examination of the forty cases in which the clinician gave a complexity score of “high” while the computer gave a lower score, to gain greater insight into such cases and ensure that the algorithm was not “missing” complexity it should have been detecting. We reviewed charts to determine what data might have influenced the providers to score the patients as “high.” In most of the cases, we found evidence of comorbidities (diagnoses or multiple medication orders). Comorbidities are considered in the paper version of the algorithm, but were not included in this first version of the computerized algorithm, i.e., they were not included in the “top 14” sub-scores that were included in HELP2 and computerized.

In some cases, chart review did not reveal the reasons for the provider’s score of “high.” We reasoned that occasionally the computer algorithm might not be able to generate a “high enough” score because it did not have access to the proper data elements, and in such situations, the provider seemed to be properly disagreeing.

In general, we hypothesized that disagreements could have stemmed from several sources:

• The clinician and the computer may have been using different sets of inputs. There may be factors that are salient to complexity that the clinician was privy to but that were not available to the classification criteria.

• The clinicians may have disagreed with the MHI researchers’ judgment that was reflected in the criteria logic.

• The clinicians may have agreed with the criteria logic, but may have committed errors, e.g., overlooked data or misapplied the criteria.

• There may be certain combinations of inputs which were not considered in the development of our classification criteria.

The rate of agreement was not a major concern at this point. There have been extensive examples of computerizing guidelines and protocols at Intermountain in which compliance is initially low. But as clinicians become more comfortable with the logic of the computer and the logic is adjusted according to clinician feedback, compliance increases. We view this effort as akin to those experiences.

Pilot

We moved forward with a pilot to learn why the clinicians were disagreeing and to see which of the hypothesized reasons for disagreement were in operation. To date, two licensed providers – a psychiatric nurse practitioner and a PhD psychologist – have participated. When they sign a Mental Health Integration note in HELP2, an alert is sent to their HELP2 message queue, indicating the complexity score the algorithm has generated. The participants acknowledge the alerts, either accept or reject the scores, and provide feedback (especially what a rejected score should have been in their judgment). This functionality is only operational for the participating clinicians.

To date, the two participants have responded to 34 scores. They agreed with the computer scores in 24 of the 34 cases (70.6%). Table 5 presents the agreement results. Table 6 presents data availability data for the ten cases in which the clinician disagreed with the computer. For each case, the table shows the computer’s assessment vs. the clinician’s and the number of data items (out of the maximum of 24 possible) that the computer actually found entered in HELP2 for that patient.

Table 5.

Clinician and computer agreement in the pilot.

Computer
		Mild	Moderate	High	TOTAL
Clinician	Mild	13	0	0	13 16 5
	Moderate	6	9	1
	High	0	3	2
	TOTAL	19	12	3	34

Table 6.

Data availability for disagreement cases in the pilot.

Case	Computer	Clinician	Number of data items available (out of 24)
1	Moderate	High	24
2	Mild	Moderate	3
3	Mild	Moderate	3
4	Mild	Moderate	6
5	Mild	Moderate	6
6	High	Moderate	6
7	Mild	Moderate	3
8	Moderate	High	24
9	Mild	Moderate	3
10	Moderate	High	22

Discussion

Summary and Findings

In the retrospective data, the clinicians would have agreed with the classification only a little more than half the time. When they disagreed, the classification tended toward greater complexity. We purposely tended toward overstating complexity because at this stage of the algorithm’s development we thought we could learn more by identifying all potential contributors to complexity, even at the cost of some “false positives.” In the pilot, the clinician has agreed with the classification even more (70.6%).

Our process of defining and refining the classification criteria naturally aligned with the hypothetico-deductive process^18,19 common in medical problem solving. From experience and data, the researchers formulated a hypothesis (the original criteria), exhibiting data-driven inductive reasoning characteristic of experts.¹⁹ By identifying different classes of data that would be helpful in formulating the criteria (S1, S2, O), our researchers were using “schemes,” a characteristic of expert problem solving.²⁰ We then tested the hypothesis against data. Computerization assisted us here, allowing us to more easily validate all the criteria against real patient data. We identified test cases where the resulting complexity score did not match the researchers’ expert judgment (i.e., they conflicted with the hypothesis) and adjusted the criteria to produce better results for those cases without impairing its performance with regard to the rest of the cases. We will continue to repeat the cycle.

The effect of “missing data” on criteria execution warrants further examination. Others have noted the impacts of incomplete data on inference and suggested strategies, both in the general case^21–24 and in the case of electronic health records and decision support.^25–27 In the pilot so far, as shown in Table 4, in six of the ten cases in which the clinician disagreed with the computer, the computer concluded “mild” while the clinician viewed the patient as “moderate” complexity. As shown in Table 5, in two of the six cases, clinicians had only entered six out of 24 possible elements into HELP2 while in the other four cases, they had only entered three of the 24 elements. As has been explained, the criteria conclude “mild” if not presented with any data that drive another conclusion. Consequently, as might have been predicted, the pilot showed that, in the face of sparse data, the criteria are unable to conclude anything but “mild.” The clinician, on the other hand, with more data available to him or her, is able to make a more educated assessment of complexity. We need to investigate:

• Why did they enter so few elements? Is it because the ones they entered are most important to complexity? Or were they the most conveniently collected from the patient?

• What are the most salient elements that the clinician is using that the computer does not have access to, either because 1) the MHI packet is not addressing them, 2) the MHI packet is addressing them but HELP2 is not storing them, or 3) the MHI packet is addressing them, HELP2 is storing them, but the clinicians are not entering them?

We recognize that interpreting missing data as “mild” complexity may appear inconsistent with our earlier stated preference to err on the side of “over-involving” the mental health specialist. However, in our tendency to over-state complexity, we were satisfied based on our data analysis that the criteria would result in a slight over-statement, but not so much that would cause skepticism toward the result (i.e., “alert fatigue”). In contrast, we felt that making sparse data case generate “moderate” or “high” would indeed cross that threshold. A better solution may have been to not generate a result at all if some pre-determined number of items were not present, with an “insufficient data” notification. A variant of this idea would be to always conclude a complexity, but also generate an accompanying confidence measure based on the amount of data available. Still another possibility is to improve MHI data entry via reminders.

Computerization of scoring algorithms has been undertaken in Intensive Care Unit (ICU) acuity, readmission risk, community acquired pneumonia risk, and other areas.^28–32 Little if any computerization of algorithms has been performed in the mental health complexity scoring. One reason is likely the subjective, self-reported, and often sensitive nature of the data. But we venture to capture and utilize this data in our automated scoring because patient engagement is a critical component of Intermountain’s MHI efforts. They in fact reflect the enterprise’s Shared Accountability strategy, which aims to engage all stakeholders – the healthcare organization, employers, caregivers, the community, and patients – in providing better care and better health at sustainable cost levels. This underscores a cultural shift at Intermountain – and in the nation as a whole – in which patients and their families are being invited to become more engaged and invested in their care. In this climate, researchers in informatics and the social sciences will need to develop innovative, engaging, and nonintrusive technical and cultural solutions for interacting with patients and collecting reliable data from them.

Limitations and Future Work

A thorough, formal examination of the criteria before computerization may have isolated computerization issues from criteria issues. But our motivation for omitting this step may be found in our view of the criteria and in the environment at Intermountain. It would be preferable to validate a diagnostic algorithm (i.e., ensure the algorithm generates a sensitive and specific outcome by comparing its result to the “gold standard”) before computerizing. In contrast, the classification criteria have been developed based on MHI researchers’ experience as an aid to the PCP. No “gold standard” exists. Our computerization is actually a quality improvement exercise aimed at refining the criteria and, when clinicians disagree, learning more about why they do. Further, continuous quality improvement and computerized data capture and decision support are so firmly engrained in our culture at Intermountain that it was natural and convenient to expedite the implementation of the criteria in HELP2, allowing us to easily collect feedback and continuously improve clinician understanding on the one hand and the criteria on the other.

After the pilot, we intend to do more thorough analysis on both the agreement cases and disagreement cases. We will explore whether there is correlation between individual clinicians or clinician roles and agreement/disagreement and whether certain sub-scores are responsible for an inordinate amount of disagreement.

The work described is analysis of retrospective data and pilot results to date. Prospective evaluation of refined criteria on larger numbers of patients is anticipated after incorporation of pilot findings. The criteria specifically address a narrow subject – MHI complexity. The classification is not intended to be applicable to any other clinical domain. However, the Foresight architecture has been used in a variety of domains.^33–35 And we can conceive of a Foresight-based framework to more generically address scoring/classification problems like the MHI complexity problem.

We performed the retrospective analysis only on patients for whom most of the 24 data elements had been entered. This did not give us insight into those cases in which data were sparse. The pilot is our passage into exploration of how to appropriately interpret and address missing data.

We focused our criteria on the data elements that experience, best practices, and external sources suggested. Starting with the pilot, we can endeavor to identify which of the elements truly contribute most to MHI complexity and which elements are not contributing.

The capstone of the effort will be incorporating the criteria into Intermountain’s production system and the clinician workflow. During the pilot, the generation of a complexity score results in an alert message to the clinician. The clinician must navigate to his or her message queue to view the generated score. The message displays the complexity score and prompts for feedback, but does not self-explain. In contrast, when implemented in production, the system will ideally generate and display the score at the point the clinician enters or reviews the data, display the sub-scoring that constituted the complexity, track clinician responses, and assist the clinician in launching workflow (for example, orders for consults) if the clinician desires. Also envisioned is direct patient-entry of the MHI data at our patient portal instead of the current transcription process.

Conclusion

We have been able to effectively computerize classification criteria to assess a primary care patient’s mental health complexity. Over time, as we analyze our data, further our understanding of the nature of MHI complexity, refine the criteria to incorporate the most important data elements, and educate clinicians on our findings, we hope the scoring will yield a true representation of the patient’s MHI complexity and assist PCPs in an area outside their expertise. The computerization of the MHI complexity scoring will provide more scores, more appropriate scores, and more consistent scoring, which we anticipate will further the MHI objective of integrating more appropriate and directed mental health care in the primary care environment and facilitate future research.