A Method for Rating Charts to Identify and Classify Patients with Medically Unexplained Symptoms

Abstract

Background

As part of conducting a randomized control trial (RCT) to treat chronically high utilizing patients with medically unexplained symptoms (MUS), we developed the chart rating method reported here to identify and classify MUS subjects.

Method

Intended at this point only as a research tool, the method is comprehensive, uses explicit guidelines, and requires clinician raters. It distinguishes primary organic disease patients from those with primary MUS, quantifies medical comorbidities in primary MUS patients, and also distinguishes subgroups among MUS patients that we call somatization {resembles DSM-IV somatoform disorders) and minor acute illness (MAI) which differs from DSM-IV somatoform definitions. Scoring rules are used to generate the diagnoses above. The rules may be set according to the investigator’s needs, from highly sensitive to highly specific.

Results

We found high levels of agreement with the gold standard for MUS vs. organic disease (97.6%) and among raters for the key individual chart elements rated (92–96%). The method identified 206 MUS subjects and the extent of their medical comorbidities for entry into a RCT, It also identified somatization and MAI; the latter supports the validity of this newly reported MAI syndrome.

Conclusion

We concluded that this method offered research potential for identifying MUS patients, for quantifying their medical comorbidities, and for classifying MUS subgroups.

Physicians have only DSM-IV to diagnose and classify patients with medically unexplained symptoms (MUS), but the vast majority of DSM-IV somatoform diagnoses have not been validated [1, 2]. Moreover, among other inadequacies [3, 4], DSM-IV fails to account for medical comorbidities which likely compound the problem [5–7].

We propose that chart review methods can potentially complement DSM-IV to more effectively classify MUS patients. This challenges the widespread belief and practice that DSM-IV alone is sufficient to identify and classify all MUS patients, including those in primary care. We suggest that using chart as well as DSM-IV data has the potential to enhance the definition of all primary care MUS subgroups, and that chart data can also provide unique data about medical comorbidities.

To identify predominantly MUS patients for a clinical trial we developed the chart rating method presented here. It also identified clinical subgroups within the MUS population as well as the proportion of organic disease-based symptoms present in MUS patients, a measure of medical comorbidities.

Chart review methodology to identify M US patients is not well defined [8–11]. No previous studies contained sufficient methodological detail to help us, nor did they report reliability studies [12, 13], but they were encouraging in providing rich clinical data [12–16]. Health services research includes medical chart review in, for example, quality of care, risk adjustment; and health care delivery and utilization work [17–23], and we took advantage of these experiences in developing our methodology. Nevertheless, we addressed a fundamentally different issue, one for which there was no precedent that we could determine in the literature: identifying clinical patterns that indicate MUS and its possible subgroups as well as identifying medical comorbidities in MUS patients. Rather than using existing diagnosis labels in charts, we identified individual symptoms, observed and assessed them over time, noted symptom types, determined the degree of medical workup for the symptoms and its results, and made MUS diagnoses from this information that often had not previously been made, Using explicit, detailed criteria, the chart review took as much as 60 min/chart, averaged 20 min after training and experience, and seldom took less than 15 min. Further, the method required the knowledge and diagnostic skills of senior internal medicine residents or experienced clinicians, We do not propose this chart-rating procedure for clinical application and focus only on its potential use for research.

Method

Three senior medical residents (E.K., M.K., R.H.) and 2 faculty (CL., F.C.D.) performed ratings of adult primary care patients’ charts in a staff model HMO. The study met the institutional review board requirements of the HMO and the university. The residents rated 85% of the charts.

Procedure

Training

At the outset, raters received approximately 40 h of training using non-study charts. During this period, the chart procedures and conceptual categories were refined. Symptom categories were clarified and decision rules developed so that the categories were both exhaustive (all symptoms could fall into a category) and mutually exclusive (based on available chart data, each symptom could be placed in only one category). When raters demonstrated over 90% agreement (18/20 charts rated) with 1 of the authors (R.C.S.) on identifying patients with a primary MUS problem (the criterion for entry into the randomized control trial (RCT)) they began rating the study charts. Once the method was finalized, training 2 additional raters (F.C.D., C.L.) required approximately 20 h.

Chart Rating

Following rater training, the study charts reported here were rated for all medical care during a specified 12-month period. However, all available chart data from any time period could be used to inform the 12-month rating. Only the primary care physician’s chart notes and those of the medical assistant, who put the patient into the room prior to the visit, were specifically rated for symptoms; assistants were instructed to systematically record all symptoms the patient had. However, all other chart data and other physicians’ notes (e.g., consultations) were used to inform ratings of symptoms in the primary care physicians’ notes.

Origin of Charts Rated

The HMO had 28,000 adult patients and approximately 60% had 1 or more visits/year. Fifteen percent made 8 or more visits/year. To obtain persistent high utilizers, we identified 1,646 patients with 8 or more visits/year in the preceding 24 months (most had at least 3 years of consecutive high utilization). We then excluded 246 cases with obvious organic diseases. Raters fully rated the remaining 1,400 charts, those reported here, for our primary purpose of identifying MUS subjects for a treatment trial. One of the authors (R.C.S.) performed detailed reviews (not using the rating procedure) of all charts designated by the rating system as primary MUS for the accuracy of this diagnosis prior to entry into the study. He excluded those cases that still represented primary organic diseases as false positives of the rating procedure. Further, during his supervision of care of the treatment group, he observed the degree of stability of the MUS diagnosis.

Evaluating Agreement

1. To evaluate the agreement of each rater with 1 of the authors (R.C.S.) on primary MUS vs. primary organic disease, each of 4 different, raters and R.C.S. rated the same randomly selected charts (from among the 1,400), varying from 9 to 13 charts for each rater-R.C.S. pair; a total of 41 charts overall were jointly rated and agreement on designation as MUS or organic was evaluated. When agreement was< 100%, we used the kappa statistic.

2. To evaluate agreement among the raters on each of the disease categories (organic, nonorganic, undocumented: see rating method later), all 3 resident raters rated the same 10 randomly selected charts (from among the 1,400). We designated disease category counts within 2 as defining agreement, while counts differing by 3 or more were designated as disagreement.

Chart Rating Method

The instructions to raters are available from the authors.

Individual Symptoms

While recording symptoms from the chart for each patient visit to the primary care physician during the 12 months rated, raters first listed each physical symptom that was mentioned in either the medical assistant’s notes or in the physician’s notes; we did not rate the rarely recorded psychological symptoms. Raters then classified each symptom into one of the following disease categories according to the following specific rules.

1. Documented organic disease required definite, diagnostic abnormalities on laboratory, radiographic or other objective testing or, rarely, from physical examination to rate a symptom as having a documented organic disease explanation and, then, only if the symptoms (and their extent) were clearly explained by the identified organic disease. We thus relied upon classic definitions of organic disease. We did not make diagnoses of organic disease in the absence of an adequate and objective data base.

2. Documented nonorganic disease required symptoms/complaints for which there is no (or incomplete) organic disease basis, as documented by negative diagnostic tests or consultants’ opinions based upon objective testing. The absence of physical examination signs only was not sufficient. To rate this category meant that the definitive testing required had been performed and showed little or no organic disease explanation, e.g., negative studies on: laparoscopy for pelvic pain, MRI/CT for low back pain, endoscopy for inflammatory bowel disease, MRI/CT for headaches.

3. Undocumented disease required that little or no laboratory or other objective diagnostic workup had been performed. Thus, insufficient investigative data existed and this precluded making a judgment of organic vs. nonorganic. These problems/symptoms typically were minor and of the type for which many patients would not seek health care at all and, hence, had little investigation. Nevertheless, treating physicians often had made diagnoses that sounded like organic disease (e.g., sciatica, bursitis, sinusitis, tendonitis, chronic infectious mononucleosis, atypical chest pain) and/or initiated treatment for organic diseases that were not objectively documented (e.g., antibiotics for an upper respiratory tract infection, release from work for a minor injury). In those instances, we rated the patient as having an undocumented explanation.

Summarizing Visits

After classifying each symptom at each visit in 1 of the 3 disease categories, raters then summarized each primary care visit by noting which and how many disease categories characterized each visit; more than 1 disease category rating/visit was common; e.g., a patient could have 1 or more organic symptoms and 1 or more nonorganic symptoms in the same visit. We then computed disease category totals across all primary care visits for the 12-month period.

For example, a patient who had 10 clinic visits had a total of 14 disease categorizations for the year; among these 14, documented organic disease was present 6 times (monitoring mild hypertension and diabetes), documented nonorganic disease 4 times (severe, disabling chronic low back pain with a negative MRI), and undocumented disease was rated 4 times (mild upper respiratory infections, none with positive physical examination signs). The 12-month category summaries were the key data used in the scoring rules. Table 1 shows the disease category profile of this patient (see ‘Total’ for final summary used in scoring rules).

Table 1.

Disease category summary for the patient summarized in the text

Primary care visit	Documented organic disease symptoms1	Documented nonorganic disease symptoms2	Undocumented disease symptoms3
1	X
2	X
3		X	X
4	X	X
5		X
6	X		X
7			X
8	X	X
9			X
10	X
Total	6	4	4

¹Marker that later determines either primary organic disease or, in MUS patients, medical comorbidity.

²Marker that later determines socialization subgroup.

³Marker that later determines MAI subgroup.

Scoring Rules

The 12-month sum (see Total; table 1) of the documented nonorganic and undocumented rating categories constituted the ‘MUS disease component’. The percent visits for the MUS disease component = the number of documented nonorganic + undocumented classifications divided by the total of all disease categories, while the remaining percent represents the ‘organic disease component’. In the patient example in the preceding section and table 1, we see that of 14 total disease categorizations (in 10 clinic visits) 8 were for the MUS disease component (4 documented nonorganic + 4 undocumented); 8/14 = 57% MUS disease component; 6/14 = 43% organic disease component.

Although any settings are possible, we now present the scoring rules we used to identify subjects for the RCT as an example. To identify the primary MUS patients we needed for study, we set the proportion of MUS low to be as inclusive as possible in screening, accepting the likelihood of a high false-positive rate (organic diseases). In addition, the rating procedure also produced data not needed to identify subjects for study but important for identifying comorbid organic disease in MUS patients and for identifying MUS subgroups of patients.

Rule 1

Primary MUS problem, comorbid organic disease. For this rule, we defined a ‘primary MUS problem’ as having 33% or more MUS disease components for the year rated (67% or less organic disease component). ‘Primary organic disease problems’ were defined as having<33% MUS disease components. Our clinical experience led to this prospectively made decision. For example, we knew that many high utilizing organic disease visits were for brief, nondemanding, and relatively straightforward medical problems, such as follow-up for diabetes or hypertension.

In the patient example given above (and table 1), with >33% MUS disease components and therefore a primary MUS patient, the subject entered the RCT for a treatment trial with the additional information that 43% (6 symptoms of a total of 14) of all symptoms had an organic disease basis, a measure of medical comorbidities.

Rule 2

Subgroups. Designated also on a priori clinical observations, we knew that, among the MUS category, documented nonorganic disease symptoms carried far more clinical implications than undocumented because the nonorganic symptoms had negative results on laboratory or other diagnostic investigations. They represented a bona fide MUS. To reflect these clinical experiences, we developed a rule to parallel DSM-IV somatoform disorders by capturing those patients with a substantial proportion of documented nonorganic symptoms; given at least 33% MUS disease components (primary MUS problem), 25% or more of the MUS disease components must be documented as nonorganic disease for a diagnosis of ‘somatization’ while, conversely, >75% of MUS disease components must be undocumented disease to diagnose ‘minor acute illness (MAI)’. MAI patients thus have very few diagnostic investigations and, in turn, this is a marker of lack of severity compared to those with somatization where the decision by the physician to perform an investigation suggests a more severe problem.

Thus, the patient example given above would have been diagnosed as somatization: with >33% for MUS disease components, rule 1 says the patient has a primary MUS problem of some type. Because, among MUS disease components, documented nonorganic was 50% and undocumented was 50%, rule 2 for somatization is met (≥25% nonorganic), and that for MAI is not (>75% undocumented). Table 2 summarizes the scoring rules.

Table 2.

Scoring rules: sensitive setting

Rule 1

Primary MUS problem: The combined number of primary care visits in which nonorganic and undocumented symptoms (together, medically unexplained symptoms or MUS) are a reason for a visit must be ≥33% of visits during the 12 months rated $\frac{\text{Nonorganic\hspace{0.17em}}+\text{\hspace{0.17em}undocumented}}{\text{Organic }+\text{ nonorganic }+\text{ undocumented}}\ge 33\%=\text{Primary MUS problem}$

Rule 2

Somatization (two conditions) 1 A primary MUS problem 2 The proportion of nonorganic symptoms to the total number of medically unexplained symptoms (MUS) is ≥25% $\frac{\text{Nonorganic}}{\text{Nonorganic\hspace{0.17em}}+\text{\hspace{0.17em}undocumented}}\ge 25\%=\text{Somatization}$

Minor Acute Illness (MAI) (two conditions) 1 A primary MUS problem 2 The proportion of undocumented symptoms to the total number of medically unexplained symptoms (MUS) is >75% $\frac{\text{Undocumented}}{\text{Nonorganic\hspace{0.17em}}+\text{\hspace{0.17em}undocumented}}>75\%\hspace{0.17em}=\hspace{0.17em}\text{MAI}$

Results

Agreement among raters with 1 of the authors (R.C.S.) for a primary MUS problem was 97.6% (40/41 cases similarly evaluated). In the 1 rater without perfect agreement, the kappa was 0.84 with agreement on 12/13 charts (92.3%). Agreement between raters themselves on yearly counts of disease categories (organic, nonorganic, undocumented) was quite high and ranged from 92 to 96% for each of the 3 categories.

Of 1,400 charts rated, using the above rule for screening MUS subjects, 375 (27%) had <33% MUS and were thereby designated as primary organic disease. Of the remaining 1,025 (73%) with at least 33% MUS, 319 were excluded by one of the author’s (R.C.S.) independent and detailed clinical reviews of the charts because the cases still represented predominant organic disease problems, with an expected high false-positive rate; 319/1,025 = 31 % false-positive rate. The author observed no instances of false-positive MUS designations during the subsequent 12 months of the treatment intervention. Of the 706/1,400 patients with a final diagnosis of primary MUS (50.4%), 204 were ineligible or refused; from the remaining 502, we recruited 206 (41% recruitment rate) for the RCT. The high prevalence of MUS in a high utilizing population is consistent with our previous report [24] and with reports in all outpatients [25].

Table 3 shows a brief example of the data output of the rating procedure for 10 subjects. The last column gives the diagnosis that is made with the scoring rule presented. Where disease categories exceed the total visit numbers, multiple categories were recorded during single visits; where disease categories are fewer than total visits, this reflects that we counted all physician visits (not just primary care visits) to determine utilization, while we rated just primary care visits to determine disease categories. Applying the sensitive rule, of 206 subjects recruited and entered into the RCT, we found the following chart diagnoses: 124 had somatization (60.2%), and 82 had MAI (39.8%).

Table 3.

Ten patients’ chart-rated material

Case No.	Total visits	Org	NonOrg	UnDoc	MUS %	NonOrg %	UnDoc %	Chart diagnosis1
1	8	3	4	4	73	50	50	1
2	14	1	7	4	92	64	36	1
3	9	6	4	6	63	40	60	1
4	15	8	4	5	53	44	56	1
5	24	3	5	1	67	83	17	1
6	11	10	0	10	50	0	100	2
7	14	2	0	6	75	0	100	2
8	16	7	2	9	61	18	82	2
9	14	3	1	8	75	11	89	2
10	10	6	2	2	40	50	50	1

Org = Organic disease category; NonOrg = nonorganic disease category; UnDoc = undocumented disease category; MUS = medically unexplained symptoms.

¹For chart diagnosis: 1 = somatization; 2 = minor acute.

Discussion

We present a systematic chart rating method to identify and classify MUS patients for research investigations. We showed that the method effectively identified 206 subjects for entry into a RCT for treating MUS patients. Verification stemmed from 1 of the author’s performance on 2 evaluations: independent and detailed chart review prior to entry into the study, and ongoing evaluation of treatment patients during the 12-month study. It is important also that the high false-positive rate was expected. With a sensitive rule, our goal was to identify as many MUS patients as possible and accept many false-positives because we could easily exclude them later (by 1 author’s, R.C.S., independent chart review prior to entry into the study). When an investigator wants a low false-positive rate, a more specific scoring rule can be formulated [26, 27]; i.e., increase the percent MUS required. Overall, fewer cases would be identified, but the ones identified would have few false positives.

This study also demonstrated that the rating method identified subgroups. These data support our recent findings by again identifying MAI. We previously identified MAI in a different MUS population from chart review and showed evidence for its validity when it persisted over time [24]. In addition, the method precisely quantifies medical comorbidities. Failure to incorporate comorbidities in research and treatment in primary care has been one of the most ignored but pressing problems [5–7, 28].

We have demonstrated good evidence of rater agreement with the senior author (‘gold standard’) and shown high interrater agreement for the key ratings of disease categories. Our high levels of agreement in identifying MUS patients are not surprising and are consistent with other studies. Two consultation-liaison psychiatrists and 1 general physician had high levels of agreement (kappa 0.74–0.76) for MUS [29], and another study showed similar concordance in identifying MUS patients from medical charts (kappa 0.78) [30].

There are many limitations with this new rating method. (1) As currently conceived, it is labor-intensive. The method requires physicians, physician assistants, or nurse practitioners to perform the ratings. We believe that in time this method can become more cost-effective and be adapted for clinical uses outside research, and we plan later to investigate its practical applications. (2) The average time for rating an individual chart is in the range of 15–20 min at present. Similarly, we will explore more simplified criteria, shorter rating periods, and efficiency styles of different raters. The present situation is not unlike the early days of any new method when developing it can be labor- and time-intensive. (3) While we value our decision rule because it stems entirely from clinical observations, rather than post hoc synthesis, further investigation of this and other rules and their clinical correlates is needed.

The method is also limited by those shortcomings of any retrospective chart review. (1) We depended on how aggressively a doctor attempted to diagnose organic disease and on how completely a busy clinician(s) recorded their findings. We attempted to improve completeness of symptom recording by including notes made by the assistant who put the patients into examining rooms just before seeing the doctor. (2) An investigator cannot be certain that MUS patients do not in fact have organic diseases if they do not investigate each patient themselves from a biomedical perspective. However, during active clinical follow-up of treatment patients for 12 months, we observed no instance of an important organic disease having been missed, which is consistent with the literature on the subject [31, 32]. (3) Finally, notwithstanding interrater agreement, chart reviews are plagued by problems regarding reliability and validity of the record itself and the data extracted from it [33]; e.g., does the chart reflect all symptoms presented, do different physicians record differently? We agree with caveats urging additional sources of information. We recommend using both chart data and DSM-IV data together [33].

The significance of this work lies in its potential to complement DSM-IV. In research on MUS patients, the additional use of our chart data has several possible advantages. (1) Data cover most possible physical symptoms - 245 compared to only 38 symptoms in DSM-IV. (2) Chart data show whether actual symptoms were documented organic disease, documented nonorganic disease, or undocumented (no workup). Because the latter 2 disease categories indicate whether or not diagnostic investigation was performed, they become measures of the severity of the MUS problem from the physician’s perspective. Because of the disease category ‘organic’, we can also identify in MUS patients how many medical comorbidities coexist with MUS. Both the severity of MUS and comorbidities are key parameters in better classifying patients with a chronic disease according to recent models developed for primary care depression, a closely related condition [5–7, 28]. (3) Chart data involve symptoms prompting health care seeking, as recorded at the time of actual health care seeking, quite in contrast to DSM-IV which may be obtained at points well removed from the actual symptoms. (4) Finally, chart data have recognized value in longitudinal evaluation, compared to cross-sectional DSM-IV data [12–14].

This potential notwithstanding, more work is needed to verify the actual research value of the method and to effectively integrate it with DSM-IV. We are presently beginning such work and will later report these results, including the following: DSM-IV profile of the chart-based diagnoses, distinction of MAI from somatization, and the stability over time of chart diagnoses.