Congruence between clinical and research-based psychiatric assessment in bariatric surgery candidates

Abstract

Background

Mental health professionals have become increasingly involved in working with bariatric surgery candidates, particularly in performing preoperative psychological evaluations clearing candidates for surgery.

Objectives

To examine the concordance of psychiatric diagnoses obtained during routine clinical evaluation prior to bariatric surgery and diagnoses obtained separately in a research facility using the Structured Clinical Interview for DSM-IV axis I disorders (SCID-I).

Setting

The study included 68 consecutively enrolled bariatric surgery candidates who participated in the Longitudinal Assessment of Bariatric Surgery-3 study (LABS-3).

Methods

The SCID data obtained from research assessments were compared to the diagnostic data of routine preoperative psychiatric evaluation. Congruence of current and lifetime diagnoses was assessed using Cohen's coefficient kappa.

Results

There was considerable variability among the major diagnostic categories, with generally poor agreement in current diagnoses. Kappa coefficients tended to be larger for lifetime diagnoses. The agreement was moderate for any lifetime mood disorder with kappa values of 0.45. With regard to any lifetime anxiety, substance use, and eating disorder, the clinical diagnoses rarely concurred with the results in the SCID interview with kappa statistics of 0.30, 0.36, and 0.32.

Conclusions

The congruence between diagnoses assigned during routine clinical psychiatric evaluation and research assessment using the SCID is surprisingly low. These conclusions are tentative given the time interval and the possibility of treatment between the two evaluations. Overall, these data raise interesting questions concerning the use of unstructured psychiatric evaluations obtained prior to bariatric surgery.

Introduction

Bariatric surgery procedures have become increasingly utilized as a safe and effective means of producing rapid weight loss in obese individuals. In the period from 1998 to 2002, the rate of bariatric surgery procedures per 100,000 patients in the United States rose from 7.0 to 38.6¹. As the rate of bariatric surgeries has risen, mental health professionals have become increasingly involved in working with bariatric surgery candidates, particularly in performing preoperative psychological evaluations clearing candidates for surgery. One recent study of practices performing bariatric surgeries indicated that 88% of these practices require patients to undergo preoperative psychological evaluations². The American Society for Metabolic and Bariatric Surgery has suggested that all bariatric surgery candidates undergo such evaluations³.

Thorough psychological and psychosocial assessment is essential because of the necessity for long-term behavioral change on the part of the bariatric surgery candidate. Theoretically assessment could help to identify strengths and weaknesses that may affect post-surgical outcome, although the number of current clinically established prognostic indicators is small^4,5. Information in pre-surgical assessments is typically collected through the use of self-report data and face-to-face interviewing^2,4,6.

Sarwer and colleagues⁷ reported that between 3% and 20% of bariatric surgery candidates are excluded from surgery due to concerns with psychological and psychosocial functioning. Illegal drug use at the time of assessment, psychotic symptoms, and severe mental retardation are among the most commonly cited reasons for preventing candidates from receiving bariatric surgery^2,6.

Zimmerman and colleagues⁸ described that nearly one in five patients were not allowed to go forward with surgery due to identified psychological difficulties that were thought to potentially negatively affect the outcome of the procedure. The most frequently cited reason for denying clearance for surgery was the patients’ reports of overeating to cope with stressors, which was present in 57% of those denied clearance⁸. Results of a survey by Walfish and colleagues⁹ suggested that psychologists differ in their preoperative evaluation practice and that there is still no consensus among clinicians regarding which bariatric surgery candidate's factors merit recommending delay or denial of bariatric surgery.

An important area of concern in the preoperative psychological evaluation of bariatric surgery patients is the lack of specific assessment instruments developed for this group or a standardized approach to assessing these individuals. The American Society of Metabolic and Bariatric Surgery has issued formal practice suggestions for preoperative psychological evaluations¹⁰; however, these guidelines contain suggestions that could be used in a wide variety of psychological evaluations and do not indicate specific evidence to support the use of the suggested assessment instruments.

It is also important to note that bariatric surgery candidates face a unique situation when being evaluated by mental health professionals prior to surgery. They doubtlessly recognize the possibility of being denied surgery based on the evaluation and thus have an incentive to try to present themselves in as favourable a light as possible. Studies examining bariatric surgery patients have countered these concerns by conducting evaluations independent of preoperative psychological evaluations usually provided to the surgical team^11-14.

A related issue relevant to the present study is the use of structured interviews in the assessment of mental disorders. A number of studies have demonstrated disagreement between results obtained from structured interviews and general clinical evaluations. One recent meta-analysis indicated that agreement between structured interviews and general clinical evaluations varied widely by disorder and diagnostic clusters¹⁵.

The purpose of this study was to examine the concordance of psychiatric diagnoses obtained during routine evaluation prior to bariatric surgery and diagnoses obtained separately in a research facility using the Structured Clinical Interviews for DSM-IV axis I disorders (SCID-I)¹⁶. We expected a low agreement between the two types of evaluation. Another aim was to explore whether symptoms indicating psychopathology were purposely underreported by patients in the routine psychiatric evaluation so that their candidacy for bariatric surgery would not be impaired. Despite the obvious importance of this question, to our knowledge this has never been examined previously.

Method

Patients

The study included 68 consecutively enrolled bariatric surgery candidates 18 years of age and older who participated in the Longitudinal Assessment of Bariatric Surgery-3 study (LABS-3) at the Neuropsychiatric Research Institute, Fargo, ND. The mean age was 46.4 years (SD 10.1, range 26-68) and the mean BMI was 46.1 kg/m² (SD 6.5, range 37.1-71.2). Fifteen participants (22.1%) were men and all participants were Caucasian. The majority were married or living with a partner (n=44, 68.8%).

Data collection

The SCID data were obtained from research assessments conducted as part of a Longitudinal Assessment of Bariatric Surgery Substudy (LABS-3) protocol designed to examine psychosocial issues in bariatric surgery patients. All research assessments were conducted at the Neuropsychiatric Research Institute in Fargo, ND, and the protocol was approved by the University of North Dakota Institutional Review Board.

These data were then compared to the diagnostic data recorded in the medical records of the LABS-3 participants who were also seen at the Eating Disorders Institute, MeritCare HealthCare Systems, Fargo, ND, for a routine preoperative psychiatric evaluation. The clinical charts were reviewed and scored by two independent raters blind to the research diagnoses. Disagreements were resolved by the first author. Current and lifetime DSM-IV axis I diagnoses were examined. The comparison of routine and SCID- based psychiatric diagnoses was not planned in advance and all data were collected prior to the time when the idea for the study was conceived. Therefore both sets of assessors were unaware that the study would be done. The assessments were conducted from February 28, 2006, through November 1, 2007. The secondary data analysis conducted for this study was approved by the Institutional Review Boards of both the University of North Dakota and MeritCare Health Systems.

The routine clinical psychiatric evaluation is shared with the surgery team and is required as a condition of eligibility for coverage by most health insurance companies. Clinical evaluations were performed by eight doctoral level psychologists who were all experienced in working with bariatric surgery candidates. They did not follow a set format. The length of clinician's notes varied from three to eight pages (Mean 5.2, SD 1.1). The research assessments were scheduled to occur after the clinical interviews and were conducted at a separate facility from the clinical evaluations on different days by four trained and supervised SCID interviewers. SCID interviews were conducted by masters or doctoral level assessors who were all experienced in working with bariatric surgery patients. All had been trained in a standardized format beginning with observations of typical SCID interviews, observations of life SCID interviews, and finally typical SCID interviews they conducted which were reviewed by the head assessor at the research Institute conducting the study. All had several years experience conducting SCID interviews in various randomized treatment trials of patients with eating disorders. In some of these trials, but not in this study, inter-rater reliability data were obtained, with rates of agreement of .91 to .99. All assessors meet weekly for supervision and discussion. Subjects were told that the SCID data would not be shared with the surgical team unless they admitted to certain high risk problems, such as suicidality or risk of substance withdrawal when hospitalized.

Data analysis

Statistical analyses were conducted with the statistical program SPSS 17.0. Congruence of current and lifetime diagnoses was assessed using Cohen's coefficient kappa within the following major categories: any affective disorder, any substance use disorder, any anxiety disorder, any somatoform disorder, any eating disorder. Cohen's coefficient kappa (κ) measures chance corrected agreement between diagnoses and can range from −1.00 to +1.00, where κ<0.20 indicates slight agreement, 0.21< κ<0.40 fair agreement, 0.41< κ<0.60 moderate agreement, 0.61< κ<0.80 substantial agreement, and 0.81< κ<1.00 reflects almost perfect agreement¹⁷.

Results

The clinical interviews were all performed prior to the SCID assessments. Table 1 summarizes the mean time intervals between the two evaluations and between evaluations and dates of surgery. The mean time period between the clinical interview and the SCID assessment was 160 days. In all 68 cases the time between the assessments exceeded four weeks. Twenty-one patients (31%) were interviewed using the SCID more than six months after the clinical interview. With regard to the time span between evaluations and surgery, almost all patients were assessed by the SCID less than four weeks prior to surgery. Only one patient answered the SCID 85 days prior to surgery. In contrast, the time interval between the clinical interviews and the date of surgery always exceeded four weeks with a mean time span of about five months.

Table 1.

Time between evaluations

Days between	N	Mean	(SD)
Clinical Evaluation and SCID	68	159.6	(85.4)
Clinical Evaluation and Surgery	661	174.7	(86.7)	t=16.4, df=65, p<0.0001
SCID and Surgery	661	15.11	(10.8)

Note.

¹two patients did not have surgery

As shown in Table 2, clinicians more often assessed any current axis I diagnosis, especially depression. The diagnosis of any current eating disorder was more often made by SCID assessors. With regard to lifetime diagnoses, in both assessments approximately 50% of the patients were diagnosed with a history of any axis I disorder (see Table 2). Again, clinicians tended to diagnose affective disorder more commonly but eating disorder less commonly within the life span.

Table 2.

Rates of Axis I disorders

Diagnoses	Current				Lifetime
	Clinical Interview		SCID		Clinical Interview		SCID
	N	(%)	N	(%)	N	(%)	N	(%)
Major Depression	20	(29.4)	5	(7.5)	25	(36.8)	23	(33.8)
Dysthymia			2	(2.9)			3	(4.4)
Any Mood Dx	21	(30.9)	6	(8.8)	28	(41.2)	26	(38.2)
Panic Disorder	1	(1.5)	2	(2.9)	1	(1.5)	6	(8.8)
Social Phobia	0	(0)	1	(1.5)	0	(0)	3	(4.4)
Specific Phobia	2	(2.9)	4	(5.9)	2	(2.9)	5	(7.4)
Obsessive Compulsive Disorder	0	(0)	0	(0)	0	(0)	1	(1.5)
Posttraumatic Stress Disorder	1	(1.5)	2	(2.9)	3	(4.4)	12	(17.6)
Generalized Anxiety Disorder	4	(5.9)	0	(0)	7	(10.3)	1	(1.5)
Any Anxiety Dx	10	(14.7)	6	(8.8)	12	(17.6)	19	(27.9)
Alcohol Dependence	0	(0)	1	(1.5)	1	(1.5)	3	(4.4)
Cannabis Dependence	0	(0)	0	(0)	2	(2.9)	3	(4.4)
Stimulants Dependence	1	(1.5)	0	(0)	1	(1.5)	1	(1.5)
Any Substance Dependence	1	(1.5)	1	(1.5)	3	(4.4)	7	(10.3)
Bulimia Nervosa	0	(0)	1	(1.5)	1	(1.5)	3	(4.4)
Binge Eating Disorder	2	(2.9)	6	(8.8)	3	(4.4)	11	(16.2)
Any Eating Dx	2	(2.9)	7	(10.3)	4	(5.9)	12	(17.6)
Adjustment Disorder	4	(5.9)	1	(1.5)	4	(5.9)	1	(1.5)
Any Axis I Dx	28	(41.2)	16	(23.5)	34	(50.0)	38	(55.9)

Congruence between current psychiatric diagnoses

Table 3 summarizes the agreement between current diagnoses. As can be seen, there was considerable variability among the major diagnostic categories, with generally poor congruence between clinical interviews and SCID assessments. The SCID appeared to diagnose current psychiatric disorders that were different from clinical diagnoses, which may have been influenced strongly by the large time span between the two assessments. Current mood disorders and current anxiety disorders were more often diagnosed in the clinical interviews than the SCID interviews. Current eating disorders were more often diagnosed in the SCID interview. Surprisingly, only two patients were diagnosed with current binge eating disorder in the clinical interview, and six patients in the SCID interview, without any overlap.

Table 3.

Congruence of current clinical and SCID diagnoses

Current Diagnoses	Only Clinical Interview		Only SCID		Clinical Interview and SCID		Kappa Coefficient
	N	(%)	N	(%)	N	(%)
Any current Mood Dx	17	(25.0)	2	(2.9)	4	(5.9)	0.18
Any current Anxiety Dx	9	(13.2)	5	(7.4)	1	(1.5)	0.02
Any current Substance Dependence	1	(1.5)	1	(1.5)	0	(0)	0.02
Any current Eating Dx	2	(2.9)	7	(10.3)	0	(0)	0.05
Any current Axis I Dx	21	(30.9)	9	(13.2)	7	(10.3)	0.03

Congruence between lifetime psychiatric diagnoses

Overall, kappa coefficients tended to be larger for lifetime compared to current diagnoses with fair to moderate agreement. The results of lifetime clinical and SCID diagnoses are shown in Table 4. The best agreement was found for any lifetime mood disorder. Among those with a diagnosis of any lifetime mood disorder per the clinical evaluation, 64% (N=18 of 28) also were identified as having a lifetime mood disorder in the SCID assessment. Among those with a lifetime mood diagnosis by the SCID, 69% (N=18 of 26) of clinical records also identified a lifetime depression. With regard to eating disorders, the clinical diagnoses rarely concurred with the results in the SCID interview. Among those with a lifetime diagnosis of any eating disorder on the SCID, only 25% (N=3 of 12) were given an eating disorder diagnosis by clinicians. SCID assessors diagnosed lifetime binge eating disorder in 11 patients, but in clinical notes a history of binge eating disorder was reported in only three of those cases. Lifetime bulimia nervosa was diagnosed by the SCID in three cases, and only in one of those patients by the clinician administered evaluation.

Table 4.

Congruence of lifetime clinical and SCID diagnoses

Lifetime Diagnoses	Only Clinical Interview		Only SCID		Clinical Interview and SCID		Kappa Coefficient
	N	(%)	N	(%)	N	(%)
Any lifetime Mood Dx	10	(14.7)	8	(11.8)	18	(26.5)	0.45
Any lifetime Anxiety Dx	5	(7.4)	12	(17.6)	7	(10.3)	0.30
Any lifetime Substance Dependence	1	(1.5)	5	(7.4)	2	(2.9)	0.36
Any lifetime Eating Dx	1	(1.5)	9	(13.2)	3	(4.4)	0.32
Any lifetime Axis I Dx	8	(11.8)	12	(17.6)	26	(38.2)	0.41

Discussion

This report is the first to examine the agreement of psychiatric diagnoses obtained during routine clinical evaluation prior to bariatric surgery and diagnoses obtained separately in a research clinic using the SCID interview. In both settings, about 50% of bariatric surgery candidates were diagnosed with at least one lifetime psychiatric disorder. These findings are consistent with previous studies suggesting high rates of psychiatric co-morbidity among bariatric surgery candidates^11-14. With regard to the results of current psychiatric comorbidity, clinicians made more diagnoses than the SCID assessors with a remarkably low agreement between the two types of evaluation. Of course, the long time interval between the two assessments of more than four weeks must be taken into account when interpreting these results. It cannot be excluded that current symptoms changed during the elapsed time. Current diagnoses are generally defined as the presence of symptoms in the past month, and the poor agreement might be explained by the large time gaps between the two separate evaluations. Moreover, the substantial time span between the clinical interview and the date of surgery is worrisome. Given the fact that the routine evaluations were conducted several months prior to surgery, it is reasonable to question whether or not the results of current clinical diagnoses are valid at the date of surgery. Also, many patients who are candidates for bariatric surgery are taking psychotropic medications. Therefore the low current prevalence rates may to some extent reflect the fact that some patients are being successfully treated due to a recommendations made following the clinical evaluations.

With regard to lifetime diagnoses, better agreement between the two evaluations was found. However, it might be recommended that lifetime diagnosis may be of less relevance, since remitted disorders would randomly not be a cause for declining surgery. The highest congruence was found for mood disorders. Kappa coefficients suggested fair agreement for lifetime anxiety, substance use, and eating disorders. It cannot be excluded that the concordance for substance dependence and eating disorders was artificially low due to low base rates¹⁸. Interestingly, compared to the clinicians, the SCID assessors tended to diagnose these disorders more often. On the other hand, there are several reasons why clinicians may fail to identify all possible diagnoses. For instance, clinical evaluations are limited to approximately 60 minutes per patient. In contrast, the SCID assessments are typically lengthy and detailed and are not time constrained. The clinical evaluations in our study had occurred before the patients were approved for surgery by their insurance company. By the time respondents were entered into the LABS-3 study they were already approved for surgery and they may have more openly reported psychiatric issues and may have been more truthful with research assessors.

In addition, we had assumed that patients might minimize symptoms, e.g. eating disturbances, due to concern that they will not be approved for surgery. Previous studies have shown that indeed some bariatric surgery candidates are excluded from surgery because of maladaptive psychological and psychosocial functioning^7,8. However, our data indicated an opposite pattern, with more frequent diagnosis in many categories in the clinical rather than the SCID interview diagnosis. The reasons for this are unclear. It certainly is possible that clinicians assign diagnoses without fully assessing the formal diagnostic criteria if certain symptoms are present. However, the finding is counterintuitive and clearly needs replications. Another way to approach this issue might be to actually score the clinical interviews based on whether or not diagnostic criteria are actually assessed. The finding that patients seem more willing to acknowledge symptoms of eating disorders to SCID interviewers rather than clinical interviewers suggests that the subjects may have understood that this specific type of symptom might be particularly relevant (and potentially detrimental) to the clinicians decisions about whether surgery is appropriate. Therefore they may have a different level of frankness in reporting this. Zimmerman and colleagues⁸ reported that one in five bariatric surgery candidates did not receive positive recommendation after the presurgical psychiatric evaluation, most commonly because of a current eating disorder. At the same time, the presence of disordered eating behavior in bariatric surgery candidates is well documented by studies using the SCID. Kalarchian and colleagues¹² reported a lifetime prevalence of eating disorders of 30%. Mühlhans and colleagues¹³ reported a rate as high as 50%, which might be due to the inclusion of EDNOS (eating disorders not otherwise specified) diagnoses. Rosenberger et al.¹⁴ and Mauri et al.¹⁹ found a lifetime prevalence rate of about 13%.

The question of whether or not presurgical psychiatric comorbidity, especially an eating disorder, is a predictor for less successful weight loss outcomes is unresolved and different studies have shown mixed results^20-24. Among bariatric surgery candidates, the most commonly diagnosed eating disorder is binge eating disorder. A diagnosis of binge eating disorder prior to surgery should be considered a predictor of less weight loss after surgery; whereas a return of loss of control eating following surgery could negatively impact outcome^21,25. Thus, binge eating prior to surgery should be considered as a warning sign for lower weight loss and, in general, lower postoperative health outcomes.

In summary, it is rather striking that findings of the two diagnostic assessments were quite divergent. The fact that the findings regarding binge eating disorder were completely divergent is particularly interesting. These findings are concerning, although not entirely surprising, given the results of previous studies where low levels of agreement were reported between routine unstructured clinical interviews and SCID interview^26,27. Rettew and colleagues¹⁵ conducted a meta-analysis and reported low to moderate agreement between diagnoses made from clinical evaluations and standardized diagnostic interviews. They analyzed the results of 38 articles published from 1995 to 2006 where Kappa coefficients for diagnostic clusters ranged from 0.14 for mood disorders to 0.70 for eating disorders.

There are several limitations to acknowledge. First, the data are limited by the small sample size. Second, the generalizability of our findings is constrained by the assessment of bariatric surgery candidates who were seen at only one center, since evaluation practices probably differ significantly among clinical centers. Third, the results may be biased by the large time intervals between the clinical and research evaluations, which significantly limit the interpretability of the findings. The conclusions are also restricted due to the possibility of treatment between the two evaluations.

Another limitation of the current data concerns the fact that all the subjects seen for the research interview had already been approved for surgery and therefore individuals who had clear psychosocial contraindications to surgery would have been excluded from the analysis.

Overall, our data indicate that clinical and research-based evaluations of bariatric surgery candidates prior to surgery often yield discrepant diagnoses. Spitzer²⁸ has proposed the LEAD standard as an operationalization of the best estimate diagnosis. LEAD is an acronym for Longitudinal Expert Evaluation using All Data and refers to criterion-based diagnoses made by experts reaching consensus after exploring results from several sources and from repeated assessments over time. With regard to previous research, structured clinical interviews are assumed to be superior to traditional clinical assessment and SCID diagnoses appear to conform more to the LEAD standard than traditional clinical interviews²⁹. A multitude of studies have used the SCID as the gold standard for clinical diagnoses^26,30. Another important clinical implication from these data is that clinical evaluations should be scheduled more proximate to the date of surgery or repeated shortly before surgery to enhance validity. The findings indicate that further effort and research are needed to ensure the valid psychiatric assessment of bariatric surgery candidates.