Meta‐analysis of self‐reported substance use compared with laboratory substance assay in general adult mental health settings

Abstract

An accurate assessment of substance use is necessary to make a correct psychiatric diagnosis and to provide appropriate treatment. This study uses meta‐analysis to establish the strength of the association between self‐reported substance use and the results of laboratory substance assay including the testing for specific substances and screening for any substance use in psychiatric hospitals and in community mental health settings.

A systematic search for published studies was supplemented by additional data required for meta‐analysis provided by several researchers in this field. Using random‐effects meta‐analysis, we calculated the pooled estimate of the odds ratio of a positive substance assay in patients reporting use or non‐use of substances and estimated the sensitivity, specificity, positive predictive value and negative predictive value.

Twenty‐six studies met the inclusion criteria. Very strong associations were found between self‐reported use and positive tests for cannabis [N = 11 studies, odds ratio (OR) = 22.3; 95% confidence interval (CI) = 10.1–49.1], amphetamines (N = 8, OR = 26.6; 95% CI = 7.9–88.9), cocaine (N = 8, OR = 39.7; 95% CI = 16.2–97.2) and opiates (N = 7, OR = 83.5; 95% CI = 26.7–260.7). Strong associations were found between self‐reported use of any substance and positive substance screening (N = 15, OR = 7.2, 95% CI = 3.6–14.1) and tests for alcohol use (N = 5, OR = 8.5; 95% CI = 2.5–28.4). Screening for any substance use had a sensitivity of 61% and a specificity of 66%. Testing for individual substances was specific but lacked sensitivity.

Screening has the potential to detect clinically relevant substances that would not be reported by the patient, whereas testing for a specific substance has little advantage over self‐report. The sensitivity of the substance assay might be improved by obtaining a sample at the earliest opportunity. Consideration should be given to the increased use of substance screening in general adult mental health settings because it could improve the accuracy of psychiatric diagnosis and increase the likelihood of patients receiving treatment for substance use disorders. Copyright © 2012 John Wiley & Sons, Ltd.

Introduction

An association between substance use and mental disorder has been demonstrated in population‐based surveys of mental illness (Grant et al., 2004; Scott et al., 2006; Teesson et al., 2009), among psychiatric inpatients (Katz et al., 2008; Lambert et al., 1996), at the time of an emergency mental health assessment (Curran et al., 2008), and during first‐episode psychosis (Barnett et al., 2007; Wade et al., 2006). The use of cannabis (Andreasson et al., 1987) and the use of amphetamines (Ellinwood, 1968) are both associated with psychosis. Symptoms of cocaine intoxication have been reported to be similar to those of schizophrenia (Serper et al., 1999). Patients with mental illness who also use substances are less adherent to prescribed treatment (Lacro et al., 2002; Perkins et al., 2008), have worse outcomes (Mauri et al., 2006) and are more likely to harm themselves or others (Fazel et al., 2009; Swanson et al., 2006). The failure to identify substance use in patients presenting to mental health facilities can result in incorrect diagnoses and inappropriate treatments (Appleby et al., 1995; Hall et al., 1977).

In clinical practice, the usual method of detecting substance use is to ask patients about substance use and to compare the patient's reply with corroborative information obtained from relatives and carers. Urine substance assay is sometimes used, although routine substance assay of all patients is usually only performed in specialized substance use treatment programmes. In clinical research settings structured or semi‐structured interviews can be used to ensure systematic enquiry about substance use (Barnaby et al., 2003; Mallin et al., 2002, Swartz et al., 2006). The results of interviews have been compared with urine assays (Galletly et al., 1993) or with the assay of blood or hair (Chung et al., 2009; Selten et al., 2002; Swartz et al., 2003), finding varying levels of agreement (Galletly et al., 1993; Helseth et al., 2005). The extent of agreement between patients’ accounts of substance use and the results of testing depend on the accuracy and sensitivity of the substance assay, the patient's knowledge of the substances they have taken, the treatment setting, the confidence of the patient in the person asking about their substance use, the potential consequences of disclosure and the prevalence of substance use in the population being studied (Chung et al., 2009). The benefit of substance assays in mental health settings has not been clearly defined due to (i) conflicting findings about the reliability of disclosure, (ii) the cost of testing, (iii) concern about patients’ privacy and (iv) patients’ capacity to consent to the assay.

In this paper we present the first meta‐analysis of the association between self‐report of substance use and the results of substance assay in general psychiatric settings. General psychiatric settings were defined as inpatient mental health facilities and community mental health settings. Both types of treatment setting were chosen because of the possible effect of the severity of patients’ mental illness on their ability to provide an accurate account of their substance use and because more psychiatric care is now provided in outpatient settings.

A systematic review and meta‐analysis of this topic has considerable advantages over individual studies because it increases the power of the analysis and reduces the likelihood of chance findings, particularly because the majority of existing studies have small sample sizes. Individual studies with a small sample size are prone to chance findings, particularly with respect to less frequently used and reported substances such as cocaine and amphetamines. Hence, meta‐analysis has the potential to more accurately define the strength of the association between self‐report of substance use and substance assay.

Based on studies of acute patients, who are believed to under‐report substance use (Test et al., 1989), we formed the null hypothesis that there would be no significant association between self‐reporting by psychiatric patients and substance assay results.

Based on the assumption that patients with a psychotic illness would have greater difficulty providing an accurate account of their substance use or would have less knowledge of the substances they had used, a second hypothesis was that there would be a weak association between self‐report and substance assay in studies of patients with a psychotic illness.

Method

Search strategies

Using a range of related terms we performed four sets of searches (see Figure 1) of multiple electronic databases for papers written or indexed in English (see Table 1).

Figure 1.

Flow chart of searches.

Table 1.

Search terms and databases

Search 1. All fields: (urine OR blood OR hair) AND (THC OR marijuana OR amphetamines OR cocaine OR opiate OR substance) AND (schizophrenia OR depression OR mental illness) in the following databases: EBM Reviews‐ ACP Journal Club (1991 to September 2009), Cochrane Central Register of Controlled Trials inception to 4th Quarter 2009, Cochrane Database of Systematic Reviews, inception to 3 rd Quarter 2009, Cochrane Methodology Register, inception to 4th Quarter 2009, Health Technology Assessment 4th Quarter 2009, Database of Abstracts of Reviews of Effects, inception to 4th Quarter, NHS Economic Evaluation Database, inception to 4th Quarter2009; EMBASE,1980 to 2009 Week 42, Ovid MEDLINE(R), 1950 to October 2009; PsycINFO, 1967 to October 2009, locating 8493 titles. Fifteen included studies were located using this search.

Search 2. (substance‐related disorders [MeSH Terms] OR (substance‐related[All Fields] AND disorders [All Fields]) OR substance‐related disorders [All Fields] OR (substance [All Fields] AND abuse [All Fields]) OR substance abuse [All Fields]) AND (mental disorders [MeSH Terms] OR (mental [All Fields] AND disorders [All Fields]) OR mental disorders [All Fields] OR (mental [All Fields] AND illness [All Fields]) OR mental illness [All Fields]) AND detection [All Fields] in the Data base PubMed to October 2009 locating 4365 titles. Five additional studies were located using this search

Search 3. (substance OR drug OR drugs OR dual diagnosis OR cannabis OR cocaine OR opiates OR amphetamine OR alcohol AND mental OR schizophrenia OR psychiatric OR depression OR psychosis) AND (detection OR testing OR self report OR self Reporting OR Self Reported OR Screen OR Screening) [all fields] in the titles of papers indexed in Medline [1950– January 2010), [PsychINFO 1967–2010], Cinahl [1980–January 2010] locating 1506 titles after the removal of duplicates. Four additional studies were located using this search.

Search 4. substance OR dual diagnosis OR cannabis OR cocaine OR opiates OR amphetamine OR alcohol AND psychosis OR schizophrenia in the titles of papers indexed in Medline [1950–January 2010], [PsychINFO 1967–2010], Cinahl [1980–January 2010] locating 1122 titles after the removal of duplicates. No additional studies were located using this search.

One study was located by hand searches of the reference lists of studies located in the searches of electronic databases. The searches were conducted in English.

Studies were included if they:

1. reported the results of samples of adult patients receiving treatment for mental disorders in hospital or community settings;

2. included measures of self‐reported substance use, either by way of a diagnosis made at a diagnostic interview or by self‐report of recent substance use;

3. included the results of substance assays for alcohol, amphetamine, cannabis, cocaine or opiates, or substance screening for the presence of any psychoactive substance;

4. reported sufficient information to extract the following data:

   1. self‐reported substance use associated with a positive substance assay;

   2. self‐reported substance use associated with a negative substance assay;

   3. no self‐reported substance use associated with a positive substance assay;

   4. no self‐reported substance use associated with a negative assay.

Studies were excluded if they:

1. reported a sample of patients receiving treatment for substance use disorders alone;

2. reported on cohorts of patients with a pre‐existing diagnoses of both substance use and a mental disorder (e.g. exclusively dual diagnosis patients) because the pattern of these patients’ substance use was likely to have been well‐established;

3. reported a sample of patients who were unrepresentative of general adult mental health settings, such as samples from child and adolescent services; or

4. provided insufficient information to calculate effect‐size data, even after correspondence with the authors.

Of 506 papers examined in full text (Table 1), 26 studies met the inclusion criteria (Table 2).

Table 2.

The characteristics of 26 included studies of the substance use and substance testing or screening in general adult mental health settings

Study	Institution	Study population	N	Diagnostic group	Diagnostic system, diagnostic method	Assessment of reports of substance use	Type of assay details	Substance assay
1. Albanese et al., 1994	Boston Massachusetts Mental Health Centre, USA	Patients admitted to a psychiatric intensive care unit	67	Mixed	DSM‐III‐R, SCID	SCID	Enzyme linked immuno‐assay of urine	amphetamines, cocaine, opiates, THC, alcohol
2. Bertolín‐Guillén, 1999	University Hospital Valencia, Spain	Admitted psychiatric patients	240	Mixed	ICD‐10, SCAN	SCAN	Radio immuno‐assay of urine	amphetamines, THC, alcohol, cocaine, opiates,
3. Bonsack et al., 2006	Lausanne University Adult Psychiatric Hospital, Switzerland	Admitted psychiatric patients	210	Mixed	ICD‐10, Clinical	ASI	Enzyme linked immuno‐assay of urine	amphetamines, cocaine, opiates, THC
4. Brady and Casto, 1989; Brady et al., 1991	South Carolina Veterans Administration Medical Centre, USA	Admitted psychiatric patients	38	Mixed	DSM‐III‐R, SCID	SCID	Radio immuno fluorescence	cocaine, THC
5. Chung et al., 2009	Rozelle Hospital, Sydney, Australia	Admitted psychiatric patients	50	Mixed	DSM‐III‐R, SCID	structured questionnaire	GCMS of serum	cocaine, opiates, THC, alcohol
6. Claassen et al., 1997; Gilfillan et al., 1998	Parkland Memorial Hospital, Texas, USA	Psychiatric emergency setting	112	Psychosis	DSM‐III‐R Clinical	Clinical	Enzyme linked immuno‐assay of urine confirmed with CGMS of urine	amphetamine, cocaine, opiates & THC
7. Covell and Essock 2008	New York Psychiatric Institute, USA	Admitted psychiatric patients and outpatients in a medication trial	40	Treatment resistant schizophrenia	DSM‐IV, SCID	SCID	Enzyme linked immuno‐assay of urine and GC/MS of hair	amphetamines, cocaine, opiates, THC
8. Crowley et al., 1974	Colorado Psychiatric Hospital, Denver, USA	Admitted psychiatric patients	50	Mixed	DSM‐II, Clinical	Clinical	Gas chromatography of serum and urine	amphetamines, opiates
9. de Beaurepaire et al., 2007	Paul‐Guiraud‐Villejuif Psychiatric Hospital, France	Admitted psychiatric patients	486	Mixed	DSM‐IV, Clinical	Structured questionnaire	Enzyme linked immuno‐assay of urine	amphetamines, cocaine, opiates & THC
10. DeQuardo et al., 1994	University of Michigan, USA	Admitted psychiatric patients in psychosis programme	67	Schizophrenia	DSM‐III‐R Clinical	Clinical	Radio immuno‐assay & GCMS of urine	cocaine, stimulants, THC
11. Elangovan et al., 1993	New York, USA	Psychiatric emergency setting	218	Mixed	DSM‐III‐R SCID	SCID	Enzyme linked immuno‐assay & GCMS of urine	cocaine
12. Flovig et al., 2009	Østmarka Psychiatric Hospital, Norway	Admitted psychiatric patients	227	Mixed	ICD‐10,Clinical	Structured questionnaire	LCMS of urine	stimulants, THC, alcohol
13. Fowler et al., 1998	Hunter Area Health Service, Australia	Out patients managed in community health centres	176	Schizophrenia	DSM‐III‐R SCID	SCID	Enzyme linked immuno‐assay or thin layer chromatography confirmed with GCMS of urine	Amphetamines opiates, THC
14. Galletly et al., 1993	Hillcrest Hospital, Adelaide, Australia	Admitted psychiatric patients	121	Mixed	DSM‐III‐R, Clinical	Clinical	Enzyme linked immuno‐assay or thin layer chromatography confirmed with GCMS of urine	amphetamines, THC
15. Helseth et al., 2005	Blackstad Hospital, Oslo, Norway	Admitted psychiatric patients	65	Mixed	DSM‐IV, SCID	ASI	Enzyme linked immuno‐assay confirmed with LCMS of serum	amphetamines, opiates, THC
16. Jackson et al., 2004	Connecticut Department of Mental Health and Addiction, USA	Out patients managed in two community centres	1930	Mixed	DSM‐IV, SCID	SCID	Membrane based immuno chromatography of urine	cocaine, THC
17. Ley et al., 2002	South Devon Healthcare Trust, UK	Admitted psychiatric patients	112	Mixed	ICD 10 Clinical	Clinical	Enzyme immuno‐assay of urine	amphetamines, cocaine, opiates, THC
18. Modestin et al., 1997	Psychiatric University Hospital, Zurich, Switzerland	Admitted psychiatric patients	353	Mixed	ICD 10, Clinical	ASI	Enzyme linked immuno‐ assay of urine	cocaine, opiates, THC
19. Perrone et al., 2001	University of Pennsylvania Hospital, USA	Psychiatric emergency setting	124	Mixed	Clinical	Structured questionnaire	Enzyme linked immuno‐assay of urine	amphetamines, cocaine, alcoholopiates, THC
20. Robinson and Wolkind, 1970	London Hospital, UK	Admitted psychiatric patients	54	Mixed	Clinical	Clinical	Gas chromatography	amphetamines
21. Schiller et al., 2000	San Francisco General Hospital, USA	Admitted psychiatric patients	198	Mixed	Clinical	Clinical	Immuno‐assay of urine confirmed with gas chromatography of urine	amphetamines, cocaine, opiates.
22. Selten et al., 2002	The Hague, The Netherlands	First episode psychosis	64	Psychosis	ICD‐9, CASH	Clinical with corroboration	GCMS of hair	amphetamines THC
23. Sembhi and Lee, 1999	Te Awamutu, New Zealand	Admitted psychiatric patients	39	Psychosis	Purpose designed checklist, BPRS	Structured questionnaire	Enzyme linked immuno‐assay of urine	THC
24. Shaner et al., 1993	West Los Angeles Veterans Administration Medical Centre, USA	Admitted psychiatric patients	100	Psychosis	DSM‐III‐R, Clinical	Structured questionnaire	Fluorescence polarization immuno‐assay and gas–liquid chromatography of urine	cocaine
25. Swartz et al., 2003	Treatment facilities in North Carolina, USA	Admitted psychiatric patients and outpatients	203	Psychosis	DSM‐IV Clinical	Clinical	Radioimmuno‐assays of hair and enzyme linked immuno‐assay of urine	amphetamines, cocaine, opiates, THC
26. Wilkins et al., 1991	West Los Angeles Veterans Administration Medical Centre, USA	Admitted psychiatric patients	56	Mixed	DSM‐III‐R	DAST	Fluorescence polarization immuno‐assay and gas–liquid chromatography of urine	amphetamines, cocaine, opiates, THC

Note: GCMS, gas chromatography mass spectrometry; LCMS, liquid chromatography mass spectrometry; CASH, comprehensive assessment of symptoms and history; ASI, Addiction Severity Index; DSAT, Drug and Alcohol Screening Test; BPRS, Brief Psychiatric Rating Scale; DSM, Diagnostic and Statistical Manual; THC, Δ9‐ tetrahydrocannabinol; ICD, International Classification of Diseases.

Data extraction and coding

Differences in the search results were resolved by further examination of the publications by GSm and then by a joint examination by GSm and ML. ML and GSa extracted the data independently.

Terminology

The terms substance assay was used to describe either substance testing or substance screening. The term screening was used for methods of simultaneous analyses for multiple commonly used substances and was compared with patient self‐report of any psychoactive substance use. The term testing was used for assay of a substance specified on clinical grounds (either as a stand‐alone test or in the context of multiple substance analyses) and was compared to patient self‐report of specific substance use.

Patients were described as “self‐reporters” or “self non‐reporters”, acknowledging that some reporting no substance use might not be aware of the substances they had used or may have been unable to accurately report substance use due to the effects of severe mental illness. The term substance use was used in preference to abuse or dependence because substance testing and screening by assay can only detect the presence of substances, not substance use disorders.

Some studies used the diagnosis of substance abuse and dependence as the measure of substance use while others utilized the patients’ account of recent substance use. The terms substance abuse and dependence are only used in relation to substance use syndromes as defined by the Diagnostic and Statistical Manual of Mental Disorders – Fourth Edition (DSM‐IV) (American Psychiatric Association, 2000) and the International Classification of Diseases, 10th Revision (ICD‐10) (World Health Organization, 2010). Studies using patient reports of recent substance use defined “recent” as the period within which the substance assay might reasonably be expected to detect a substance, variably defined as between a day and a week.

True and false positive tests were defined by self‐report rather than the substance assay result. When any substance use was examined, patients who reported no substance use but returned a positive assay were regarded as having a false‐negative self‐report, despite the possibility that the substance assay had produced a biological false‐positive result. Patients who reported substance use but returned a negative assay were regarded as having false‐positive self‐reports, despite the probability that the substances taken by some had not been detected due to time delays or because the assay methods were not sufficiently sensitive. Sensitivity was therefore defined as the proportion of self‐reporters with a positive assay (true positives) among all the patients with a positive assay. Specificity was defined as the proportion of non‐reporters with a negative assay among all those with a negative assay. Positive predictive value (PPV) was defined as the proportion of positive assays among those reporting substance use. Negative predictive value (NPV) was defined as the proportion of negative assays among those not reporting substance use.

For the purpose of this paper we assumed that the substance assay was insensitive when the patient reported substance use and the substance assay was negative. In the reverse situation when the substance assay was positive and the patient denied substance use we assumed that the patient was inaccurate in the self‐report. Hence, we acknowledge both that some patients might report substance use that did not occur and that inevitably a small proportion of positive tests will be biological false positives.

Meta‐analysis

A random‐effects meta‐analysis was performed using Comprehensive Meta‐analysis (CMA Biostat; USA) version 2.2. Specifically the analysis calculated the odds ratio (OR) of positive substance tests among patients who reported and those who denied substance use. Using a single group meta‐analysis we also calculated pooled estimates of the sensitivity, specificity, PPV and NPV.

CMA is able to assess heterogeneity using both the I‐square and Q‐value statistics and employs the same computational algorithms to assess treatment effects and weight the studies by the inverse variance method used by the Cochrane Collaboration. A random effects model was chosen because the samples studied differed in terms of treatment settings and substance detection methods. Furthermore it is possible that using the current search strategy, we only located a sub‐sample of all existing studies on this topic. Cells with an N of zero were scored as one to allow log odds calculations. Effect sizes were classified as weak (OR between 1.5 and 2.5), moderate (OR between 2.5 and 4), strong (OR between 4 and 10) or very strong (OR greater than 10) (Rosenthal, 1996).

Using meta‐regression and between groups tests, we investigated sources of between study heterogeneity. We used random effects meta‐regression with a method of moments model to assess the association between the proportion of subjects with a psychotic illness and the effect size for the association between self‐reports and substance assay. A random effects meta‐regression was also used to assess the association between the year of the study and the effect size. Between‐group heterogeneity was used to compare the group of studies employing different methods of measuring self‐reports and detection of substances. These included comparisons of:

1. studies using recent substance use as the self‐report measure versus studies using a substance abuse or dependence diagnosis based on a clinical history or the results of a structured or semi‐structured clinical interview;

2. studies using a structured interview for substance use versus studies using unstructured interviews or a chart review;

3. studies using chromatography and spectrometry versus those using immuno‐assay or other methods of substance detection;

4. studies from different geographic regions.

Results

Included and excluded studies

Twenty‐six studies met the inclusion criteria (Figure 1 and Table 2). Fifteen of these examined patients’ reports of any recent substance use and any detected psychoactive substance. Eleven studies reported testing for one or more specific substance, but no data for any substance use. Eleven studies reported data for cannabis, in addition to which eight reported data for amphetamines, eight for cocaine, seven for opiates and five for alcohol. The studies were conducted in the United States (13 studies), Europe (nine studies) and Australasia (four studies).

Screening for any substance

Meta‐analysis of the results of the 15 studies considering psychoactive substance screening for any substance demonstrated a strong association [odds ratio (OR) = 7.2, 95% confidence interval (CI) = 3.6–14.1] between self‐reporting and substance screening (Figure 2) with a high degree of between‐study heterogeneity (I‐square = 86.1, Q‐value = 100.6, df = 14, p ≤ 0.001). There was a stronger association between reported substance use and substance screening in studies with a higher proportion of psychotic patients (N = 14 studies, slope = 0.027, standard error = 0.012, p = 0.021). There was no evidence that the between‐study heterogeneity was explained by the year the study was performed (N = 15 studies, slope = 0.003, standard error = 0.06, p = 0.96). Six studies using a structured interview to assess substance use reported a significantly stronger association between any reported substance use and substance screening (OR = 19.7, 95% CI = 6.1–63.8) than did the nine studies assessing substance use using clinical methods (OR = 4.0, 95% CI = 1.65–9.8; Q‐value = 4.48, df = 1, p = 0.034).

Figure 2.

Forest plot of studies of the association between self‐reports of substance use and substance assay.

There was a potentially important interaction between the independent variables of the proportion of subjects with psychosis and the use of a systematic measure of substance use, in that none of the six studies with fewer than 30% of subjects with a psychotic disorder used a systematic measure to assess substance use. By contrast, six of the eight studies with more than 30% of subjects suffering from psychosis used a systematic method to assess substance use. Eight studies using self‐report of recent substance use reported a non‐significantly weaker association with a positive substance screen (OR = 3.5, 95% CI = 1.2–10.4) than did seven studies using a diagnosis based on the patients’ longitudinal history (OR = 17.2, 95% CI = 5.1–58.2; Q‐value = 3.653, df = 1, p = 0.056). Ten studies using chromatography and spectrometry reported a non‐significantly weaker association between self‐reporting and substance screening (OR = 6.1, 95% CI = 2.1–17.5) than those relying on immuno‐assay or other tests (OR = 9.2, 95% CI = 2.3–37.3; Q‐value = 0.202, df = 1, p = 0.53). It was not possible to examine the effect of inpatient versus outpatient setting on the heterogeneity of the association between self‐reporting and substance screening because all but two of the studies included only inpatients. None of the between study heterogeneity could be explained by the region in which the study was conducted. There was no significant group difference between the pooled ORs from three Australian studies (OR = 3.2. 95% CI = 0.3–34.0), four European studies (OR = 10.0, 95% CI = 2.0–50.6) and eight studies from the United States (OR = 7.2, 95% CI = 2.0–24.8) (Q‐value = 0.61, df = 2, p = 0.74). Moreover, there appear to be between study differences arising from the way health care was funded. For example, two studies conducted in Australia, where there is universal government funded health care, reported a lack of any association between self‐reports of substance use and substance assay (Chung et al., 2009; Galletly et al., 1993) while a third Australian study found a very strong association (Fowler et al., 1998).

Testing for cannabis

Meta‐analysis of the 11 papers demonstrated a very strong association (OR = 22.3; 95% CI = 10.1–49.1) between self‐reporting of cannabis and cannabis testing with a low degree of between‐study heterogeneity (I‐square = 12.3, Q‐value = 12.4, df = 10, p = 0.26; see Figure 2).

There was no evidence that the between‐study heterogeneity in the association between reported cannabis and a positive test was explained by the proportion of patients with psychosis (slope = −0.005, standard error = 0.005, p = 0.52) or by the year the study was performed (N = 11 studies, slope = 0.08, standard error = 0.053, p = 0.36). The eight studies assessing substance use through a structured interview reported a similar association between cannabis use and testing (OR = 22.7, 95% CI = 12.7–40.5; Q‐value = 0.098, df = 1, p = 0.754) to those studies assessing substance use using clinical methods (OR = 19.6, 95% CI = 9.5–40.5). Five studies using self‐report of recent substance use reported a similar association with a positive test (OR = 22.5, 95% CI = 11.1–45.3) to studies using a diagnosis of substance abuse or dependence (OR = 20.8, 95% CI = 11.8–35.6; Q‐value = 0.027, df = 1, p = 0.868). Six studies using chromatography and spectrometry reported a similar association between self‐reporting and screening (OR = 25.6, 95% CI = 9.9–66.2) to those relying on immuno‐assay (OR = 20.4, 95% CI = 12.7–32.8; Q‐value = 0.176, df = 1, p = 0.676).

Testing for amphetamines, cocaine, opiates and alcohol

Meta‐analysis of eight papers demonstrated a very strong association (OR = 26.6; 95% CI = 8.0–88.9) between self‐reporting of amphetamine and positive assay for the substance, with a low degree of between‐study heterogeneity (I‐square = 2.8, Q‐value = 7.2, df = 7, p = 0.41; Figure 2).

Meta‐analysis of eight papers demonstrated a very strong association (OR = 39.7; 95% CI = 16.2–97.2) between self‐reported cocaine use and substance testing, with a high degree of between‐study heterogeneity (I‐square = 79.5, Q‐value = 34.2, df = 7, p < 0.001; Figure 2).

Meta‐analysis of seven papers demonstrated a very strong association (OR = 83.5; 95% CI = 26.7–260.7) between self‐reported opiate use and positive assay, with a high degree of between‐study heterogeneity (I‐square = 64.1, Q‐value = 16.7, df = 6, p = 0.01; Figure 2).

There was a strong association between self‐reporting and substance assay for illicit substance testing (an overall result for amphetamines, cannabis, cocaine and opiates) (OR = 35.6, 95% CI = 18.3–69.1, Z = 10.56, p < 0.001) with significant between group heterogeneity associated with a higher OR for opiates when compared with amphetamines, cannabis and cocaine (I‐square = 62.7, Q‐value = 88.6, df = 33, p < 0.001; Figure 2).

Meta‐analysis of five papers demonstrated a strong association (OR = 8.5; 95% CI = 2.5–28.4) between self‐reported alcohol use and substance assay, with a low degree of between‐study heterogeneity (I‐square = 40.6, Q‐value = 6.7, df = 4, p = 0.15; Figure 2).

Sensitivity, specificity, PPV and NPV of substance assays when compared with self‐report

Assuming that patients’ self‐report of any substance use were accurate, screening was not a highly specific or a highly sensitive method for detecting any substance use; approximately two‐thirds of patients reporting recent substance use had a positive substance screen and one‐quarter of those who did not report substance use had a positive screen (Table 3).

Table 3.

Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) estimated by meta‐analysis

Substance	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
Any substance	61	66	66	76
Cannabis	71	88	52	96
Amphetamines	30	98	53	94
Cocaine	59	89	74	93
Opiates	68	97	61	97
Alcohol	65	79	27	95

Testing for cannabis, cocaine or opiates in patients who reported taking those substances was not especially sensitive. Testing for amphetamine was quite insensitive. Approximately half of those patients reporting cannabis or cocaine use had a negative test and only approximately a quarter of those who reported recent alcohol use had a positive test. By contrast, a negative self‐report was quite specific, because about 95% of patients denying substance use had a negative test for that substance (Table 3).

Discussion

We report the first meta‐analysis of studies examining the strength of the association between self‐reported substance use and substance assays in general adult mental health settings. Our main findings are:

1. a strong association between self‐report of any substance use and a positive substance screen for any substance, with an overall sensitivity of 61% and a specificity of 66%;

2. a strong association between self‐report of the use of alcohol, amphetamine, cannabis, cocaine and opiates and a subsequent positive test for those substances, with a sensitivity varying from 30% for amphetamine to 71% for cannabis and a specificity varying from 79% for alcohol to 98% for amphetamine.

These findings lead us to reject our null hypotheses that there is no significant association between self‐report of substance use by psychiatric patients and substance assay results.

With respect to our second hypothesis, we found a stronger association between self‐report and testing in studies with a higher proportion of patients diagnosed as having a psychotic illness which was unexpected.

The strength of the association between self‐report and substance assay was largely due to the high specificity of testing. The sensitivity of substance assay was lower because no substances were detected in a third of patients who did report use of a substance, although the strength of association varied for different substances. In almost half the patients who reported using cannabis or amphetamine substance assay was negative, whilst a third of patients who reported using opiates and a quarter of those who reported using cocaine or alcohol returned negative assays.

In contrast, with testing for specific substances, self‐report of the use of any substance was less strongly associated with the results of screening, because at least one substance was detected in 27% of patients who denied any substance use.

The finding of a low sensitivity for testing could have been due to the lack of sensitivity of the substance assays, particularly for cannabis, or because the substance had been excreted in the period between ingestion and sampling. Hence, the main limitation of testing appears to be false‐negative tests, rather than false‐positives. The results also suggest if the aim of substance testing is to detect a particular type of substance use, testing for individual substances has little to offer over asking the patient if they had taken the substance.

However, there does appear to be a case for routine screening, because about a quarter of those who deny substance use will have detectable psychoactive substances, knowledge of which could assist in both diagnosis and treatment.

There are several possible reasons for negative testing in patients reporting substance use. The patient might have believed that the substance they used was different from the substance detected. For example, patients who believe that they had bought cocaine might have been sold amphetamine, resulting in a false‐positive report for cocaine, a false‐negative for amphetamines and a true positive if asked about any substance use. A second reason for the failure to detect a reported substance is that the substance had been metabolized or excreted before the sample could be collected. This is especially true for alcohol, metabolized in a linear way and cleared from the body in a matter of hours. Stimulant substances are also rapidly excreted.

The low level of detection of cannabis metabolites among patients reporting recent cannabis use is less readily explained. Regular use of cannabis results in a concentration of cannabis and its metabolites in fatty tissues, and cannabis can be detected for at least four weeks after its most recent use (Swatek, 1984). By contrast, occasional use of low doses of cannabis is not necessarily associated with detectable concentration of cannabis metabolites, even by more sensitive methods of substance assay such as gas chromatography mass spectrometry GCMS (Fritch et al., 2009). Other possible explanations include the high threshold for cannabis in some screening tests, which results in a negative test despite the presence of traces of cannabis. It is also possible that patients over‐report the use of cannabis to emphasize the role of substance use in their symptoms and to minimize the extent of an underlying mental illness, or they admit to cannabis use while not reporting the use of less acceptable substances.

The finding of a stronger association between self‐reporting and screening in studies where there were more patients with a diagnosis of psychotic illness was unexpected, although our analysis was unable to show that patients with psychosis were more accurate in reporting substance use than non‐psychotic patients. Possible explanations include the greater effect of intoxication by some substances on patients with underlying psychotic illness, triggering an acute presentation and hence a ready acknowledgement of the role of recent substance use. The likelihood of admitting to substance use would also depend on the level of cultural acceptance of substance use and the nature of community sanctions which could be incurred. The interaction between the independent variables (of the proportion of subjects with psychosis and the use of a systematic measure of substance use) might also be responsible for a spurious positive association between the proportion of subjects with psychosis and the strength of the association between self‐reporting and substance screening.

An investigation of the sources of heterogeneity in agreement between the results of self‐reporting and screening did not find a stronger association between substance use and substance testing in more recently conducted studies, but did suggest that using structured interviews about substance use might be more accurate than other methods of clinical assessment. Heterogeneity tests also suggested that studies including patients meeting the criteria for the diagnosis of a substance use disorder are more likely to have positive tests for substances than studies of other groups of patients who reported recent substance use.

A strength of this systematic review and meta‐analysis is that it allowed the calculation, in a large sample from a range of settings, the pooled estimates of measures of agreement between patient self‐reporting and the results of substance assay. Hence, the results are less likely to include chance findings and are more likely to be representative of the results found in clinical practice. However, this also raises a limitation of this study: that the results might not be typical of all mental health settings. The between study variation in the strength of the association between self‐reporting of substance use and substance assay could be due to the differences in the ability and willingness of patients from different populations to disclose substance use (Chung et al., 2009; Galletly et al., 1993).

A further limitation of the meta‐analysis is that the 26 studies utilized 17 different substance detection methods or combinations of methods, as well as a number of different methods of enquiry about, and definitions of, substance use. Newer, improved methods for substance assay were not available at the time of earlier studies. Saliva testing has been recommended by some authors because it is readily observed, requires less patient cooperation than urine assay and can now be assayed using better methods such as liquid chromatography‐tandem mass spectrometry, potentially overcoming the low sensitivity of many currently used tests (Badawi et al., 2009; Fritch et al., 2009).

An important finding of this study was the significant number of psychoactive substances detected by substance screening among mental health patients who had denied using any substances. A discussion of the cost and benefits of mental health practice regarding routine screening versus selective testing or no assay at all is beyond the scope of this paper. We have not considered issues related to patient consent to provide samples, or the potential harms of substance assay. For example, not all patients will consent to a substance assay, and a positive assay might harm some patients in the longer term, by prejudicing access to health insurance, employment or by the discovery of substance use in court cases. These harms might be particularly problematic in cases of a biological false positive substance assay incorrectly indicating substance use in a non‐user. However, routine substance screening at the time of presentation for mental health care could improve both the accuracy of diagnoses and the appropriateness of treatment for many patients, as some patients do not have accurate knowledge of the substances they have used and others might not report relevant substance use.

Conclusion

This study emphasizes both the uses and limitations of substance assay in mental health settings. It shows that the sensitivity of substance assay is not very high, and that a negative substance assay in a patient who denies substance use does not necessarily mean that the patient has not used substances. The low sensitivity of substance assay confirms the need to take a detailed and systematic substance use history in every patient as well as the importance of obtaining a sample soon after presentation if substance assay is planned.

The findings of this study should assist in development of clinical practice recommendations with regard to the use of substance screening and testing in the diagnosis of substance use in mental health settings and might stimulate research into more sensitive and specific methods of substance assay pertinent to this population.

Declaration of interest statement

ON has received fees for lectures (of his own material) from Astrazeneca, Janssen Cilag and Sanofi Aventis in the last five years. He has also received a research grant (to study survivors of violent suicide attempts) from Janssen Cilag. The other authors have no competing interests.