Abstract
PURPOSE The short, validated, self-administered, Case-finding and Help Assessment Tool (CHAT) for lifestyle and mental health assessment of adult patients in primary health care addresses inactivity, tobacco use, alcohol and other drug misuse, problem gambling, depression, anxiety and stress, abuse, and anger problems. For each issue patients are asked whether they would like help, either during the consultation or at a later date. This study aims to assess the value of the help question.
METHODS Validation of the CHAT was conducted according to the STAndards for Reporting of Diagnostic accuracy studies statement for diagnostic tests. The setting was Auckland primary care practices with populations ranging from socioeconomically advantaged to deprived. Participants were 755 consecutive primary care patients who completed the CHAT plus the help question and reference standards. Sensitivity, specificity, and likelihood ratios with and without the addition of help the question were calculated.
RESULTS Sensitivity ranged from 80% to 98% for the more-common conditions (depression, nicotine dependency, anxiety, problematic drinking). For each condition, specificity increased with the addition of the help question: depression increased from 73% to 98%; anxiety 77% to 99%; drinking 85% to 99%; verbal anger 92% to 99%; verbal abuse 97% to 99%; problematic drinking and gambling 98% to 99%.
CONCLUSIONS The help question increased specificity without compromising sensitivity and reduced false positives, thereby increasing the positive predictive value. It allowed patients with comorbidities to prioritize issues they wished to address, indicate their readiness to change, promote self-determination, and give the clinician an indication of which topics to pursue.
Keywords: Sensitivity and specificity, primary health care, mass screening, life style, mental health, risk reduction behavior, validation studies
INTRODUCTION
There is a progressive trend for primary care to be a continuing health care process, improving community health through preventive services, disease prevention, screening, and generalist first-level interventions.1 Many at-risk behaviors and mental health issues, however, remain unidentified in routine practice, with an estimated only 30% of those needing treatment receiving it.2 To this end, we developed the short, self-administered, Case-finding and Help Assessment Tool (CHAT) for lifestyle and mental health assessment of adult patients (aged 16 years and older) in primary care.3 The tool, which assesses for inactivity, tobacco use, alcohol and other drug misuse, problem gambling, depression, anxiety and stress, abuse and anger problems, and insomnia, was designed by a multidisciplinary team from general practice, nursing, psychology, and university academia. For each item (1 or 2 questions), patients are asked whether they would like help with the issue, either during the consultation or at a later date. The tool has now undergone testing for acceptability, reliability, and validity. Initially the tool included questions assessing for eating disorders, but these were found to be imprecise and replaced by a question on insomnia.4 For the current CHAT see the Supplemental Appendix, available at http://annfammed.org/cgi/content/full/7/3/239/DC1.
Patients might object to being asked or be embarrassed by sensitive questions about their lives. For example, studies looking at whether women object to being asked about domestic violence report a wide variability, ranging from 15% to 57%, of patients who are unhappy about being asked.5 It was hypothesized that combining issues in a self-administered tool would increase acceptability, reduce the possibility of patients being offended by a particular question, and inform patients that their primary care practitioners are interested in these issues and can offer assistance.
The acceptability of the CHAT was evaluated in a study involving more than 2,500 patients from 20 randomly selected urban and 11 rural general practice physicians and 20 practice nurses.3 Patients wanting help the day of the consultation (0.5% to 13.5%) did not overwhelm the practitioners. The tool was well accepted by patients, with less than 1% objecting to any of the questions. Both physicians and nurses were keen to use the tool once it became available. It takes most patients less than 2 minutes to complete the CHAT.
In separate studies of the 2 depression questions6,7 and of the anxiety question8 plus the help question, the addition of the question inquiring whether help is needed was found to increase test specificity (reducing false positives) while maintaining sensitivity.
Validation of the tool was conducted with 1,000 consecutive primary care patients completing both the CHAT and a composite reference standard.4 The aim of our current analysis was to assess the additional value of the help question for each of the individual items in terms of estimating diagnostic accuracy.
METHODS
The CHAT was validated in primary care practices serving socioeconomically deprived populations in South Auckland and socioeconomically advantaged populations in Auckland’s North Shore in 2006–2007.
Consecutive primary health care patients aged 16 years and older were recruited from waiting room situations to complete both the CHAT with the help questions and the composite reference standard (Table 1 ▶).9–15 Although the Diagnostic and Statistical Manual-IV diagnostic interviews might be ideal, selection of the reference standards was pragmatic after considering the time restraints of recruited patients in the waiting room and the prohibitively large sample size required if patients completed only 1 diagnostic instrument on a random basis.
Table 1.
Case Prevalence by CHAT and Reference Standards
| Condition | CHAT- Positive Cases | Reference Standard | Case Criteria | Reference Standard–Positive Cases % (n/N)a |
| Nicotine dependency | 38 | Heavy Smoking Index (HSI)9 | >2 | 6.8 (51/755) |
| Problematic drinking | 67 | Alcohol Use Disorders Identification Test (AUDIT)10 | >7 | 11.3 (84/746) |
| Problematic drug use | 9 | Drug Abuse Screening Test (DAST)11 | >5 | 1.9 (14/750) |
| Problematic gambling | 4 | South Oaks Gambling Screen (SOGS)12 | ≥4 | 0.7 (5/688) |
| Major depression | 30 | Patient Health Questionnaire – Depression (PHQ-9)13 | ≥15 | 4.1 (30/737) |
| Anxiety | 58 | Hospital Anxiety & Depression Scale (HADS)14 | A >10 | 9.3 (68/728) |
| Being verbally abused | 8 | Conflict Tactic Scale 1 (CTS-1)15 | VA ≥15 | 2 (12/688) |
| Being physically abused | 1 | Conflict Tactic Scale 1 (CTS-1)15 | PA ≥24 | 0.3 (2/594) |
| Being verbally angry | 10 | Conflict Tactic Scale 1 (CTS-1)15 | VA ≥15 | 2.2 (13/594) |
| Being physically angry | 1 | Conflict Tactic Scale 1 (CTS-1)15 | PA ≥24 | 0.3 (2/594) |
A=anxiety score; CHAT=Case-finding and Help Assessment Tool; PA = physical aggression scale; VA = verbal aggression scale.
a No. of cases/No. responding.
The CHAT and the composite reference standard forms were self-administered by patients in the waiting room. There was a research assistant available to assist with consent and form collection, and they were advised not to look at the screening tool answers when the patients were completing the reference standard. Where the CHAT showed a risk factor that the patient wanted addressed, the family physician could either deal with the problem at the time of the consultation or schedule an appointment for a later date.
The validation was conducted according to the STAndards for Reporting of Diagnostic accuracy studies (STARD) statement for diagnostic tests.16 Approval for the study was obtained from the Auckland Ethics Committee (REF Study AKY/04/10/367).
Data analysis using Microsoft Excel was conducted on all cases where participants had completed both the base CHAT questions, including the help question, and the reference standard. Reference standard scores were dichotomized as ”case” or “not a case.” Sensitivities, specificities, positive and negative predictive values, and likelihood ratios were calculated using the Center for Evidence-Based Medicine online statistical calculator (http://www.cebm.utoronto.ca/practise/ca/statscal/).
The sensitivity and specificity of CHAT questions compared with those for the reference standards were calculated for items relating to tobacco use, alcohol and other drug misuse, problem gambling, depression, anxiety and stress, abuse, and anger problems. We omitted from this analysis the question relating to inactivity, because the initial format of the question in the validation study was reversed (a yes response meant that the person was physically active), which was confusing and led to inverted responses in some cases. The current version of CHAT has addressed this issue. Specificity was also calculated for those items for which patients answered yes to a help question (either wanting help today or later).
RESULTS
Although 1,000 patients were recruited for the validation study,4 only 755 completed the CHAT forms, including the help questions, and the reference standards. There was a 2% decline from consecutive eligible patients invited to participate. The case prevalence detected by the reference standards ranged from 0.3% for being a victim of verbal abuse or for having difficulty controlling one’s physical anger to 11.3% for problematic drinking (Table 1 ▶).
Table 2 ▶ displays the percentage of positive cases with a request for help. Requests for help ranged from 11% for those with problematic drinking to 57% for those with major depression. Most patients wanting help, however, did not request it for the current consultation but were prepared to return later to address the issue.
Table 2.
Percentage of CHAT-Positive Patients Wanting Help
| Condition | Positive Cases No. | Wanting Help No. (%) | Wanting Help Not Today No. (%) | Wanting Help Today No. (%) |
| Nicotine dependency | 51 | 23 (45) | 18 (35) | 5 (10) |
| Problematic drinking | 84 | 9 (11) | 5 (6) | 4 (5) |
| Problematic drug use | 14 | 3 (21) | 2 (14) | 1 (7) |
| Problematic gambling | 5 | 1 (20) | 1 (20) | 0 (0) |
| Major depression | 30 | 17 (57) | 11 (37) | 6 (20) |
| Anxiety | 68 | 27 (40) | 12 (18) | 15 (26) |
| Being verbally abused | 14 | 2 (14) | 1 (7) | 1 (7) |
| Being verbally angry | 13 | 4 (31) | 3 (23) | 1 (8) |
CHAT=Case-finding and Help Assessment Tool.
Sensitivity ranged from 80% to 98% for the more-common conditions (depression, nicotine dependency, anxiety, problematic drinking) (Table 3 ▶). For less-common conditions, such as problematic gambling and drug use, abuse, and difficulty controlling anger, sensitivities are lower and confidence intervals are wider.
Table 3.
Comparison of CHAT With Reference Standards: Sensitivity and Specificity
| Condition | % Sensitivity (95% CI) | Positive CHAT Responses/ Positive Reference Standard | % Specificity (95% CI) | Negative CHAT Responses/ Negative Reference Standard |
| Nicotine dependency | 88 (77–95) | 38/51 | 91 (89–93) | 61/704 |
| Problematic drinking | 80 (70–87) | 67/84 | 85 (82–87) | 101/662 |
| Problematic drug use | 64 (39–84) | 9/14 | 98 (97–99) | 3/736 |
| Problematic gambling | 80 (38–99) | 4/5 | 98 (97–99) | 13/683 |
| Major depression | 98 (86–100) | 30/30 | 73 (70–76) | 191/707 |
| Anxiety | 85 (75–92) | 58/68 | 77 (73–80) | 155/660 |
| Being verbally abused | 57 (33–79) | 8/14 | 97 (96–98) | 16/580 |
| Being verbally angry | 77 (50–92) | 10/13 | 92 (89–94) | 47/581 |
CHAT=Case-finding and Help Assessment Tool; CI = confidence interval.
For some conditions, particularly depression and anxiety, specificity is relatively low (73% and 77%, respectively), meaning that along with true positives, responses to the CHAT included a number of false positives. Even so, because the CHAT in effect has 2 tiers (patients only respond to the help question if they have indicated problem condition), the help question can improve the specificity without reducing the sensitivity. Once a patient has indicated yes to a CHAT issue, that patient is identified as CHAT positive, and only CHAT-positive patients indicate whether they want help. Table 4 ▶ shows that for each condition the specificity increases with the addition of the help question (patients who say they want help are much more likely to have the condition in question).
Table 4.
CHAT Specificity With and Without the Help Questions
| Condition | Specificity No Help Question % (95% CI) | Specificity With Help Question % (95% CI) |
| Nicotine dependency | 91 (89–93) | 99 (98–99) |
| Problematic drinking | 85 (82–87) | 99 (99–100) |
| Problematic drug use | 98 (97–99) | 99 (99–100) |
| Problematic gambling | 98 (97–99) | 99 (99–100) |
| Major depression | 73 (70–76) | 98 (97–99) |
| Anxiety | 77 (73–80) | 99 (98–100) |
| Being verbally abused | 97 (96–98) | 99 (98–100) |
| Being physically abused | 94 (92–95) | 99 (98–100) |
| Being verbally angry | 92 (89–94) | 99 (98–100) |
| Being physically angry | 91 (88–93) | 99 (98–100) |
CHAT=Case-finding and Help Assessment Tool; CI = confidence interval.
Table 5 ▶ displays the likelihood ratio of the reference standard being positive when patients request help (either today or later) for the more-common conditions of nicotine dependency, problematic drinking, major depression, and anxiety. When calculating the likelihood ratios of being a positive case if the help question answer is positive, wanting help today indicates a very high likelihood of the person being positive for that particular condition. Positive likelihood ratios for all 4 conditions show a consistent downward progression from wanting help today to wanting help at a later date. When wanting help today or later are combined, the likelihood ratios range between 9.4 and 11.8.
Table 5.
Effect of Help Question on Likelihood Ratio of Being CHAT Positive for Common Conditions
| Positive Condition | Help Today LR (95% CI) | Help But Not Today LR (95% CI) | Not Requesting Help LR (95% CI) | Help Requested (Either Today or Later) LR (95% CI) |
| Nicotine dependency | 11.5 (3.6–36.4) | 10.4 (6.0–17.4) | 0.6 (0.5–0.7) | 10.6 (6.7–16.8) |
| Problematic drinking | 70.2 (3.8–1,292.6) | 6.6 (2.0–21.0) | 0.9 (0.8–1.0) | 11.8 (4.3–32.4) |
| Major depression | 21.6 (10.4–45.0) | 5.2 (2.3–11.7) | 0.5 (0.3–0.7) | 10.3 (6.6–15.9) |
| Anxiety | 18.2 (8.0–41.0) | 5.8 (3.0–11.4) | 0.6 (0.5–0.8) | 9.4 (5.9–14.9) |
CHAT=Case-finding and Help Assessment Tool; CI = confidence interval; LR = likelihood ratio.
DISCUSSION
The findings indicate that the help question serves as a second step for case finding and increases the specificity of the test without compromising its sensitivity. The help question serves to reduce the false positives in that patients who have a CHAT-positive response and who indicate they would like help are very likely to have the condition being assessed. This finding is consistent with those of previous studies in which we compared a composite international reference standard with the 2 screening questions for depression6 and the anxiety questions7 with and without the help questions.
Using the help question reduces the consulttion time needed to determine which patients require immediate intervention. The results also show a progressive increase in the likelihood ratio for positive case findings from wanting help some time in the future to wanting help today, which raises the posttest probability.
It should be noted that false-positive CHAT responses may represent patients with subsyndromal conditions. Time is a tool, and once an issue has been highlighted, physician and patient may explore the issue further at a later date.
Implications
Asking patients whether they would like help with an issue allows primary care clinicians to identify those patients with problem conditions. It also allows patients who may have comorbidities (eg, problem gambling, drinking, and depression) to prioritize the issues they wish to address. An added benefit is not overloading the clinician with multiple problems during a single consultation. Similarly, the option of wanting help at a later date enables scheduling a specific appointment. Furthermore, patients’ indication that they want help is likely to correlate with their readiness to change.17
The small percentage of CHAT-positive patients requesting immediate help indicates that routine use of the CHAT would not lead to an overwhelming increase in the need for immediate care. Conversely, patients who score is positive but who do not want help can still be offered a brief intervention. For example, nicotine-dependent smokers who say no to help can be told that, when they do decide to address their smoking, there is help available.
The help question is already being incorporated in health services delivery. For example, in the United Kingdom primary care Quality and Outcomes Framework, the help question is a recommended indicator for a case finding of depression.18
Study Limitations
This study was limited by inclusion of some less-common conditions; hence, sensitivity and specificity for these conditions were low and confidence intervals were wide. Reduction from the original 1,000 participants4 to the 755 who returned completed CHAT forms, help questions, and reference standard forms further magnified this issue. Calculating specificity for help questions in those patients with positive CHAT scores further diminished the denominators. Furthermore, we used reference standard comparison instruments, because conducting diagnostic interviews for all conditions would have been excessively time-consuming in the primary health care setting.
Strengths
There was a consistent finding that use of the help question improves specificity while maintaining sensitivity, that the likelihood ratios of having a condition increase when patients indicate they would like help, and that the likelihood ratios increase more so when patients identify that they would like help that day. Because patients complete the CHAT before their consultation, the help question allows them to indicate whether they have any issues they wish to address in that day’s consultation. CHAT is unlikely to inhibit or hinder patients from discussing their own agenda rather than their doctor’s prevention agenda.
The CHAT is an important tool for routine use in primary health care settings for lifestyle and mental health domains, where strong argument can be made for case finding and subsequent intervention. Its use is now being adopted in a variety of settings, such as the entry criterion into a New Zealand primary mental health initiative and as a resource guide for lifestyle prescriptions in South Australian general practices. Some NZ practices are using it with all new patients and are asking adult patients to complete it if more than 2 years have elapsed since their last visit. An electronic version is being developed to be integrated in the electronic medical record and self-administered on touch-screens in waiting rooms. During the consultation, if a patient has a positive response on the CHAT, second-tier tools (eg, the Alcohol Use Disorders Identification Test) will be available for clinicians to administer using a computer that includes automatic scoring.
Future Research
Because it is quick to use and well-accepted, the CHAT can be used for follow-up after intervention for identified problems. Our next research step is to conduct a randomized trial to compare CHAT results with clinical outcomes, which will establish whether systematic use of the CHAT in the primary health care setting leads to better health outcomes for patients.
The CHAT embodies many of the principles of the Alma-Ata declaration.1 As a self-administered tool that invites respondents to consider whether they want help with the issues it raises, the CHAT promotes self-reliance and self-determination. It can be offered or administered by a range of primary care clinicians, including physicians, nurses, and community workers. Because it addresses a number of lifestyle behaviors, as well as disturbed mood, its informs patients that primary care physicians are concerned about social and community activities that affect their lives and their health.
The CHAT can be used to identify at-risk patients for whom education, primary prevention, and early intervention can be provided to improve health. As a simple, efficient, and validated tool well-suited to the resource- and time-strapped primary care environment, it allows health care clinicians to assess rapidly the important mental and social needs of their patients. The help question reduces the numbers of false-positive findings and identifies issues about which patients indicate concerns and their readiness to change.
Acknowledgments
The study involved collaboration between primary health care researchers with specific lifestyle or mental health interests and expertise in the Department of General Practice and Primary Health Care, the University of Auckland in the development of the tool. We thank all those who have made contributions to this work over the several years of its development.
Conflict of interest: none reported
Funding support: This study was funded by the Oakley Mental Health Foundation and the Charitable Trust of the Auckland Faculty of the Royal New Zealand College of General Practitioners. All researchers were completely independent of the funders.
Disclaimer: The study sponsor had no role in the study design, data collection, data analysis, data interpretation, or writing of the report. All authors had access to the data.
REFERENCES
- 1.International Conference on Primary Health Care. Declaration of Alma-Ata. Alma-Ata, USSR, 1978. http://www.who.int/hpr/NPH/docs/declaration_almaata.pdf.
- 2.Andrews G, Sanderson K, Slade T, Issakidis C. Why does the burden of disease persist? Relating the burden of anxiety and depression to effectiveness of treatment. Bull World Health Organ. 2000;78(4):446–454. [PMC free article] [PubMed] [Google Scholar]
- 3.Goodyear-Smith F, Arroll B, Sullivan S, Elley R, Docherty B, Janes R. Lifestyle screening: development of an acceptable multi-item general practice tool. NZ Med J. 2004;117(1205):U1146. [PubMed] [Google Scholar]
- 4.Goodyear-Smith F, Coupe N, Arroll B, Elley C, Sullivan S, McGill A. Case-finding of lifestyle and mental health problems in primary care: validation of the ‘CHAT’. Br J Gen Pract. 2008;58(546):26–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Goodyear-Smith F, Arroll B. Screening for domestic violence in general practice: A way forward? Br J Gen Pract. 2003;53(492):515–518. [PMC free article] [PubMed] [Google Scholar]
- 6.Arroll B, Goodyear-Smith F, Kerse N, Fishman T, Gunn J. Effect of the addition of a “help” question to two screening questions on specificity for diagnosis of depression in general practice: diagnostic validity study. BMJ. 2005;331(7521):884.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Goodyear-Smith F, Arroll B. Short screening tools for depression. Br J Gen Pract. 2007;57(538):412–413. [PMC free article] [PubMed] [Google Scholar]
- 8.Puddifoot S, Arroll B, Goodyear-Smith F, Kerse N, Fishman T, Gunn J. A new case-finding tool for anxiety: a pragmatic diagnostic validity study in primary care. Int J Psychiatry Med. 2007;37(4):371–381. [DOI] [PubMed] [Google Scholar]
- 9.Kozlowski LT, Porter CQ, Orleans CT, Pope MA, Heatherton T. Predicting smoking cessation with self-reported measures of nicotine dependence: FTQ, FTND, and HSI. Drug Alcohol Depend. 1994;34(3):211–216. [DOI] [PubMed] [Google Scholar]
- 10.Wennberg P, Escobar F, Espi F, et al. The alcohol use disorders identification test (AUDIT): a psychometric evaluation. Reports from the Department of Psychology, U. Stockholm. No. 1996; 811:1–14. [Google Scholar]
- 11.Dawe S, Loxton N, Hides L, Kavanagh D, Mattick R. Review of Diagnostic Screening Instruments for Alcohol and Other Drug Use and Other Psychiatric Disorders. 2nd ed. Commonwealth of Australia; 2002:172.
- 12.Lesieur HR, Blume SB. The South Oaks Gambling Screen (SOGS): a new instrument for the identification of pathological gamblers. Am J Psychiatry. 1987;144(9):1184–1188. [DOI] [PubMed] [Google Scholar]
- 13.Spitzer RL, Williams JB, Kroenke K, et al. Utility of a new procedure for diagnosing mental disorders in primary care. The PRIME-MD 1000 study. JAMA. 1994;272(22):1749–1756. [PubMed] [Google Scholar]
- 14.Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand. 1983;67(6):361–370. [DOI] [PubMed] [Google Scholar]
- 15.Straus MA. Measuring intrafamily conflict and violence: the conflict tactic (CT) scales. In: Straus MA, Gelles RJ, eds. Physical Violence in American Families: Risk Factors and Adaptations to Violence in 8,145 Families. New Brunswick: Transaction Publishers; 1990: 29–47.
- 16.Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Fam Pract. 2004;21(1):4–10. [DOI] [PubMed] [Google Scholar]
- 17.DiClemente CC, Prochaska JO. Toward a comprehensive, transtheoretical model of change: stages of change and addictive behaviors. In: Miller WR, Heather N, eds. Treating Addictive Behaviors. 2nd ed. New York, NY: Plenum; 1998.
- 18.Kendrick T, Lester H, Society for Academic Primary Care. Quality and Outcomes Framework (QOF): Depression. Southampton and Manchester: National Primary Care Research and Development Centre; 2007.