Skip to main content
NCBI home page
International Journal of Methods in Psychiatric Research logoLink to International Journal of Methods in Psychiatric Research
. 2008 Sep 3;17(3):159–173. doi: 10.1002/mpr.255

Methods for measuring utilization of mental health services in two epidemiologic studies

Douglas K Novins 1,✉, Janette Beals 1, Calvin Croy 1, Spero M Manson 1
PMCID: PMC4530964  NIHMSID: NIHMS710113  PMID: 18767205

Abstract

Objectives of Study: Psychiatric epidemiologic studies often include two or more sets of questions regarding service utilization, but the agreement across these different questions and the factors associated with their endorsement have not been examined. The objectives of this study were to describe the agreement of different sets of mental health service utilization questions that were included in the American Indian Service Utilization Psychiatric Epidemiology Risk and Protective Factors Project (AI‐SUPERPFP), and compare the results to similar questions included in the baseline National Comorbidity Survey (NCS).

Methods: Responses to service utilization questions by 2878 AI‐SUPERPFP and 5877 NCS participants were examined by calculating estimates of service use and agreement (κ) across the different sets of questions. Logistic regression models were developed to identify factors associated with endorsement of specific sets of questions.

Results: In both studies, estimates of mental health service utilization varied across the different sets of questions. Agreement across the different question sets was marginal to good (κ = 0.27–0.69). Characteristics of identified service users varied across the question sets.

Limitations: Neither survey included data to examine the validity of participant responses to service utilization questions.

Recommendations for Further Research: Question wording and placement appear to impact estimates of service utilization in psychiatric epidemiologic studies. Given the importance of these estimates for policy‐making, further research into the validity of survey responses as well as impacts of question wording and context on rates of service utilization is warranted. Copyright © 2008 John Wiley & Sons, Ltd.

Keywords: survey methodology, mental health service utilization, Indians, North American

Full Text

The Full Text of this article is available as a PDF (233.9 KB).

REFERENCES

  1. Alegria M, Kessler RC, Bijl R, Lin E, Heeringa SG, Takeuchi DT, Kolody B (2000). Comparing data on mental health service use between countries In Andrews G, Henderson S. (eds) Unmet Need in Psychiatry: Problems, Resources, Responses. Cambridge, MA: Cambridge University Press; pp. 97–118. [Google Scholar]
  2. Alegria M, Takeuchi D, Canino G, Duan N, Shrout P, Meng XL, Vega W, Zane N, Vila D, Woo M, Vera M, Guarnaccia P, Aguilar‐Gaxiola S, Sue S, Escobar J, Lin KM, Gong F (2004). Considering context, place and culture: the National Latino and Asian American Study. Int J Methods Psychiatr Res 13: 208–20. DOI: 10.1002/mpr.178 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Alonso J, Codony M, Kovess V, Angermeyer MC, Katz SJ, Haro JM, De Girolamo G, De Graaf R, Demyttenaere K, Vilagut G, Almansa J, Lepine JP, Brugha TS (2007). Population level of unmet need for mental healthcare in Europe. Br J Psychiatry 190: 299–306. DOI: 10.1192/bjp.bp.106.022004 [DOI] [PubMed] [Google Scholar]
  4. Ascher B, Farmer E, Burns B, Angold A (1996). The Child and Adolescent Services Assessment (CASA): description and psychometrics. J Emot Behav Dis 4: 12–20. [Google Scholar]
  5. Beals J, Manson SM, Shore JH, Friedman M, Ashcraft M, Fairbank J, Schlenger W (2002). The prevalence of posttraumatic stress disorder among American Indian Vietnam veterans: disparities and context. J Traumatic Stress 15: 89–97. DOI: 10.1023/A:1014894506325 [DOI] [PubMed] [Google Scholar]
  6. Beals J, Manson SM, Mitchell CM, Spicer P, The AI‐SUPERPFP Team (2003). Cultural specificity and comparison in psychiatric epidemiology: walking the tightrope in American Indian research. Cult Med Psychiatry 27: 259–89. DOI: 10.1023/A:1025347130953 [DOI] [PubMed] [Google Scholar]
  7. Beals J, Novins DK, Spicer P, Orton HD, Manson SM, The AI‐SUPERPFP Team (2004). Challenges in operationalizing the DSM‐IV clinical significance criteria. Arch Gen Psychiatry 61: 1197–207. DOI: 10.1001/archpsyc.61.12.1197 [DOI] [PubMed] [Google Scholar]
  8. Beals J, Manson SM, Whitesell NR, Spicer P, Novins DK, Mitchell CM, The AI‐SUPERPFP Team (2005). Prevalence of DSM‐IV disorders and attendant help‐seeking in two American Indian reservation populations. Arch Gen Psychiatry 62: 99–108. DOI: 10.1001/archpsyc.62.1.99 [DOI] [PubMed] [Google Scholar]
  9. Box‐Steffensmeier JM, Jacobson GC, Grant JT (2000). Question wording and the house vote choice: some experimental evidence. Public Opin Q 64: 257–70. DOI: 10.1086/317988 [DOI] [PubMed]
  10. Canino G, Shrout PE, Alegria M, Rubio‐Stipec M, Chavez LM, Ribera JC, Bravo M, Bauermeister JJ, Fabregas LM, Horwitz S, Martinez‐Taboas A (2002). Methodological challenges in assessing children's mental health services utilization. Ment Health Serv Res 4: 97–107. DOI: 10.1023/A:1015252217154 [DOI] [PubMed] [Google Scholar]
  11. Canino G, Shrout PE, Rubio‐Stipec M, Bird HR, Bravo M, Ramirez R, Chavez L, Alegria M, Bauermeister JJ, Hohmann A, Ribera J, Garcia P, Martinez‐Taboas A (2004). The DSM‐IV rates of child and adolescent disorders in Puerto Rico: prevalence, correlates, service use, and the effects of impairment. Arch Gen Psychiatry 61: 85–93. DOI: 10.1001/archpsyc.61.1.85 [DOI] [PubMed] [Google Scholar]
  12. Duan N, Alegria M, Canino G, McGuire T, Takeuchi D (2007). Survey conditioning in self‐reported mental health service use: randomized comparison of alternative instrument formats. Health Serv Res 42: 890–907. DOI: 10.1111/j.1475‐6773.2006.00618.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Farmer EMZ, Angold A, Burns BJ, Costello EJ (1994). Reliability of self‐reported service use: Test–retest consistency of children's responses to the Child and Adolescent Services Assessment (CASA). J Child Fam Stud 3: 307–25. DOI: 10.1007/BF02234688 [Google Scholar]
  14. Fleiss JL (1981). Statistical Methods for Rates and Proportions. New York: Wiley. [Google Scholar]
  15. Horwitz SM, Hoagwood K, Stiffman AR, Summerfeld T, Weisz JR, Costello EJ, Rost K, Bean DL, Cottler L, Leaf PJ, Roper M, Norquist G (2001). Reliability of the services assessment for children and adolescents. Psychiatric Serv 52: 1088–94. DOI: 10.1176/appi.ps.52.8.1088 [DOI] [PubMed] [Google Scholar]
  16. Hosmer DW, Lemeshow S (1989). Applied Logistic Regression. New York: John Wiley & Sons. [Google Scholar]
  17. Jensen PS, Watanabe HK, Richters JE (1999). Who's up first? Testing for order effects in structured interviews using a counterbalanced experimental design. J Abnorm Child Psychol 27: 439–45. DOI: 10.1023/A:1021927909027 [DOI] [PubMed] [Google Scholar]
  18. Kessler RC (2000). National Comorbidity Survey, 1990–1992. Ann Arbor, MI: Inter‐university Consortium for Political and Social Research, University of Michigan. [Google Scholar]
  19. Kessler RC, Ustun TB (2004). The World Mental Health (WMH) Survey Initiative Version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI). Int J Methods Psychiatr Res 13: 93–121. DOI: 10.1002/mpr.168 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Kessler RC, Olfson M, Berglund PA (1998). Patterns and predictors of treatment contact after first onset of psychiatric disorders. Am J Psychiatry 155: 62–9. [DOI] [PubMed] [Google Scholar]
  21. Kessler RC, Frank RG, Edlund M, Katz SJ, Lin E, Leaf P (1997a). Differences in the use of psychiatric outpatient services between the United States and Ontario [see comments]. N Engl J Med 336: 551–7. DOI: 10.1056/NEJM199702203360806 [DOI] [PubMed] [Google Scholar]
  22. Kessler RC, Zhao S, Katz SJ, Kouzis AC, Frank RG, Edlund M, Leaf P (1999). Past‐year use of outpatient services for psychiatric problems in the National Comorbidity Survey. Am J Psychiatry 156: 115–23. [DOI] [PubMed] [Google Scholar]
  23. Kessler RC, McGonagle KA, Zhao S, Nelson CB, Hughes M, Eshleman S, Wittchen H‐U, Kendler KS (1994). Lifetime and 12‐month prevalence of DSM‐III‐R psychiatric disorders in the United States. Results from the National Comorbidity Survey. Arch Gen Psychiatry 51: 8–19. [DOI] [PubMed] [Google Scholar]
  24. Kessler RC, Anthony JC, Blazer DG, Bromet E, Eaton WW, Kendler K, Swartz M, Wittchen HU, Zhao S (1997b). The US National Comorbidity Survey: Overview and future directions [editorial]. Epidemiologia E Psichiatria Sociale 6: 4–16. [DOI] [PubMed] [Google Scholar]
  25. Kish L (1965). Survey Sampling. New York: John Wiley & Sons. [Google Scholar]
  26. Landis JR, Koch GG (1977). The measurement of observer agreement for categorical data. Biometrics 33: 159–74. DOI: 10.2307/2529310 [PubMed] [Google Scholar]
  27. Narrow WE, Regier DA, Rae DS, Manderscheid RW, Locke BZ (1993). Use of services by persons with mental and addictive disorders. Arch Gen Psychiatry 50: 95–107. [DOI] [PubMed] [Google Scholar]
  28. Olfson M, Kessler RC, Berglund PA, Lin E (1998). Psychiatric disorder onset and first treatment contact in the United States and Ontario. Am J Psychiatry 155: 1415–22. [DOI] [PubMed] [Google Scholar]
  29. Rice DP, Miller LS (1998). Health economics and cost implications of anxiety and other mental disorders in the United States. Br J Psychiatry, Supplement 34: 4–9. [PubMed] [Google Scholar]
  30. SAS Institute Inc (2002–2003). SAS Language. Cary, NC: SAS Institute. [Google Scholar]
  31. Schuman H, Presser S (1981). Questions and answers in attitude surveys: experiments on question form, wording, and context. New York: Academic Press. [Google Scholar]
  32. Sedikides C, Skowronski J (1991). The law of cognitive structure activation. Psychol Inq 2: 169–84. DOI: 10.1207/s15327965pli0202‐_18 [Google Scholar]
  33. Spicer P (1998). Narrativity and the representation of experience in American Indian discourses about drinking. Cult Med Psychiatry 22: 139–69. DOI: 10.1023/A:1005305309191 [DOI] [PubMed] [Google Scholar]
  34. Spitznagel EL, Helzer JE (1985). A proposed solution to the base rate problem in the kappa statistic. Arch Gen Psychiatry 42: 725–8. [DOI] [PubMed] [Google Scholar]
  35. StataCorp (2003). STATA. College Station, TX: Stata Corporation. [Google Scholar]
  36. Stiffman AR, Horwitz SM, Hoagwood K, Compton W, 3rd, Cottler L, Bean DL, Narrow WE, Weisz JR (2000). The Service Assessment for Children and Adolescents (SACA): adult and child reports. J Am Acad Child Adolesc Psychiatry 39: 1032–9. DOI: 10.1097/00004583‐200008000‐00019 [DOI] [PubMed] [Google Scholar]
  37. Todorov A (2000a). The accessibility and applicability of knowledge: predicting context effects in national surveys. Public Opin Q 2000a; 64: 429–51. [DOI] [PubMed] [Google Scholar]
  38. Todorov A (2000b). Context effects in national health surveys: effects of preceding questions on reporting serious difficulty seeing and legal blindness. Public Opin Q 2000b; 64: 65–76. [PubMed] [Google Scholar]
  39. US Department of Health and Human Services (1999). Mental Health: A Report of the Surgeon General. Rockville, MD: US Department of Health and Human Services. [Google Scholar]
  40. US Department of Health and Human Services (2000). Epidemiologic Catchment Area Study, 1980–1985: [United States]. Ann Arbor, MI: Inter‐university Consortium for Political and Social Research, University of Michigan. [Google Scholar]
  41. Vega WA, Kolody B, Aguilar‐Gaxiola S, Catalano R (1999). Gaps in service utilization by Mexican Americans with mental health problems. Am J Psychiatry 156: 928–934. [DOI] [PubMed] [Google Scholar]
  42. Vega WA, Kolody B, Aguilar‐Gaxiola S, Alderete E, Catalano R, Caraveo‐Anduaga J (1998). Lifetime prevalence of DSM‐III‐R psychiatric disorders among urban and rural Mexican Americans in California. Arch Gen Psychiatry 55: 771–8. DOI: 10.1001/archpsyc.55.9.771 [DOI] [PubMed] [Google Scholar]
  43. Wang PS, Berglund P, Olfson M, Pincus HA, Wells KB, Kessler RC (2005a). Failure and delay in initial treatment contact after first onset of mental disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 62: 603–13. DOI: 10.1001/archpsyc.62.6.603 [DOI] [PubMed] [Google Scholar]
  44. Wang PS, Lane M, Olfson M, Pincus HA, Wells KB, Kessler RC (2005b). Twelve‐month use of mental health services in the United States: results from the National Comorbidity Survey Replication. Arch Gen Psychiatry 62: 629–40. DOI: 10.1001/archpsyc.62.6.629 [DOI] [PubMed] [Google Scholar]
  45. Wang PS, Demler O, Olfson M, Pincus HA, Wells KB, Kessler RC (2006). Changing profiles of service sectors used for mental health care in the United States. Am J Psychiatry 163: 1187–98. DOI: 10.1176/appi.ajp.163.7.1187 [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Wu LT, Kouzis AC, Leaf PJ (1999). Influence of comorbid alcohol and psychiatric disorders on utilization of mental health services in the National Comorbidity Survey. Am J Psychiatry 156: 1230–6. [DOI] [PubMed] [Google Scholar]

Articles from International Journal of Methods in Psychiatric Research are provided here courtesy of Wiley

RESOURCES