Skip to main content
PMC home page
International Journal of Methods in Psychiatric Research logoLink to International Journal of Methods in Psychiatric Research
. 2006 Mar 20;15(1):46–56. doi: 10.1002/mpr.28

A comparison of agency‐based and self‐report methods of measuring services across an urban environment by a drug‐abusing homeless population

David E Pollio 1,2,✉, Carol S North 3, Karin M Eyrich 4, Douglas A Foster 2, Edward L Spitznagel 5
PMCID: PMC6878718  PMID: 16676685

Abstract

The purpose of this paper is to advance the methodology for studying service assessment by comparing self‐report and agency‐generated methods. This study compares 30‐day self‐reported service use for homeless individuals (N = 229) randomly recruited from a single urban environment (St Louis, Missouri) with similar data collected from a broad array of service agencies providing homeless, substance abuse and outpatient mental health services across the same environment. Comparisons were made between self‐report and agency‐based data on shelter use, outpatient mental health service use (case management, psychiatric treatment, group therapy), outpatient substance abuse service use (case management, counselling, group therapy) and drop‐in/day treatment use. Consistently low levels of kappa scores (all under 0.4) and correlation coefficients (only shelter use demonstrated significant agreement) were found. Findings demonstrated that the two methods of collecting service data are generally not concordant at the individual level. Certain demographic characteristics (increased age, being male, non‐white ethnicity) and diagnoses (cocaine abuse/dependence, mania, schizophrenia) were associated with decreased reliability between the two methods of data collection. The two methods of assessment appeared to capture overlapping but not identical information. Each method of assessment has different utility to researchers and providers wishing to assess service use. Copyright © 2006 John Wiley & Sons, Ltd.

Keywords: services, service measurement, substance abuse, homelessness

Full Text

The Full Text of this article is available as a PDF (156.2 KB).

REFERENCES

  1. Barrow SM, Hellman F, Lovell AM, Plapinger JD, Struening EL. Effectiveness of Programs for the Mentally Ill Homeless. New York: New York Psychiatric Institute, 1989. [Google Scholar]
  2. Blankertz LE, Cnaan RA, Saunders M. Assessing the impact of serving the long‐term mentally disabled homeless. Journal of Sociology and Social Welfare 1992; 19: 199–220. [Google Scholar]
  3. Calsyn RJ, Allen G, Morse GA, Ruth, S , Templehoff B. Can you trust self‐report data provided by homeless mentally ill individuals? Eval Rev 1993; 17(3): 353–66. [Google Scholar]
  4. Chermack ST, Roll J, Reilly M, Davis L, Kilaru U, Grabowski J. Comparison of patient self‐reports and urinalysis results obtained under naturalistic treatment conditions. Drug Alcohol Depend 2002; 59: 43–9. [DOI] [PubMed] [Google Scholar]
  5. Chinman MJ, Rosenheck R, Lam JA. The case management relationship and outcomes of homeless persons with serious mental illness. Psychiatr Serv 2000; 51(9): 1142–7. [DOI] [PubMed] [Google Scholar]
  6. Cohen C, Onserud H, Monaco C. Outcomes for the mentally ill in a program for older homeless persons. Hospital and Community Psychiatry 1993; 44: 650–6. [DOI] [PubMed] [Google Scholar]
  7. Cook RF, Bernstein AD, Andrews CM. Assessing drug use in the workplace: A comparison of self‐report, urinalysis, and hair analysis In Harrison L, Hughes A. (eds) The Validity of Self‐Reported Drug Use: Improving the Accuracy of Survey Estimates. NIDA Research Monograph ♯167. Washington D.C.: National Institute on Drug Abuse, 1997. [PubMed] [Google Scholar]
  8. Cottler LB, Compton WC. Advantages of the CIDI family of instruments in epidemiological research on substance use disorders. Int J Meth Psychiatr Res 1993; 3: 109–19. [Google Scholar]
  9. Cox GB, Walker RD, Freng SA, Short BA, Meijer L, Gilchrist L. Outcome of a controlled trial of the effectiveness of intensive case management for chronic public inebriates. J Stud Alcohol 1998; (September): 523–32. [DOI] [PubMed] [Google Scholar]
  10. Dixon L, Stewart B, Krauss N, Robbins J, Hackman A, Lehman A. The participation of families of homeless persons with severe mental illness in an outreach intervention. Community Ment Health J 1998; 34(3): 251–9. [DOI] [PubMed] [Google Scholar]
  11. Falck R, Siegel HA, Forney MA, Wang J, Carlson RG. The validity of injection drug users' self‐reported use of opiates and cocaine. J Drug Issues 1992; 22(4): 823–32. [Google Scholar]
  12. Fendrich M, Vaughn CM. Diminished lifetime substance use over time: an inquiry into differential under‐reporting. Publ Opin Q 1994; 58: 96–123. [Google Scholar]
  13. Harrell AV. The validity of self‐reported drug use data: the accuracy of responses on confidential self‐administered answered sheets In Harrison L, Hughes A. (eds) The Validity of Self‐reported Drug Use: Improving the Accuracy of Survey Estimates. NIDA Research Monograph ♯167. Washington D.C.: National Institute on Drug Abuse, 1997, pp. 37–58. [PubMed] [Google Scholar]
  14. Humphreys K, Rosenheck R. Treatment involvement and outcomes for four subtypes of homeless veterans. Am J Orthopsychiatry 1998; 68(2): 285–94. [DOI] [PubMed] [Google Scholar]
  15. Kashner TM, Suppes T, Rush AJ, Altshuler KZ. Measuring use of outpatient care among mentally ill individuals: a comparison of self‐reports and provider records. Evaluation and Program Planning 1999; 22: 31–9. [Google Scholar]
  16. Kessler RG, McGonagle KA, Zhao S, Nelson CB, Hughes M, Eshelman S, Wittchen HU, Kendler KS. Lifetime and 12‐month prevalence of DSM‐III‐R psychiatric disorders in the United States: Results from the National Comorbidity Study. Arch Gen Psychiatr 1994; 51: 8–19. [DOI] [PubMed] [Google Scholar]
  17. Klinkenberg WD, Calsyn RJ, Morse GA. The helping alliance in case management for homeless persons with severe mental illness. Community Ment Health J 1998; 34(6): 569–78. [DOI] [PubMed] [Google Scholar]
  18. Koegel P, Sullivan G, Burnam A, Wenzel S. Utilization of mental health and substance abuse services among homeless adults in Los Angeles. Med Care 1999; 37: 306–17. [DOI] [PubMed] [Google Scholar]
  19. Lam JA, Rosenheck R. The effect of victimization on clinical outcomes of homeless persons with serious mental illness. Psychiatr Serv 1998; 49(5): 678–83. [DOI] [PubMed] [Google Scholar]
  20. Magura S, Kang S. The validity of self‐reported cocaine use in two high‐risk populations In Harrison L, Hughes A. (eds) The Validity of Self‐Reported Drug Use: Improving the Accuracy of Survey Estimates. NIDA Research Monograph ♯167. Washington D.C.: National Institute on Drug Abuse, 1997, pp. 227–46. [PubMed] [Google Scholar]
  21. Mallon GP. After care, then where? Outcomes of an independent living program. Child Welfare 1998; 77(1): 61–78. [PubMed] [Google Scholar]
  22. McGrew JH, Wright ER, Pescosolido BA, McDonel EC. The closing of central state hospital: Long‐term outcomes for persons with severe mental illness. The Journal of Behavioral Health Services and Research 1999; 26(3): 246–61. [DOI] [PubMed] [Google Scholar]
  23. Morrall AR, McCaffrey D, Iguchi MY. Hardcore drug users claim to be occasional users: Drug use frequency underreporting. Drug Alcohol Depend 2000; 57: 193–202. [DOI] [PubMed] [Google Scholar]
  24. Morse GA, Calsyn RJ, Allen G, Kenny DA. Helping homeless mentally ill people: what variables mediate and moderate program effects? Am J Community Psychol 1994; 22(5): 661–83. [DOI] [PubMed] [Google Scholar]
  25. North CS, Eyrich KM, Pollio DE, Foster DA, Cottler LB, Spitznagel EL. The homeless supplement to the Diagnostic Interview Schedule: test‐retest analysis. Int J Meth Psychiatr Res 2004; 13(3): 184–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Nunnally JC, Bernstein IH. Psychometric Theory. 3rd edn New York: McGraw‐Hill, 1994. [Google Scholar]
  27. Page WF, Davies JE, Ladner RA, Alfasso J, Tennis H. Urinalysis screened versus verbally reported drug use: The identification of discrepant groups. Int J Addiction 1977; 12(4): 439–50. [Google Scholar]
  28. Pollio DE, North CS, Eyrich KM, Foster DA, Spitznagel EL. Modeling service access in a homeless population. J Psychoactive Drugs 2003; 35(4): 487–96. [DOI] [PubMed] [Google Scholar]
  29. Pollio DE, North CS, Thompson SJ, Paquin JW, Spitznagel EL. Predictors of achieving stable housing in a mentally ill homeless population. Psychiatr Serv 1997; 44(4): 528–30. [DOI] [PubMed] [Google Scholar]
  30. Pollio DE, Spitznagel EL, North CS, Thompson SJ, Foster DA. Service use over time and achievement of stable housing in a mentally ill homeless population. Psychiatr Serv 2000; 51(12): 1536–43. [DOI] [PubMed] [Google Scholar]
  31. Rife JC, First RJ, Greenlee RW, Miller LD, Feichter MA. Case management with homeless mentally ill people. Health and Social Work 1991; 16(1): 58–67. [DOI] [PubMed] [Google Scholar]
  32. Robins L, Cottler L, Bucholz K, Compton W. Diagnostic Interview Schedule for DSM‐IV. St Louis MO: Washington University, 1995. [Google Scholar]
  33. Rosenheck R, Morrissey J, Lam J, Calloway M, Johnsen M, Goldman H, Randolph F, Blasinsky M, Fontana A, Calsyn R, Teague G. Service system integration, access to services, and housing outcomes in a program for homeless persons with severe mental illness. Am J Publ Health 1998; 88(11): 1610–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Schumacher JE, Milby JB, Caldwell E, Raczynski J, Engle M, Michael M, Carr J. Treatment outcome as a function treatment attendance with homeless persons abusing cocaine. J Addict Dis 1995; 14(4): 73–85. [DOI] [PubMed] [Google Scholar]
  35. Smith EM, North CS, Spitznagel EL. Are hard‐to‐interview street dwellers needed in assessing psychiatric disorders in homeless men? Int J Meth Psychiatr Res 1991; 1: 69–78. [Google Scholar]
  36. Smith EM, North CS, Spitznagel EL. A systematic study of mental illness, substance abuse and treatment in 600 homeless men. Ann Clin Psychiatr 1992; 4: 111–20. [Google Scholar]
  37. Smith EM, North CS, Spitznagel EL. Alcohol, drugs, and psychiatric comorbidity among homeless women: an epidemiologic study. J Clin Psychiatr 1993; 54: 82–7. [PubMed] [Google Scholar]
  38. Spitznagel EL, Helzer JE. A proposed solution to the base rate problem in the kappa statistic. Arch Gen Psychiat 1985; 42: 725–8. [DOI] [PubMed] [Google Scholar]
  39. Susser E, Conover S, Struening EL. Problems of epidemiologic method in assessing the type and extent of mental illness among homeless adults. Hospital and Community Psychiatry 1989; 40: 261–5. [DOI] [PubMed] [Google Scholar]
  40. Susser E, Conover S, Struening EL. Mental illness in the homeless: problems of epidemiologic method in surveys of the 1980s. Community Ment Health J 1990; 26: 391–412. [DOI] [PubMed] [Google Scholar]
  41. Weinreb L, Goldberg R, Bassuk E, Perloff J. Determinants of health and service use patterns in homeless and low‐income housed children. Pediatrics 1998; 102(3): 554–62. [DOI] [PubMed] [Google Scholar]
  42. Wish ED, Hoffman JA, Nemes S. The validity of self‐reports of drug use at treatment admission and at follow‐up: Comparisons with urinalysis and hair assays In Harrison L, Hughes A. (eds) The Validity of Self‐Reported Drug Use: Improving the Accuracy of Survey Estimates. NIDA Research Monograph ♯167. Washington D.C.: National Institute on Drug Abuse, 1997, pp. 200–26. [PubMed] [Google Scholar]
  43. Wolff N, Helminiak TW, Morse GA, Calsyn RJ, Klinkenberg WD, Trusty ML. Cost‐effectiveness evaluation of three approaches to case management for homeless mentally ill clients. Am J Psychiatr 1997; 154(3): 341–8. [DOI] [PubMed] [Google Scholar]

Articles from International Journal of Methods in Psychiatric Research are provided here courtesy of Wiley

RESOURCES