Abstract
The use of services for mental problems is generally reported as being relatively low. However, the methods used for data collection in surveys may have influenced the quality of self‐reported service use. This study compares the information on recourse to physicians for mental problems reported in different sections of a survey conducted in six European countries. Thus, 5545 respondents were asked questions on contacts with physicians at least twice: (1) after the symptoms checklist in any completed diagnostic section, and (2) in a section devoted to use of care for mental problems. Of these 39.3% reported contacts with physicians about mental problems in the diagnostic sections, whereas 29.5% did so in the use‐of‐care section. Inconsistencies concerned 20.1% of participants, among whom those reporting consultations in diagnostic sections without reporting them in the use‐of‐care section represented the majority (74.4%). Multiple logistic regression analysis revealed that age, marital status, educational level and country were associated with under‐reporting in the use‐of‐care section, as well as having mood or sleep problems. In conclusion, services used for mental health reasons when measured through a question referring to use of care due to the presence of a mental problem may underestimate the care people received for their problems. Copyright © 2011 John Wiley & Sons, Ltd.
Keywords: use of care, methods of data collection, survey, mental health
Introduction
Many studies on health care use are based on survey data obtained from self‐administered questionnaires, or from telephone or face‐to‐face interviews, in which people provide details on the different services they have used to deal with their health problems, in particular their visits to health professionals. Given the role of such studies in the design of health policy, especially for health care resource allocation and planning, a better understanding of the quality of such information is crucial.
Few studies have assessed the quality of consumer responses to use‐of‐care surveys and many of them have focused on specific groups of people, such as the elderly (Bush et al., 1989; Carsjö et al., 1994; Glandon et al., 1992; Green et al., 1979; Lubeck and Hubert, 2005; Raina et al., 2002; Rozario et al., 2004; Wallihan et al., 1999), the mentally ill (Byford et al., 2007; Clark et al., 1996; Goldberg et al., 2002; Kashner et al., 1999; Killeen et al., 2004), and the homeless (Calsyn et al., 1997; Pollio et al., 2006), for whom such issues may be particularly pertinent. Some of these studies have assessed the reliability of self‐reported data measuring the repeatability of questionnaires on health care use using essentially test–retest or inter‐interviewer techniques (Chung et al., 2008; Dubois et al., 2007; Fabricant and Harpham, 1993; Goldberg et al., 2002; Santelli et al., 2002). However, most of these studies have focussed on evidence for validity assessed by consistency with other sources of data such as health insurance claims, medical records or survey data collected from health care professionals.
Validity assessments of this kind generally indicate that survey participants' responses are biased toward under‐reporting of use of services (Clark et al., 1996; Jobe et al., 1990; Petrou et al., 2002; Ritter et al., 2001), although there are some exceptions (Kashner et al., 1999). In addition, the validity of self‐report may be related to the type of health care services considered, with lower agreement between survey data and administrative or medical data for visits to physicians, and better consistency for events such as emergency room visits and hospitalization (Brown and Adams, 1992; Byford et al., 2007; Mirandola et al., 1999; Petrou et al., 2002; Ritter et al., 2001; Roberts et al., 1996; Ungar et al., 1998), the extent of care use, with a tendency for the discrepancy to increase with the amount of services used (Kashner et al., 1999; Ritter et al., 2001; Roberts et al., 1996), the period of time covered, with less consistency observed when the recall period is longer (Petrou et al., 2002; Roberts et al., 1996), and the time elapsed since the last episode of care, patient reports being less accurate the longer the time between service use and reporting (Clark et al., 1996). Generally, no such association with reporting consistency has been observed for socio‐demographic characteristics (Reijneveld and Stronks, 2001; Ritter et al., 2001), with the exception of lower consistency for immigrants (Reijneveld, 2000). In addition, differences in reporting quality were found in individuals with certain mental disorders, poorer consistency being observed, for example, for schizophrenic patients (Clark et al., 1996; Mirandola et al., 1999). No such difference was reported for physical illnesses (Ritter et al., 2001).
Surveys on use of care for mental health problems have traditionally yielded particularly low rates of contact with the health care system for these problems. This has led several authors to study the extent of unmet mental health care needs for those who are the most in need (Alonso et al., 2007; Demyttenaere et al., 2004). As described earlier, in all surveys, self‐reported data may be subject to an under‐reporting bias. However, we hypothesize that this bias may be more important in mental health surveys, since these surveys require respondents not only to remember their use of services, but also to label their problems as mental. Indeed, information on service use for mental problems is generally collected in a specific section of the questionnaire beginning with a broad question asking participants to report the services they used for their mental problems. This is the case, for instance, in the World Mental Health (WMH) survey in Europe (ESEMeD) on which this article is based(The World Mental Health Survey Initiative). In this survey, similar questions on use of mental health care are present in different sections of the interview. It is possible that information of health care use obtained from the use‐of‐care section of the interview, which is generally used to provide rates of contact, may yield lower estimates than information derived from the section devoted to the diagnosis of mental disorders. The study presented here, which was restricted for technical reasons to the measurement of physician contacts, evaluated whether inconsistencies could be detected between the diagnostic and use‐of‐care sections, and assessed possible demographic and health‐related factors associated with these inconsistencies.
Method
The ESEMeD survey is part of the World Health Organization (WHO) WMH survey initiative(The World Mental Health Survey Initiative). A detailed description of this survey has been published in Alonso et al. (2004). Briefly, ESEMeD is a one‐shot cross‐sectional survey of non‐institutionalized adults aged 18 and over from six European countries selected using a stratified, multistage, clustered area, probability sample design. Eligible individuals were interviewed face‐to‐face by trained lay interviewers at their homes after they had given consent. The questionnaire, subdivided into 38 different sections, was administered on the same day using computer‐assisted interview (CAPI) techniques. Internal sub‐sampling was used to optimize the interviewing process and reduce cost by dividing the interview in two parts. Part 1 assessed diagnosis of the most common mood and anxiety disorders, health related quality of life, health services utilization and demographics. Part 2 included, inter alia, an in‐depth interview about additional mental disorders, self‐reported chronic physical conditions, and risk factors. All respondents completed Part 1. All Part 1 respondents who reported more than a pre‐specified number of symptoms of specific mood and anxiety disorders, as well as a random 25% of the rest of the sample, were administered Part 2.
Sample
In total, 21,425 individuals were interviewed between January 2001 and August 2003. The overall response rate in the six countries was 61.2%, with the highest rates being observed in Spain (78.6%) and Italy (71.2%) and the lowest in Germany (57.8%), the Netherlands (56.4%), Belgium (50.6%) and France (45.9%). In this analysis, only subjects participating in Part 2 were considered (n = 8796). Because the objective of this study was to compare the responses to the questions on use of services for mental health reasons provided at different moments of the interview, we selected the 5545 (40.7%) respondents in the Part 2 sample who completed at least one diagnostic section in which questions on use of services were asked, as well as the use‐of‐care section of the interview, which was completed by all participants in the survey.
Use of health services for mental health problems
One section of the interview was devoted to the use of care for “problems with emotions or mental health” and was administered to all participants in the survey after the diagnostic sections had been administered. Respondents were first asked if they had consulted for such problems. Individuals reporting any contact with a provider were asked to select whom they had consulted from a list of health professionals (including psychiatrists; non‐physician mental health professionals, such as psychologists, psychotherapists, social workers and counsellors; general practitioners; other medical specialists; other health professionals, such as nurses and nutritionists), and of non‐health care professionals. In addition, in all diagnostic sections, with the exception of those on post‐traumatic stress, substance use, conduct disorder, or separation anxiety, individuals who screened positive for the disorder and who declared sufficiently significant symptoms to complete the whole section had previously reported if they had talked to a physician concerning the symptoms about which they were interviewed. The 5545 respondents we selected for our study gave information on their use of services for mental health reasons in both sections (Table 1).
Table 1.
Diagnostic sectionsa | Use‐of‐care section |
---|---|
D29. Did you ever talk to a medical doctor about your episodes of sadness or discouragement or lack of interest? | SR1. The next question is about speaking with professionals about problems with emotions or mental health. By “professionals” we mean medical doctors, nurses, psychologists, social workers, spiritual advisors, herbalists, and any other healing professionals. With this definition in mind, did you ever in your life talk to a medical doctor or other professional about any problem with your emotions or mental health? |
SR2. Did you ever in your life talk to any of the following types of professionals about problems with your emotions or mental health: | |
SR2a. a psychiatrist? | |
… | |
SR2c. a general family doctor? | |
SR2d. any other doctor? | |
… |
Example of the diagnostic section on depression.
Mental disorder status
Information on mental health was collected using the third version of the Composite International Diagnostic Interview (CIDI 3.0), which was developed and adapted by the WHO Coordinating Committee for their WMH survey initiative (Kessler and Ustun, 2004). The diagnostic sections have been expanded in CIDI 3.0 to include a measure of impairment, which allows the severity of any mental disorders to be assessed. The diagnosis of mental disorders was made using the criteria of the Diagnostic and Statistical Manual of Mental Disorders Fourth Edition (DSM‐IV). A clinical reappraisal conducted in parallel to the main ESEMeD survey showed that diagnoses assigned with the CIDI 3.0 were concordant with clinical diagnoses based on the SCID (Haro et al., 2006). The mental disorders considered in this study, which correspond to the diagnostic sections comprising questions on use of services, were mood disorders (major depression, dysthymia, and pre‐menstrual dysphoric disorder), anxiety disorders (panic disorder, specific phobia, social phobia, agoraphobia with or without panic disorder, generalized anxiety disorder, and obsessive‐compulsive disorder), eating disorders (anorexia nervosa, bulimia nervosa, and binge‐eating disorder), and childhood disorders (attention‐deficit/hyperactivity disorder and oppositional defiant disorder). For other disorders (post‐traumatic stress, substance use, conduct disorder and separation anxiety), questions about contact with any health care professional were asked, but as it was not possible to discern if a physician had been contacted, these diagnostic sections were not considered in this analysis.
Other variables
In a series of questions about chronic conditions adapted from the US Health Interview Survey, respondents were asked about the presence of chronic pain, including arthritis or rheumatism, chronic back or neck problems, frequent or severe headaches, or any other chronic pain. Information on the use of sedatives was collected through two questions, the first one asking respondents if they had used a sedative or tranquilizer on their own without a doctor's recommendation and the second one asking them if they had used a sedative or tranquilizer that a doctor prescribed for them. A list of sedatives and tranquilizers including the three most commonly prescribed ones in the country was provided as an example.
Statistical analysis
The chi square test was used to evaluate differences in categorical variables between countries. The kappa coefficient was used to evaluate the concordance between reporting in the diagnostic and use‐of‐care sections of the interview. The closeness of the comparison for the estimated kappa values was determined using the Landis and Koch benchmark (Landis and Koch, 1977). Multiple logistic regression analysis was used to assess the association between socio‐demographic and clinical factors, on the one hand, and under‐reporting of service use in the use‐of‐care section of the interview, on the other hand. A probability level of 0.05 was deemed statistically significant.
Data for individual subjects were weighted to account for the known probabilities of selection as well as to restore the age and gender distribution of the population within countries and the relative sample size between countries (Alonso et al., 2004). The Taylor linearized variance estimator was used for statistical inference to take into account the complexity of the sampling design. The statistical analysis was performed using StataSE 9.1 software.
Results
The socio‐demographic characteristics of the study sample are presented in Table 2. In all countries, participants were most frequently female (60.1%) and aged 35 to 64 years (51.4%). All other characteristics differed between countries, notably the proportion of individuals with a post‐secondary education. Overall, 57.6% of respondents reporting symptoms of mental problems qualified for a diagnosis of mental disorder, this proportion varying significantly (p < 0.001) from 46.6% in Italy to 66.8% in France.
Table 2.
Characteristics | Belgium(n = 677) | France(n = 1105) | Germany(n = 780) | Italy(n = 1026) | Netherlands(n = 748) | Spain(n = 1209) | Overall(n = 5545) | p |
---|---|---|---|---|---|---|---|---|
Gender | ||||||||
Male | 41.0 | 39.3 | 41.1 | 39.0 | 41.0 | 39.0 | 39.9 | 0.926 |
Female | 59.0 | 60.7 | 58.9 | 61.0 | 59.0 | 61.0 | 60.1 | |
Age | ||||||||
18–34 years | 33.3 | 32.9 | 31.6 | 29.5 | 31.9 | 37.9 | 32.3 | 0.167 |
35–64 years | 48.9 | 51.8 | 53.6 | 51.3 | 55.1 | 44.5 | 51.4 | |
≥65 years | 17.8 | 15.3 | 14.8 | 19.2 | 13.0 | 17.6 | 16.3 | |
Marital status | ||||||||
Married, cohabiting with someone | 66.0 | 67.6 | 58.6 | 64.6 | 70.3 | 62.2 | 63.8 | <0.001 |
Separated, widowed, divorced, single | 12.9 | 13.7 | 17.5 | 9.6 | 12.7 | 11.2 | 13.6 | |
Single | 21.1 | 18.7 | 23.9 | 25.8 | 17.0 | 26.6 | 22.6 | |
Income level | ||||||||
Low | 15.6 | 25.0 | 19.4 | 18.6 | 26.5 | 17.9 | 20.9 | 0.036 |
Average | 69.8 | 59.1 | 62.4 | 65.8 | 59.2 | 66.7 | 62.8 | |
High | 14.6 | 15.9 | 18.2 | 15.6 | 14.3 | 15.4 | 16.3 | |
Educational level | ||||||||
Primary, secondary education | 54.8 | 63.4 | 74.8 | 58.3 | 43.9 | 63.9 | 64.2 | <0.001 |
Post‐secondary education | 45.2 | 36.6 | 25.2 | 41.7 | 56.1 | 36.1 | 35.8 | |
Any mental disorderb | ||||||||
Yes | 59.1 | 66.8 | 54.3 | 46.6 | 66.7 | 59.9 | 57.6 | <0.001 |
No | 40.9 | 33.2 | 45.7 | 53.4 | 33.3 | 40.1 | 42.4 |
Data are presented as percentages. All values were obtained after weighting. For each socio‐demographic variable, potential differences in prevalence rates between countries were assessed using the chi square test.
The proportion of subjects who reported having sought help from physicians for mental health problems varied significantly between the two sections of the interview (diagnosis sections and use‐of‐care section), both overall and in each individual participating country (Table 3). In the diagnostic section, 39.3% of participants reported having talked to a physician about mental problems, whereas 29.5% reported visits to physicians for mental health problems in the use‐of‐care section. The percentage agreement between the two sections and the kappa coefficients were lowest in France and highest in Germany, this range of values corresponding to moderate to substantial agreement.
Table 3.
Country | n | Diagnostic sectionsb | Use‐of‐care section | p | Percentage agreement | Kappa | ||
---|---|---|---|---|---|---|---|---|
Percentage | Confidence interval | Percentage | Confidence interval | |||||
Belgium | 677 | 46.0 | [40.0; 52.0] | 37.3 | [31.4; 43.6] | 0.004 | 76.4 | 0.52 |
France | 1105 | 50.4 | [45.8; 54.9] | 38.1 | [33.9; 42.4] | <0.001 | 72.4 | 0.45 |
Germany | 780 | 33.7 | [29.6; 38.1] | 27.2 | [23.5; 31.2] | <0.001 | 84.6 | 0.64 |
Italy | 1026 | 27.5 | [24.5; 30.7] | 15.2 | [13.0; 17.6] | <0.001 | 83.0 | 0.51 |
Netherlands | 748 | 55.0 | [49.1; 60.9] | 44.8 | [39.5; 50.3] | <0.001 | 77.4 | 0.55 |
Spain | 1209 | 38.8 | [34.9; 42.9] | 31.0 | [27.5; 34.8] | <0.001 | 82.2 | 0.61 |
Overall | 5545 | 39.3 | [37.3; 41.3] | 29.5 | [27.6; 31.3] | <0.001 | 79.9 | 0.56 |
All values were obtained after weighting.
Any diagnostic section.
The inconsistencies observed in self‐reported use of care between the two sections are presented in Table 4. Out of the 5545 participants who replied to at least one diagnostic section as well as the use‐of‐care section, 20.1% provided contradictory information, this proportion being highest in France (27.6%) and lowest in Germany (15.4%). Moreover, these contradictory responses most often corresponded to a positive answer in the diagnostic sections and a negative answer in the use‐of‐care section (74.4%).
Table 4.
Country | Respondents with contradictory responses | Types of inconsistencies in respondents with contradictory results | |||
---|---|---|---|---|---|
n | Percentage***, b | n | Percentage ofc | ||
Yes in diagnostic sections | No in diagnostic sections | ||||
No in use‐of‐care section | Yes in use‐of‐care section | ||||
Belgium | 677 | 23.6 | 185 | 68.4 (16.2) | 31.6 (7.5) |
France | 1105 | 27.6 | 307 | 72.2 (20.0) | 27.8 (7.7) |
Germany | 780 | 15.4 | 148 | 71.2 (11.0) | 28.8 (4.4) |
Italy | 1026 | 17.0 | 243 | 86.3 (14.7) | 13.7 (2.3) |
Netherlands | 748 | 22.6 | 183 | 72.6 (16.4) | 27.4 (6.2) |
Spain | 1209 | 17.8 | 266 | 71.9 (12.8) | 28.1 (5.0) |
Overall | 5545 | 20.1 | 1332 | 74.4 (15.0) | 25.6 (5.2) |
All values were obtained after weighting.
Potential differences in prevalence rates between countries were assessed using the chi square test.
Figures in parentheses correspond to prevalence rates in our interviewee population, the size of which being given in the second column.
p < 0.001.
In order to identify factors associated with under‐reporting of medical consultations, multiple logistic regression was used to compare the respondents who reported consultations with physicians for psychological problems in the diagnostic sections only (n = 1097) to those who reported their contacts with physicians in both types of sections (n = 2105). Only subjects reporting having talked to a physician about mental problems in response to at least one question in the diagnostic section were selected for the analysis, in order to exclude bias from medical consultations for mental problems other than those explored in the diagnostic section (for example, substance use or dependence problems). For this reason, only inconsistencies of the type “yes in the diagnostic section and no in the use‐of‐care section”, which represent around three‐quarters of the contradictory responses observed (74.4%; Table 4), were studied.
This multiple logistic regression analysis identified certain personal and social factors, although sometimes of borderline significance, associated with under‐reporting of utilization of services in the use‐of‐care section of the interview (Table 5). Older age was associated with an increased likelihood of under‐reporting, whereas people who were separated, widowed or divorced and those with a better education were less likely to under‐report service use. The analysis also identified country as being associated with under‐reporting, this being less frequent in Germany, the Netherlands and Spain, and more frequent in France. Italy, where the proportion of contradictory responses was the highest, was the country in which under‐reporting was most marked.
Table 5.
Characteristics | Odds ratio | 95% Confidence interval |
---|---|---|
Gender | ||
Male | 1.00 | |
Female | 0.95 | [0.72; 1.26] |
Age | ||
18–34 years | 1.00 | |
35–64 years | 1.07 | [0.74; 1.55] |
≥65 years | 1.61 | [1.00; 2.63]* |
Marital status | ||
Married, cohabiting with someone | 1.00 | |
Separated, widowed, divorced | 0.78 | [0.56; 1.08] |
Single | 1.10 | [0.78; 1.55] |
Income level | ||
Low | 1.21 | [0.87; 1.67] |
Average | 1.00 | |
High | 0.89 | [0.61; 1.29] |
Educational level | ||
Primary, secondary education | 1.00 | |
Post‐secondary education | 0.75 | [0.56; 1.02] |
Country c | ||
Belgium | 0.98 | [0.74; 1.30] |
France | 1.20 | [0.96; 1.51] |
Germany | 0.66 | [0.50; 0.88]** |
Italy | 2.08 | [1.65; 2.62]*** |
Netherlands | 0.72 | [0.57; 0.91]** |
Spain | 0.86 | [0.70; 1.04] |
Multiple logistic regression adjusted for all clinical variables. All values were obtained after weighting.
Participants having reported contacts with physicians in the diagnostic sections and not in the use‐of‐care section (n = 1097) versus those having reported contacts with physicians in both sections (n = 2105).
Compared to the overall mean value.
p < 0.05;
p < 0.01;
p < 0.001.
When considering health‐related factors (Table 6), under‐reporting of medical consultations for mental problems in the use‐of‐care section of the interview was lowest in participants suffering from mood disorders and in those reporting use of sedative drugs, which can be taken as a proxy of sleep problems.
Table 6.
Characteristics | Odds ratio | 95% Confidence interval |
---|---|---|
Mood disorders | ||
Yes | 0.44 | [0.33; 0.58]*** |
No | 1.00 | |
Anxiety disorders | ||
Yes | 0.99 | [0.73; 1.33] |
No | 1.00 | |
Eating disorders | ||
Yes | 1.37 | [0.82; 2.27] |
No | 1.00 | |
Childhood disorders | ||
Yes | 0.63 | [0.33; 1.21] |
No | 1.00 | |
Use of sedative drugs | ||
Yes | 0.44 | [0.33; 0.59]*** |
No | 1.00 | |
Chronic pain | ||
Yes | 0.96 | [0.74; 1.26] |
No | 1.00 | |
Number of years sincethe last symptoms | ||
0–5 years | 1.00 | |
6–15 years | 0.94 | [0.66; 1.34] |
>15 years | 0.76 | [0.45; 1.27] |
Number of diagnosticsessions completed | ||
1 | 1.00 | |
≥2 | 0.90 | [0.66; 1.24] |
Multiple logistic regression adjusted for all socio‐demographic variables. All values were obtained after weighting.
Participants having reported contacts with physicians in the diagnostic sections and not in the use‐of‐care section (n = 1097) versus those having reported contacts with physicians in both sections (n = 2105).
p < 0.001.
Discussion
Our study shows first that visits to physicians for mental health problems appear to be less reported in the use‐of‐care section of the interview when compared to the diagnostic sections. Moreover, some features were found to be associated with a lower prevalence of use in the use‐of‐care section, notably age and the type of mental symptoms reported. Finally, although the same inconsistency was identified in all countries, the probability of under‐reporting in the use‐of‐care section varied from one country to another.
Most published studies on the validity of responses to use‐of‐care surveys have noted the issue of under‐reporting. In these studies, a clear under‐reporting bias exists when data is collected from direct participant response compared with other sources of data, particularly administrative data on health care utilization. In our study, differences in prevalence of use are apparent within the same survey, depending on how the questions are structured. In particular, physician consultation is less reported in the use‐of‐care section of the interview compared with other sections dealing with diagnosis of mental disorders.
Several factors may contribute to this difference in reporting sensitivity. Firstly, many studies assessing the reproducibility of responses to questions on use of care have shown that the same question asked twice to a given individual may generate different responses, and that response stability may vary according to the type of questions asked (for example, sentence complexity, time frame, or proposed response modalities) (Fabricant and Harpham, 1993; Santelli et al., 2002) and according to the type of services for which information is collected (Chung et al., 2008; Dubois et al., 2007). Moreover, several reports have highlighted the importance of question wording on the respondent's understanding of the question (Bhandari and Wagner, 2006; Del Boca and Noll, 2000; Schwarz, 1999), which is the case in the present study, where the questions asked in the use‐of‐care section and in the diagnostic sections of the interview are slightly different (Table 1). Secondly, and perhaps most importantly, in the use‐of‐care section of the interview, respondents were asked if they had already consulted a physician for any problem with their emotions or mental health. This implies that respondents identify and label mental health problems correctly (Jorm, 2000; Jorm et al., 2006). In our study, the observation that respondents with mood problems have less difficulty reporting consultations in the use‐of‐care section, compared to respondents with eating disorders is consistent with this interpretation. In the same way, one would have expected that respondents with sleep problems or chronic pain would have more difficulty reporting their medical visits in the use‐of‐care section since these disorders are generally perceived as physical problems, although they are most often symptoms of a more mental condition. In addition, in the use‐of‐care section of the interview, respondents are implicitly expected to recognize that they have suffered from a mental health problem. Given the stigma which is attached to these problems, respondents may want to present themselves in a favourable fashion (Del Boca and Noll, 2000) and thus not admit that they have sought help. This is consistent with the fact that in Italy, where discrimination against the mentally ill has been shown to be marked (European Commission, 2006), under‐reporting of service use is relatively more frequent. Finally, in the diagnostic sections, talking about symptoms from the outset may help respondents to identify these as mental problems and trigger recall of the care received. It is well known that the context in which questions are asked may have an impact on responses (Bhandari and Wagner, 2006; Del Boca and Noll, 2000; Schwarz, 1999).
Our survey has a number of limitations which should be taken into account when interpreting these results. Firstly, the questions asked regarding visits to physicians in the different sections were not formulated in exactly the same way (Table 1), which may bias to some extent the comparison, given the importance of questionnaire design on participants' responses (Bhandari and Wagner, 2006; Del Boca and Noll, 2000; Fabricant and Harpham, 1993; Santelli et al., 2002; Schwarz, 1999). Secondly, the 5545 participants evaluated in the second phase of the survey represent a subset of the Part 2 sample, since they were required to report sufficiently significant symptoms in order to be eligible for the diagnostic sections. Concerning the extent of this potential bias, our 5545 participants include 71.6% of all Part 2 respondents having reported any contact with a physician in the survey, the remaining 28.4% corresponding to individuals who reported contacts with physicians in the use‐of‐care section only as they were not asked any other question on use of services in the rest of the interview. Thirdly, our results only address visits to physicians and are thus not generalizable to all health care services. Indeed, many previous studies of the quality of survey data, in terms of stability (Chung et al., 2008; Dubois et al., 2007) or agreement with other data sources (Brown and Adams, 1992; Byford et al., 2007; Mirandola et al., 1999; Petrou et al., 2002; Reijneveld and Stronks, 1999; Ritter et al., 2001; Roberts et al., 1996; Ungar et al., 1998), have shown that the quality of the data obtained varies according to the type of services considered. Fourthly, use‐of‐care data collected in our survey was not validated by any comparison with outside sources of data. In this way, discussion of under‐reporting and bias may be considered as inappropriate since it implicitly assumes that when people reported service use in any section, they had effectively received care for their problems. Finally, the participation rate was relatively low in one country (France).
Conclusion
Surveys are generally the only way to identify all care received by a given population for a given problem. For example, insurance claims or prescription databases do not provide information on services which are not covered by health insurance (for example, psychologists in Belgium, France and Italy), and medical records do not provide information on recourse to professionals outside the health care system such as human services. Both situations are particularly frequent when it comes to the care of mental problems (Kovess‐Masfety et al., 2007; Kovess‐Masfety et al., 2010; Sevilla‐Dedieu et al., 2010; Wang et al., 2005). However, the issue of under‐reporting in such surveys, which has been extensively documented in the literature, may be particularly relevant to the reporting of care of mental problems due to the large number and wide variety of mental health issues, which are not well known by the lay public. Our study reveals that collecting information on use of services for mental problems through a single question without previously reminding respondents about the different mental disorders or any of their symptoms, which is commonly done in mental health surveys in a section devoted to use of care, seems to underestimate contacts with professionals. Given the impact these estimates may have on the design of health policies, such a bias should be borne in mind when interpreting such data. This kind of consideration has led the WMH to generate a composite variable to measure use of care, bringing together information collected in all sections of the survey interview (i.e. diagnostic and use‐of‐care sections) (Wang et al., 2007). However, for individuals who do not have enough symptoms to complete questions on use of care in at least one diagnostic section, the use‐of‐care section might be the only way to assess service use. Therefore, for future research, efforts should be made to improve reporting of use of services in the use‐of‐care section, for example by making more explicit what is meant by mental health problems, in particular for conditions less likely to be considered as mental problems, such as eating disorders or alcohol problems. In addition, questions on mental health, using at least a brief instrument such as the MH5 (Mental Health 5), even though this deals only with emotions, should be completed by all survey respondents before answering questions on use of care to give them a basic awareness of the range of mental health issues.
Declaration of interest statement
The authors declare that they have no competing interests.
Acknowledgements
The ESEMeD project (http://www.epremed.org) was supported by the European Commission (Contracts QLG5‐1999‐01042; SANCO 2004123), the Piedmont Region (Italy), Fondo de Investigación Sanitaria, Instituto de Salud Carlos III, Spain (FIS 00/0028), Ministerio de Ciencia y Tecnología, Spain (SAF 2000‐158‐CE), Departament de Salut, Generalitat de Catalunya, Spain, and other local agencies and by an unrestricted educational grant from GlaxoSmithKline. ESEMeD is carried out in conjunction with the World Health Organization World Mental Health (WMH) Survey Initiative. We thank the WMH staff for assistance with instrumentation, fieldwork, and data analysis. These activities were supported by the US National Institute of Mental Health (R01MH070884), the John D. and Catherine T. MacArthur Foundation, the Pfizer Foundation, the US Public Health Service (R13‐MH066849, R01‐MH069864, and R01 DA016558), the Fogarty International Centre (FIRCA R03‐TW006481), the Pan American Health Organization, the Eli Lilly & Company Foundation, Ortho‐McNeil Pharmaceutical, Inc., GlaxoSmithKline, Bristol‐Myers Squibb, and Shire.
References
- Alonso J., Angermeyer M.C., Bernert S., Bruffaerts R., Brugha T.S., Bryson H., de Girolamo G., Graaf R., Demyttenaere K., Gasquet I., Haro J.M., Katz S.J., Kessler R.C., Kovess V., Lepine J.P., Ormel J., Polidori G., Russo L.J., Vilagut G., Almansa J., Arbabzadeh‐Bouchez S., Autonell J., Bernal M., Buist‐Bouwman M.A., Codony M., Domingo‐Salvany A., Ferrer M., Joo S.S., Martinez‐Alonso M., Matschinger H., Mazzi F., Morgan Z., Morosini P., Palacin C., Romera B., Taub N., Vollebergh W.A. (2004) Sampling and methods of the European Study of the Epidemiology of Mental Disorders (ESEMeD) project. Acta Psychiatrica Scandinavica. Supplementum, 109(s420), 8–20. [DOI] [PubMed] [Google Scholar]
- Alonso J., Codony M., Kovess V., Angermeyer M.C., Katz S.J., Haro J.M., De Girolamo G., De Graaf R., Demyttenaere K., Vilagut G., Almansa J., Lepine J.P., Brugha T.S. (2007) Population level of unmet need for mental healthcare in Europe. British Journal of Psychiatry, 190(4), 299–306. [DOI] [PubMed] [Google Scholar]
- Bhandari A., Wagner T. (2006) Self‐reported utilization of health care services: Improving measurement and accuracy. Medical Care Research and Review, 63(2), 217–235. [DOI] [PubMed] [Google Scholar]
- Brown J.B., Adams M.E. (1992) Patients as reliable reporters of medical care process. Recall of ambulatory encounter events. Medical Care, 30(5), 400–411. [DOI] [PubMed] [Google Scholar]
- Bush T.L., Miller S.R., Golden A.L., Hale W.E. (1989) Self‐report and medical record report agreement of selected medical conditions in the elderly. American Journal of Public Health, 79(11), 1554–1556. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Byford S., Leese M., Knapp M., Seivewright H., Cameron S., Jones V., Davidson K., Tyrer P. (2007) Comparison of alternative methods of collection of service use data for the economic evaluation of health care interventions. Health Economics, 16(5), 531–536. [DOI] [PubMed] [Google Scholar]
- Calsyn R.J., Morse G.A., Klinkenberg W.D., Trusty M.L. (1997) Reliability and validity of self‐report data of homeless mentally ill individuals. Evaluation and Program Planning, 20(1), 47–54. [Google Scholar]
- Carsjö K., Thorslund M., Wärneryd B. (1994) The validity of survey data on utilization of health and social services among the very old. Journal of Gerontology, 49(3), S156–S164. [DOI] [PubMed] [Google Scholar]
- Chung S., Domino M.E., Jackson E.W., Morrissey J.P. (2008) Reliability of self‐reported health service use: Evidence from the Women with Co‐occurring Disorders, and Violence Study. Journal of Behavioral Health Services and Research, 35(3), 265–278. [DOI] [PubMed] [Google Scholar]
- Clark R.E., Ricketts S.K., McHugo G.J. (1996) Measuring hospital use without claims: A comparison of patient and provider reports. Health Services Research, 31(2), 153–169. [PMC free article] [PubMed] [Google Scholar]
- Del Boca F.K., Noll J.A. (2000) Truth or consequences: The validity of self‐report data in health services research on addictions. Addiction, 95(11s3), 347–360. [DOI] [PubMed] [Google Scholar]
- Demyttenaere K., Bruffaerts R., Posada‐Villa J., Gasquet I., Kovess V., Lepine J.P., Angermeyer M.C., Bernert S., de Girolamo G., Morosini P., Polidori G., Kikkawa T., Kawakami N., Ono Y., Takeshima T., Uda H., Karam E.G., Fayyad J.A., Karam A.N., Mneimneh Z.N., Medina‐Mora M.E., Borges G., Lara C., de Graaf R., Ormel J., Gureje O., Shen Y., Huang Y., Zhang M., Alonso J., Haro J.M., Vilagut G., Bromet E.J., Gluzman S., Webb C., Kessler R.C., Merikangas K.R., Anthony J.C., Von Korff M.R., Wang P.S., Brugha T.S., Aguilar‐Gaxiola S., Lee S., Heeringa S., Pennell B.E., Zaslavsky A.M., Ustun T.B., Chatterji S. (2004) Prevalence, severity, and unmet need for treatment of mental disorders in the World Health Organization World Mental Health Surveys. Journal of the American Medical Association, 291(21), 2581–2590. [DOI] [PubMed] [Google Scholar]
- Dubois M.F., Raîche M., Hébert R., Gueye N.R. (2007) Assisted self‐report of health‐services use showed excellent reliability in a longitudinal study of older adults. Journal of Clinical Epidemiology, 60(10), 1040–1045. [DOI] [PubMed] [Google Scholar]
- Commission European (2006) Special Eurobarometer 248: Mental Well‐being, Luxembourg, European Commission. [Google Scholar]
- Fabricant S.J., Harpham T. (1993) Assessing response reliability of health interview surveys using reinterviews. Bulletin of the World Health Organization, 71(3–4), 341–348. [PMC free article] [PubMed] [Google Scholar]
- Glandon G.L., Counte M.A., Tancredi D. (1992) An analysis of physician utilization by elderly persons: Systematic differences between self‐report and archival information. Journal of Gerontology, 47(5), S245–S252. [DOI] [PubMed] [Google Scholar]
- Goldberg R.W., Seybolt D.C., Lehman A. (2002) Reliable self‐report of health service use by individuals with serious mental illness. Psychiatric Services, 53(7), 879–881. [DOI] [PubMed] [Google Scholar]
- Green S., Kaufert J., Corkhill R., Creese A., Dunt D. (1979) The collection of service utilisation data: A research note on validity. Social Science & Medicine, 13A(2), 231–234. [PubMed] [Google Scholar]
- Haro J.M., Arbabzadeh‐Bouchez S., Brugha T.S., de Girolamo G., Guyer M.E., Jin R., Lepine J.P., Mazzi F., Reneses B., Vilagut G., Sampson N.A., Kessler R.C. (2006) Concordance of the Composite International Diagnostic Interview version 3.0 (CIDI 3.0) with standardized clinical assessments in the WHO World Mental Health surveys. International Journal of Methods in Psychiatric Research, 15(4), 167–180. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Jobe J.B., White A.A., Kelley C.L., Mingay D.J., Sanchez M.J., Loftus E.F. (1990) Recall strategies and memory for health‐care visits. Milbank Quarterly, 68(2), 171–189. [PubMed] [Google Scholar]
- Jorm A.F. (2000) Mental health literacy. Public knowledge and beliefs about mental disorders. British Journal of Psychiatry, 177(5), 396–401. [DOI] [PubMed] [Google Scholar]
- Jorm A.F., Christensen H., Griffiths K.M. (2006) The public's ability to recognize mental disorders and their beliefs about treatment: Changes in Australia over 8 years. Australian and New Zealand Journal of Psychiatry, 40(1), 36–41. [DOI] [PubMed] [Google Scholar]
- Kashner T.M., Suppes T., Rush A.J., Altshuler K.Z. (1999) Measuring use of outpatient care among mentally ill individuals: A comparison of self reports and provider records. Evaluation and Program Planning, 22(1), 31–39. [Google Scholar]
- Kessler R.C., Ustun T.B. (2004) The World Mental Health (WMH) survey initiative version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI). International Journal of Methods in Psychiatric Research, 13(2), 93–121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Killeen T.K., Brady K.T., Gold P.B., Tyson C., Simpson K.N. (2004) Comparison of self‐report versus agency records of service utilization in a community sample of individuals with alcohol use disorders. Drug and Alcohol Dependence, 73(2), 141–147. [DOI] [PubMed] [Google Scholar]
- Kovess‐Masfety V., Alonso J., Brugha T.S., Angermeyer M.C., Haro J.M., Sevilla‐Dedieu C, the ESEMeD/MHEDEA 2000 Investigators . (2007) Differences in lifetime use of services for mental health problems in six European countries. Psychiatric Services, 58(2), 213–220. [DOI] [PubMed] [Google Scholar]
- Kovess‐Masfety V., Dezetter A., de Graaf R., Haro J.M., Bruffaerts R., Briffault X., Gilbert F., Codony M., Alonso J, the ESEMeD/MHEDEA 2000 Investigators . (2010) Religious advisors' role in mental health care in the European Study of the Epidemiology of Mental Disorders survey. Social Psychiatry and Psychiatric Epidemiology, 45(10), 989–998. [DOI] [PubMed] [Google Scholar]
- Landis J.R., Koch, G.G. (1977) The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. [PubMed] [Google Scholar]
- Lubeck D.P., Hubert H.B. (2005) Self‐report was a viable method for obtaining health care utilization data in community‐dwelling seniors. Journal of Clinical Epidemiology, 58(3), 286–290. [DOI] [PubMed] [Google Scholar]
- Mirandola M., Bisoffi G., Bonizzato P., Amaddeo F. (1999) Collecting psychiatric resources utilisation data to calculate costs of care: A comparison between a service receipt interview and a case register. Social Psychiatry and Psychiatric Epidemiology, 34(10), 541–547. [DOI] [PubMed] [Google Scholar]
- Petrou S., Murray L., Cooper P., Davidson L.L. (2002) The accuracy of self‐reported healthcare resource utilization in health economic studies. International Journal of Technology Assessment in Health Care, 18(3), 705–710. [DOI] [PubMed] [Google Scholar]
- Pollio D.E., North C.S., Eyrich K.M., Foster D.A., Spitznagel E.L. (2006) A comparison of agency‐based and self‐report methods of measuring services across an urban environment by a drug‐abusing homeless population. International Journal of Methods in Psychiatric Research, 15(1), 46–56. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Raina P., Torrance‐Rynard V., Wong M., Woodward C. (2002) Agreement between self‐reported and routinely collected health‐care utilization data among seniors. Health Services Research, 37(3), 751–774. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Reijneveld S.A. (2000) The cross‐cultural validity of self‐reported use of health care: A comparison of survey and registration data. Journal of Clinical Epidemiology, 53(3), 267–272. [DOI] [PubMed] [Google Scholar]
- Reijneveld S.A., Stronks K. (1999) The impact of response bias on estimates of health care utilization in a metropolitan area: The use of administrative data. International Journal of Epidemiology, 28(6), 1134–1140. [DOI] [PubMed] [Google Scholar]
- Reijneveld S.A., Stronks K. (2001) The validity of self‐reported use of health care across socioeconomic strata: A comparison of survey and registration data. International Journal of Epidemiology, 30(6), 1407–1414. [DOI] [PubMed] [Google Scholar]
- Ritter P.L., Stewart A.L., Kaymaz H., Sobel D.S., Block D.A., Lorig K.R. (2001) Self‐reports of health care utilization compared to provider records. Journal of Clinical Epidemiology, 54(2), 136–141. [DOI] [PubMed] [Google Scholar]
- Roberts R.O., Bergstralh E.J., Schmidt L., Jacobsen S.J. (1996) Comparison of self‐reported and medical record health care utilization measures. Journal of Clinical Epidemiology, 49(9), 989–995. [DOI] [PubMed] [Google Scholar]
- Rozario P.A., Morrow‐Howell N., Proctor E. (2004) Comparing the congruency of self‐report and provider records of depressed elders' service use by provider type. Medical Care, 42(10), 952–959. [DOI] [PubMed] [Google Scholar]
- Santelli J., Klein J., Graff C., Allan M., Elster A. (2002) Reliability in adolescent reporting of clinician counseling, health care use, and health behaviors. Medical Care, 40(1), 26–37. [DOI] [PubMed] [Google Scholar]
- Schwarz N. (1999) Self‐reports: How the questions shape the answers. The American Psychologist, 48(2), 93–105. [Google Scholar]
- Sevilla‐Dedieu C., Kovess‐Masféty V., Haro J.M., Fernandez A., Vilagut G., Alonso J., ESEMeD‐Mental Health Disability: A European Assessment in Year 2000 Investigators (2010) Seeking help for mental health problems outside the conventional health care system: Results from the European Study of the Epidemiology of Mental Disorders (ESEMeD). Canadian Journal of Psychiatry, 55(9), 586–597. [DOI] [PMC free article] [PubMed] [Google Scholar]
- The World Mental Health Survey Initiative. http://www.hcp.med.harvard.edu/wmh/ [Accessed 1 September 2009]., authors
- Ungar W.J., Coyte P.C., the Pharmacy Medication Monitoring Program Advisory Board (1998) Health services utilization reporting in respiratory patients. Journal of Clinical Epidemiology, 51(12), 1335–1342. [DOI] [PubMed] [Google Scholar]
- Wallihan D.B., Stump T.E., Callahan C.M. (1999) Accuracy of self‐reported health services use and patterns of care among urban older adults. Medical Care, 37(7), 662–670. [DOI] [PubMed] [Google Scholar]
- Wang P.S., Aguilar‐Gaxiola S., Alonso J., Angermeyer M.C., Borges G., Bromet E.J., Bruffaerts R., de Girolamo G., de Graaf R., Gureje O., Haro J.M., Karam E.G., Kessler R.C., Kovess V., Lane M.C., Lee S., Levinson D., Ono Y., Petukhova M., Posada‐Villa J., Seedat S., Wells J.E. (2007) Use of mental health services for anxiety, mood, and substance disorders in 17 countries in the WHO world mental health surveys. Lancet, 370(9590), 841–850. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wang P.S., Lane M., Olfson M., Pincus H.A., Wells K.B., Kessler R.C. (2005) Twelve‐month use of mental health services in the United States: Results from the National Comorbidity Survey Replication. Archives of General Psychiatry, 62(6), 629–640. [DOI] [PubMed] [Google Scholar]