Abstract
Background
Patient self-report allows collecting comprehensive data for the purpose of performing economic evaluations. The aim of the current study was to assess the feasibility, reliability and a part of the construct validity of a commonly applied questionnaire on healthcare utilization and productivity losses in patients with a psychiatric disorder (TiC-P).
Methods
Data were derived alongside two clinical trials performed in the Netherlands in patients with mental health problems. The response rate, average time of filling out the questionnaire and proportions of missing values were used as indicators of feasibility of the questionnaire. Test-retest analyses were performed including Cohen’s kappa and intra class correlation coefficients to assess reliability of the data. The construct validity was assessed by comparing patient reported data on contacts with psychotherapists and reported data on long-term absence from work with data derived from registries.
Results
The response rate was 72%. The mean time needed for filling out the first TiC-P was 9.4 minutes. The time needed for filling out the questionnaire was 2.3 minutes less for follow up measurements. Proportions of missing values were limited (< 2.4%) except for medication for which in 10% of the cases costs could not be calculated. Cohen’s kappa was satisfactory to almost perfect for most items related to healthcare consumption and satisfactory for items on absence from work and presenteeism. Comparable results were shown by the ICCs on variables measuring volumes of medical consumption and productivity losses indicating good reliability of the questionnaire.
Absolute agreement between patient-reported data and data derived from medical registrations of the psychotherapists was satisfactory. Accepting a margin of +/− seven days, the agreement on reported and registered data on long-term absence from work was satisfactory. The validity of self-reported data using the TiC-P is promising.
Conclusions
The results indicate that the TiC-P is a feasible and reliable instrument for collecting data on medical consumption and productivity losses in patients with mild to moderate mental health problems. Additionally, the construct validity of questions related to contacts with psychotherapist and long-term absence from work was satisfactory.
Keywords: Patient-reported questionnaire, Healthcare utilization, Productivity losses, Reliability, Validity
Background
Economic evaluations examine both costs and benefits of alternative healthcare interventions and are increasingly used to inform decision-makers for reimbursement in healthcare. Due to the international interest in economic evaluations, transparency and generalizability of the results are important issues. Additionally, the validity of evaluations also depends on the method of collecting data.
Several arguments support the adoption of a societal perspective in performing economic evaluations [1]. A number of national health economic guidelines recommend performing these evaluations from a societal perspective, including all costs, regardless of who bears the costs and who receives the benefits [2,3]. So, besides the direct medical costs also direct and indirect non-medical costs should be included. Direct costs outside the healthcare sector are directly related to the intervention, but generally do not incur within the formal healthcare system, such as travelling costs and time costs of patients and their informal caregivers. Indirect costs outside the healthcare sector are costs incurred outside the scope of the formal healthcare system arising as a secondary effect of the intervention, such as productivity costs due to absence from work or reduced efficiency during paid or unpaid work [4,5].
Quantities of resources are often collected alongside clinical studies by means of the traditional ‘case report form’ (CRF). The advantage of clinical studies is that the quantities of resources and effectiveness data can be collected simultaneously. However, the data collection is restricted to direct medical costs of the intervention. Thus, other medical cost that may be related to patients’ health and direct and indirect non-medical costs need to be collected from alternative data sources. Administrative databases of healthcare institutions, such as hospitals, other healthcare providers, occupational health institutions, and insurers, are considered to contain the most accurate patient-level data. However, an important limitation of these databases is that they commonly serve other purposes than economic evaluations and the accessibility of these databases may be limited [6]. Furthermore, data needs to be collected from multiple sources which necessitate the cooperation of different institutions and patients’ informed consent at each of these institutions [7,8]. In addition, registrations may be incomplete. For example, databases of health insurers are restricted to the healthcare interventions that are covered by the health insurance system and registrations of occupational health institutions only comprise data on sick leave. More importantly, routine data supporting the cost calculation of informal care and reduced efficiency during paid work are commonly not recorded.
In economic evaluations, data on resource use may also be collected directly from the patient by means of patient self-reports (e.g. diaries, interviews or questionnaires). The reliability of the collected data, however, may be compromised by the degree to which patients recall quantities of resources consumed [9,10]. Furthermore, the validity of the collected data may be restricted by lack of standardized instruments which jeopardizes the meaningful comparison of economic evaluations [7].
The questionnaire on healthcare consumption and productivity losses for patients with a Psychiatric disorder (TiC-P) is a comprehensive questionnaire focused on establishing direct medical costs and productivity costs due to absence from work or reduced efficiency during paid or unpaid work and is widely used in the Netherlands for economic evaluations in mental health [11-14]. In this paper, the performance of the TiC-P is evaluated by addressing the following issues: i) testing the feasibility (practicality) of the questionnaire; ii) assessing the reliability of the questionnaire; and iii) assessing the construct validity of the TiC-P.
Methods
TiC-P
The TiC-P is questionnaire designed for self-report in patients with a mental disorder. A translation in English of the questionnaire is available at http://www.imta.nl[15]. The TiC-P is a generic questionnaire, meaning that the items are not related to a target disease. Distinguishing between healthcare consumption and production losses as a consequence of the target disease and comorbidity is difficult, especially in psychiatric disorders, as patients also may have physical symptoms that are connected to the psychiatric illness. Moreover, psychiatric comorbidity is a common occurrence in psychiatric illness.
Before the introduction of the TiC-P the questionnaires’ feasibility in daily practice was assessed by interviewing 20 respondents with a psychiatric disorder who were treated in a specialisedcentre for psychiatry. This resulted in a number of textual changes [16].
The TiC-P consists of two parts, both can also be used separately. Additionally, a number of general questions may be added for collecting data on respondents’ demographic characteristics and co-morbidity.
The first part of the TiC-P includes 14 structured no/yes questions on relevant medical resource items each followed by a question on the volume of medical consumption (see Figure 1).
Figure 1.
Example of an item on medical resource use.
The questions include contacts within the mental healthcare sector (regional mental healthcare organisation, psychiatrist/psychologist or psychotherapist in private practices or outpatient hospital, institutional day-care treatment, Consultation Agency for Alcohol and Drug addiction (CAD), self-help group and contacts with general healthcare providers (general practitioner, paramedical and social worker, alternative practices, outpatients visits to medical specialists, hospital admission and contacts with an occupational practitioner) and the use of medication. The inclusion of questions related to different types of contacts in mental healthcare makes the TiC-P suitable for broad application, i.e. for various psychiatric disorders. Additionally, depending on the relevance for the target population the questionnaire allows adding or leaving out specific items of resource utilisation.
Part two consists of the Short Form-Health and Labour Questionnaire (SF-HLQ), a generic instrument to collect data on productivity losses due to health problems and is based on the HLQ [17]. The SF-HLQ aims to measure absence from work and reduced efficiency of paid and unpaid work. Absence from work is measured by two questions related to short-term absence and long-term absence (< 2 weeks and > 2 weeks respectively ) from work, see Figure 2.
Figure 2.
Questions related to absence from work.
Additionally, three questions are included for measuring productivity losses due to reduced efficiency during paid work (see Figure 3).
Figure 3.
Questions related to reduced efficiency during work.
Study design and subjects
The performance of the TiC-P was assessed using data from two unrelated patient groups with mental disorders in two different studies. Firstly, the study of the TiC-P’s performance was accommodated as part of a Dutch cost-utility study performed alongside a randomized clinical trial evaluating alternative feedback mechanisms during psychotherapy in patients treated in ambulant mental healthcare treatments settings private practice or a regional mental healthcare institution. We refer to this study as the Monitoring Study. The study was approved by the Medical Ethical Committee of the Erasmus University Medical Centre (NL11529.078.06). The study started in 2006 and included a total of 891 patients starting psychotherapy. Inclusion of new patients ended in January 2011 [18]. The TiC-P was used to collect data on medical resource use and productivity losses. The TiC-P was filled out at baseline and monthly during the first 3 months of the study, and every 3 months thereafter. The follow-up ended at treatment completion. In this study a recall period of 4 weeks was applied for collecting data on medical consumption and 2 weeks for collecting data on productivity losses. Generally, the questionnaire was filled out online by the patient at the healthcare setting where psychotherapy was provided.
For practical reasons, data from a second study were obtained to assess the construct validity of long term absence from work data of the TiC-P. This study is referred to as the Collaborative Care Study. The study was approved by the Medical Ethical Committee and the EMGO scientific committee of the VU University Medical Centre (ISRCTN 78462860). In this study patients with a major depressive disorder were randomized to usual care or collaborative care in an occupational healthcare setting [19]. Patients were recruited from workers who were on a sick list for between 4 and 12 weeks. Patient’ characteristics of both studies are presented in Table 1.
Table 1.
Characteristics of the study populations
| Monitoring study (n=631) | Collaborative care study (n=126) | |
|---|---|---|
| Age (mean) |
37.7 |
42.6 |
| Female (%) |
66.8 |
54.0. |
| Education (%) | ||
| Low |
6.9 |
31.4 |
| Secondary |
43.3 |
33.1 |
| High |
49.9 |
35.5 |
| Diagnoses (%) | ||
| Mood disorder |
27 |
100* |
| Adjustment disorder |
21 |
|
| Anxiety disorder |
14 |
|
| Relational problems |
14 |
|
| Other1 | 24 | |
* major depressive disorder; 1 Other disorders included: disorders usually first diagnosed in infancy, childhood or adolescence, impulse control disorders, eating disorders, dissociative disorders, sexual disorders, substance-related disorders and psychotic disorders (in order of frequency).
Analyses
Feasibility and reliability were evaluated using data from the Monitoring study. Feasibility included the TiC-P response rate, respondent’s time of filling out the TiC-P and data completeness on the items of the questionnaire. Completeness is reported as proportions of missing values. Additionally, completeness on reported medication was evaluated including the medication name, the dose per intake, the number of doses per day, and the number of days that the medication was taken during the previous 4 weeks. For this evaluation a random sample (n=283) was taken from the questionnaires submitted by patients who actually reported use of medication. Completeness on medication name was checked manually and was defined missing if no name was reported or in cases that the medication was described in general terms (f.e. sleeping pill, antibiotic). Respondents’ time for filling out the TiC-P was assessed as follows. Patients in the Monitoring study filled out online a number of different questionnaires at the same time. Consequently, information was available of the time of filling out for successive submitted questionnaires. The average time for filling out the TiC-P was calculated by subtracting the times of these questionnaires according to the web-based dataset. Identically, we estimated the mean time for filling out the TiC-P of the first measurement and for the next measurements. Differences in time for filling out the TiC-P at baseline and during follow-up measurements were evaluated using a paired sample t-test.
Reliability was assessed using a test-retest design. Test-retest reliability analyses were performed to evaluate consistency of the data reported. For these analyses a subsample was invited to fill out the TiC-P again (retest) two weeks after submission of the original measurement. A cover letter explained the purpose of the retest and we offered a gift voucher of €10 if the retest questionnaire would be returned. Consistency of categorical (yes/no) variables was assessed with percentages of absolute agreements indicating the proportion of cases with the same value on the test and retest questionnaire. To adjust for the fact that a number of these agreements may arise by chance alone, chance-corrected agreements were assessed using Cohen’s kappa coefficients (κ values). The following values were attached to the coefficients: modest (0.21-0.40); moderate (0.41-0.60); satisfactory (0.61-0.80) and almost perfect (0.81-1.00) [20]. Consistency of data on interval level was evaluated by computing intra class correlation coefficients (ICC) (two-way mixed models; absolute agreement).
The construct validity of the TiC-P was evaluated by assessing the agreement with reported and registered data for the items ‘contacts with a psychotherapist’ and ‘long-term absence from work’ including the percentage absolute agreement, absolute differences between reported data en registered data and Spearman rank correlation coefficient (rho). Reported data on contacts with therapists were compared with registration data of the therapists. Additionally, reported data on long-term absence from work was compared to registration data from the occupational health service. As registration data on absence from work of the Monitoring study were not accessible, the latter were derived from the Collaborative Care Study [19]. All statistical analyses were performed in SPSS (V. 17.0; Chicago, IL). Significance was set at a p-value of 0.050.
Results
Feasibility
The gross response rate of the TiC-P was 72% meaning nearly three quarters of the respondents submitted at least one TiC-P questionnaire during the study.
Overall, the time for filling out was on average 7.8 minutes (SD 5.1). The average time measured for filling out the first TiC-P was 9.4 minutes (SD 5.5) and significantly less for questionnaires that were filled out during follow-up measurements (7.1 minutes; SD 4.7; p < 0.001). Overall, proportions of missing values related to the quantification of medical resource use and productivity losses were small. The most frequently missing information concerned the number of hospital outpatient contacts (2.3%).
To calculate the costs of medications, 4 items are required: i.e. the name of the medication, dose per intake, the daily dose and the number of days that the medication was used. In 29% of cases, at least one of these items was missing. Table 2 presents the share of missing values per item. Most frequently missing information concerned the name of the medication and the dose per intake. For specified medications, substituting missing data concerning the dose of intake with daily defined doses (DDD) decreased the number of cases of missing cost data for this item to 10.2%.
Table 2.
Completeness of information on medication (n=283)
| Missing (%) | |
|---|---|
| Specification of medication |
12.4 |
| Dose per intake |
20.2 |
| Number of intake per day |
2.8 |
| Number of days of medication intake |
|
| during the preceding 4 weeks | 4.6 |
Reliability
A total of 111 respondents filled out the retest questionnaire (response rate retest 61.8%). Of these, 99 retest questionnaires were eligible to assess consistency of the data reported. The period between test and retest questionnaire of the remainder 12 respondents was more than one month and was considered too large for assessing the reliability. The retest was submitted on average 17 days (SD 7.7; range 1–28) after submission of the initial questionnaire.
Absolute agreement on medical resource use (yes/no questions) ranged from 82% to 99%. Agreement according to Cohen’s kappa of items related to medical consumption was as follows: almost perfect n=2; satisfactory n=6; moderate n=4. Generally, ICCs of test-retest measurements on the number of contacts with the healthcare providers showed good agreement. Non-significant agreement was found on measurements related to the number of contacts with a social worker, medical specialist and alternative medicine practice. Due to the small number of patients reporting on the number of contacts with specific healthcare provider ICCs of 5 items (e.g. the duration of hospital admission, number of hospital day-care treatment, contacts with a self help group and CAD, and outpatient visits to mental healthcare providers) could not be calculated (see Table 3).
Table 3.
Reliability: consistency of data on medical resource use (n=99)
| Absolute agreement (%) | Cohen’s Kappa | Use of service (%) | ICC (95% CI) | |
|---|---|---|---|---|
| General practitioner |
81.8 |
0.597 |
25.3 |
0.804 |
| (0.608-0.908) | ||||
| Therapist in regional mental healthcare |
91.9 |
0.649 |
12.1 |
0.769 |
| (0.341-0.929) | ||||
| Mental healthcare therapist with private practice |
88.9 |
0.618 |
75.8 |
0.740 |
| (0.613-0.829) | ||||
| Mental healthcare therapist hospital outpatient |
0.0 |
- |
1.0 |
* |
| Occupational physician |
88.9 |
0.600 |
12.1 |
0.884 |
| (0.652-0.965) | ||||
| Hospital outpatient visit to a medical specialist |
96.0 |
0.645 |
4.0 |
0.000 |
| (−0.878-0.878) | ||||
| Paramedical healthcare provider |
93.9 |
0.805 |
16.2 |
0.825 |
| (0.569-0.935) | ||||
| Social worker |
97.0 |
0.711 |
4.0 |
0.571 |
| (−0.475-0.965) | ||||
| Alternative medicine practice |
94.9 |
0.519 |
6.1 |
0.231 |
| (−0.164-0.775) | ||||
| Self help group |
98.0 |
0.492 |
1.0 |
* |
| Consultation Agency Alcohol & Drugs (CAD) |
100.0 |
- |
1.0 |
* |
| Hospital day care treatment |
99.0 |
0.795 |
2.0 |
* |
| Hospital admission duration? |
99.0 |
0.662 |
2.0 |
* |
| Use of medication | 91.9 | 0.839 | 49.5 | 0.684 |
| (0.499-0.810) |
* Too few cases to analyze; - Not applicable.
Agreement between test and retest on nominal variables related to productivity losses was satisfactory. Almost perfect consistency was found on the reported number of sick leave days during the preceding 2 weeks (ICC 0.83; CI 0.55-0.94). Consistency on the number of days at work while impeded between test and retest measurement was moderate. However, consistency of the efficiency rates at work was satisfactory (see Table 4).
Table 4.
Reliability: consistency of data on productivity losses (paid work: n=79)
| Absolute agreement (%) | Cohen’s Kappa | Yes (%) | ICC (95% CI) | |
|---|---|---|---|---|
| Were you on sick leave? |
87.3 |
0.654 |
25.3 |
- |
| Number of days off work during last 2 weeks |
|
|
25.3 |
0.825 |
| (0.553-0.940) | ||||
| Sick leave episode > 2 weeks? |
93.7 |
0.762 |
11.1 |
- |
| Were you impeded during paid work? |
81.3 |
0.646 |
- |
- |
| Number of days at work while impeded |
- |
- |
27.3 |
0.556 |
| (0.224-0.771) | ||||
| Efficiency rate* (at work while impeded) | - | - | 27.3 | 0.729 |
| (0.492-0.866) |
- Not applicable; * Rate ranging from 1 to 10 indicating much to no influence of health problems on work performance.
Construct validity
The construct validity of part one of the TiC-P was evaluated by comparing the patient reported number of contacts with psychotherapist of 114 responders (corresponding to a total of 365 measurements) with the registration of the psychotherapists. The reported number of contacts in the preceding 4 weeks was 1.9 (SD 1.5) ranging from 0 to 8 and was highly correlated with the number of contacts according to the registration data (rho= 0.791). Absolute agreement between the number of reported and registered contacts was 76.7%. The difference between reported and registered data was on average 0.01 (SD 0.97). Occupational registry data and reported data of absence from work data were available of 117 respondents derived from the Collaborative Care Study. Absence from work was on average 274 (calendar) days (SD 167). Absolute agreement on reported and registered data was 54.2%. Accepting a margin of +/− 7 days as acceptable between the reported and registered date of onset of absence from work resulted in an agreement of 70.9%. The difference between reported and registered date of absence from work was on average 5.7 days (SD 20.6). The correlation between reported and registered days of absence was 0.663.
Discussion
In this study the feasibility and reliability of the TiC-P were evaluated. Additionally, the construct validity of two items of the TiC-P was assessed. The small number of missing data indicated that the questionnaire was generally well understood. The response rate was at an acceptable level (72%). The average completion time of the questionnaire (on average below 10 minutes) was acceptable. The test-retest analyses showed good agreement on items of medical resource use that were frequently reported by the respondents. Absolute agreement between reported data on contacts with psychotherapists and long-term absence from work and registration data was around 70-75%. Additionally, the construct validity of the items on contacts with psychotherapists and long-term absence from work was satisfactory.
Thus, the TiC-P, including the SF-HLQ, seems a feasible and reliable instrument for measuring healthcare utilization and productivity loss. Additionally, the findings regarding the construct validity of the items ‘contacts with psychotherapist’ and ‘long term absence from work’ are promising.
Qualitative feedback on the questions was evaluated on the draft version of the TiC-P. For the current study, feasibility was operationalized as the response rate, the time for filling out the questionnaire and data completeness. Generally, patients in the Monitoring study filled out the questionnaire at the healthcare setting. This may have positively influenced the response rate and data completeness of the questionnaire in comparison to filling out the TiC-P at home. Proportions of missing values on the items were small with exception of missing values related to the measurement of the use of medication. The findings on incompleteness of self-reported data on medication are in line with findings in another study among elderly patients [21]. Cost calculation of medication requires relatively comprehensive information of the respondents. It may be feasible to reduce the number of questions on medication using daily defined dose for the calculation of the costs. If costs of medication are expected to contribute substantially we recommend considering alternative sources for collecting these data, e.g. patient records.
The relatively easy way of filling out questionnaires online may present an underestimation of the time needed for filling out the paper version of the questionnaire. Also, the relatively young population participating in the Monitoring study may underestimate the time to fill out the TiC-P in general. Additionally, in the online version respondents were pointed out to the missing quantification in case of reporting ‘yes’ on the items. Despite this, respondents were able to ignore the automatic signal and continue the questionnaire. However, this might have decreased the number of missing values.
Consistency between test and retest measurement was relatively high. Generally, the reliability of test retest analyses require similar circumstances of the successive measurements and an appropriate time interval between the measurements moments. For this study we chose to send the retest questionnaire after t=1 (the second measurement) since we assumed that changes in medical consumption and productivity losses are limited at the start of therapy. A number of 180 respondents were invited for filling out a retest questionnaire. The response rate was relatively low. Additionally, 10% of the retest questionnaires were filled out after a relatively long period. Due to the relatively small sample of the retest, the generalizability and the interpretation of the results warrant some caution. Despite of this, the ICCs related to contacts with healthcare providers that were contacted most frequently (i.e. > 12% of the respondents) seem satisfactory and have relatively small confidence intervals. A number of healthcare providers who were contacted less frequently (e.g. contacts reported by < 12% of the respondents) had lower agreement scores. Consequently, whether the frequencies of these contacts are relatively constant over time may be argued. More research is necessary to further assess the reliability of these items.
The reliability of short-term absence from work was satisfactory. Test and retest figures related to the number of days at work while impeded by health problems were moderate. It can be assumed that it is more complicated to remember the exact number of days at work with impediment. However, alternative methods for measuring reduced efficiency are currently not available. More research on this topic is warranted.
Agreement between reported and registered data on the number of contacts with psychotherapists was satisfactory, indicating an acceptable construct validity of this item. This finding is in line with other studies indicating that self-reports provide accurate data for more important and for less frequent events [9,22]. It was not possible to study resource utilization other than contacts with psychotherapists who participated in the clinical study. Consequently, further research is necessary for assessing the construct validity of measuring the other items of health care utilization. Agreement between the number of days of absence from work based on data derived from the occupational health service and self-report of the patient was satisfactory. Our results are in line with previous studies of patient-reported absence from work [23-25]. A limitation of our study is that we were only able to compare reported data with registered data on long-term absence from work. However, Severens et al. [23] found that 95% of patient reported data matched the registered data on absence from work perfectly applying a recall period of 2 and 4 weeks.
Another limitation is that our study was performed in patients with mental disorders treated in ambulant settings e.g. among patients with less severe mental disorders. We expect that the findings in our study can be generalized to other groups of patients. However, future research on this is desired. Currently, the default version of the TiC-P uses a recall period of three months for measuring medical consumption and one month for measuring productivity losses is applied. This interval is in line with commonly applied measurement intervals in clinical studies. In the Monitoring study, a recall period of four and two weeks was applied respectively. This may limit the generalizability of the results of this study for the current version of the TiC-P. Further research should indicate the impact of a longer recall period on the different items of the TiC-P.
Finally, the current version of the TiC-P is translated by a professional language institution into a patient-friendly version using more simple language. We assume that this will enhance the feasibility and validity of the TiC-P.
Conclusions
In economic evaluations, data on resource use is frequently collected directly from the patient by means of patient self-reports. Therefore, the reliability and validity of the thus collected data are important for the validity of the evaluation. Additionally, the application of standardized instruments allows a meaningful comparison of economic evaluations.
The results of our study indicate that the TiC-P is a fairly feasible and reliable alternative for collecting data on resource use data in comparison to collecting data from registries. Additionally, the results on the validity of the reported data related to contacts with psychotherapists and long term absence from work for the validity of the TiC-P are promising.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
RT and LH were project leaders of the Monitoring Study and contributed to the study design and monitoring of the study. CFC contributed to the study design and monitoring of the Collaborative Care Study. KdJ, MV and SST were involved in the collection of data. CB wrote the manuscript and performed the analyses. All authors contributed to the interpretation of the analyses and commented on the manuscript. All authors read and approved the final manuscript.
Pre-publication history
The pre-publication history for this paper can be accessed here:
Contributor Information
Clazien Bouwmans, Email: bouwmans@bmg.eur.nl.
Kim De Jong, Email: k.dejong.3@erasmusmc.nl.
Reinier Timman, Email: r.timman@erasmusmc.nl.
Moniek Zijlstra-Vlasveld, Email: MVlasveld@trimbos.nl.
Christina Van der Feltz-Cornelis, Email: c.m.vdrfeltz@tilburguniversity.edu.
Siok Swan Tan, Email: tan@bmg.eur.nl.
Leona Hakkaart-van Roijen, Email: hakkaart@bmg.eur.nl.
Acknowledgements
The study was supported by a grant (no. 945064149) of the Council of Health and Health Service Research from the Netherlands (ZonMW). The funding organisation had no role in the conceptualization of the study, the collection and analyses of the data, or in the reporting of the results. The authors gratefully thank J. Alberts for his practical work on the data collection and K. Kooiman (RIAGG Rijnmond), H. Morres, G. Bakker- van Zeil, M. Verheul, J. Hoekstra and P. Hoffman for putting contact data from their registration of patients included in the Monitoring study at the authors’ disposal.
The data that were used for assessing construct validity of absence from work was obtained from a study initiated by the Depression Initiative, a National Initiative aimed at improved depression management in the Netherlands, funded by the InnovatiefondsZorgverzekeraars.
References
- Jonsson B. Ten arguments for a societal perspective in the economic evaluation of medical innovations. Eur J Health Econ. 2009;10(4):357–359. doi: 10.1007/s10198-009-0173-2. [DOI] [PubMed] [Google Scholar]
- Knies S, Severens JL, Ament AJ, Evers SM. The transferability of valuing lost productivity across jurisdictions. Differences between national pharmacoeconomic guidelines. Value Health. 2010;13(5):519–527. doi: 10.1111/j.1524-4733.2010.00699.x. [DOI] [PubMed] [Google Scholar]
- Hjelmgren J, Berggren F, Andersson F. Health economic guidelines–similarities, differences and some implications. Value Health. 2001;4(3):225–250. doi: 10.1046/j.1524-4733.2001.43040.x. [DOI] [PubMed] [Google Scholar]
- Tan SS, Bouwmans CAM, Rutten FF, Hakkaart-van Roijen L. Update of the Dutch Manual for Costing in Economic Evaluations. Int J Technol Assess Health Care. 2012;28(2):152–158. doi: 10.1017/S0266462312000062. [DOI] [PubMed] [Google Scholar]
- Drummond MF, Sculpher MJ, Torrence GW, O’Brien BJ, Stoddart GL. Methods for the Economic Evaluation of health Care Programmes: third ed. Oxford: Oxford University Press; 2005. [Google Scholar]
- Riley GF. Administrative and claims records as sources of health care cost data. Med Care. 2009;47(7 Suppl 1):S51–S55. doi: 10.1097/MLR.0b013e31819c95aa. [DOI] [PubMed] [Google Scholar]
- Sevilla-Dedieu C, Kovess-Masfety V, Angermeyer M, Bruffaerts R, Fernandez A, Girolamo GD, De Graaf R, Haro JM, Konig HH. The ESEMED/MHEDEA 2000 Investigators. Measuring use of services for mental health problems in epidemiological surveys. Int J Methods Psychiatr Res. 2011;20(3):182–191. doi: 10.1002/mpr.346. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wang PS, Aguilar-Gaxiola S, Alonso J, Angermeyer MC, Borges G, Bromet EJ, Bruffaerts R, De Girolamo G, De Graaf R, Gureje O, Haro JM, Karam EG, Kessler RC, Kovess V, Lane MC, Lee S, Levinson D, Ono Y, Petukhova M, Posada-Villa J, Seedat S, Wells JE. Use of mental health services for anxiety, mood, and substance disorders in 17 countries in the WHO world mental health surveys. Lancet. 2007;370(9590):841–850. doi: 10.1016/S0140-6736(07)61414-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Van den Brink M, Van den Hout WB, Stiggelbout AM, Putter H, Van de Velde CJ, Kievit J. Self-reports of health-care utilization: diary or questionnaire? Int J Technol Assess Health Care. 2005;21(3):298–304. doi: 10.1017/s0266462305050397. [DOI] [PubMed] [Google Scholar]
- Goossens ME, Rutten-van Molken MP, Vlaeyen JW, Van der Linden SM. The cost diary: a method to measure direct and indirect costs in cost-effectiveness research. J Clin Epidemiol. 2000;53(7):688–695. doi: 10.1016/S0895-4356(99)00177-8. [DOI] [PubMed] [Google Scholar]
- Hakkaart-van Roijen L, Van Straten A, Al M, Rutten F, Donker M. Cost-utility of brief psychological treatment for depression and anxiety. Br J Psychiatry. 2006;188:323–329. doi: 10.1192/bjp.188.4.323. [DOI] [PubMed] [Google Scholar]
- Polak AR, Witteveen AB, Visser RS, Opmeer BC, Vulink N, Figee M, Denys D, Olff M. Comparison of the effectiveness of trauma-focused cognitive behavioral therapy and paroxetine treatment in PTSD patients: design of a randomized controlled trial. BMC Psychiatry. 2012;12:166. doi: 10.1186/1471-244X-12-166. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Berghout CC, Zevalkink J, Hakkaart-Van Roijen L. The effects of long-term psychoanalytic treatment on healthcare utilization and work impairment and their associated costs. J Psychiatr Pract. 2010;16(4):209–216. doi: 10.1097/01.pra.0000386907.99536.75. [DOI] [PubMed] [Google Scholar]
- Horn EK, Van Benthem TB, Hakkaart-van Roijen L, Van Marwijk HW, Beekman AT, Rutten FF, Van der Feltz-Cornelis CM. Cost-effectiveness of collaborative care for chronically ill patients with comorbid depressive disorder in the general hospital setting, a randomised controlled trial. BMC Health Serv Res. 2007;7:28. doi: 10.1186/1472-6963-7-28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Institute for Medical Technology Assessment. Questionnaire Healthcare consumption, illness and work (TIC-P) http://www.bmg.eur.nl/english/imta/publications/questionnaires_manuals.
- Hakkaart-Van Roijen L. Manual. Trimbos/iMTA questionnaire for Costs associated with Psychiatric Illness (TiC-P adults) 2010. http://www.bmg.eur.nl/english/imta/publications/questionnaires_manuals.
- Roijen VL, Essink-Bot M, Koopmanschap M, Bonsel G, Rutten F. Labor and health status in economic evaluation of health care. The Health and Labor Questionnaire. Int J Technol Assess Health Care. 1996;12(3):405–415. doi: 10.1017/S0266462300009764. [DOI] [PubMed] [Google Scholar]
- De Jong K. A chance for change: building an outcome monitoring feedback system for outpatient mental health care. PhD thesis.Leiden University, Department of Clinical Health and Neuropsychology; 2012. [Google Scholar]
- Vlasveld MC, Van der Feltz-Cornelis CM, Ader HJ, Anema JR, Hoedeman R, Van Mechelen W, Beekman AT. Collaborative care for major depressive disorder in an occupational healthcare setting. Br J Psychiatry. 2012;200(6):510–511. doi: 10.1192/bjp.bp.111.095687. [DOI] [PubMed] [Google Scholar]
- Altman D. Practical statistics for medical research. London: Chapman & Hall; 1991. [Google Scholar]
- Al Mahdy H, Seymour DG. How much can elderly patients tell us about their medications? Postgrad Med J. 1990;66(772):116–121. doi: 10.1136/pgmj.66.772.116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Pinto D, Robertson MC, Hansen P, Abbott JH. Good agreement between questionnaire and administrative databases for health care use and costs in patients with osteoarthritis. BMC Med Res Methodol. 2011;11:45. doi: 10.1186/1471-2288-11-45. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Severens J, Mulder J, Laheij R, AL AV. Precision and accuracy in measuring absence from work as a basis for calculating productivity costs in The Netherlands. Soc Sci Med. 2000;51(2):243–249. doi: 10.1016/S0277-9536(99)00452-9. [DOI] [PubMed] [Google Scholar]
- Van Poppel MN, De Vet HC, Koes BW, Smid T, Bouter LM. Measuring sick leave: a comparison of self-reported data on sick leave and data from company records. Occup Med (Lond) 2002;52(8):485–490. doi: 10.1093/occmed/52.8.485. [DOI] [PubMed] [Google Scholar]
- Ferrie JE, Kivimaki M, Head J, Shipley MJ, Vahtera J, Marmot MG. A comparison of self-reported sickness absence with absences recorded in employers’ registers: evidence from the Whitehall II study. Occup Environ Med. 2005;62(2):74–79. doi: 10.1136/oem.2004.013896. [DOI] [PMC free article] [PubMed] [Google Scholar]