Abstract
PURPOSE Psychological distress may decrease adherence to medical treatments and lead to poorer health outcomes of chronic diseases. The aim of this study was to evaluate the relationship between psychological distress and multimorbidity among patients seen in family practice after controlling for potential confounding variables and taking into account the severity of diseases.
METHODS We evaluated 238 patients to construct quintiles of increasing multimorbidity based on the Cumulative Illness Rating Scale (CIRS), which is a comprehensive multimorbidity index that takes into account disease severity. Patients completed a psychiatric symptom questionnaire as a measurement of their psychological distress. In the first model of logistic regression analyses, we used the counted number of chronic diseases as the independent variable. In subsequent models, we used the quintiles of CIRS.
RESULTS After adjusting for confounding factors, multimorbidity measured by a simple count of chronic diseases was not related to psychological distress (OR, 1.12; 95% CI, 0.97–1.29; P = .188), whereas multimorbidity measured by the CIRS remained significantly associated (OR, 1.67; 95% CI, 1.19–2.37; P = .002). The estimate risk of psychological distress by quintile of CIRS was as follows: Q1/2 = 1.0; Q3 = OR, 1.72; 95% CI, 0.53–5.86; Q4 = OR, 2.99; 95% CI, 1.01–9.74; Q5 = OR, 4.67; 95% CI, 1.61–15.16.
CONCLUSIONS Psychological distress increased with multimorbidity when we accounted for disease severity. Clinicians should be aware of the possible presence of psychological distress, which can further complicate the comprehensive management of these complex patients.
Keywords: Multimorbidity, comorbidity, chronic disease, family practice, psychological stress
INTRODUCTION
Psychological distress is a key indicator of emotional problems. It refers to subjective general distress rather than disorders, describing human psychological responses in adapting to the environment.1 Psychological distress is correlated with stress and several mental disorders2 and has a negative impact on a person’s working capacity, family life, and welfare activities.3
Among persons with chronic diseases, such as arthritis or diabetes, psychological distress can intensify the effect of illness by increasing pain, functional limitations, and disability and by decreasing adherence to medical treatment protocols,4,5–7 all of which may lead to difficult clinical courses, such as poorer health outcomes and increased risks of complications.8–10 Moreover, psychological distress is associated with premature mortality among adults registered in general practice.11 Very little is known, however, about the relationship between psychological distress and multimorbidity, defined as having more than 1 chronic disease, despite the high prevalence of multimorbidity in family practice12–14 and a growing concern about multimorbidity among primary care workers.
We are aware of only 3 reports3,15,16 about the relationship between psychological distress and multimorbidity. All 3 studies reported that psychological distress increased with the number of chronic diseases. Methodological limitations question the validity of these results, however. One study was a general population survey that took into account a limited number of health problems, and the results described were only descriptive.3 A second study was restricted to an elderly population in Northern India16 and did not account for potential confounders. The third study,15 carried out in a sample of the black American population, used an index in which the number of health problems considered was limited. Moreover, none of the studies took into account disease severity.
We conducted this study to evaluate the relationship between psychological distress and multimorbidity among patients seen in family practice after controlling for potential confounding variables and taking into account not only the number of illnesses but also their severity.
METHODS
This study was included in the second part of a 2-phase project aimed at better understanding multimorbidity in patients treated in primary care. Details of the methods and sampling strategies used are described in the report of the first phase of this project,14 conducted in the Saguenay region of Québec, Canada, from January to July 2003. In brief, the participation of adult patients was solicited during consecutive consultation periods from 21 family physicians’ practices. This sample of 16 physicians in private practice and 5 working in institutions scattered in the region (population approximately 150,000 living in 1 major city and several smaller centers) was considered representative of the practice setting. We included all patients attending appointments and giving informed consent for a period of several weeks.
For all patients enrolled in the study, diagnoses of chronic diseases were collected from a chart review, based on the World Health Organization’s definition of chronic conditions, namely, “health problems that require ongoing management over a period of years or decades.”17 Multimorbidity was measured in 2 ways: first with a simple count of chronic diseases in each patient, and second with the Cumulative Illness Rating Scale (CIRS),18–20 an index that considers the severity of each health problem. The simple count included all chronic diseases in each patient, but just the number of them without any information about their severity. The CIRS uses a scoring system comprising 14 domains (Table 1 ▶). The scoring system is flexible and simple, making it easy to use. The CIRS requires assigning a value from 0 (no problem in this domain) to 4 (extremely severe problem) to determine a severity score for each domain. In the case of multiple diseases affecting a particular domain, if more than 1 score was assigned to that domain, the highest score was given to the domain. Trained research nurses reviewed the medical records and completed the data extraction. Our research group recently validated the use of the CIRS as a tool for quantifying multimorbidity in primary care patients.21
Table 1.
Characteristic | Scale |
Domain | 1. Cardiac |
2. Vascular | |
3. Hematological | |
4. Respiratory | |
5. Ophthalmological and ORL | |
6. Upper gastrointestinal | |
7. Lower gastrointestinal | |
8. Hepatic and pancreatic | |
9. Renal | |
10. Genitourinary | |
11. Musculoskeletal and tegumental | |
12. Neurological | |
13. Endocrine, metabolic, breast | |
14. Psychiatric | |
Weight | All domains weighted from 0 to 4: |
0. No problem | |
1. Mild | |
2. Moderate | |
3. Severe | |
4. Extremely severe | |
Final score | Sum of weights assigned to each domain |
Patient Recruitment and Data Collection
For the second, current phase of the project, we randomly selected patients from the 980 recruited during the first phase, stratified according to their CIRS scores. To ensure a good dispersion of the independent variable, we arbitrarily decided to divide the range of measured CIRS (0 to 30) in quintiles. Then, our goal was to recruit 60 patients for each CIRS quintile to provide sufficient power for multivariate analyses. In this process, we kept contacting patients until reaching the target sample size.
A research associate met the patients at their convenience, either at home or in our office. After signing an informed consent form, patients completed the French version of the 14-item questionnaire “Indice de Détresse Psychologique de l’Enquête Santé Québec” (IDPESQ14)22 to measure their psychological distress. The IDPESQ14 is an index covering 2 of the most important syndromes in mental health: depression and anxiety. The IDPESQ14 has good construct validity for men and women of different age-groups. The internal consistency of the scale is excellent (Cronbach’s α = .92), and it has good criterion validity.23,24 Higher scores represent greater psychological distress. Age-sex thresholds for the IDPESQ14 have been defined, and normal values for the general population are available.25 Psychological distress is considered high level when a person’s score is greater than the 80th percentile for the general population of the same age and sex category. For this study, psychological distress was classified as either present or absent, based on a comparison of the patient’s score with the appropriate age-sex threshold.
The research associate, blinded to all patients’ CIRS scores, stayed with the patients while they completed the IDPESQ14 to ensure that all items of the questionnaire were answered. Patients were given the opportunity to ask questions to clarify the items, when necessary. Consequently, the reject rate for the questionnaires was zero. Data for potential confounders were collected either from the patient’s record in the first phase of the study either by study (age, sex) or by questionnaire (all other variables). Self-perceived social support was measured with the Social Provisions Scale.26
The IDPESQ14 data were collected from November 2003 to February 2004. As mentioned above, the CIRS was measured during the first phase of the project from January to July 2003.
The research ethics board of the Centre de Santé et de Services Sociaux de Chicoutimi approved this study.
Statistical Analysis
We first conducted bivariate analyses between psychological distress and several characteristics of the participants, namely, age, sex, self-perceived social support, educational level, household income, self-perceived economic status, marital status, and number of persons living in the same dwelling. A variable was retained for multivariate analyses as a potential confounder if it had a significant relationship with psychological distress (P <.05) in the bivariate analyses. Afterward, we constructed 3 multivariate logistic regression models that related psychological distress to multimorbidity, adjusting simultaneously for potential confounders. We used unconditional logistic regression models and computed odds ratios (ORs) with their 95% confidence intervals (CIs), based on the likelihood-ratio test (Logistic procedure, SAS version 8.02, SAS Institute, Inc, Cary, NC). In the first model, we used the number of chronic diseases as the measure of multimorbidity and the odds ratio for the risk associated with an increase of 1 disease. In the 2 subsequent models, we used the quintiles of the CIRS as the measure of multimorbidity. We created dummy variables in the second model to index the CIRS quintiles and calculated the odds ratio for each quintile. In the third model, we treated the CIRS quintiles as an ordinal variable and obtained a single odds ratio that indicated the risk associated with changing from one quintile to the next. We examined various graphic displays and diagnostic statistics to detect the presence of influential observations.27 We also used hierarchical modeling to take into account the clustering of patients within physicians.
No consensus about the appropriate measures of explained variation proposed for the logistic model has yet emerged, so we relied on 3 statistic measures to estimate how accurately multimorbidity predicts psychological distress: (1) a measure of explained variation based on the correlation between the binary outcome and the predictors,28 (2) a generalized R2 statistic involving the likelihood ratio test,29 and (3) the area under the receiver operating characteristic curve (ROC curve). We calculated these accuracy indices, first for a baseline model that included only the confounding variables, then again after adding multimorbidity to the baseline model, to determine whether multimorbidity added significant information about psychological distress beyond that contributed by knowing the patient’s attributes that are related to both psychological distress and multimorbidity.
RESULTS
Of the 419 patients whom we tried to contact by telephone, 66 could not be reached despite repeated attempts. Telephone calls were made at any time of the day, including evenings. Of the remaining 353 patients, 238 agreed to participate (67% participation rate). The 115 patients who refused to participate did not have time available (66 patients), were not interested (42 patients), or were too ill (7 patients). Because the highest number of refusals came from patients in the first and second CIRS quintiles, we decided to group them together. The final number of patients in the CIRS groups (mean CIRS ± SD) was 64 in the first and second quintiles combined (3.8 ± 1.56), 55 in the third quintile (8.2 ± 0.8), 55 in the fourth quintile (11.3 ± 0.97), and 64 in the fifth quintile (17.9 ± 3.59). The characteristics of respondents and nonrespondents are summarized in Table 2 ▶. All patients were white French Canadians, a group representative of our population.
Table 2.
Characteristic | Refusals (n = 115) | Participants (n = 238) | PValue |
Age, years, mean (SD) | 56.5 (17.4) | 59.0 (14.3) | .169* |
CIRS score, mean (SD) | 10.3 (6.2) | 10.3 (5.7) | .998* |
No. of diagnoses, mean (SD) | 5.5 (3.2) | 5.3 (2.8) | .485* |
Male, % | 33.9 | 29.0 | .389† |
Educational level, % | |||
<8 years | 21.8 | ||
8 to 12 years | 38.2 | ||
Higher level (college or university) | 39.5 | ||
Missing data | 0.5 | ||
Household income in Canadian dollars, % | |||
<$10,000 | 8.8 | ||
$10,000–$29,999 | 35.3 | ||
$30,000–$49,999 | 20.5 | ||
≥$50,000 | 20.2 | ||
Missing data | 15.2 | ||
Self-perceived economic status, % | |||
Poverty | 15.6 | ||
Sufficient | 61.3 | ||
Wealthy | 22.7 | ||
Missing data | 0.4 | ||
Marital status, % | |||
Married or cohabiting | 68.1 | ||
Divorced or separated | 10.5 | ||
Widower | 12.6 | ||
Never married | 8.8 | ||
Persons living in the same dwelling, % | |||
0 | 18.9 | ||
1 | 46.2 | ||
≥2 | 34.9 |
CIRS = Cumulative Illness Rating Scale.
* t test.
† χ2 test.
Bivariate analyses showed that multimorbidity, whether measured by a simple count of chronic diseases or by the CIRS, was significantly associated with psychological distress. Associations with age, educational level, household income, marital status, and number of persons living in the same dwelling were not significant. Consequently, the multivariate models were adjusted only for sex, self-perceived social support, and self-perceived economic status.
Multivariate analyses (Table 3 ▶) showed that multimorbidity measured by a simple count of chronic diseases and adjusted for confounding factors was not related to psychological distress (OR, 1.12; P = .188). Only multimorbidity measured by the CIRS, whether treated as a categorical (OR, 1.72–4.67; P = .024) or an ordinal variable (OR, 1.67; P = .002), was significantly associated with psychological distress. The magnitude of the relation is given by the finding that the highest burden of disease in the uppermost quintile was associated with psychological distress, with an odds ratio of 4.67, as compared with the lowest quintile. There was a 3.7-fold increase of the proportion of patients with psychological distress from the lowest quintile, Q1/2 (9.4% of patients), of the CIRS measure of multimorbidity to the highest, Q5 (34.4%). Clustering patients by physician had no effect on the results.
Table 3.
Psychological Distress (IDPESQ14 score) | |||
Multimorbidity Measure | OR (95% CI) | P* | Pcorr† |
Count of chronic diseases | 1.12 (0.97–1.29) | .113 | .188 |
Quintiles of CIRS | |||
1/2 | 1.00 | .036 | .024 |
3 | 1.72 (0.53–5.86) | ||
4 | 2.99 (1.01–9.74) | ||
5 | 4.67 (1.61–15.16) | ||
Changing CIRS quintile | 1.67 (1.19–2.37) | .003 | .002 |
IDPESQ14 = Indice de Détresse Psychologique de l’Enquête Santé Québec; OR = odds ratio; CI = confidence interval; Pcorr = P corrected ; CIRS = Cumulative Illness Rating Scale.
* When patients clustered by physician was not taken into account.
† When patients clustered by physician was taken into account.
A comparison of the predictive accuracy of models of multimorbidity for psychological distress is displayed in Table 4 ▶. The number of chronic diseases did not add significant information about psychological distress beyond that contributed from knowledge of the patient’s sex, self-perceived social support, and self-perceived economic status. On the other hand, a model including the CIRS quintiles, whether treated as a categorical or ordinal variable, better predicted psychological distress than did a model restricted to the 3 confounding factors. The areas under the ROC curves, ranging from 0.747 to 0.777, all fall in the “fair” predictive range of 0.7 to 0.8.
Table 4.
Additions to Baseline Model | ||||
CIRS Quintile Variables | ||||
Statistical Measures | Baseline Model* | No. of Chronic Diseases | Categorical | Ordinal |
Deviance | 216.96 | 214.44 | 207.81 | 207.83 |
Likelihood ratio test | ||||
χχ2 (df) | n/a | 2.53 (1) | 9.15 (3) | 9.13 (1) |
P value | .112 | .027 | .003 | |
Measures of explained variation | 0.0962 | 0.1004 | 0.1299 | 0.1298 |
r2(y, p̂) | 0.1100 | 0.1197 | 0.1446 | 0.1445 |
R2 analogue | ||||
Area under the ROC curve | 0.747 | 0.756 | 0.776 | 0.777 |
CIRS = Cumulative Illness Rating Scale; n/a = not applicable; ROC = receiver operating characteristic. * Including only the confounding variables (ie, sex, self-perceived social support, and self-perceived economic status).
DISCUSSION
In a sample representative of a population consulting family physicians, we found that the presence of psychological distress increased with the severity of multimorbidity measured by the CIRS index. The simple count of chronic diseases was not associated with psychological distress after controlling for confounding variables.
To our knowledge, our study is the first to analyze the association between multimorbidity and psychological distress in a family practice context and using an index that takes the severity of illnesses into account. The rationale for using the CIRS, in addition to a simple count of health problems, was that different chronic diseases, even if numerically equal, may represent a very different burden for patients. We believed it was important to take into account the severity of diseases, and the CIRS provides a comprehensive evaluation of medical problems by organ system validated in a family practice context.21
We considered several factors (age, sex, self-perceived social support, educational level, household income, self-perceived economic status, marital status, and number of persons living in the same dwelling) to eliminate their potential confounding effect on the relationship between multimorbidity and psychological distress. Among these potential confounding variables, only patients’ sex, self-perceived social support, and self-perceived economic status had an effect on the association. These findings are consistent with previous data that linked lower socioeconomic status15 with greater psychological distress and multimorbidity. Our results, however, were significant only for self-perceived economic status, not for household income.
After we controlled for sex, self-perceived social support, and self-perceived economic status in our sample, multimorbidity measured by a simple count of chronic diseases was no longer associated with psychological distress. Joshi and colleagues16 also found a relationship between psychological distress and the number of chronic diseases, but their analysis stopped at that point and did not control for any confounding factor. Johnson and Arbor15 did control for indicators of socioeconomic status, but at variance with our result they still found a relationship between the number of health problems and psychological distress. These authors, however, used an 8-item index composed of a list of 7 health problems and an eighth item that was “any other health problem,” which made the range of the index from 0 to 8. If a particular patient did not have any of the 7 health problems included in the index but several other problems instead, all of them were counted as 1. This limited assessment of multimorbidity could partly explain the discrepancy between their finding and our result. Another possible reason for the disagreement between our result and that of Johnson and Arbor is their sample included only 1 ethnic group in which, as a whole, the prevalence of diseases and socioeconomic conditions may be different.
The current study has some limitations. First, the lack of simultaneous measurement of psychological distress (IDPESQ14) and multimorbidity (CIRS) may have affected our measure of their relationship. Consequently, 1 or both variables may have been modified during the time elapsed between measurements. We believe that any error introduced by the difference in the timing of the measurements would be minor because the time elapsed was relatively short compared with the length of time patients had had these chronic diseases. Moreover, patients in an acute state of their chronic illnesses at the second phase were not included (7 patients too ill).
That the highest number of refusals to participate was in the 2 lower CIRS quintiles could have introduced a selection bias. Our results indicate that the proportion of patients with psychological distress increased with the quintiles of increasing CIRS. Grouping the first and second quintiles together should increase the proportion of patients with psychological distress in the resulting combined group, which would make any difference between this reference group and the others more difficult to identify. In any case, if selection bias were present, it would have reduced the differences we found between quintiles.
The measurement tool used to assess psychological distress has limitations. The IDPESQ14 identifies patients with a high level of psychological distress, but it does not provide information about the nature or seriousness of the underlying problem.2 Elevated scores indicate that something is wrong, but not what is wrong. Like all other scales measuring psychological distress, this index gives only general indications of distress1; the index does not take the severity of the symptoms into account. All items are given the same weight; the global score considers only the number and frequency of reported symptoms during a defined period (up to 7 days before the measurement, in the case of the IDPESQ14).
The presence of the evaluator while the patients completed their IDPESQ14 questionnaires may also have influenced the outcome. Patients may have had hidden psychological problems that could harm their social acceptability or others’ opinions of them, a phenomenon known as “social desirability,”30 which would result in an underestimation of the scores of the IDPESQ14. This limitation, however, would affect all CIRS quintiles equally.
In conclusion, our results show that psychological distress of patients in family practice increased with multimorbidity measured by the CIRS. The estimate risk of psychological distress increased almost 5 times in the group with the highest burden of disease. A simple count of diseases was found to be an inappropriate measure of multimorbidity to study this association, because its relationship with psychological distress was no longer observed after controlling for sex, self-perception of social support, and self-perceived economic status. Clinicians should be aware of possible psychological distress in patients with multimorbidity, which can further complicate the comprehensive management of these complex patients.
Acknowledgments
Sharon Nancekivell, Guelph, Ontario, provided editorial assistance.
Conflicts of interest: none reported
Funding support: Drs Fortin, Bravo and Hudon were supported by a grant (#24300-2028) from the Fonds de la recherche en santé du Québec (Grant number: 24300-2028). Drs Fortin and Hudon were also supported by an independent research grant from Pfizer Canada.
REFERENCES
- 1.McDowell I, Newell C. Measuring Health. A Guide to Rating Scales and Questionnaires. 2nd ed. New York, NY: Oxford University Press; 1996.
- 2.Perreault C. L’enquête Santé Québec et la santé mentale des québécois: cadre conceptuel et méthodologie. Santé mentale au Québec. 1989;14:132–143. [PubMed] [Google Scholar]
- 3.Daveluy C, Pica L, Audet N, Courtemanche R, Lapointe F. Enquête Sociale et de Santé 1998. 2nd ed. Québec: Institut de la statistique du Québec; 2000.
- 4.Strine TW, Hootman JM, Okoro CA, et al. Frequent mental distress status among adults with arthritis age 45 years and older, 2001. Arthritis Rheum. 2004;51:533–537. [DOI] [PubMed] [Google Scholar]
- 5.Serious psychological distress among persons with diabetes--New York City, 2003. MMWR Morb Mortal Wkly Rep. 2004;53:1089–1092. [PubMed] [Google Scholar]
- 6.Parker JC, Wright GE. The implications of depression for pain and disability in rheumatoid arthritis. Arthritis Care Res. 1995;8:279–283. [DOI] [PubMed] [Google Scholar]
- 7.Ciechanowski PS, Katon WJ, Russo JE. Depression and diabetes: impact of depressive symptoms on adherence, function, and costs. Arch Intern Med. 2000;160:3278–3285. [DOI] [PubMed] [Google Scholar]
- 8.Kruse J, Schmitz N, Thefeld W. On the association between diabetes and mental disorders in a community sample: results from the German National Health Interview and Examination Survey. Diabetes Care. 2003;26:1841–1846. [DOI] [PubMed] [Google Scholar]
- 9.Lustman PJ, Anderson RJ, Freedland KE, et al. Depression and poor glycemic control: a meta-analytic review of the literature. Diabetes Care. 2000;23:934–942. [DOI] [PubMed] [Google Scholar]
- 10.Anderson RJ, Grigsby AB, Freedland KE, et al. Anxiety and poor glycemic control: a meta-analytic review of the literature. Int J Psychiatry Med. 2002;32:235–247. [DOI] [PubMed] [Google Scholar]
- 11.Robinson KL, McBeth J, Macfarlane GJ. Psychological distress and premature mortality in the general population: a prospective study. Ann Epidemiol. 2004;14:467–472. [DOI] [PubMed] [Google Scholar]
- 12.Hoffman C, Rice D, Sung HY. Persons with chronic conditions. Their prevalence and costs. JAMA. 1996;276:1473–1479. [PubMed] [Google Scholar]
- 13.Metsemakers JF, Hoppener P, Knottnerus JA, Kocken RJ, Limonard CB. Computerized health information in The Netherlands: a registration network of family practices. Br J Gen Pract. 1992;42:102–106. [PMC free article] [PubMed] [Google Scholar]
- 14.Fortin M, Bravo G, Hudon C, Vanasse A, Lapointe L. Prevalence of multimorbidity among adults seen in family practice. Ann Fam Med. 2005;3:223–228. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Johnson EH. Psychiatric morbidity and health problems among black Americans: a national survey. J Natl Med Assoc. 1989;81:1217–1223. [PMC free article] [PubMed] [Google Scholar]
- 16.Joshi K, Kumar R, Avasthi A. Morbidity profile and its relationship with disability and psychological distress among elderly people in Northern India. Int J Epidemiol. 2003;32:978–987. [DOI] [PubMed] [Google Scholar]
- 17.World Health Organization. Innovative Care for Chronic Conditions. Building blocks for action. Available at: http://www.who.int/chronic_conditions/en/icccglobalreport.pdf. Accessed: 23 November 2005.
- 18.Linn BS, Linn MW, Gurel L. Cumulative illness rating scale. J Am Geriatr Soc. 1968;16:622–626. [DOI] [PubMed] [Google Scholar]
- 19.Miller MD, Towers A. A Manual of Guidelines for Scoring the Cumulative Illness Rating Scale for Geriatrics (CIRS-G). Pittsburgh, Pa: University of Pittsburgh; 1991.
- 20.Harboun M, Ankri J. [Comorbidity indexes: review of the literature and application to studies of elderly population]. Rev Epidemiol Sante Publique. 2001;49:287–298. [PubMed] [Google Scholar]
- 21.Hudon C, Fortin M, Vanasse A. Cumulative Illness Rating Scale was a reliable and valid index in a family practice context. J Clin Epidemiol. 2005;58:603–608. [DOI] [PubMed] [Google Scholar]
- 22.Santé Q. Et la Santé, Ca Va? Rapport de L’enquête Santé Québec 1987. Québec: Ministère de la Santé et des Services sociaux, Gouvernement du Québec, Les publications du Québec; 1988.
- 23.Préville M, Boyer R, Potvin L, Perreault C, Légaré G. La Détresse Psychologique: Détermination de la Fiabilité et de la Validité de la Mesure Utilisée dans L’enquête Santé Québec, Enquête Santé Québec 1987. Québec: Ministère de la Santé et des Services sociaux; 1992.
- 24.Préville M Étude de la Validité et de la Fiabilité de la Mesure de Détresse Psychologique Utilisée dans l’Enquête Santé Québec. Montreal: Université de Montréal. T95-01; 1995.
- 25.Boyer R, Preville M, Legare G, Valois P. [Psychological distress in a noninstitutionalized population of Quebec: normative results of the Quebec health survey]. Can J Psychiatry. 1993;38:339–343. [DOI] [PubMed] [Google Scholar]
- 26.Cutrona CE, Russell DW. The provisions of social support and adaptation to stress. Adv Pers Relationships. 1987;1:37–67. [Google Scholar]
- 27.Hosmer DW, Lemeshow S. Applied Logistic Regression. New York, NY: John Wiley and Sons; 1989.
- 28.Mittlbock M, Schemper M. Explained variation for logistic regression. Stat Med. 1996;15:1987–1997. [DOI] [PubMed] [Google Scholar]
- 29.Cox DR, Snell EJ. Analysis of Binary Data. London: Chapman and Hall; 1989.
- 30.Allaire D. Questionnaires: mesure verbale du comportement. In: Robert M, Ed. Fondements et Etapes de la Recherche Scientifique en Psychologie. 3rd ed. St-Hyacinthe: Edisem; 1988:229–275.