Abstract
Background
Patient reported outcome measures (PROMs) are used to evaluate lifestyle interventions but little is known about differences between patients returning valid and invalid responses, or of potential for bias in evaluations. We aimed to examine the characteristics of patients who returned valid responses to lifestyle questionnaires compared to those whose responses were invalid for evaluating lifestyle change.
Methods
We conducted a secondary data analysis from the SPHERE Study, a trial of an intervention to improve outcomes for patients with coronary heart disease in primary care. Postal questionnaires were used to assess physical activity (Godin) and diet (DINE) among study participants at baseline and 18 month follow-up. Three binary response variables were generated for analysis: (1) valid Godin score; (2) valid DINE Fibre score; and (3) valid DINE Total Fat score. Multivariate analysis comprised generalised estimating equation regression to examine the association of patients’ characteristics with their return of valid responses at both timepoints.
Results
Overall, 92.1% of participants (832/903) returned questionnaires at both baseline and 18 months. Relatively fewer valid Godin scores were returned by those who left school aged <15 years (36.5%) than aged 18 and over (50.5%), manual workers (39.5%) than non-manual (49.5%) and those with an elevated cholesterol (>5 mmol) (34.7%) than those with a lower cholesterol (44.4%) but multivariate analysis identified that only school leaving age (p = 0.047) was of statistical significance.
Relatively fewer valid DINE scores were returned by manual than non-manual workers (fibre: 80.8% v 86.8%; fat: 71.2% v 80.0%), smokers (fibre: 72.6% v 84.7%; fat: 67.5% v 76.9%), patients with diabetes (fibre: 75.9% v 82.9%; fat: 66.9% v 75.8%) and those with cholesterol >5 mmol (fat: 68.2% v 76.2%) but multivariate analysis showed statistical significance only for smoking (fibre: p = 0.013; fat: p = 0.045), diabetes (fibre: p = 0.039; fat: p = 0.047), and cholesterol (fat: p = 0.039).
Conclusions
Our findings illustrate the importance of detailed reporting of research methods, with clear information about response rates, respondents and valid outcome data. Outcome measures which are relevant to a study population should be chosen carefully. The impact of methods of outcome measurement and valid response rates in evaluating healthcare requires further study.
Keywords: PROMs, Lifestyle, Self report questionnaires, Diet, Physical activity, Research methods
Background
A large body of evidence, from both clinical and observational studies, has been built up over recent decades to support the principle that a lifestyle which includes regular physical activity and a balanced, healthy diet helps to prevent disease [1]. The effectiveness of interventions incorporating different approaches to lifestyle change has been reported for both the primary and secondary prevention of coronary heart disease (CHD) [2, 3]. However, gaps remain in the evidence regarding optimal intervention design. Crucial to evaluating the effectiveness of lifestyle interventions are the methods chosen to assess changes in physical activity and diet.
Self-report measures of physical activity, such as questionnaires or diaries, are used widely although they are subject to recall bias [4, 5] and may have limited accuracy [6]. Objective measures such as accelerometers, pedometers or heart rate monitors [7] can also be used to assess physical activity but these are more expensive and study participants require instruction in their use. Similarly, a variety of measures can be used to assess diet, including specialist techniques involving doubly labelled water, direct observation, detailed food diaries and questionnaires: measurement errors can exist in every method [8]. Food frequency questionnaires are widely used, given that they are relatively easily administered, inexpensive and provide useful information [9] although multiple issues are relevant to their validity [10]. Higher numbers of items, giving greater detail, may increase validity [11] but higher response rates are more likely with shorter questionnaires [12].
The SPHERE Study (Secondary Prevention of Heart Disease in General Practice) [13, 14] aimed to evaluate individually tailored care plans to improve outcomes, including health-related behaviours, for patients with CHD in primary care. The study chose self-report measures to assess behaviour, using the Godin Leisure Time Questionnaire [15] to assess levels of physical activity and the Dietary Instrument for Nutrition Education (DINE) questionnaire [16] to assess diet.
The Godin questionnaire [15] provides a measure of type, duration and intensity of leisure-time exercise and is designed to be straightforward to use [17]. It has been validated [18] and tested for reliability [19] in studies in which subjects had opportunity to ask questions but other studies have reported posting it to participants [20, 21] without offering this facility. It is a four-item questionnaire, which gives examples of mild, moderate and strenuous exercise and asks how many times per week the respondent undertakes 15 minutes of each exercise category.
The DINE questionnaire [16] is a short questionnaire which is considered an acceptable alternative to more detailed diet recall questionnaires and food diaries. It gives a measure of dietary intake of fibre and fat, focusing on these because of their important association with cardiovascular disease and cancer and excluding other diet components in order to keep the questionnaire as short as possible. It was validated amongst workplace employees [16], to whom nurses administered the questionnaire: scores derived from it were compared with 4-day diet records. Face-to-face administration allowed ambiguous or unclear responses to be clarified immediately but in other studies DINE has been posted to participants [22, 23].
Within the SPHERE study, questionnaires were posted to participants and returned to practices by post, without opportunity for advice or review. Whilst the study achieved 92.8% follow-up [24], assessment of change in physical activity and in diet was possible only for, respectively, 38% and 75% of the total sample (these were the proportions of participants who provided valid responses at both time points). Thus, potential respondent bias may have implications for the interpretation of the evaluation. Whilst people who live in affluent areas are more likely to respond to questionnaires than those who live in deprived areas [12] differences in characteristics of patients with CHD who do and do not return valid responses to lifestyle questionnaires, administered by post, are not known.
This study aimed to examine possible respondent bias within the SPHERE Study by conducting a secondary data analysis to determine if characteristics of those who returned responses to postal questionnaires which allowed valid assessment of their physical activity and diet differed from those who did not. We also incorporated qualitative observations from a later follow-up study during which the researcher directly observed participants completing questionnaires.
Methods
Background: the SPHERE study
This study is set in the context of the SPHERE Study, a large cluster randomised trial conducted in the two different primary healthcare systems on the island of Ireland [13]. Briefly, participants were recruited from three different centres (Belfast, Northern Ireland (NI), UK; Galway, Republic of Ireland (RoI) and Dublin (RoI). Forty eight practices (16 from each centre) were randomly selected by an individual independent of the research team, using computer generated random numbers.
Lists of patients with known CHD were compiled by practice staff and those with significant mental or physical illness likely to impair capacity to participate in the intervention were excluded. Patients were invited, by post, from random order lists until 20 from each practice agreed to take part. They were posted questionnaires, including Godin and DINE, with reply paid envelopes which were posted back to the practice, for collection by the researchers, without review by practice staff. Six weeks were allowed for non-response, during which reminders were posted.
Baseline data collection was completed before randomisation into intervention and control groups to minimise potential recruitment bias. During the following 18 months (2005–2007) those in the intervention group attended their practice nurse or general practitioner for 4-monthly consultations at which lifestyle, risk factors and medication were reviewed. Goals for lifestyle change and targets were set, for review at each subsequent consultation. After 18 months, in addition to objective measurement of bio-physical risk factors, participants were again posted questionnaires following the same process as at baseline.
The SPHERE Study was granted ethical approval by the Irish College of General Practitioners and the Queen’s University research ethics committee.
Qualitative study
A follow-up qualitative study of barriers and facilitators affecting lifestyle change was conducted after 4 years and has been reported previously [25]: 45 participants were interviewed and at the end of their interview they were invited to self-complete a Godin and a DINE questionnaire in the presence of the researcher who addressed any queries. Observational data relating to the completion of these questionnaires were extracted from that study in order to help contextualise the quantitative analyses described below.
Ethical approval for this follow-up study was granted by the Office for Research Ethics Committees (Northern Ireland), ref 10/NIR03/11 (April 2010), and the Irish College of General Practitioners Research Ethics Committee (May 2010).
Data management
The aim of this current paper was to use data from the SPHERE Study to compare characteristics of participants who did or did not return questionnaires from which valid outcome measurements of physical activity and diet could be calculated at both baseline and follow-up.
An invalid response to the Godin questionnaire was considered to include any response other than a clear number of 15-minute PA sessions. Responses were scored to indicate the level of health benefit resulting from the reported exercise level. A score of <14 units indicates an insufficient level of activity with low health benefits; 14–23 indicates moderate activity with some health benefits; ≥24 indicates sufficient activity with substantial health benefits [17]. An invalid response to the DINE questionnaire was attributed to anything other than a stipulated answer [16]. Questions relate to the frequency of consumption of 19 food groups which contribute about 70% of the fat and fibre in a common Western diet. Each food group is given a score to reflect the nutrient content of a standard portion size, and scores are weighted according to frequency of consumption. No attempt was made to interpret unclear or missing responses. Within the current study, analysis was confined to DINE scores relating to fibre and total fat.
Statistical analyses
Three binary response variables were generated for the statistical analysis: (1) valid Godin score; (2) valid DINE Fibre score; and (3) valid DINE Total Fat score. The statistical analyses examined rates of valid scores for each response variable for study participants with varying characteristics using a multivariate approach in which the choice of regression model was informed by the nature of the response variables under consideration and the hierarchical nature of the SPHERE dataset. In the case of the latter, there is a natural classification to the observations at the level of an individual general practice surgery, such that data are organised or clustered according to the practice with which a person is registered. With clustered data individual observations are not independent and multilevel analytical approaches are required for regression analysis [26]. In this study, generalised estimating equation (GEE) multivariate regression analysis was used to explore the effect of a range of independent variables on the three response variables of interest. When estimating a GEE model, in addition to identifying the appropriate linear predictor, it is also necessary to specify a suitable variance function, link function, and correlation structure [26]. The approach adopted in the current analysis was a binomial variance function and a log link function (both based on the binary nature of the response variables), and an exchangeable correlation structure (based on recommendations from the literature [26]). The patient characteristics which were included as independent variables in the multivariate regression models were: age, gender, years since diagnosis, history of myocardial infarction, systolic blood pressure, diastolic blood pressure, total cholesterol, smoking status, body mass index, SPHERE treatment allocation, age left school, marital status and occupational status.
The results of the multivariate analyses allow us to identify statistically significant associations for three response variables of interest, with statistical significance set at p < 0.05 for all analyses. The statistical analyses were conducted using the statistics package Stata 13.
Results
In total, 903 patients took part in the SPHERE Study; all returned questionnaires at baseline and 832 (92.1%) completed response questionnaires at follow-up after 18 months.
Response rates
Table 1 shows that the rates of return of valid Godin scores at both baseline and 18 month follow-up were similar (approximately 60%); a higher percentage returned valid DINE scores at baseline (95% fibre; 90% fat) than at follow-up (86% fibre; 81% fat).
Table 1.
Numbers who returned valid and invalid questionnaire scores at different time points during the SPHERE Study
| Total questionnaires returned | ||
|---|---|---|
| Valid scores | Invalid scores | |
| N (%) | N (%) | |
| Godin | ||
| Baseline* | 541 (59.1) | 362 (40.9) |
| Follow-up** | 523 (62.9) | 309 (37.1) |
| DINE fibre | ||
| Baseline* | 861 (95.4) | 42 (4.6) |
| Follow-up** | 716 (86.1) | 116 (13.9) |
| DINE total fat | ||
| Baseline* | 814 (90.1) | 89 (9.90) |
| Follow-up** | 673 (80.9) | 159 (19.1) |
*Baseline total N = 903; **Follow-up N = 832; 71 lost to follow-up due to: death; withdrawn consent; left practice; too ill; too busy; admitted to nursing home (44 intervention; 27 control).
Participants who did not return a follow-up questionnaire (n = 71) were excluded from our analyses. For the remaining 832 participants, characteristics were compared between those who returned questionnaires for which valid scores could be calculated at both time points and those who did not. Valid Godin scores were returned by 41.9% of the original sample (349/832), valid DINE fibre scores by 82.5% (686/832) and valid DINE total fat scores by 74.0% (616/832) at both time points.
Descriptive comparisons of patient characteristics
In relation to the Godin questionnaire, fewer of the participants who left school at age 14 or under returned valid scores (36.5%), compared to those who left school aged 18 and over (50.5%) (Table 2); 39.1% of manual workers returned valid scores, whilst 49.8% of participants in non-manual occupations did so. Fewer of those whose total cholesterol was >5 mmol/l returned valid Godin scores (34.7%) than did those with lower cholesterol levels (44.4%).
Table 2.
Characteristics of patients at baseline and of those with valid and invalid responses to GODIN, DINE fibre and DINE fat questionnaires
| GODIN | DINE fibre | DINE fat | ||||||
|---|---|---|---|---|---|---|---|---|
| Total study population (N = 832)* | Valid response | Invalid response | Valid response | Invalid response | Valid response | Invalid response | ||
| N (%) | 349 (41.9) | 483 (58.1) | 679 (81.6) | 153 (18.4) | 616 (74.0) | 216 (26.0) | ||
| Patient characteristics** | N (%)‡ | N (%)‡ | N (%)‡ | N (%)‡ | N (%)‡ | N (%)‡ | ||
| Age (years) (Mean; SD) | Mean 67.2; SD 9.53 (Range 29–91) | 66.6 (9.77) | 67.6 (9.33) | 67.3 (9.57) | 66.7 (9.37) | 67.2 (9.58) | 67.3 (9.41) | |
| 70 years and older | 340 (40.9%) | 138 (40.6) | 202 (59.4) | 280 (82.4) | 60 (17.6) | 247 (72.6) | 93 (27.4) | |
| <70 years | 492 (59.1%) | 211 (42.9) | 281 (57.1) | 399 (81.1) | 93 (18.9) | 369 (75.0) | 123 (25.0) | |
| Gender | Male | 582 (70.0) | 250 (42.9) | 332 (57.1) | 479 (82.3) | 103 (17.7) | 435 (74.7) | 147 (25.3) |
| Female | 250 (30.0) | 99 (39.6) | 151 (60.4) | 200 (80.0) | 50 (20.0) | 181 (72.4) | 69 (27.6) | |
| History of MI | Yes | 415 (49.9) | 182 (43.9) | 233 (56.1) | 338 (81.4) | 77 (18.6) | 310 (74.7) | 105 (25.3) |
| No | 417 (50.1) | 167 (40.0) | 250 (60.0) | 341 (81.8) | 76 (18.2) | 306 (73.4) | 111 (26.6) | |
| History of diabetes | Yes | 145 (17.4)) | 58 (40.0) | 87 (60.0) | 110 (75.9) | 35 (24.1) | 97 (66.9) | 48 (33.1) |
| No | 685 (82.3) | 290 (42.3) | 395 (57.7) | 568 (82.9) | 117 (17.1) | 519 (75.8) | 166 (24.2) | |
| Elevated systolic BP*** | Yes | 281 (33.8) | 115 (40.9) | 166 (59.1) | 229 (81.5) | 52 (18.5) | 202 (71.9) | 79 (28.1) |
| No | 548 (65.9) | 234 (42.7) | 314 (57.3) | 447 (81.6) | 101 (18.4) | 412 (75.2) | 136 (24.8) | |
| Elevated diastolic BP*** | Yes | 98 (11.8) | 37 (37.8) | 61 (62.2) | 77 (78.6) | 21 (21.4) | 69 (70.4) | 29 (29.6) |
| No | 733 (88.1) | 312 (42.6) | 421 (57.4) | 601 (82.0) | 132 (18.0) | 546 (74.5) | 187 (25.5) | |
| Elevated cholesterol*** | Yes | 173 (20.8) | 60 (34.7) | 113 (65.3) | 137 (79.2) | 36 (20.8) | 118 (68.2) | 55 (31.8) |
| No | 621 (74.6) | 276 (44.4) | 345 (55.6) | 514 (82.8) | 107 (17.2) | 473 (76.2) | 148 (23.8) | |
| Smoker | Smoker | 117 (14.1) | 48 (41.0) | 69 (59.0) | 85 (72.6) | 32 (27.4) | 79 (67.5) | 38 (32.5) |
| Non-smoker | 685 (82.3) | 296 (43.2) | 389 (56.8) | 580 (84.7) | 105 (15.3) | 527 (76.9) | 158 (23.1) | |
| BMI >25 kg/m2 | Yes | 673 (80.9) | 278 (41.3) | 395 (58.7) | 551 (81.9) | 122 (18.1) | 493 (73.3) | 180 (26.7) |
| No | 156 (18.6) | 70 (44.9) | 86 (55.1) | 126 (80.8) | 30 (19.2) | 121 (77.6) | 35 (22.4) | |
| Study arm | Intervention | 400 (48.1) | 159 (39.8) | 241 (60.3) | 330 (82.5) | 70 (17.5) | 295 (73.8) | 105 (26.3) |
| Control | 432 (51.9) | 190 (44.0) | 242 (56.0) | 349 (80.8) | 83 (19.2) | 321 (74.3) | 111 (25.7) | |
| Marital status | Married/living with partner | 565 (67.9) | 250 (44.2) | 315 (55.8) | 471 (83.4) | 94 (16.6) | 427 (75.6) | 138 (24.4) |
| Single/widowed/separated/divorced | 241 (28.9) | 92 (38.2) | 149 (61.8) | 200 (82.9) | 41 (17.1) | 180 (74.7) | 61 (25.3) | |
| Age left school (years) | 14 and under | 337 (40.5) | 123 (36.5) | 214 (63.5) | 280 (83.1) | 57 (16.9) | 244 (72.4) | 93 (27.6) |
| 15 to 17 | 335 (40.3) | 147 (43.9) | 188 (56.1) | 279 (83.3) | 56 (16.7) | 257 (76.7) | 78 (23.3) | |
| 18 and over | 95 (11.4) | 48 (50.5) | 47 (49.5) | 82 (86.3) | 13 (13.7) | 78 (82.1) | 17 (17.9) | |
| Occupation | Manual | 281 (33.8) | 110 (39.1) | 171 (60.9) | 227 (80.8) | 54 (19.2) | 200 (71.2) | 81 (28.8) |
| Non manual | 325 (39.1) | 162 (49.8) | 163 (50.2) | 282 (86.8) | 43 (13.2) | 260 (80.0) | 65 (20.0) | |
*Total N = 832; numbers for some characteristics are less due to incomplete data; ‡percentages refer to row totals.
**Baseline data; ***Elevated SBP defined as systolic blood pressure >140 mmHg; elevated DBP defined as diastolic blood pressure >90 mmHg; elevated cholesterol defined as serum cholesterol >5.0 mmol/l.
Rates of return of valid scores observed for participants of different gender, age, marital status, study arm, BMI, blood pressure, history of MI or diabetes, or smoking status were similar.
In relation to the DINE fibre questionnaire, a smaller proportion of participants in manual occupations (80.8%) than in non-manual occupations (86.8%) returned valid scores. There was a trend towards those who left school at an older age being more likely to return valid scores than those who left school earlier. More of those who reported being non-smokers returned valid fibre scores than smokers (84.7% v 72.6%), Fewer of those without a history of diabetes than of those with diabetes returned valid responses (82.9% v 75.9%). Rates of valid and invalid DINE total fat scores were similar to those for DINE fibre in relation to history of diabetes, total cholesterol, smoking status and occupation (Table 2).
Multivariate analysis of patient characteristics
Table 3 presents the results from the multivariate regression analyses. In the Godin score analysis, after controlling across the full set of patient characteristics, only a school leaving age of 18 or over was found to be statistically significant. The regression coefficient indicates a positive and statistically significant response effect (0.508; p = 0.047). In the DINE fibre analysis, a history of diabetes (−0.506; p = 0.039) and smoking (−0.682; p = 0.013) were both associated with negative and statistically significant response effects. Similarly, in the DINE total fat score analysis, a history of diabetes (−0.457; p = 0.047), smoking (−0.529; p = 0.045), and high total cholesterol (−0.455; p = 0.039) were associated with negative and statistically significant response effects.
Table 3.
Multivariate analysis of characteristics of patients with and without valid GODIN, DINE fibre and DINE fat scores at baseline
| Instrument | GODIN | DINE fibre | DINE total fat | ||||
|---|---|---|---|---|---|---|---|
| Patient characteristics** | Coefficient | p-value | Coefficient | p-value | Coefficient | p-value | |
| Age | −0.010 | 0.238 | −0.01 | 0.695 | −0.006 | 0.616 | |
| Gender | Male | −0.044 | 0.801 | 0.174 | 0.440 | −0.070 | 0.744 |
| Female | Ref* | Ref | Ref | ||||
| History of MI | Yes | 0.040 | 0.790 | −0.258 | 0.198 | −0.235 | 0.208 |
| No | Ref | Ref | Ref | ||||
| History of diabetes | Yes | −0.037 | 0.854 | −0.506 | 0.039 | −0.457 | 0.047 |
| No | Ref | Ref | Ref | ||||
| Elevated systolic BP*** | Yes | 0.074 | 0.680 | 0.082 | 0.735 | 0.009 | 0.966 |
| No | Ref | Ref | Ref | ||||
| Elevated diastolic BP*** | Yes | −0.318 | 0.218 | −0.379 | 0.238 | −0.330 | 0.267 |
| No | Ref | Ref | Ref | ||||
| Elevated cholesterol*** | Yes | −0.283 | 0.140 | −0.324 | 0.178 | −0.455 | 0.039 |
| No | Ref | Ref | Ref | ||||
| Smoker | Smoker | −0.109 | 0.636 | −0.682 | 0.013 | −0.529 | 0.045 |
| Non-smoker | Ref | Ref | Ref | ||||
| BMI >25 kg/m2 | Yes | −0.239 | 0.222 | −0.005 | 0.983 | −0.225 | 0.368 |
| No | Ref | Ref | Ref | ||||
| Study arm | Intervention | −0.159 | 0.293 | 0.141 | 0.488 | −0.023 | 0.909 |
| Control | Ref | Ref | Ref | ||||
| Marital status | Married/living with partner | 0.233 | 0.204 | −0.228 | 0.366 | 0.025 | 0.911 |
| Single/widowed/separated/divorced | Ref | Ref | Ref | ||||
| Age left school (years) | 14 and under | Ref | Ref | Ref | |||
| 15 to 17 | 0.240 | 0.147 | −0.004 | 0.987 | 0.024 | 0.905 | |
| 18 and over | 0.508 | 0.047 | 0.134 | 0.712 | 0.475 | 0.179 | |
| Occupation | Manual | Ref | Ref | Ref | |||
| Non manual | 0.226 | 0.243 | 0.306 | 0.267 | 0.275 | 0.250 | |
*Total N = 832; numbers for some characteristics are less due to incomplete data; Ref refers to reference category for valid responses for each characteristic.
**Baseline data; ***Elevated SBP defined as systolic blood pressure >140 mmHg; elevated DBP defined as diastolic blood pressure >90 mmHg; elevated cholesterol defined as serum cholesterol >5.0 mmol/l.
Figures in bold are statistically significant.
Qualitative study data Of the 45 interviewees 2 declined to complete either a Godin or a DINE questionnaire due to time constraints. Of the remainder, 72.1% (31/43) had valid scores for the Godin questionnaire; 93.0% (40/43) had valid scores for DINE fibre; scores for total fat were not computed. Questions asked by participants related to advice as to whether certain activities could be counted as ‘exercise’, help in translating activities into 15 minute periods of different intensities of exercise, whether certain foods ‘fitted’ the categories presented in DINE and translation of their diet recall into numbers of servings per week. It was observed that some participants did not take time to read questions carefully or take account of different question construction requiring a different format of answer.
Discussion
Summary of main findings
These findings show that study participants with a lower level of education returned relatively fewer valid Godin scores for assessment of change in their self-reported physical activity. Relatively fewer valid DINE scores were returned by smokers, people with diabetes and total cholesterol >5 mmol/l.
The Godin questionnaire seemed to be challenging for respondents, all of whom had CHD: more than 50% returned questionnaires which did not allow a calculation of change of their level of physical activity. The DINE seemed to be more straightforward to answer, with almost all at baseline and 80% at follow-up giving responses which allowed calculation of change in their fibre intake.
Comparison with existing literature
Details of the methods of administration, response rates and valid completion rates of lifestyle behaviour questionnaires have often not been reported clearly in previous studies of behaviour change interventions. For example, Hunt-Shanks et al. [20] investigated the effect of exercise among cardiac patients by posting Godin questionnaires, but response rates were unclear. Trinh et al. [21] examined correlates of physical activity among kidney cancer survivors by posting a survey ‘package’, including a Godin questionnaire. Completed questionnaires were returned by 42.5% (703/1,654) which is similar to our findings of the rate of valid returns: they found no differences in age, sex or surgery rate between responders and non-responders but, in contrast to our report, they did not report if any incomplete or invalid responses were received.
Rosenburg et al. [27] used postal Godin questionnaires among a disabled population and commented that the examples of exercise were inappropriate but did not report clearly details of responses received. A reason for poor return of valid scores in our study may be that participants considered that the questions were not relevant to them because of impaired mobility. Evaluation of intervention effectiveness is only meaningful if the outcome measures are appropriate to the study population which, in secondary intervention studies, often includes older and multimorbid people, with cognitive or physical impairments. Involvement of patients from the target study population in all stages of research design planning and in interpreting findings should help to ensure that relevant outcome measures are chosen and measured appropriately.
Our finding that return of valid scores is associated with higher educational attainment is in keeping with findings of Cash et al. [28] who surveyed diet and physical activity behaviours among employees who were mostly high school graduates or higher. Godin scores were reported for approximately 80% of the sample but reasons for the full sample not being included are not given or discussed.
A previous study [6] which used a self administered Godin questionnaire, sent and returned by post, in testing the effectiveness of an intervention for patients with multiple sclerosis offered a $10 incentive for a completed return. The report suggests that 90.5% (19/21) completed questionnaires successfully. Thus, the offer of an incentive may boost response rates but there needs to be recognition that an incentive which is more attractive to some participants than others may also introduce a further source of response bias in evaluating an intervention.
In keeping with our findings, good response rates have been reported for DINE in previous studies. John et al. [23] mailed a DINE questionnaire to participants in a study of a healthy eating intervention before they attended appointments with a nurse. Whilst a 95% (655/690) response rate was reported, it is unclear if the DINE was reviewed during the appointment and it was noted that participants were of a higher socioeconomic class than the UK average. Steptoe et al. [29] also used DINE to examine the effect of eating behaviour counselling but their report did not state if patients completed this in the presence of a researcher or health professional from whom they could seek advice. At 12-month review it was completed by 80% of the original participants (218/271) but no comments were made regarding invalid responses. Further detail of methods of administration and completion of outcomes in these studies would be helpful.
Strengths and limitations
The SPHERE Study involved a large sample (n = 903) with opportunity to compare participants’ responses to self report questionnaires at baseline and follow-up time points. The completeness of follow-up (98.2%) allowed comparisons between those who did and did not return valid scores in relation to many different characteristics.
Administering the questionnaires by post without giving participants an opportunity to ask advice if they had uncertainty regarding any questions may be considered to be a weakness of the original study, which limited the possibility of maximising the number of valid responses. Analysing returned questionnaires, however, without making assumptions regarding responses which were not in strict accordance with instructions reflects accurately the way in which the questions were answered without assistance and without the possible influence of observer bias.
A further strength of this study is its integration of quantitative and qualitative data. The researchers’ observations of the process of questionnaire completion confirmed how study participants struggled to understand the content and construct of some questions and to respond appropriately. Some had difficulty following the format of the forms, illustrating how completion of questionnaires requires literacy and an ability to follow instructions correctly [30] and the relevance of our findings which associate higher educational status with the return of valid responses.
Implications
Our results suggest that research study participants may have difficulty in providing appropriate responses to the Godin and DINE questionnaires when these are posted and completed without access to support, resulting in a lower valid response rate than when administered face-to-face. Having someone available to give advice is not always practical because of time, travel and financial constraints and whilst it may increase the number of meaningful responses, completeness of evaluation and confidence in conclusions regarding intervention effectiveness, it may also introduce bias due to the influence of the source of advice. However, self-completion of questionnaires, with review by a health professional or researcher and revision of responses if indicated, may reflect a pragmatic approach which is relevant to real-world clinical settings. For example, some respondents may leave boxes blank when an appropriate response would be to record a zero, indicating no relevant food consumption or activity. Careful consideration may be given to the possibility of making assumptions to interpret some responses in postal questionnaires, to allow these to be included in analyses.
The importance of reporting information about valid response rates is indicated by our findings that those with higher levels of cardiovascular risk, associated with manual occupational status and lower educational attainment, with higher cholesterol levels and with diabetes were more likely to return invalid responses. Those who are aware of their higher level of risk may have feelings of guilt about their eating and physical activity behaviours which may influence valid completion of questionnaires. However, exclusion of such individuals from analysis of the outcomes of lifestyle change interventions has implications for conclusions regarding their overall effectiveness.
Our findings also illustrate the importance of ensuring that outcome measures used in evaluating interventions are appropriate for the target population. There is an increasing focus on developing health services that are patient centred [31] and patient reported outcome measures (PROMs) can be used to achieve this aim [32]. However, whilst these can be useful in monitoring the quality of care delivery [33], potentially harmful results may arise from inappropriate methodology, relating to the design of measures chosen or the method of their administration and data collection, leading to evaluations which are applicable only to a skewed sample of the eligible population. Attention should be paid to patients’ literacy [30] and health literacy [34]. The readability of information and patients’ views are of paramount importance in developing management plans [35] and should be key determinants of the choice and design of appropriate PROMs, such as lifestyle questionnaires.
Conclusion
More detailed reporting of research methods and findings, with clear information about response and valid completion rates is important to inform the evaluation of interventions to change diet and physical activity behaviour. Close attention should be paid to the setting and context of studies which have been used to validate outcome measures chosen to evaluate healthcare interventions. Further research is required to determine which lifestyle questionnaires are most suitable and how they should be administered in order to maximise valid outcome data, particularly for populations such as older adults with chronic illness and those with socio-economic disadvantage or lower levels of education. The impact of methods of outcome measure administration and valid response rates in evaluating healthcare requires further study.
Acknowledgements
This study was supported by funding from the Centre of Excellence for Public Health (Northern Ireland), Queen’s University Belfast. The SPHERE Study was funded by the Health Research Board, Ireland and the Irish Heart Foundation. We wish to thank the SPHERE Study team for their conduct of the SPHERE Study, Edel Murphy and Claire Leathem for their support of this study and staff at all general practices involved, for their help in participant recruitment.
Footnotes
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
JC collected data, carried out the analysis and drafted the manuscript. SS and MB were involved in the conception and conduct of the SPHERE Study and in revising successive drafts of this manuscript. PG was involved in data analysis and revision of drafts. AM conceived the SPHERE Study and was involved in intellectual review of the draft manuscript. MC was involved in the conception of the SPHERE Study, conceived the current study, and shared in the analysis and writing of successive drafts of the manuscript. All authors read and approved the final manuscript.
Contributor Information
Judith A Cole, Email: jcole04@qub.ac.uk.
Paddy Gillespie, Email: paddy.gillespie@nuigalway.ie.
Susan M Smith, Email: susansmith@rcsi.ie.
Molly Byrne, Email: molly.byrne@nuigalway.ie.
Andrew W Murphy, Email: andrew.murphy@nuigalway.ie.
Margaret E Cupples, Email: m.cupples@qub.ac.uk.
References
- 1.De Backer G, Ambrosioni E, Borch-Johnsen K, Brotons C, Cifkova R, Dallongeville J, Ebrahim S, Faergeman O, Graham I, Mancia G, Cats VM, Orth-Gomer K, Perk J, Pyorala K, Rodicio JL, Sans S, Sansoy V, Sechtem U, Sibler S, Thomsen T, Wood D. European guidelines on cardiovascular disease prevention in clinical practice. Eur Heart J. 2003;24:1601–1610. doi: 10.1016/S0195-668X(03)00347-6. [DOI] [PubMed] [Google Scholar]
- 2.Ebrahim S, Taylor F, Ward K, Beswick A, Burke M, Davey SG: Multiple risk factor interventions for primary prevention of coronary heart disease.Cochrane Database Syst Rev 2011.,19(1): CD001561. doi:10.1002/14651858.CD001561.pub3 [DOI] [PubMed]
- 3.Cole JA, Smith SM, Hart N, Cupples ME. Cardiol Res Pract. 2011. Systematic review of the effect of diet and exercise lifestyle interventions in the secondary prevention of coronary heart disease. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Tudor-Locke CE, Myers AM. Challenges and opportunities for measuring physical activity in sedentary adults. Sports Med. 2001;31(2):91–100. doi: 10.2165/00007256-200131020-00002. [DOI] [PubMed] [Google Scholar]
- 5.Sallis JF, Saelens BE. Assessment of physical activity by self-report: status, limitations, and future directions. Res Q Exerc Sport. 2000;71(2):S1–S14. [PubMed] [Google Scholar]
- 6.Dlugonski D, Motl RW, McAuley E. Increasing physical activity in multiple sclerosis: replicating Internet intervention effects using objective and self-report outcomes. J Rehabil Res Dev. 2011;48(9):1129–1136. doi: 10.1682/JRRD.2010.09.0192. [DOI] [PubMed] [Google Scholar]
- 7.Warren JM, Ekelund U, Bessond H, Mezzanif A, Geladasg N, Vanhees L. Assessment of physical activity – a review of methodologies with reference to epidemiological research: a report of the exercise physiology section of the European Association of cardiovascular prevention and rehabilitation. Eur J Cardiovasc Prev Rehabil. 2010;17:127–139. doi: 10.1097/HJR.0b013e32832ed875. [DOI] [PubMed] [Google Scholar]
- 8.Kroke A, Kilpstein-Grobusch K, Voss S, Moseneder J, Thielecke F, Noack R, Boeing H. Validation of a self-administered food-frequency questionnaire administered in the European Prospective Investigation into Cancer and Nutrition (EPIC) Study: comparison of energy, protein and macronutrient intakes estimated with the doubly labelled water, urinary nitrogen and repeated 24-h dietary recall methods. Am J Clin Nutr. 1999;70:439–447. doi: 10.1093/ajcn/70.4.439. [DOI] [PubMed] [Google Scholar]
- 9.Willett WC, Sampson L, Stampfer MJ, Rosner B, Bain C, Witschi J, Hennekens CH, Speizer FE. Reproducibility and validity of a semiquantitative food frequency questionnaire. Am J Epidemiol. 1985;122(1):51–65. doi: 10.1093/oxfordjournals.aje.a114086. [DOI] [PubMed] [Google Scholar]
- 10.Cade JE, Burley DL, Warm RL, Margetts BM. Food-frequency questionnaires: a review of their design, validation and utilisation. Nutr Res Rev. 2004;17(1):5–22. doi: 10.1079/NRR200370. [DOI] [PubMed] [Google Scholar]
- 11.Molag ML, de Vries JHM, Ocke MC, Dagnelie PC, van den Brandt PA, Jansen MCJF, van Staveren WA, van’t Veer P. Design characteristics of food frequency questionnaires in relation to their validity. Am J Epidemiol. 2007;166(12):1468–1478. doi: 10.1093/aje/kwm236. [DOI] [PubMed] [Google Scholar]
- 12.Sahlqvist S, Song Y, Bull F, Adams E, Preston J, Ogilvie D. Effect of questionnaire length, personalisation and reminder type on response rate to a complex postal survey: randomised controlled trial. BMC Med Res Methodol. 2011;11:62. doi: 10.1186/1471-2288-11-62. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Murphy AW, Cupples ME, Smith SM, Byrne M, Leathem C, Byrne MC. The SPHERE Study. Secondary prevention of heart disease in general practice: protocol of a randomised controlled trial of tailored practice and patient care plans with parallel qualitative, economic and policy analyses. Curr Control Trials Cardiovasc Med. 2005;6:11. doi: 10.1186/1468-6708-6-11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Byrne M, Cupples ME, Smith SM, Leathem C, Corrigan M, Byrne MC. Development of a complex intervention for secondary prevention of coronary heart disease in primary care using the UK medical research council framework. Am J Manag Care. 2006;12(5):261–266. [PubMed] [Google Scholar]
- 15.Godin G, Shephard RJ. A simple method to assess exercise behavior in the community. Can J Appl Sport Sci. 1985;10(3):141–146. [PubMed] [Google Scholar]
- 16.Roe L, Strong C, Whiteside C, Neil A, Mant D. Dietary intervention in primary care: validity of the DINE method for diet assessment. Fam Pract. 1994;11:375–381. doi: 10.1093/fampra/11.4.375. [DOI] [PubMed] [Google Scholar]
- 17.Godin G. The godin-shephard leisure-time physical activity questionnaire (Commentary) Health Fit J Canada. 2011;4(1):2011. [Google Scholar]
- 18.Miller DJ, Freedson PS, Kline GM. Comparison of activity levels using the Caltrac accelerometer and five questionnaires. Med Sci Sports Exerc. 1994;26(3):376–382. doi: 10.1249/00005768-199403000-00016. [DOI] [PubMed] [Google Scholar]
- 19.Jacobs DR, Jr, Ainsworth BE, Hartman TJ, Leon AS. A simultaneous evaluation of 10 commonly used physical activity questionnaires. Med Sci Sports Exerc. 1993;25(1):81–91. doi: 10.1249/00005768-199301000-00012. [DOI] [PubMed] [Google Scholar]
- 20.Hunt-Shanks T, Blanchard C, Reid RD. Gender differences in cardiac patients: a longitudinal investigation of exercise, autonomic anxiety, negative affect and depression. Psychol Health Med. 2009;14(3):375–385. doi: 10.1080/13548500902866939. [DOI] [PubMed] [Google Scholar]
- 21.Trinh L, Plotnikoff RC, Rhodes RE, North S, Courneya KS. Correlates of physical activity in a population-based sample of kidney cancer survivors: an application of the theory of planned behaviour. Int J Behav Nutr Phys Act. 2012;9:96. doi: 10.1186/1479-5868-9-96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Huxley RR, Lean M, Crozier A, John JH, Neil HAW. Effect of dietary advice to increase fruit and vegetable consumption on plasma flavonol concentrations: results from a randomised controlled intervention trial. J Epidemiol Community Health. 2004;58:288–289. doi: 10.1136/jech.2003.014274. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.John JH, Ziebland S, Yudkin P, Roe LS, Neil HA, Oxford Fruit and Vegetable Study Group Effects of fruit and vegetable consumption on plasma antioxidant concentrations and blood pressure: a randomised controlled trial. Lancet. 2002;359(9322):1969–1974. doi: 10.1016/S0140-6736(02)98858-6. [DOI] [PubMed] [Google Scholar]
- 24.Murphy AW, Cupples ME, Smith SM, Byrne M, Byrne MC, Newell J. Effect of tailored practice and patient care plans on secondary prevention of heart disease in general practice: cluster randomised controlled trial. Br Med J. 2009;339:b4220. doi: 10.1136/bmj.b4220. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Cole JA, Smith SM, Hart N, Cupples ME. Do practitioners and friends support patients with coronary heart disease in lifestyle change? A qualitative study. BMC Fam Pract. 2013;14:126. doi: 10.1186/1471-2296-14-126. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Hardin JW, Hilbe JM. Generalized Estimating Equations. Boca Raton, FL: Chapman & Hall/CRC; 2003. [Google Scholar]
- 27.Rosenburg DE, Bombardier CH, Artherholt S, Jensen MP, Motl RW. Self-reported depression and physical activity in adults with mobility impairments. Arch Phys Med Rehabil. 2013;94:731–736. doi: 10.1016/j.apmr.2012.11.014. [DOI] [PubMed] [Google Scholar]
- 28.Cash SW, Beresford SAA, Henderson JA, McTiernan A, Xiao L, Wang CY, Patrick DL. Dietary and physical activity behaviours related to obesity-specific quality of life and work productivity: baseline results from a worksite trial. Br J Nutr. 2012;108(6):1134–1142. doi: 10.1017/S0007114511006258. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Steptoe A, Perkins-Porras L, McKay C, Rink E, Hilton S, Cappuccio FP. Behavioural counselling to increase consumption of fruit and vegetables in low income adults: randomised trial. Br Med J. 2003;19:326. doi: 10.1136/bmj.326.7394.855. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Al-Tayyib AA, Rogers SM, Gribble JN, Villarroel M, Turner CF. Effect of low medical literacy on health survey measurements. Am J Public Health. 2002;92(9):1478–1480. doi: 10.2105/AJPH.92.9.1478. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Valderas JM, Fitzpatrick R, Roland M. Using health status to measure NHS performance: another step into the dark for the health reform in England. BMJ Qual Saf. 2012;21(4):352–353. doi: 10.1136/bmjqs-2011-000184. [DOI] [PubMed] [Google Scholar]
- 32.Valderas JM, Kotzeva A, Espallargues M, Guyatt G, Ferrans CE, Halyard MY, Revicki DA, Symonds T, Parada A, Alonso J. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008;17(2):179–193. doi: 10.1007/s11136-007-9295-0. [DOI] [PubMed] [Google Scholar]
- 33.Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ. Routine use of patient reported outcome measures in healthcare settings. Br Med J. 2010;340(7744):464–467. doi: 10.1136/bmj.c186. [DOI] [PubMed] [Google Scholar]
- 34.Osborne RH, Batterham RW, Elsworth GR, Hawkins M, Buchbinder R. The grounded psychometric development and initial validation of the Health Literacy Questionnaire (HLQ) BMC Public Health. 2013;13:658. doi: 10.1186/1471-2458-13-658. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Leathem CS, Cupples ME, Byrne MC, Byrne M, Corrigan M, Smith SM, Murphy AW. Using the opinions of coronary heart disease patients in designing a health education booklet for use in general practice consultations. Prim Health Care Res Dev. 2009;10(3):189–199. doi: 10.1017/S1463423609001224. [DOI] [Google Scholar]