Acceptability measures for evaluating smoking cessation interventions among individuals with diabetes

Abstract

Background

The literature indicates that individuals with diabetes do not easily adopt smoking cessation interventions. Given that the success of such interventions depends on patient involvement and attitudes, assessing intervention acceptability, including patient satisfaction and perceived usefulness, is crucial before implementing a smoking cessation intervention. This paper reports the preliminary validation of the satisfaction and perceived usefulness questionnaires for evaluating smoking cessation interventions among individuals with diabetes.

Study design

Validity study.

Methods

The satisfaction questionnaire contained eight statements while the perceived usefulness questionnaire had fourteen; both rated on a 5-point Likert scale. Content validation involved five tobacco cessation facilitators rating item relevance using a 4-point ordinal rating scale, suggesting improvements. The questionnaires were also translated into Maltese for local use and assessed for translation validity using a similar scale. Unanimous agreement among experts was required for item relevance and equivalence. Thirty-four individuals with type 1 or type 2 diabetes, attending a diabetes-specific smoking cessation intervention, received either the Maltese or English versions of the questionnaires. Internal consistency was measured using Cronbach's alpha.

Results

After two rounds of content validation, the experts unanimously agreed on item relevance and conceptual equivalence. Fifteen and sixteen participants completed the Maltese and English versions of the questionnaires, respectively. Both questionnaires’ versions were found to have a high internal consistency (>0.8).

Conclusions

These findings provide the initial validation of these instruments for assessing the acceptability of smoking cessation interventions among individuals with diabetes. Further validation in different settings using a larger sample is suggested.

1. What this study adds

This study reports on the initial validation of the satisfaction and perceived usefulness questionnaires for evaluating smoking cessation interventions among individuals with diabetes.

2. Implications for policy and practice

The use of these measures is recommended to assess the acceptability of a smoking cessation intervention amongst individuals with diabetes.

3. Introduction

Diabetes mellitus (DM), characterised by chronic hyperglycaemia which can lead to the development of various macro- and micro-vascular complications, is estimated to affect 537 million people worldwide [1]. While glycaemic control is key in diabetes management, to prevent the associated diabetic complications people living with DM require medical care and education that also go beyond glucose management. The cessation (and prevention) of tobacco smoking, is a crucial aspect of diabetes management. In addition to causing endothelial dysfunction and altering plasma viscosity, tobacco smoking has been found to increase insulin resistance and worsen glycaemic and lipid control in individuals with DM [2]. This increases the risk for both macro- and microvascular complications of DM. Compared to non-smokers with diabetes, both individuals with type 1 and type 2 DM who smoke are at approximately 50% higher risk of developing cardiovascular events such as coronary heart disease and stroke [3]. Additionally, a higher risk for cardiovascular mortality and for total mortality among smokers with type 1 and type 2 DM has been identified [3]. Tobacco use may also increase the likelihood of microvascular complications associated with diabetes, such as diabetic nephropathy, neuropathy and retinopathy, particularly amongst individuals with type 1 DM [2]. However, despite these risks, many individuals with diabetes continue to smoke even after diagnosis. Durlach et al. estimated that on average 20% of individuals with type 2 DM and 30% of individuals with type 1 DM smoke [2].

The provision of tailored smoking cessation interventions for individuals with diabetes has been recommended at large [2,4]. However, the literature suggests that individuals living with diabetes do not easily adopt smoking cessation interventions, and often smoking cessation success rates are low [4]. Given that the success of healthcare interventions is very much dependent on patient involvement and their attitudes to them, the assessment of patients’ views of an intervention is crucial before implementing an intervention into practice [5]. Assessing the acceptability of any smoking cessation intervention to patients, in terms of their satisfaction with the intervention and their perceptions of its usefulness, can help fine-tune interventions to improve uptake and success [5]. While the use of qualitative research, being flexible and explorative in nature, has been widely used to improve interventions before further evaluation and implementation, the additional use of quantitative methods, such as questionnaires, has also been recommended [6].

Despite the availability of valid satisfaction questionnaires, such as the widely used Client Satisfaction Questionnaire (CSQ-8) [7], and the UK National Health Service (NHS) Stop Smoking Service Client Satisfaction Survey (which also investigates perceived usefulness) [8], these were deemed inadequate to help assess the satisfaction with and perceived usefulness of a smoking cessation intervention among individuals with diabetes. This is because such tools either measure satisfaction as a broad concept, without referring to smoking cessation (e.g., the CSQ-8) [7], or are too specific, referring to a specific context/smoking cessation service (e.g., the UK National Health Service Stop Smoking Service Client Satisfaction Survey) [8]. Since no quantitative measures which evaluate satisfaction and perceived usefulness of smoking cessation interventions among individuals with diabetes have been identified [4], this study aimed to validate the content of two self-developed instruments. These instruments were designed to measure satisfaction and perceived usefulness of a smoking cessation intervention among individuals with diabetes. Additionally, this study aimed to establish the conceptual equivalence of these measures after translation into Maltese for local use. It also sought to assess the internal consistency of both versions of the instruments.

4. Methods

4.1. Questionnaires

Prior to the development of the satisfaction and perceived usefulness questionnaires, the literature, including the ePROVIDE™ platform (a centralised platform for Patient-Centered Outcomes, particularly for Clinical Outcome Assessments) [9], was screened to identify existing valid and reliable acceptability measures for evaluating smoking cessation interventions/programmes. However, only the UK NHS Smoking Service Client Satisfaction Survey [8] was identified. Both instruments were initially developed in English by JG, after consideration of the literature and the satisfaction questionnaires identified [7,8,10,11], and were reviewed by the other authors. The questionnaires were devised to evaluate diabetes-specific or general smoking cessation face-to-face interventions which may not include the provision of pharmacotherapy for smoking cessation.

The satisfaction questionnaire consisted of eight statements covering the main elements of smoking cessation interventions [8], i.e., the support received, the setting, the appointment times given, the waiting time for having the first session, the duration of each session, the frequency and the number of follow-up sessions, and the method used to help the smoker quit. Conversely, the perceived usefulness questionnaire consisted of 14 items. While the first two items were about the ability of the smoking cessation intervention in meeting the participant's expectations and his/her needs, the other 12 items were about the ability of the intervention in providing the necessary information, motivation, and behavioural skills required to quit smoking as per the Information-Motivation-Behavioural Skills (IMB) model of behaviour change [10]. Both instruments were rated by a 5-point Likert scale, ranging from (1) ‘very unsatisfied’ to (5) ‘very satisfied’ or ‘strongly disagree’ to ‘strongly agree,’ respectively. Three open-ended questions, asking participants to explain which aspects of the smoking cessation intervention they were most and least satisfied with, and for suggestions for improvement, complemented these instruments. A close-ended question (‘yes’ or ‘no’ answer) asking participants whether they would recommend the intervention to others, was also added.

Following the development of the questionnaires, two individuals with diabetes, who had attended general stop-smoking services, were asked to review the questionnaires for comprehension and appropriateness. However, no concerns were expressed.

4.2. Content validity

Basing expertise on clinical experience, all the smoking cessation facilitators within the Maltese National Health (NHS) Stop Smoking Services (n = 7, excluding JG) and a former smoking cessation facilitator who still provided ad hoc smoking cessation support services, were invited to participate in the content validation process. Using the 4-point ordinal rating scale by Lynn [12], ranging from (1) ‘not relevant’ to (4) ‘very relevant and succinct,’ they were asked to independently rate the extent to which the items and the instruments measure the concepts of interest. They were also invited to suggest additional items, item rewording/deletion, or provide comments [12]. Despite sending several reminders three facilitators did not reply. All the facilitators who replied held a Master of Science degree in a healthcare-related subject, with professional backgrounds in podiatry, nursing, and occupational therapy. One facilitator also held a Doctor of Philosophy degree. Three experts had over five years of experience in tobacco cessation services, while the other two had less than one year and three years, respectively. Two experts were also lecturers, teaching public health and research methods, and tobacco cessation and control modules, respectively, at the University of Malta. Given that there were less than six experts in the panel, a 100% agreement by all experts in rating the items as (3) ‘relevant but needs minor alteration’ or (4) ‘very relevant and succinct’ was required [12].

4.3. Translation validity

Once validated, the questionnaires were translated into Maltese and back-translated into English and compared to the original versions by bilingual translators who ensured their literal and syntactic equivalence. Minor edits were required.

To ensure that the original concepts were still being measured, the same panel of experts was asked to assess each item of the Maltese questionnaires for conceptual equivalence using the 4-point ordinal rating scale by Tang and Dixon [13], ranging from (1) ‘totally different to (4) ‘equivalent.’ Again unanimous agreement by all experts in rating the items as (3) ‘equivalent but needs minor modification,’ or (4) ‘equivalent’ was required [13].

4.4. Internal consistency

In addition to establishing content validity and equivalence of the questionnaires, both sets of questionnaires required at least the assessment of internal reliability as part of this initial validation process [14]. Given that the shorter instrument had eight items, in assuming that the coefficient of Cronbach's alpha in the null hypothesis and alternative hypothesis be equal to 0.0 and 0.7, respectively, based on an alpha value fixed at 0.05, a minimum sample size of 15 was required to achieve a power of 80.0% [15]. Therefore, a minimum of 30 participants were required for assessing the internal consistency of both versions of the questionnaires.

Between November 2022 and July 2023, a pilot study was conducted to test and refine a diabetes-specific smoking cessation intervention [16], which included the provision of Nicotine Replacement Therapy (NRT). The pilot study involved a small sample of individuals with type 1 or type 2 DM (n = 34), recruited from the diabetic clinics within the two main acute public hospitals in Malta. All the participants filled out a questionnaire on their general characteristics, and their diabetes and smoking profiles on recruitment. These were then invited to complete the satisfaction and perceived usefulness questionnaires at the end of their study period (at 12 weeks). Participants were randomly given the English or Maltese versions of these questionnaires, ensuring an equal distribution.

The mean scores (SD) and Cronbach alpha values for both sets of questionnaires were calculated. To identify any items which detracted from the overall reliability, Cronbach alpha (and scale mean) was also computed repeatedly, each time eliminating one item from the analysis. The correlation of each item with the sum of the remaining items (item-to-total correlation), was also calculated. The findings from the open-ended questions and the additional close-ended question are not reported in this paper.

5. Results

5.1. Content validity

All experts rated almost all the items from both questionnaires as relevant (3–4 ratings), suggesting minor edits. However, one expert (Expert 1) suggested the deletion of the item ‘Made you aware of severe diabetic complications caused by smoking’ in the perceived usefulness questionnaire, remarking that such information may not be needed if the client is sufficiently informed. Given that a 100% agreement was required for establishing content validity, this item was removed. Conversely, another expert suggested the addition of the following items for the same questionnaire, ‘provided you with options on how to quit smoking,’ and ‘helped you to set a specific date to quit.’ Thus, a second round of content validation was conducted. All experts found almost all items relevant, however, Expert 1 suggested the deletion of the recently added item, ‘Helped you to set a specific date to quit,’ stating that from experience most individuals do not like to set a quit date and so might not perceive its utility. Hence, this was removed. The final version of the questionnaires is available in the Supplementary file.

5.2. Translation validity

All experts unanimously agreed that the items were equivalent, providing three or four ratings. When items were rated as ‘three,’ suggestions were provided. These were then discussed with the bilingual translators and revised accordingly.

5.3. Internal consistency

Fifteen participants completed the Maltese versions of the questionnaires, while sixteen completed the English versions. Descriptive statistics of the respondents, including the demographics, diabetes, and smoking profiles (at baseline and end-of-study period) and the number of smoking cessation support sessions provided are available in the Supplementary file. The mean scores of the satisfaction and perceived usefulness questionnaires (both versions) were high, denoting that most participants were satisfied/agreed with the posed statements (Table 1). Cronbach's alpha scores were 0.87 and 0.91 for the Maltese and English versions of the satisfaction questionnaire, and 0.94 and 0.96 for the Maltese and English versions of the perceived usefulness questionnaire. On eliminating the items one at a time from the analysis, the Cronbach alpha (and scale mean) remained relatively stable (Supplementary file, Tables 2–5). All item-scale correlations were >0.4.

Table 1.

Questionnaires' scores and internal consistency assessment.

Instrument	No. of items	Total score range	n	Mean (SD)	Cronbach's alpha
Satisfaction questionnaire - English version	8	8–40	16	36.0 (3.35)	0.91
Satisfaction questionnaire - Maltese version			15	32.9 (4.22)	0.87
Perceived usefulness questionnaire - English version	14	14–70	16	62.3 (9.57)	0.96
Perceived usefulness questionnaire - Maltese version			15	56.1 (9.76)	0.94

6. Discussion

This paper reports on the initial validation of the satisfaction and perceived usefulness questionnaires for evaluating smoking cessation interventions among individuals with diabetes. Following content validation of the English questionnaire, the conceptual equivalence of the translated Maltese questionnaires was established. Both versions of the questionnaires were found to have a high internal consistency (>0.8). Furthermore, all items correlated well with the total.

These questionnaires present a better alternative to current standard satisfaction questionnaires for assessing the acceptability of smoking cessation interventions among individuals with diabetes. This satisfaction questionnaire is more specific than a general satisfaction questionnaire, e.g., the CSQ-8 [7]. Conversely, it is not as specific as the UK NHS Smoking Service Client Satisfaction Survey [8], thus allowing its use in different face-to-face settings for comparative purposes. The perceived usefulness questionnaire adds to the use of the satisfaction questionnaire. The use of the perceived usefulness questionnaire can help researchers assess acceptability further by investigating participants’ perceptions of a smoking cessation intervention in providing the necessary information, motivation and behavioural skills, which are required to quit smoking as per the Information-Motivation-Behavioural Skills (IMB) model of behaviour change [10]. The use of both questionnaires, with or without use of additional methods, such as semi-structured interviews, is thus recommended to help researchers investigate the acceptability of a smoking cessation intervention amongst individuals with diabetes to improve its uptake and success.

In this study, participants were satisfied with the intervention provided, perceiving it as useful. While the evaluation of the tested diabetes-specific smoking cessation intervention was beyond the scope of this study, the findings obtained appear promising for its potential use. Thus, further research is required to assess the feasibility of implementing this smoking cessation intervention in practice.

7. Strengths and limitations

This study's findings provide the initial validation of the satisfaction and perceived usefulness questionnaires for assessing the acceptability of diabetes-specific or general smoking cessation interventions among individuals with diabetes. However, internal reliability assessment was only conducted among a small sample of participants, the majority of whom were males with type 2 diabetes, as expected (see supplementary file) [2], who attended a diabetes-specific smoking cessation intervention. Hence, further validation, which may include assessing the questionnaires' stability over time and factor analyses, in different settings using a larger sample is suggested.

Ethical approval

Ethical approval was obtained from the Faculty of Health Sciences Research Ethics Committee on behalf of the University Research Ethics Committee of the University of Malta (UREC FORM V_15062020 8618, Date: October 5, 2022). Participants provided informed consent.

Funding

The research work disclosed in this publication is funded by the Tertiary Education Scholarships Scheme, Ministry for Education, Sport, Youth, Research and Innovation, Malta. The funders had no role in the study design, in the collection, analysis and interpretation of data, in the writing of the report, or in the decision to submit the article for publication.

Consent

Informed consent was obtained from all individual participants included in the study.

Data availability

The data supporting this research are available from the authors on reasonable request.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article.