Minimal dataset for quitlines: a best practice

Abstract

Objectives

This paper discusses the development of a minimal dataset (MDS) for tobacco cessation quitlines across North America. The goal was to create a standardised instrument and protocol that would allow for comparisons and pooling of data across quitlines for evaluation and research purposes. Principles of utilisation focused evaluation were followed to achieve consensus across diverse stakeholder groups in two countries.

Methods

The North American Quitline Consortium (NAQC) assembled a working group with representatives from quitline service providers, funders, evaluators and researchers from Canada and the United States. An extensive, iterative consultation process over two years led to consensus on the evaluation domains, indicators and specific items. Descriptive information on quitline service models, data collection protocols and methodological issues were addressed.

Results

The resulting minimal dataset (MDS) includes 15 items collected from eligible callers at intake and eight items collected from smokers participating in evaluation. Recommendations for selecting evaluation participants, length of follow‐up and repeat callers were developed. Full MDS questions and technical documents are available on the NAQC website.

Conclusion

Adoption and implementation of the MDS occurred in the majority of North American quitlines by the end of 2006. Key success factors included a focus on utility and feasibility, a commitment to meeting multiple and varied needs, sensitivity to situational factors and investment in working interactively with stakeholders. The creation and implementation of a MDS across two countries is an important “first” in tobacco control which will help speed the creation of practice based evidence and facilitate practice based research.

To achieve meaningful and rapid reductions in tobacco related morbidity and mortality, we must increase both the number of individuals who attempt to quit, and the success rate of those attempts. Among the many available cessation programmes, telephone based cessation services, also known as quitlines, offer the opportunity to achieve both goals because of their broad reach and effectiveness.¹,²,³,⁴,⁵ Two Cochrane reviews have supported quitlines, particularly those that offer proactive services, as an effective population based approach to tobacco cessation.¹,² Studies have also shown quitlines to be cost effective⁶ and cost beneficial.⁷ In fact, quitlines are increasingly being recognised as a key component of comprehensive tobacco control strategies in Australia,⁸ Canada, the United Kingdom, the United States⁹,¹⁰ and the European Union.

The emergence of quitlines as a population based approach to smoking cessation has followed the classic innovation diffusion curve,¹¹ at least in North America. The first North American quitline service at a population level began in California in 1992; by 2002 there were 33 state and provincial quitlines and by 2006 all smokers in North America had access to quitline services.¹² Despite the proliferation of quitlines there were no agreed upon metrics or standards for evaluation, making it difficult for quitlines to learn from each other, and comparative studies impossible.

The first North American Conference of Smoking Cessation Quitlines was held in Arizona in 2002. Participants at that meeting asked for a common approach to evaluation. The North American Quitline Consortium (NAQC), which was established two years later to provide leadership and a unified voice for quitlines, led the initiative to create a common evaluation approach. This paper describes the development of an evaluation framework and standardised set of measures (now known as the minimal dataset or MDS) designed to inform quitline operations, facilitate comparative evaluation studies and enhance research opportunities.

Evaluation framework

All evaluations are concerned with utility, generalisability, scientific rigor and relevance. While there are many approaches to evaluation, utilisation focused evaluation is distinguished by its emphasis on utility, relevance and stakeholder involvement.¹³ What fundamentally distinguishes utilisation focused evaluation is that “the evaluator does not carry the burden for making choices about the nature, purpose, content and methods of evaluation alone. These decisions are shared by an identifiable and organised group of intended users.” Guiding principles described by Patton include active involvement of users, a commitment to meeting multiple and varied needs, a concern for utilisation as a driving force, sensitivity to situational factors affecting utilisation and investment in working interactively with stakeholders.

This framework and accompanying principles were especially relevant for the creation of a common dataset for evaluating quitlines. There were a large and diverse number of stakeholders with an interest and enthusiasm for learning what works in what settings and for whom. The rich diversity of quitline service delivery models, clients and target populations in different settings across North America made comparative studies possible via the adoption of a set of common measures.

Development process

The North American Quitline Consortium to establish a 14‐member working group including quitline researchers, service providers, evaluators and funders from across North America. Members of the working group had linkages to leading edge quitlines and key organisational affiliations (National Cancer Institute, US Centers for Disease Control and Prevention, Health Canada, NCIC Centre for Behavioural Research and Program Evaluation). From the outset, the vision was to develop a set of measures that would provide valid, standardised data on a few important indicators.

The process, illustrated in table 1, occurred over two years. The working group operated via teleconference and email, with key face to face meetings attached to other meetings and conferences, all supported by funding from NAQC sponsors, Health Canada and the Canadian Tobacco Control Research Initiative. Members of the working group provided leadership and momentum for the MDS, both internationally and within their own organisations. In addition, researchers with the European Smoking Cessation Helplines Evaluation Research (ESCHER) group of the European Network of Quitlines (ENQ) participated in key meetings.

Table 1 Chronology and timeline of minimal dataset development and implementation.

Date		Event/actions		Accomplishments
May 2002		First North American Conference of Smoking Cessation Quitlines		Identified need for:
				Organisation to provide leadership and unified voice for quitlines
				Common evaluation framework to promote shared learning
June 2003		NAQC planning meeting held in Chicago, USA		Planners agreed to begin processes to create NAQC and a standard dataset
				Agreed to address need for standard dataset as first NAQC project
September 2003		Health Canada hosts quitline meeting in Ottawa, Canada		Purpose and content of a minimal dataset discussed. Research and Evaluation (R&E) Working Group established by NAQC—joint Canadian and US leadership
November 2003 to February 2004		NAQC R&E working group met by teleconference		Extensive consultation with quitline stakeholders, tobacco control researchers
				Existing indicators and measures identified
				Draft MDS completed, stakeholder consultation
June 2004		NAQC meeting in San Diego, USA		R&E working group convened to review input from consultations
				MDS revised and only “essential” indicators included. Process for standardising optional questions developed
				NAQC formally launched
				MDS sent to stakeholders for final input
February 2005		NAQC R&E working group meeting in Phoenix, Arizona		MDS items finalised, definitions adopted
		Included representation from ESCHER team working with ENQ		Methodological issues reviewed and guidelines prepared
March 2005 to May 2005		Vermont quitline pilot tested MDS		Pilot experience shared at NAQC annual meeting
		NAQC annual meeting		MDS launched with suggested implementation deadline of September 2005
June 2005 to September 2005		NAQC prepares for September 2005 launch of MDS		MDS questions, technical support, frequently asked questions, MDS teleconferences offered by NAQC
September 2005 onwards		North American quitlines begin conversion to MDS		Voluntary implementation of MDS

ENQ, European Network of Quitlines; ESCHER, European Smoking Cessation Helplines Evaluation Research; MDS, minimal dataset; NAQC, North American Quitline Consortium.

To begin, a generic logic model was created to clarify the causal relation between quitline inputs (for example, resources, staffing models), activities (for example, promotion, counselling protocols), reach (client characteristics) and outcomes (quit attempts, quit rates). The logic model helped identify areas for indicator development. Next, existing evaluation tools were gathered from Canadian and US quitlines and formed the pool from which the selection of indicators and items was made. This respected the previous work of individual quitlines. The third step was an extensive consultation process to get agreement on the evaluation domains, to differentiate between “essential” and “important, nice to have” indicators and to select relevant intake and follow‐up questions. This process was critical to getting buy in from the diverse set of stakeholders interested in quitlines' success. By linking domains, questions and decisions, quitline funders, service providers, evaluators and researchers could see the relevance and importance of each question and the benefits of standardising questions across quitlines.

The Working Group adopted a set of guiding principles to ensure the MDS would facilitate service provision, evaluation and research, make comparisons possible and not impose undue resource burdens on quitlines. These principles were operationalised as follows:

• indicators must inform decisions important to the improvement of quitlines

• whenever possible, preservation of questions and wording already in use to allow quitlines to continue historical comparisons

• preference given to measures with acceptable reliability and validity, endorsed by scientific bodies (for example, Society For Research On Nicotine And Tobacco, SRNT), or used in national surveys (for example, census demographic questions)

• size (total number of items) of the MDS must not create barriers to meeting the needs of smokers calling for help with quitting.

After decisions on evaluation domains and question wording were approved, the working group addressed protocols for data collection, issues pertaining to length of follow‐up, repeat callers and information needed to understand the uniqueness of each quitline.

The final MDS and protocols were approved by the Research and Evaluation Working Group in February 2005. To facilitate implementation, NAQC shared the MDS with all NAQC members and interested parties by email, on the NAQC website and during the first NAQC annual meeting in May 2005. NAQC also hired a technical expert and offered assistance via conference calls, online resources and individual consultation for a six‐month period.

By the end of 2005, all quitlines reported that voluntary implementation of the MDS was planned or under way. A NAQC survey of MDS implementation is planned for 2007 to determine implementation, fidelity, areas for update and new information needs.

Minimal dataset

The minimal dataset consists of a set of 15 intake (baseline) questions asked of eligible callers at the time of their first call and eight follow‐up questions collected during follow‐up interviews seven months after intake. As per the logic model, indicators included those designed to evaluate quitline promotion activities (for example, client awareness and response to promotion strategies, previous calls to the quitline), reach (type of caller, client demographic characteristics, geographic location and tobacco behaviours), service delivery (client satisfaction) and outcomes (changes in tobacco behaviours, including quit attempts, quit rates and switching to other forms of tobacco). As with any evaluation, callers can refuse to answer questions without penalty, although almost all intake questions also informed the counselling intervention.

Three factors that have consistently been shown to predict quit success (nicotine dependence, readiness to quit, use of other quit aids) are also captured. Although many quitlines use stages of change to inform their counselling interventions, the working group did not include it in the MDS given the equivocal evidence of its' predictive validity for quit success when other factors are taken into account.¹⁴,¹⁵,¹⁶

Table 2 outlines the MDS intake and follow‐up questions, the corresponding evaluation indicators and examples of the types of decisions informed by each domain. The MDS questions and technical documents are posted on the NAQC website at www.naquitline.org/index.asp?dbid = 2&dbsection = research.

Table 2 Minimal dataset items, evaluation indicators and related decisions.

MDS question		Evaluation indicators	Informs decisions about:
Questions to be asked at the intake call (first call client speaks to quitline staff)
(1) How can I help you?		Reason for calling	Effectiveness of promotion and linkages with other components of comprehensive tobacco strategy.
Help quitting
Help staying quit
Refer someone else
General information about service
(2) Are you calling for:		Caller profile: who is calling	Measure degree to which smokers or proxy callers respond and extent to which potential referral agents are aware of quitline services.
self
someone else? Are you
health professional
friend or family member
community organisation, worksite, insurance
(3) How did you hear about the quitline?		Awareness of quitline	Effectiveness of promotion
media
other advertising
referral
(4) Is this your first call to the quitline in the past year?		Status: first time or repeat caller	Distinguish first time and repeat callers
(5) Do you currently smoke cigarettes: every day, some days, not at all?If not at all when was last time you smoked a cigarette, even a puff?		Caller profile: tobacco behaviours	Profile of the types and frequency of tobacco use among callers
(6) How many cigarettes do you smoke per day on the days you smoke?
(7) Do you currently use other tobacco products? How much other tobacco do you use per week?
(8) How soon after you wake up do you smoke your first cigarette?			Level of addiction and readiness to quit are predictors of success in quitting. These can also help tailor service delivery.
(9) Do you intend to quit within the next 30 days?
(10) Are you male or female?		Caller profile: Demographic Characteristics	Profile of who is calling the quitline.
(11) What year were you born?
(12) What is your zip (postal) code?			Allows comparison with characteristics of target population.
(13) What is the highest level of education you have completed?
(14) USA: Are you Hispanic or Latino?			Can link zip code/postal code with national census tracts and health services data.
Which of these groups (race) best describes you?
(15) Canada: To which of the following ethnic or cultural groups did your ancestors belong?			Can target promotion strategies to specific demographic, ethnic or geographic groups.
Questions to be asked at the follow‐up interview (conducted seven months after intake)
(1) Overall how satisfied were you with the service received from the quitline?		Client satisfaction	Caller satisfaction with services (useful to service providers and to inform funders)
(2) Do you currently smoke cigarettes:		Changes in tobacco behaviours	Changes in tobacco behaviours, particularly quit attempts and quit rates, provide evidence of quitline effectiveness.
Every day, some days, not at all? If not at all when was last time you smoked a cigarette, even a puff?
How many cigarettes do you smoke per day on the days you smoke?			Questions about changes in the type of tobacco used can help identify whether smokers are switching tobacco products rather than quitting.
(3) Do you currently use other tobacco products?
How much other tobacco do you use per week?			Q4 and Q5 provide information on explanatory factors, such as level of addiction and readiness to quit. This is important because quitlines serve different types of smokers.
(4) How soon after you wake up do you smoke your first cigarette?
(5) Do you intend to quit within the next 30 days?
(6) Since you first called the quitline on [date] did you quit using tobacco for 24 hours or longer?			Q6 measures quit altempts.Q7 can be used to determine length of quit as per SRNT recommendations. Current SRNT recommendations for prolonged abstinence are difficult for quitlines to implement, given nature of intervention.
Optional: When was last time you smoked a cigarette, even a puff?
(7) Have you smoked any cigarettes, even a puff, or used other tobacco in the last 30 days?
(8) Since you first called the quitline on [date] have you used anything to help you quit? (eg, medication, self help, physician, etc)		Other cessation aids	Predicts quit success.

In addition to the MDS questions, the working group recognised that accurate descriptions of quitlines would be required to understand evaluation results. Recommendations on key administrative and service data were added to the technical documents accompanying the MDS (posted at http://naquitline.org/index.asp?dbid = 3&dbsection = research).¹⁷ In 2005 NAQC revised its annual survey of North American quitlines and Cummins et al present results showing the tremendous variability in quitline models.¹⁸ These data reinforce the importance of context in evaluation. In future comparative studies, the MDS can be linked to (current) descriptive information about participating quitlines to help answer questions about the effect of different delivery models on abstinence.

There is also considerable variability in quitlines' evaluation mandates and resources. With respect to evaluation methodology, the working group concluded: “Each quitline will need to determine how follow‐up will be conducted and on which population. …[quitlines] should strive to survey enough people to draw valid conclusions about their outcomes, but it will be up to the individual quitline to determine whether census surveying, random sampling, cohort sampling, or some other sampling method will be most appropriate.”¹⁷ While not ideal from a research perspective, this approach made implementation of the MDS possible with the expectation that in future common evaluation methodologies could also be implemented.

The working group followed expert guidelines for abstinence measures,¹⁹,²⁰ recommending at minimum, six‐month follow‐up and 30‐day point prevalence. This measure was meaningful to funders and follow‐up was feasible for quitlines. Quitlines already measuring prolonged abstinence and 12‐month outcomes were encouraged to continue. NAQC also recommended that quitlines report abstinence rates using both an “intent to treat” analysis and analysis of only those who completed the evaluation, with the recognition that the true quit rate lies somewhere between the two measures. It was also determined that a one‐month intervention period would permit completion of most or all proactive counselling calls for most quitlines. Thus the working group recommended the follow‐up interviews occur seven months after the intake date. This allowed one month for the full intervention (or at least the majority of proactive calls), plus the standard six‐month follow‐up period.

The final issue tackled by the working group was how to treat repeat callers. A small percentage of callers make extensive use of quitlines and the dilemma was at what point to consider them as making a new quit attempt versus continuing to act on the original counselling intervention. After considerable debate, it was agreed that a 12‐month period from the first quitline contact should be considered a new quit attempt. Thus smokers who relapsed but called back for a second quit attempt 13 or more months after their original call to the quitline would be considered as new callers for evaluation purposes.

Discussion

The minimal dataset provides the basis for a North American laboratory or “community of practice” for research and evaluation and enables us to capitalise on the diversity across quitlines and the large numbers of callers served. Quitlines themselves represent a success story of the translation of research evidence to public health practice. The MDS represents a best practice in quitline evaluation. It is a living document and will be revised and expanded as evidence, experience and capacity allow.

A desire to keep the MDS small and operationally feasible took paramount importance in discussions. It was recognised that the MDS must be easy to implement and be respectful of quitlines' service mandate. The multi‐stakeholder working group enabled satisfactory resolution between researchers' desire for comprehensive baseline data and service providers' concerns about the time required to collect those data. As one participant noted “What made this work was willingness on the part of the researchers to balance scientific rigor with practicality, coupled with a respect for and interest in scientific integrity on the part of program and policy decision makers.”

A second key success factor was accommodating differences across quitlines. Individual quitlines each had their own evaluation questions to preserve and reasons for wanting to do so. Acknowledgement that the MDS was an adjustment to existing protocols authenticated individual quitline evaluation processes while at the same time moved the field as a whole toward a core set of items. The resulting MDS replaced some but not all pre‐existing questions.

Though time intensive, the multi‐stakeholder collaborative approach was another key success factor. As noted by one quitline service provider, “The collaborative approach was unique and progressive. By bringing together such a range of stakeholders, all perspectives were represented from the very beginning, which allowed informed and rapid input, critiquing and feedback to produce a MDS that will be relevant and manageable to implement”. Also important to success was the momentum, started by Ossip‐Klein in a presentation at the North American Conference of Smoking Cessation Quitlines and continued by the co‐chairs and members of the working group. Finally, NAQC's role in communicating and providing technical assistance to address implementation issues and support quitlines was critical.

There are several early indicators of the success of this endeavour. As of December 2006, the majority of US states and almost all Canadian provinces are implementing the MDS at some level (Bailey, personal communication). Other indicators of success include adoption of the MDS as a template for data collection by the University of Massachusetts Tobacco Treatment Specialist Training Program. The US Centers for Disease Control and Prevention asked quitlines they fund to report on the use of the MDS.⁹ Other institutions have recognised the value of standardised data and are proposing to adapt the MDS or create a similar standardised dataset (Web Assisted Tobacco Intervention Initiative; University of Rochester Medical Center; the Ontario Tobacco Research Unit; and the Association for the Treatment of Tobacco Use and Dependence).

Conclusion

The North American Quitline Consortium developed, disseminated and oversaw implementation of a minimal dataset for quitlines. This was a joint effort of multiple stakeholders (service providers, funders, evaluators, researchers) in Canada and the United States, with advice from leading research groups in North America (SRNT) and Europe (ESCHER). This achievement should be useful to other areas of tobacco control programming, as the MDS represents a best practice in tobacco control evaluation. The NAQC encourages other components of comprehensive tobacco control strategies to consider building on this experience so we can accelerate reductions in tobacco related mortality and morbidity.

What this paper adds

• Quitlines have provided a successful translation of intervention research to public health impact, with a range of models and services offered around the world. This rich diversity of quitline models has offered the potential for practice based research and evaluation; however, this opportunity has not been realised because of a lack of standardisation in measures across services and a lack of venues for multi‐stakeholder collaborations.

• The North American Quitline Consortium developed and disseminated a minimal dataset (MDS) for quitlines in North America as a joint effort of multiple stakeholders involved in quitlines in the United States and Canada (service providers, funders, evaluators and researchers) with links to leading edge quitlines and key organisations.

• This paper documents the development of the MDS, the resulting intake and follow‐up items and the decisions they help inform. The resulting MDS opens the door to research and evaluation collaborations across quitlines; the process of developing the MDS provides a model that can be replicated for other tobacco control programmes that involve complex partnerships.

Abbreviations

ENQ - European Network of Quitlines

ESCHER - European Smoking Cessation Helplines Evaluation Research

MDS - minimal dataset

NAQC - North American Quitline Consortium

SRNT - Society For Research On Nicotine And Tobacco