Table 2.

Monitoring and evaluation of the CEASE intervention using the RE-AIM framework

RE-AIM Component	Process Evaluation (Level of assessment)	Monitoring to identify problems and actions to solve them	Lessons Learned
Reach	Parent level and practice level assessment • Monitor number of people screened (RedCap screener data) • Monitor the number of people watching the video embedded in the screener (Google analytics data on you tube)	• Some practices had busy front-desk or did not have families wait in waiting area so they had a drop in iPad handout numbers; those practices decided to hand out the iPads to parents to screen for household tobacco use in the examination rooms • When iPad screener numbers dropped, a video was added to the screener • When fewer views were observed, a “rate the video” question was added to screener to engage the families	• Be flexible according to the needs of the practice and parents • Keep the intervention fresh for repeat users. For example, change videos every month
Effectiveness	Parent level and practice level assessment • Parents requesting prescription for NRT and those who received it (RedCap screener data) • Parents requesting connection to the quitline and those who received the quitline enrollment form (RedCap screener data) • Monthly phone conversation with the practice leader or office manager to discuss any issues related to screener handout, medication or quitline connection (Documented on a spreadsheet)	• When a low percentage of parents receiving prescriptions was observed, CEASE staff called and/or emailed the practice to encourage clinicians to write prescriptions for all parents who requested for medication • Clinicians did not have time to sign NRT prescriptions: All practices chose to have electronic signatures on NRT patch and gum prescriptions so that nurses and medical assistants could also hand out the prescriptions • When a low percentage of parents receiving qutiline enrollment form was observed, CEASE staff called and/or emailed the practice to encourage staff to hand out the enrollment form, help parents complete the form as needed, and fax the form to the quitline using the same process as faxing referrals to other practices • Practices reported that parents were hesitant to answers calls from an unknown number. To address this concern, information was added to the screener for parents who request for a tobacco quitline form about when to expect a call from the quitline and/or how the phone number would likely appear on their caller ID (when available) • Some practices requested additional materials about the quitline to raise awareness of the services offered; additional materials were created by the CEASE team or ordered for free from the state’s tobacco control services	• Connect practices with free, existing health education materials, when possible • Use a multi-disciplinary approach to deliver care to reduce the time and effort burden on doctors
Adoption	Practice level and individual staff member level assessment • Practice-level agreement to adopt the CEASE intervention was an inclusion requirement for the study; as a result, at the practice level, 100% of the five practices adopted the intervention. • Monitor practices for new staff in need of training at the individual staff member level through routine phone calls and emails with the practice leader or office manager, documented on a spreadsheet	• When new staff members joined a practice, training in the CEASE intervention was offered; this offered training was tailored to the needs of the staff member and included telephone training calls, paid access to an online UMASS Medical school course (Basic Skills for Working with Smokers), an online American Academy of Pediatrics course (EQIPP: Eliminating Tobacco Use and Exposure to Secondhand Smoke), a training video, and a training manual	• Offer a mixture of training and education options about the intervention to suit the needs of the practice and of the individual. • Provide free or low cost training to reduce barriers to access • Provide asynchronous training options so that staff members can engage in training at a time that works best for them. • Provide access to courses that have CMEs as incentives
Implementation	Practice level assessment • Monitor staff handing out the CEASE Action Sheets to the eligible families (RedCap screener data) • Monitor staff handing out the quitline forms to the parents requesting them (RedCap screener data) • Monitor number of smokers requesting for NRT prescriptions and referral to the quitline (RedCap screener data) • Monthly phone conversation with the practice leader or office manager to discuss the intervention implementation in their office (documented on a spreadsheet) • Yearly phone meeting with the whole office staff to discuss sustainability of the intervention (documented on a spreadsheet)	• One practice had a very high Spanish speaking population and parents had trouble completing the screener: Spanish translation was added to the screener. • One practice had a high percentage of smokers who did not want to talk about harms of using tobacco; this practice chose to display a presentation about harms of tobacco smoke on the body on their television screen in the waiting rooms • Parents in a practice did not know about the quitline so did not request it: Videos were sometimes tailored based on requests from the practice staff or clinicians or based on a new quitline video available from that state’s quitline	• Seek feedback from the implementing partners like clinicians and office staff at regular intervals to improve or tailor the intervention • Use already available free media messages or resources to encourage families to quit smoking
Maintenance	Practice level assessment • Monitor staff handing out CEASE Action Sheet to repeat parents (RedCap screener data) • Monitor staff handing out quitline enrollment forms to repeat parents who requested quitline connection (RedCap screener data)	• When monitoring data revealed that practices were not handing out the CEASE Action Sheet to repeat parents, the CEASE team encouraged practice staff to hand out the CEASE Action Sheet to them • When monitoring data revealed that practices were not handing out the CEASE Action Sheet to repeat parents, the CEASE team encouraged practice staff to connect parents to the quitline • When a quitline offered free nicotine replacement therapy, this information was shared with practice staff through phone calls and emails to motivate referral to the quitline. The information was also shared in real-time with the parents through the RedCap screener. • The CEASE staff sent data about quitline connection to four practices in order to show practice staff the outcome of their quitline referral activities	• To promote the sustained adoption of an intervention, encourage each practice to have a trained local tobacco control champion • Motivate practice staff to refer repeat parents to the quitline by informing them of relevant external health promotion programs, such as free NRT from a quitline