Table 2.

Characterization of behavioral intervention technology implementation outcomes.

Outcome and definition [23]		Level of analysis in BITa studies	Measurement objective and process	Example of BIT outcome measurement
Acceptability
	Perception among stakeholders that a given evidence-based practice is useful or satisfactory	Individual provider, consumer, or administrator	Objective: assessment of the extent to which BIT aligns with expectations of an agreeable user experience Process: survey, interview, focus group, and direct observation usability testing	Mares et al (2016): qualitative methods used to assess initial consumer and provider expectations [33] Milward et al (2017): focus groups assessed the extent to which features were acceptable in terms of content, features, and design [34]
Adoption
	Intention, decision, or initiation to use an evidence-based practice	Individual provider, consumer, or administrator	Objective: assessment of actual system use or behavioral intention to use Process: passive data collection of BIT use [35]	Gilbody et al (2016): to measure consumer-level adoption, log-in records were used to identify number of participants who accessed programs [22]
Appropriateness
	Perceived fit, relevance, or compatibility of the evidence-based practice to a given context	Individual provider, consumer, or administrator Organization	Objective: assessment of perceived BIT fit with the context Process: survey, interview, focus group, direct observation usability testing, and workflow studies	Lyon et al (2016): evaluated school-based practitioner workflows and current technology use practices to determine the appropriateness of a digital measurement feedback system and identified areas for BIT redesign [36]
Feasibility
	Extent to which an evidence-based practice can be successfully used or conducted within a given context	Individual provider, consumer, or administrator Organization	Objective: in vivo assessment of the extent to which a BIT can be used by consumers or providers in a specific setting Process: Passive data collection of BIT use, survey, and structured observation studies	Kumar et al (2018): using program use data collected via BIT, the feasibility of implementing a mobile app for consumers and a provider-facing dashboard was tested in 4 outpatient clinics [37]
Fidelity
	Extent to which implementation results in an evidence-based practice being delivered as intended	Individual consumer or provider Organization	Objective: measuring adherence, dose, or quality of BIT use with respect to the developer’s intentions for use Process: passive data collection of BIT use	Calear et al (2013): reported high adherence associated with improved clinical outcomes in a hybrid implementation effectiveness study [7] Sineath et al (2017): developed and tested a fidelity protocol for a diet and lifestyle monitoring BIT that involved coaching [38]
Implementation cost
	Costs associated with implementing an evidence-based practice	Organization	Objective: intervention development costs, maintenance and versioning costs, implementation strategy costs, and operational costs Process: cost analysis, interviews, and budgetary or administrative databases	Quanbeck al (2018): measured implementation strategy costs for implementation coaching time and site visits needed to help 3 organizations integrate BIT into practice [39]
Penetration
	The integration of an evidence-based practice within a service setting (organization) and its subsystems	Organization	Objective: measuring the number of consumers or providers using BITs among those eligible or trained to engage in BIT Process: passive data collection of BIT use and electronic health record data	Titov et al (2015): measured the proportion of individuals in a defined consumer population completing lessons in 4 different BITs [40]
Sustainability
	The extent to which a newly implemented evidence-based practice is maintained or institutionalized within a service setting’s ongoing, stable operations	Administrators Organization	Objectives: measuring ongoing BIT use, change in funding streams, saturation within the organization, and inclusion in routine reports Process: passive data collection of BIT use, administrative or budgetary databases, oversight committee reports, and policy and training documents	Carlfjord et al (2013): measured continued BIT delivery after active implementation [32] Quanbeck et al (2018): measured whether or not the health system continued to offer BIT after research funding for an implementation trial ended [39]

^aBIT: behavioral intervention technology.