Table 4.

Qualities, category, and number of studies qualities were reported in

Group of quality	Quality	Category	Number of studies reported in	Fidelity studies	Engagement studies
Psychometric qualities
Use of multiple researchers	Coding	R	11	20,26,27,29,33,34,45,51,58,64	47
	Data collection		3	6,29,31
	Develop measures		3	14,26,60
	Data analysis		2	10,42
	Data entry		1	26
	Validate coding frame		1	26
Validity of measures	Validated	V	9	21,22,34,48,51	4,17,25,51
	Not validated		8	2,10,34,35,41,42,50	13
Use of independent researchers	Used – coding	R	12	20,22,26,27,29,34,38,45,51,55,63,64
	Not used – coding		1	58
	Used – develop measures		1	14
	Used – analysis		1	42
	Not used	V	1	20
Measurement of conditions	All conditions (result output)	V	8	7,50	4,13,17,18,51,53
	All conditions (reported)		5	2,48,51	2,3,35
	Intervention only		3	2,24	24,25
Reliability of measures	Reliable	R	6	21,22,48	4,17,51
	Not reliable		5	2,14,23,34,50	2,23
Random selection of data	Randomly selected	V	9	31,40,51,55,57,58,63,64	52 (data entry)
	Not randomly selected		2	45,48
Reporting of inter‐rater agreement	Reported – high	R	3	26,59	17
	Not reported		2	29,33
	Reported – poor to fair		2	27,58
	Reported – fair to excellent		1	58
	Reported – no coder drift		1	26
Coding of sessions	A percentage	V	7	33,45,51,55,57,58,63
	All		1	27
Calculated inter‐rater agreement		R	8	20,26,27,29,33,58,59	17
Use of experts	Coding	V	5	10,21,22,36,38
	Develop measures		1	27
	Not used – coding		1	27
	Checked % of data input	R	1	10
Blinding	Coders	V	3	7,26,48
	Not blinded		2	2	52
	Researchers		1		15
	Participants		1	2
Measurement of content of intervention	Some aspects of intervention	V	3	20,38	36,38
	All aspects of intervention		2	33,63
Problems with scoring criteria	Scoring criteria not sensitive	V	2	20,26
	No success cut‐off point		1	14
	Dichotomized responses reduce variability		1		25
	Measures may capture different aspects of fidelity		1	26
Standardization of procedure	Script	V	2	34,66
	Data entry		1		52
	Coding guidelines		1	64
	Not used standardized procedure		1	33
	Not used standardized measure		1		52
Self‐report bias		V	4	10,26,26,30
		R	2	5	4
Sampling	Across all providers	V	2	27,45
	Across all sites		1	10
	Across all sites (purposively)		1	33
	Across all participants		1	27
	Balanced facilitator and gender (purposively)		1	26
Audit	Data collection	R	1	6
	Data analysis		1		6
	Coding		1	20	20
	Data entry	V	1	23
	Recordings		1	40
Missing responses	Missing responses	V	1		15
Trained researchers	Trained coders	V	3	7,27,58
	Trained researcher (data collection)		1		52
Observation effects		V	4	22,26,27,34
Use of one researcher	Coding	R	1	38
	Trained observers		1	34
Revised coding guidelines		R	3	20,26,48
		V	1	33
Team meetings		R	4	1,6,23,36	23
Recording of sessions	All sessions	V	2	40,55
	% of sessions		1	35
Triangulation	Method	V	2	34,42
	Researcher		1	42
Problems with analysis plan	Did not control for provider	V	1	36
	Missing responses excluded		1	10
Social desirability		V	3	22	13,52
Objective verification		V	2		15,43
		R	1		12
Used coding guidelines		R	2	20,27
Analysis consideration – coded missing responses as no adherence		V	1		15
Independently validated coding frame		V	1	26
Measurement differences – observation and self‐report		V	1	26
Measurement period – year after intervention		V	1		25
Piloted coding guidelines		V	1	26
Practice period before recording		V	1	27
Pre‐specified dates for recordings		V	1	27
Statistician involved in sampling (stratified)		V	1	10
Training before recording may overestimate adherence		V	1	58
Piloted measure		V	1	34
Provided a reason for inter‐rater agreement		R	1	27
Supervision		R	1	58
Measures were internally consistent indicating content validity		R+V	1	27
Implementation qualities
Resource challenges	Time restrictions	P	4	5,20,27,62
	Technical difficulties	P	3	5,5,58
	Financial restrictions	P	2	5,27
	Sharing Dictaphones	P	1	45
Providers’ attitudes	Dislike paperwork	A	1	10
	Fear of discouraging participants	A	1	27
	Nerves	A	1	27
	Report participants behaving differently	A	1	27
	Positive attitudes	A	1	42
	Additional work	A	1	62
	Not enthusiastic	A	1	62
Measurement of content of intervention	Telephone calls not assessed due to difficulty	P	1	38
	Measure cannot capture non‐verbal data	P	1	20
Problems with documentation	No record of responses	P	2	10,58
	Providers did not document everything		1	10
	No record of refusals	A+P	1	27
Missing responses	Missing responses	P	1	10,10 (different aspects)
Problems with sampling	Low recruitment	P	1	60
Problems with analysis plan	Analysis not feasible	P	1	10
Incentives	Incentives used	P	2		15,52
	Incentives required	P	1	62
Feedback to providers		P	2	21,27
Feedback delay		P	1	38
Forgetting to return data		P	1		15
Logbook showed that not all steps were applied		P	1	42
Paper and digital version of measures given		P	1		5
Need simpler coding guidelines to achieve agreement		P	1	27
Reviewed fidelity after trial		P	1	45
Participants – dislike paperwork		A	1		15
Did not do a cost analysis		C	1		13
Cost of materials		C	1		37
Both psychometric and implementation qualities
Problems with scoring criteria	Lack of clarity on items	V+P	1		25
Missing responses	Missing responses	V+P	1	58
Use of one researcher	Data collection	R+P	2	5	52
Problems with sampling	Selection bias	V+A	1	2	2
	Not randomly selected	V+P	1	27

This table is ordered by the number of studies that reported a quality that fits into the ‘group of quality’ column (e.g., ‘use of multiple researchers’). Most frequent → Least frequent. The numbers in this table will not add up to the total number of studies included, as some studies included information on multiple qualities.

R = reliability; V = validity; A = acceptability; P = practicality; C = cost.