Table 3.

Evaluation methods and data sources for the trans-NIH ACSI project

Method/Data Source	Primary Content	Planned Coverage (actual n)
Review of secondary data
Website review	Coding of a variety of website characteristics	All sites (61)
ForeSee pre-implementation worksheets	Coding of team’s responses to pre-implementation questions	All sites (48)
ACSI data for sites generating sufficient response for model data	Satisfaction results per quarter	All sites collecting data during evaluation period: Q4 2004 (8) through Q1 2006 (42)
ACSI site-level data aggregated to NIH level	Standard custom question results Secondary analysis results	All sites using standard custom question; all sites using similar questions (varied by type of analysis)
Surveys
Initial survey	Site background Site evaluation before ACSI Reasons for joining the trans-NIH ACSI evaluation	All sites (57)
Final survey	Intermediate outcomes Longer term outcomes Trans-NIH benefits	All sites (51)
Interviews
Initial in-depth interview (primary focus: processes)	Implementation process Receipt and use of ACSI results Trans-NIH benefits	Subset of sites (14 in 2005; 6 in 2006)
Final in-depth interview (primary focus: outcomes)	Intermediate outcomes Longer term outcomes	Subset of sites (20)
Final brief interview	Benefits of ACSI use without full activities and data for full model	Subset of sites with less ACSI experience (5)
Observations		Coverage(number of meetings)
Observation of implementation and feedback meetings	How teams: Implemented ACSI Received and reacted to feedback	Sample of meeting types – implementation, initial feedback, follow-up feedback (15)
Observation of trans-NIH meetings	Attendee questions and issues Discussion topics Case studies	All trans-NIH meetings (5)
Observation of leadership team meetings	Management of trans-NIH effort Perceptions about ACSI use across sites	Biweekly meetings (all meetings during evaluation period)