Empowering Patients and Community Online: Evaluation of the AIDS Community Information Outreach Program

Abstract

The AIDS Community Information Outreach Program (ACIOP) was created in 1994 to assist the affected community in utilizing electronic HIV/AIDS information resources. Nearly 300 competitive awards have been made to mostly community-based organizations. A formal evaluation was undertaken to determine the performance and impact of the ACIOP. A mixed methods design combined quantitative abstractions and summarization of 47 awardee final reports from 44 organizations, and qualitative telephone interviews with 17 individuals representing 20 projects. Findings revealed that project objectives were mostly met; high-risk populations were reached; low resource organizations were funded; community partnerships were significant; projects built on existing efforts; information resources and training were tailored to local needs; and most projects overcame barriers experienced. Needed modifications centered on: 1) enhancing evaluation capacity at the individual project level and 2) revising project reporting requirements to increase the amount of information available to assess the ACIOP; both have been implemented.

1. INTRODUCTION

The National Library of Medicine's (NLM) core mission includes a vigorous outreach program to help assure that its users, including researchers, health professionals, and the lay public, are aware of and make effective use of health information resources available from NLM [17]. This goal is rooted in the conviction that improved heath information access and use provides a foundation for healthier behaviors and outcomes. For the HIV/AIDS community, this explicitly includes patients, their caregivers, and the affected community. The NLM's AIDS Community Information Outreach Program (ACIOP), initiated in 1994, is the principal vehicle for reaching the HIV/AIDS community. A comprehensive evaluation of the ACIOP's performance, impact and continued relevance was conducted by the NLM and Columbia University in 2011–2012. The purpose of this paper is to report on key findings and recommendations from this evaluation.

The ACIOP was launched at a critical time in the AIDS epidemic: newly effective but complex multi-drug treatments were emerging rapidly but an HIV diagnosis still predicted a shortened life span. At the same time, new information and computer technologies (ICTs) – notably the Internet and the World Wide Web – were also emerging and they foreshadowed a new era in patient empowerment. Patients could now access rapidly the latest authoritative research and treatment information that included complex dosing protocols to which strict adherence was required.

These twin developments were brought together in a landmark conference [10] sponsored and convened by NLM and the NIH Office of AIDS Research (OAR) in 1993. There it became apparent how quickly the HIV/AIDS academic research and clinical communities were embracing the Internet. What was also apparent was the extent to which the affected community, and the organizations and individuals trying to care for them, were similarly motivated to use the new ICTs, but they faced considerable barriers – organizational, financial, and logistical. Moreover, while it was clear that many patients and their caregivers were already using the new technologies, many others in the community, particularly members of underserved populations, were unaware of the ICT revolution that was taking place around them [10]. Privacy in seeking AIDS information was paramount, because of the stigma associated with the disease. Internet users, therefore, frequently took advantage of the privacy-protecting computer terminals of a public library that provided free access to the Internet.

NLM initiated the ACIOP in response to the urgent need identified at the 1993 Conference by the HIV/AIDS affected community to reduce the identified barriers to Internet access and a growing wealth of online information resources made available by NLM. This included content from the National Institutes of Health (NIH), Federal Drug Administration (FDA), Centers for Disease Control and Prevention (CDC), the World Health Organization (WHO), national health research and services agencies globally, and leading grass-roots efforts by community-based organizations (CBOs) throughout the US.

The other stakeholders at the Conference – researchers, clinicians, and the scientific and public news media – strongly supported the response from NLM to this call to action. That NLM did accept the challenge to help remedy this situation of uneven awareness and use of increasingly important printed – and especially newly accessible electronic – HIV/AIDS information resources was recognized as a bold but logical extension of NLM’s new focus on outreach [18].

NLM responded first with a policy change that made access to all NLM HIV/AIDS information available to everyone free of charge over the Internet. This marked a significant departure from previous practice in which NLM charged for searching on all of its databases. This change did have a strong impact on underserved populations that previously could not afford access to these resources.

The ACIOP itself was structured to enable CBOs to succeed in competing for small contract awards that did not require the level of staff expertise and supporting infrastructure usually associated with larger organizations skilled in the practice of navigating the traditional award-making process. This was a relatively risky undertaking by a federal agency, but the critical health circumstances of the day and the considerable value of the HIV/AIDS information that NLM had to offer, seemed to justify the risk. Taking this step opened up a funding opportunity to a large number of organizations serving minority and underrepresented groups.

The ACIOP was designed and implemented to allow applicants for funding a relatively wide latitude in the range of projects that could be supported. There were four categories: 1) Improving organizational and community access to electronic HIV/AIDS information (e.g., creating a computer laboratory in a CBO); 2) Providing training to develop information access skills (e.g., holding classes in both Internet use and finding quality information on it); 3) Developing specific educational materials targeting local needs (e.g., creating "translations" of new research findings suitable for low-literacy clients); and 4) Providing access to HIV/AIDS-related documents (e.g., inter-library loans, purchasing posters).

Almost 300 projects have been funded in the 19 years since the start of the ACIOP. Awardees have included secular and faith-based CBOs, patient advocacy groups, clinics, public and health sciences libraries, public health departments, and academic institutions working in partnership with community groups.

Several internal evaluations have been performed by NLM over the years that resulted in modest shifts in the ACIOP's emphasis and application procedures. However, in the face of significant changes in information and communication technologies in recent years (e.g., social networking, mobile communications devices) [3,6], along with new HIV/AIDS research findings leading to revised treatment and prevention policies [4], changing demographics [22], etc., along with the prospect of tightened government budgets in coming years, it was deemed appropriate to mount a new and comprehensive evaluation of the ACIOP.

The purpose of the evaluation as defined by NLM was to determine the ACIOP’s 1) performance, 2) impact, 3) continued relevance as presently structured given the current status of the epidemic, new tools available, decreasing stigmatization of the disease, and current needs of the affected community, and 4) needed program modifications.

NLM engaged a team from Columbia University with experience in health information outreach and specifically with HIV/AIDS information outreach to develop a suitable evaluation plan and to implement it with a purposive sample of past and current ACIOP awardees. The Columbia team interacted with the NLM/ACIOP team by means of bi-weekly conference calls from the initiation of the engagement and the selection of awards to be included in the study to the development of this paper.

2. METHODS

2.1 Evaluation Study Design

The mixed methods evaluation design combined a) quantitative abstraction and summarization of the selected awardee final reports that were scanned by NLM from its archives; and b) qualitative interviews with principals from the selected projects. Data collection and analysis were informed by the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, Maintenance) framework [1,7,8]:

• ▪Reach – the absolute number, proportion, and representativeness of participants in a given program (e.g., number of participants in ACIOP project by category)
• ▪Efficacy/Effectiveness - the impact of the program on important outcomes (e.g., ACIOP project goal achievement)
• ▪Adoption - the absolute number, proportion, and representativeness of settings willing to offer a program (e.g., number of ACIOP projects led by low resource, community-based organizations)
• ▪Implementation – consistency of delivery of program components and predisposing and enabling factors (e.g., ACIOP projects’ technology infrastructure, project management strategies)
• ▪Maintenance - the extent to which a program or policy becomes part of routine organizational practices/policies (e.g., number of ACIOP projects maintained with or without additional resources)

RE-AIM was selected because of its public health perspective and emphasis on looking beyond efficacy/effectiveness to identify other significant findings of relevance to evaluating the ACIOP. Each of the RE-AIM dimensions was associated with variable names and data sources that relate specifically to content that could be expected to be found in the awardee final reports and/or in the qualitative interviews. Appendix A (online only) contains a comprehensive summary containing all variable names, methods of analysis, and data sources. To align with NLM's specific objectives for the evaluation, the findings were condensed into four overarching evaluation components: 1) performance (Implementation, Maintenance); 2) impact (Reach, Efficacy/Effectiveness, Adoption); 3) relevance to today's needs as structured, and 4) needed modifications. Only the first two were informed by the RE-AIM model.

2.2 Sample

2.2.1 Reports

A Columbia team member purposively selected 47 awardee final reports out of 294 projects to represent the population of awardees in terms of geographical region, type of awardee organization, and type of award (Standard vs. Express). Standard awards are longer in length and receive more funding ($25K to $60K) in contrast to Express Awards which are one year in length and receive up to $10K. The reports reflected projects funded from 1994–2010 with the majority (70%) of reports abstracted from the last decade. Two reports were selected for three of 44 awardee organizations to facilitate examination of activities over time from organizations that had received multiple ACIOP awards. Forty reports were from organizations that had received Standard Awards and 7 were from organizations that had received Express Awards.

2.2.2 Interviews

The target sample for interviews was individuals from organizations whose awardee final report was selected for inclusion in the report abstraction sample. An ACIOP team member contacted each awardee organization, identified an appropriate individual for interview, assessed the individual's willingness to participate in an interview, gathered contact information, and informed the awardee representative that a member of the Columbia evaluation team would be in contact for interview scheduling. A member of the Columbia evaluation team called or emailed all individuals who agreed to be interviewed. Seventeen individuals representing 20 projects participated in interviews that were recorded, transcribed, and coded. Due to staff turnover for some of the older projects, no one in the organization had sufficient familiarity with the project to be interviewed, but the reports from those projects were included in the quantitative analysis.

2.3 Evaluation Instruments

2.3.1 Reports

Report abstraction variables were initially developed and defined by associating four dimensions of the RE-AIM framework (Reach, Efficacy/Effectiveness, Adoption, Implementation) with the aspects of the ACIOP Awardee Final Report Template that NLM provided to awardees, The Final Report Template changed over the life of the ACIOP; the open-ended questions in the most recent Template are displayed in Table 1. The complete Awardee Final Report Template is in Appendix B (online only) and the report abstraction process and variable definitions can be found in Appendix C (online only). For example, Reach addresses the absolute number, proportion, and representativeness of participants in a given project and the ACIOP Awardee Final Report Template has a required table that delineates the primary and secondary direct beneficiaries of the project. As another example, one question related to the Implementation dimension in RE-AIM has to do with ACIOP project components being delivered as intended and the ACIOP Awardee Final Report Template indicates that awardees should address whether or not their planned objectives were achieved.

Table 1.

ACIOP Awardee Final Report Template Open-ended Questions

Describe your organization Goals and objectives of project Key milestones and dates the objectives were accomplished Did you accomplish your goals? If not, why not? Were there unanticipated accomplishments? What do you consider to be your most significant accomplishments? Services developed or expanded (e.g. collection development, online searching, reference services, web access, training, publication, etc.) Novel features of the services (if any) Quantity and quality of services provided How services were promoted (Please send NLM copies of all promotional materials developed AND all publicity received such as video clips from TV news reports, articles in newspapers or newsletters, photographs, etc.) Effectiveness of promotion Target populations (indicate if different for each service) Partnerships/collaborations (originally proposed and others) What was accomplished from partnerships and collaborations Problems and barriers encountered Define “Impact” as it relates to HIV/AIDS Information Access Describe examples of how you know there was an impact based on this project Lessons learned Future plans

2.3.2 Interviews

The semi-structured interview guide (Table 2) was developed based upon Efficacy/ Effectiveness, Adoption, Implementation, and Maintenance dimensions. The heaviest emphasis was on processes associated with Adoption and Implementation to facilitate gathering of insights and lessons learned in regards to barriers and enabling factors that might not be captured in awardee reports. In addition, interviewees were explicitly queried regarding unintended positive and negative consequences of their projects.

Table 2.

Semi-structured Interview Guide

We understand that the primary focus of your project was …. and the target populations were…Please share some of your general thoughts about your project. (Warm Up Question) How was your project consistent or inconsistent with your organization’s mission, values and priorities? (RE-AIM Adoption) How did you implement your ACIOP project (Probes – collaborations, project leadership, management, and staffing; marketing)? (RE-AIM Implementation). How did you facilitate the use of the services or resources developed in your project (Probes – marketing, training, coaches)? (RE-AIM Adoption and Implementation [Enabling Factors]) What were some of the challenges to use of the services or resources developed in your project (Probes – inconvenience, technical, staff turnover)? (RE-AIM Adoption and Implementation [Predisposing Factors]) How did you overcome these barriers (Probes related to barriers identified in question 4)? (RE-AIM Adoption and Implementation [Enabling Factors]) How did the services or resources you created in your project increase access to HIV information? (RE-AIM Effectiveness) What were some of the unintended positive (i.e., serendipitous) or negative consequences of your project? (RE-AIM Efficacy/Effectiveness) Once the funding for the project was over, how did you continue to support use of your information service or resources? (RE-AIM Maintenance [Reinforcing Factors]) Based on your experiences, what would you tell other organizations about what is needed to implement an NLM AIDS Community Information Outreach Project? Are there any other thoughts about your project that you did not get a chance to share that you would like to share now? (Clean Up Question)

2.4 Data Collection Procedures

After familiarization with the abstraction tool (Appendix C, online only) and abstraction of several reports to assess consistency of abstraction, two Columbia team members abstracted variables from the 47 project reports and entered data into the report abstraction database. Two Columbia team members conducted interviews with 17 individuals representing 20 awardees using the semi-structured interview guide. Seventeen interviews were audiorecorded and professionally transcribed to create verbatim transcripts.

2.5 Data Analysis

The quantitative report abstraction data were summarized with descriptive statistics and organized according to dimensions of the RE-AIM framework.

The development of the initial coding framework for the qualitative interview data started with four of the five RE-AIM dimensions (Efficacy/Effectiveness, Adoption, Implementation, Maintenance). Reach was not included because it is primarily a quantitative assessment that was expected to be found in the report abstraction process. Twelve of 14 themes associated with the four dimensions were developed from questions associated with the original RE-AIM framework [7,8] or with its extension by Bakken and Ruland [1]. Two themes –Community Involvement and Marketing Strategies – were added to Adoption based upon their relevance to the ACIOP and organizational literature that suggested their importance. Three Columbia team members coded individual awardee interviews. Half of the interviews were reviewed by a second coder to ensure consistent application of the initial coding framework.

After the qualitative interview data were coded into 12 of 14 themes from the initial coding framework, four team members inductively generated 51 categories within the themes (details are available in Appendix D, online only) and coded interview data into 51 categories and three themes that did not have associated categories. Differences were resolved by discussion to achieve consensus.

When both awardee report quantitative abstraction data and awardee qualitative interview data were available for RE-AIM dimensions, the data were triangulated to compare and contrast findings. The triangulated findings were then condensed into four overarching evaluation components in accord with the NLM's areas of interest for the evaluation: 1) performance (Implementation, Maintenance); 2) impact (Reach, Efficacy/Effectiveness, Adoption); 3) relevance; and 4) needed modifications in the ACIOP.

3. FINDINGS

Key evaluation findings are summarized in the following paragraphs according to Performance, Impact, and Relevance. Illustrative interviewee comments are integrated into the findings. Additional examples of interviewee comments for each theme and category are in Appendix E (online only).

3.1 Performance

3.1.1 Project Objectives Were Achieved (RE-AIM Implementation)

According to awardee reports, more than 75% of the projects addressed each of first three project categories in the ACIOP solicitation: improving access to electronic HIV/AIDS information, training and skills development, and developing educational materials. Only about one-third focused on accessing HIV/AIDS related documents.

Of 194 planned objectives in the 47 projects, 164 (84.5%) were achieved with no more than minimal variation. Fifteen projects (31.9%) had one or more objectives that were not achieved as intended. Only one project in the sample did not achieve any objectives. Among the 30 planned objectives that were not achieved, objectives related to evaluation and training were reported most frequently with 9 instances each. Unmet objectives related to development of educational materials were reported 8 times. Other unmet objectives related to improving access to electronic HIV/AIDS information or increasing awareness of HIV/AIDS information.

3.1.2 Projects Built Upon Existing Efforts (RE-AIM Implementation)

Interviewees also identified the importance of building upon existing efforts. For example, an interviewee from a public library whose project focused on HIV outreach and awareness noted that “So we have Healthlink, but we also have a program called the New Americans Program, and we also have our Adult Learning Center. All of them are critical in implementing a program on HIV/AIDS outreach and awareness when you need to put into different languages.” Interviewees from CBOs also emphasized this as indicated in a statement from an organization that received multiple ACIOP awards “…because it layered on existing efforts, we were able to include it in a number of ways” and “The way when we originally wrote the first proposal for the resource room, …it was a part of our hotline.”

3.1.3 Most Projects Overcame Barriers (RE-AIM Implementation)

Awardee barriers identified were of two types: 1) barriers internal to an awardee's organization, which were the most frequently reported; and 2) barriers external to the awardee, which were typically associated with the target populations served. Barriers were identified in 38 (80.9%) reports (Table 3). Three categories of awardee barriers occurred in at least one-fourth of reports: changes in staff, technology infrastructure, and other which included partnership issues particularly in regards to recruitment of participants. The interview data confirmed awardee barriers identified in reports related to staffing, recruitment, technology infrastructure, budget, and project management. The interview data revealed additional barriers including evaluation capacity, lack of expertise in disseminating sensitive information, mismatch between user needs and project approaches, matching resource to clients in crisis, client privacy/confidentiality, and lack of logistical support for client participation. Exemplary quotes from the interviews are displayed in Table 3.

Table 3.

Organizational Barriers Identified in Reports and Awardee Interviews

Type of Barrier	Reports N (%)	Interviews
Identified in Reports and Interviews
Other (e.g., recruitment, retention, changes in partnering organization that affected primary organization’s ability to achieve objectives)	29(59.2%)	-“Well, I think one of the earlier challenges was actually recruiting people to come to workshops… But now that we really invite organizations, it’s a little bit more stable.” (Standard Award Inner City Academic Institution) -“…people were interested but to get them to take that next step to call in, to listen to our webinars, all of that, really, I found to be one of the major challenges.” (Standard Award Urban CBO) -“the clinic’s budget was cut, so we didn’t have as many physicians seeing patients during the course of the project (which affected number of patients who could be recruited).” (Standard Award Urban Academic Institution)
Staff change/turnover	14(28.6%)	-“Staff turnover is a huge issue. Sometimes it’s literally people moving between organizations, but I do think that it’s high burnout working in this field and so people come and may stay for a few years and look for other work.” (Standard Award Inner City CBO) -“Here in Kentucky we had turnover among the social work staff. And as I said, they lost two internal medicine physicians and one infectious disease doc position during the course of the project.” (Standard Award Academic Institution)
Technology infrastructure	13(26.5%)	“… the biggest challenge is you’re all set, you’re ready to go, you’ve got everything lined up you go to do a training and the power goes out.” (Standard Award CBO serving urban and rural constituencies) “Computer technology availability, so you can’t – at least you couldn’t then – get into Second Life on just any low-end computer.” (Standard Award Library serving urban, suburban, and rural constituencies) [Note: “Second Life” is an online virtual world.] “And then, you know definitely challenges with like equipment working and like some organizations blocked YouTube or they blocked PDFs. And you know it’s always like trying to get the IT people to come in and really pay attention. And we’ve had maybe three computers stolen over the years. Yes, but you know, now we lock them down so hard, we can hardly remove them when they need to be repaired.” (Standard Award Inner City Academic Institution)
Inadequate staff	10(20.4%)	-“So, I think some of our challenge is that while we have created this access, it’s fairly limited because we’re only able to staff it three days a week. We don’t yet have late evening or weekend time. So that’s a challenge. And most of that is simply around being able to fund the human resources and the time we need for that.” (Standard Award Urban CBO) -“ But going out to do, even though the community, for example, even you know going out to the community and doing the HIV training for our daycares was good, it took me away from the center. And I wasn’t able to be here with the clients.” (Standard Award Urban CBO)
Inadequate budget	9 (18.4%)	-“I don't think it's realistic to be able to manage and implement a program, certainly if it's new, with $60,000.” (Standard Award Inner City Academic Institution) -“I mean we don’t have the capacity to probably do as much as we could, or as much as people would want us to do. And so that’s one reason we don’t really market that extensively.” (Standard Award Inner City Academic Institution)
Identified in Reports Only
Change in leadership	3 (6.1%)
Identified in Interviews Only
Inadequate project management		-“So I think the management side of things is always the more challenging. It’s a lot of effort organizing any kind of program…you know that…then just trying to keep momentum going.” (Express Award Inner City CBO) -“You know, you just get it going. You find out you got funded, you get it going, and then you’re actually going to have, I think, a smaller window to do the work and allow for any contingencies.” (Standard Award CBO serving urban and rural constituencies)
Lack of evaluation capacity		-“I think another challenge in all of these projects is the evaluation piece, which is what you’re grappling with also, like what are the outcomes that are realistic and pertinent to NLM. We have here a lot of different outcomes we’d like to see and they’re hard to measure.” (Express Award Inner City CBO) -“So, I mean the weakness of our work, and I’m sure it’s hard for everybody, is…and my mostly focus is on trainings, is once you have a workshop or a discussion and if you don’t see the people again, you don’t really know how they’re using the information.” (Standard Award Inner City Academic Institution)
Lack of expertise in disseminating sensitive information		-“we had to try to figure out a way in which to really talk about the issue without scaring people. … it really was more of an issue of how do we even promote the fact that we’re offering these workshops and offering the materials in the library setting around HIV and AIDS. So it was a…we needed a whole different type of professional and a whole different type of community coordination around that.” (Express Award Urban Public Library) -“I don’t think we were honestly organizationally at that point very adept at creating accessible tools for survivors themselves, just the language we would use, the amount of detail. You know our ability to translate what we knew to something that was useful to survivors was limited at that time. And we were, I mean I still think it’s a struggle, but we’re much more aware of the kinds of things we would need to think about before we would put out something that was sort of public than we used to be.” (Standard Award Inner City CBO) -“We didn’t think the camaraderie and the soft skills would be so critical. So, the initial issues was with our IT staff, you know, because they’re very IT. …we realized quickly that they were not the people to be providing technical support to members. That just wasn’t their skills… So, that was one challenge, but once we pulled that back and really got the staff from the hotline, and then also the AmeriCorps people in place, they addressed that pretty quickly.” (Express Award Urban Public Library)
Mismatch between user needs and project approach		-“They’re calling things up on their phone. They’re not at their computer often. You know when they’re searching for information; it’s from a handheld in their office and so on and so forth.” (Express Award Urban CBO) -“And the other thing is that people’s lives are such that trying to do something on a scheduled basis was somewhat of a challenge. You know the telephone conferences were actually at a set time on a set day.” (Standard Award Urban CBO)
Matching resource to clients in crisis		-“And we serve people who, you know, it’s only been a couple of hours since they were assaulted to people who were assaulted thirty years ago and maybe are wondering when their HIV was a result of the rape that they experienced. You know it could be a whole gamut of things, so we have to really narrow the audience for the tool.” (Standard Award Inner City CBO)
Client privacy/confidentiality		-“It was our own creation with social network because there were issues around confidentiality related to using Facebook that we didn’t anticipate initially. And so we created our own social networking page that still is up and running and it’s still active, and John keeps it up-to-date.” (Standard Award Urban CBO)) -“For example, one of the women is a domestic violence survivor. And she didn’t want anything of her face or imagery on the web because she didn’t want her former abuser to be able to find her or to use her pictures, or anything like that. (Standard Award Inner City CBO)
Lack of logistical support for client participation		-“Another challenge is the transportation for the participant and the distance from where they live. You know, public transportation is not really the greatest, and sometimes they don't have time.” (Standard Award Urban CBO) -“We work with Latinas, and most of them are housewives or single parents, no income. I'm exploring the possibility to entertain their kids so they will have more time for them to practice all the computer skills, and learn more, and so on and so forth” (Standard Award Urban CBO)

External awardee barriers related to target population (clients, general public, healthcare professionals, library staff) were identified less frequently (<15%) than awardee barriers of an organizational nature. With the exception of literacy, they were primarily described in reports rather than interviews. Literacy issues such as functional literacy and computer literacy were identified for clients (14.3%) and the general public (4.1%) in awardee reports. Interviewees also identified functional literacy as a challenge with one individual from an urban mid-western CBO stating “…and then the other challenge was the literacy level of the clients. And so, we had to bring it down a little. And we really have to meet the client where the client’s at.” Computer literacy was also a challenge. For example, another interviewee stated, “Because a lot of them are not computer savvy, they're not familiar with computers, we have to show them how to do that.”

Most projects overcame barriers and 46% of awardee final reports described some type of enabling factor. These included technology infrastructure (21%), marketing strategies (19%), organizational commitment (17%), budget (17%), staffing (17%), leadership (8%), and project management (18%). Interview data confirmed the importance of the enabling factors reported. Eleven additional categories of enabling factors were discussed in awardee interviews. In terms of project development and implementation, these included: needs assessment, pre-submission planning, use of NLM resources, partnerships (with organizations at the front line), build on existing organizational efforts, engage expertise, and plan for evolving technologies. Three factors were specific to clients – creating a safe environment, tailoring resources to user needs, and providing logistical support for participation. Consideration of the context in which the project work is placed was only mentioned by a few interviewees.

3.1.4 Project Sustainability Varied (RE-AIM Maintenance)

Several patterns emerged among the interviewees who addressed sustaining of their NLM project. Projects were categorized as "maintained with existing resources" if the project funding resulted in the creation of resources that had a life beyond the funding period without additional funding. In contrast, projects that secured additional resources from NLM or elsewhere to build upon the existing project were categorized as "maintained with new resources." Projects that maintained relationships but not activities associated with the NLM award were labeled "maintained relationships." If the interviewee indicated that the project activities or resources were not maintained, the project was considered "not maintained." Illustrative examples:

• ▪Maintained with existing resources –A CBO interviewee stated “It’s one of those programs that because funding ended, the program didn’t go away. The program is still as viable, you know, maybe we have more limited resources, but it just has really been a real good addition for our agency’s list of resources that we have available for our clients.”
• ▪Maintained with new resources – An awardee from a CBO that has received multiple awards emphasized the importance of building upon investments – in their instance, an online educational management system. She said that “I feel like this grant actually was the impetus for a series of other grants that we ended up being able to get both through the National Library of Medicine and through some other sources as well because we sort of invested in creating the system in this class so then we could say we have the system in this class and we’re ready to go.”
• ▪Maintained relationships – In some instances, relationships were maintained even when specific project-funded activities ended. An awardee from an educational institution stated that “We’ve maintained relationships with many of the libraries, and of course the college has the relationship with the Lutheran Church downtown, so we’re able to do some clinics for the clinical clerkships, we’re able to do those, and then, the libraries, of course, can contact us…”
• ▪Not maintained – Some awardees did not maintain or seek further funding for project activities either because of difficulty balancing current activities or change in organizational priorities. For example, an interviewee from a CBO noted that “When the project ended, it was decided by the management that we would not continue to search out funding, since it wasn’t a priority…”

3.2 Impact

Impact was examined from several perspectives taking into account Reach, Adoption, and Efficacy/Effectiveness dimensions of the RE-AIM framework.

3.2.1 Awardees were Community-based and Community-engaged Organizations Serving High Risk Populations (RE-AIM Reach, Adoption)

The predominant type of organization to lead a project was a CBO (50%) followed by an academic (20.5%). CBO was also the most frequently occurring type of partner organization. Healthcare and academic organizations as well as public libraries and health departments were more likely to be partner organizations than to be lead organizations.

In terms of high-risk populations, the primary and secondary populations served (Table 4) provide strong evidence that the projects focused on high-risk populations including racial and ethnic minorities, substance users, people living with HIV (PLWH), and the lesbian, gay, bisexual, and transgender (LGBT) populations. In addition, more than half of the awardees reported inner city (22.7%) or urban (31.8%) as the primary geographical area of focus.

Table 4.

Populations Served (N=44)

At Risk Population1	N (%) Primary	N (%) Secondary
African American	21 (47.7)	21 (47.7)
Alaska Native	2 (4.5)	0 (0)
Asian American	6 (13.6)	3 (6.8)
American Indian	6 (13.6)	3 (6.8)
Hawaiian/Pacific Islander	4 (9.1)	2 (4.5)
Hispanic American	18 (40.9)	7 (15.9)
White	1 (2.3)	0 (0)
Youth/Teen	13 (29.5)	7 (15.9)
Senior	9 (20.5)	5 (11.4)
Inner City	16 (36.4)	1 (2.3)
Rural	8 (18.2)	1 (2.3)
PLWH	29 (65.9)	5 (11.4)
LGBT	16 (36.4)	7 (15.9)
Other	12 (27.3)	3 (6.8)
Missing	3 (6.4)	32 (65.9)

¹Select all that apply

Patients and families were the primary direct beneficiary in more than half of the projects, followed by general public (48.9%). Health professionals were primary direct beneficiaries in more than one quarter of the projects. All categories were also secondary direct beneficiaries for at least one project among the 16 reporting.

3.2.2 Difficult to Assess Impact Due to Limited Evaluation Activities Reported (RE-AIM Efficacy/Effectiveness)

Awardees from 2006 onward were asked to describe explicitly the quality and quantity of services provided as well as evidence of impact in their final reports. The type of evaluation for the 20 awardee (14 Standard, 6 Express) final reports for the time period of 2006–2010 is summarized in Table 5.

Table 5.

Type of Evaluation (n=20)

Type of Evaluation	Standard (n=14) N (%)	Express (n=6) N (%)
Quantitative
Offerings	10 (71.4)	5 (83.3)
Attendees	9 (64.3)	4 (66.7)
Web site/ online resource use	4 (28.6)	1 (16.7)
Course evaluation	1 (7.1)	-
Circulation data	1 (7.1)	1 (16.7)
Document delivery	1 (7.1)	-
Pre- and post-training assessments	2 (14.3)	-
HIV-related behavior	-	1 (16.7)
Narrative
Interviews	1 (7.1)	1 (16.7)
Informal participant feedback	1 (7.1)	1 (16.7)
Survey open-ended	1 (7.1)	1 (16.7)
Resource use log comments	1 (7.1)	-
Unspecified method	3 (21.4)	2 (33.3)

In terms of quantitative data, most projects simply tabulated the number of offerings and attendees. Five (25%) projects measured web site or online resource use. For example, a project that implemented Second Life, an online virtual world, was able to capture the number of avatars (i.e., people) within sensor range. Only two Standard awardees (and no Express awardees) implemented pre- and post-tests of educational offerings to assess changes in factors such as awareness, knowledge, and skills related to NLM resources. A single awardee assessed the relationship between training and behavior and found that the training resulted in youth completing HIV testing at a rate of about 50/month (as compared to none before the training) among high risk individuals who were unlikely to have been previously tested.

In some final reports, awardees reported interviews, informal participant feedback, open-ended surveys, and resource-use-log comments as sources of qualitative data to document impact. Examples of impact described from these sources included: increased awareness of NLM resources, improved computer skills and online information seeking behaviors, change in HIV knowledge, and advancement of advocacy skills. This was often in the form of describing individual clients or peer educators rather than through formal evaluation methods.

Although the evaluation efforts documented in the reports were limited, unintended positive consequences, defined as positive outcomes that were not the planned outcomes of a project, were richly described in awardee interviews.

Four categories of unintended positive consequences related to clients: 1) improved knowledge about health in general, 2) improved education and skills beyond those taught in the project, 3) client empowerment (e.g., use of information from web resources in a clinician visit), and 4) social engagement – “people feeling like they have a community.” Several interviewees also described situations in which access to accurate resources resulted in an individual’s intent to change their stigmatizing behavior toward PLWH.

The first of two categories of unintended positive consequence at the organizational level related to changing the traditional role played by libraries and librarians. The second category comprised instances in which components of the project expanded beyond project intent. This included re-use of project infrastructure for other purposes and having locally-developed project resources “go national.”

3.3 Relevance

The continued relevance of the ACIOP is supported through evidence that documents the need for a) community involvement, and b) tailoring of activities related to developing mechanisms for accessing online information, training, and resource development to meet local needs. These are key aspects of relevance that are congruent with the ACIOP's purpose.

3.3.1 Community Involvement and Partnerships Were Significant (RE-AIM Adoption, Implementation)

Thirty-four organizations explicitly provided information in their awardee final reports on community involvement: partnering with community groups (54.5%), use of volunteers (38.6%), participatory client involvement (18.2%), and community advisory board (4.5%). The importance of community partnerships and going out into the community was reinforced in the qualitative interviews. For example, one interviewee noted “We found that the easiest way, as opposed to trying to bring them first into the library was for us to actually go out to their sites." Another said “…and initially, we also had a partner with a survivor project, which is no longer in existence, but it was an agency that really focused on information for people living with HIV.”

3.3.2 Information Resources and Training Were Tailored to Local Needs (RE-AIM Implementation)

Awardee final reports described the necessity for tailoring both information resources and associated training to meet local needs. This was reinforced in the awardee interviews. Tailoring took different forms. In some instances, it involved simplifying information available in government resources: “Sometimes the information that the government or some entity puts together is so cumbersome. So we just try to make it plain.” Having materials available in appropriate languages such as Spanish was also mentioned. Other interviewees described the importance of personalizing the information with one stating that “…it was a number of personal stories of women who were dealing with HIV and making decisions about going on treatment, the challenges they had adhering treatment and things like that, so it was putting a personal face on HIV so what we did is we wrote easy to read.” A keen understanding of the audience to be reached was identified as a prerequisite to these types of tailoring approaches.

4. DISCUSSION

Overall, the ACIOP evaluation provides evidence of successful awardee outcomes consistent with the purpose of the program. Fully 85 percent of project objectives were found to have been achieved with minimal variation. This high success rate was accomplished despite the numerous barriers encountered. The barriers were both internal to the organization (such as staff turnover) and external (such as those related to the target population served). This is a commendable benchmark that underscores the resourcefulness of the awardees, their efforts to overcome barriers, and their realistic goal setting within the constraints of available resources. It also demonstrates the sound judgment of the NLM reviewers, who were required to fund entities that could best make a difference at the community level even though they might be novices in applying for and carrying out outreach projects with NLM support.

The results of this evaluation have been subjected to an external review for relevance. NLM convened a Workshop in December 2012 that addressed various aspects of the ACIOP and included on the agenda a Columbia study principal who presented a formal summary of the independently-derived evaluation findings. Invited participants included experts in HIV/AIDS clinical services, AIDS research, and a strong contingent of representatives of awardee organizations that actually participated in the evaluation. The Workshop participants uniformly agreed that the findings accurately reflect their own project experiences, and supported the actions NLM intended to take with respect to two lessons learned that emerged in the study: 1) remediating a lack of evaluation capacity at some awardee institutions; and 2) revising the Awardee Final Report Template. Both measures are described below and are expected to strengthen awardees' project planning and documentation of results going forward.

4.1 Remediating a Lack of Evaluation Capacity

One of the lessons learned from the evaluation is that the ACIOP should provide additional guidance to awardees regarding program evaluation to improve the quality of the evaluations in awardee reports. This guidance could take different forms such as establishing minimal technical requirements for project evaluation including evidence of evaluation capacity, providing sample evaluation plans, providing technical assistance on evaluation, establishing a peer-to-peer mechanism for sharing of evaluation materials, and facilitating access to web-based resources for evaluation such as utilization tracking, satisfaction surveys, and skills assessments. The current requirements for award recipients have been changed in response to these recommendations.

Community-based organizations focused on HIV/AIDS have had an important role in the dissemination of HIV prevention materials and tools, including HIV testing services [20,21], and information on HIV prevention activities [14]. Moreover, several leading HIV/AIDS-focused CBOs have developed new knowledge that addresses emerging issues encountered in their practices with clients that could be of value to other CBOs and to policy makers [5]. It is therefore fitting that the NLM initially targeted and to this day continues to emphasize the importance and relevance of CBOs as key beneficiaries of ACIOP funding support.

However, CBOs are often thinly staffed, and their range of technical expertise can be quite limited. This was identified by several project telephone interviewees as a barrier to achieving and successfully documenting project outcomes and impact. Other evidence for this issue is the fact that more than 25% of the objectives that were reported as not achieved in the awardee reports related to evaluation. It was also independently observed by the Columbia team as a likely cause of missing information in the Awardee Final Report Template (Appendix B, online only) that was the source of the data elements necessary for populating cells in the Report Abstraction Tool (Appendix C, online only). This absence of quantitative data was most keenly felt when reporting on project efficacy and effectiveness.

When an ACIOP project includes a partnership between a CBO and an academic health sciences library, for example, it may be found that this limitation – a lack of evaluation capacity – is substantially lessened. In other circumstances, however, it is desirable that NLM explore means that include providing evaluation assistance to awardees that is commensurate with project objectives and the available infrastructure that is proposed in the application to carry out the project. In so doing, it is also important to avoid unnecessarily increasing the sophistication, complexity and cost of a project that is proposed for funding in order to produce evaluation data that, while potentially valuable, exceed the scope and capability of the awardee's staff to carry out the functional objectives of the project as envisioned in the proposal. Balance is needed.

A successful strategy for doing this presently exists. In 2000, NLM published Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach [2]. It was developed in collaboration with the staff of the Pacific Northwest Region of NLM's National Network of Libraries of Medicine (NN/LM), and was explicitly intended to assist local organizations that lack evaluation expertise and are receiving outreach funding from NLM and the NN/LM. Subsequently, a complementary series of booklets derived from the Guide were produced and include easy-to-understand case studies and worksheets to assist with project planning and evaluation. All materials are available online free of charge from the NN/LM Outreach Evaluation Resource Center (OERC) (http://nnlm.gov/evaluation/guide/). The OERC also offers limited one-on-one expert consultation services that can assist projects desiring additional help.

In response to the need for remediating a lack of evaluation capacity, the 2013 ACIOP solicitation calls for future ACIOP awardees beginning in 2013 to be informed of the availability of the Guide and encouraged to acquire it as a potentially helpful tool that may be used to plan and evaluate their projects. Additionally, a subset of funded projects will also be offered on an experimental basis the opportunity to avail themselves of a technical evaluation expert for one-on-one consultation that is focused specifically on objectives-setting, selecting an appropriate evaluation methodology, guiding the analyses, and interpreting the results. (The administrative costs and effectiveness of NLM offering this assistance will be separately reviewed and evaluated.)

A new webinar will also be developed and offered to awardees that focuses specifically on prospective measures and tools that capture data for a timely assessment of impact within each of the four information access category objectives in the 2013 ACIOP solicitation: information retrieval; skills development; resource development; and a new category – equipment acquisition.

Assessing project impact is especially deserving of enhanced emphasis going forward because it is important for overall management of the ACIOP by NLM, and as an essential feedback mechanism for local projects' own understanding of the effectiveness of their efforts. Sharing information widely and informally amongst ACIOP awardees can also be helpful in generating new insights and identifying commonly encountered evaluation challenges and discussing their potential solutions. All project awardees will be encouraged to avail themselves of a new listserv recently established voluntarily for ACIOP by two current awardees -- Philadelphia FIGHT and the Internet Sexuality Information Services (ISIS).

4.2 Revising the Awardee Final Report Template

The Awardee Final Report Template (Appendix B, online only) is a critical document and tool for the ACIOP. It is intended to provide a succinct record of a project's activities and accomplishments within a predefined set of categories useful for evaluating both process and outcome. It also serves as a convenient tool for mining data across similar project categories. The template has evolved over the years, and most recently in 2006 introduced the requirement that awardees describe how a project effectively creates awareness and makes use of NLM's information resources, both as a destination for answering HIV/AIDS-related questions and as an authoritative and timely resource that can be customized to create targeted information services that are ideally suited to the needs of a local client population.

The present evaluation study has identified several desirable changes to the Awardee Final Report Template, perhaps the most important of which are elaborations on the need for awardees to plan, assess, and report project accomplishments and impact. Accordingly, the 2013 ACIOP award solicitation announcement now includes a requirement that prospective awardees provide evidence of evaluation capacity which may be augmented by means of the new NLM-supported evaluation initiatives described above.

Additionally, the Awardee Final Report Template will now explicitly include sample descriptions of qualitative and, whenever possible, quantitative measurable objectives that are appropriate for each of the four award categories. There will also be the requirement that awardees submit copies of the evaluation tools used and the results achieved that document project impact. For example, projects whose focus is 'Information Retrieval' will be asked to describe specific activities and their frequency associated with creating awareness (e.g., marketing efforts), and means of client use (e.g., desktop vs. mobile devices). Projects seeking to improve 'Skills Development' will report on training (e.g., new user skills acquired, satisfaction with the training experience, knowledge gained) and utilization (e.g., use of new electronic social media, email, mobile, and more traditional printed resources). 'Resource Development' projects will describe specific needs of the target population (e.g., offerings that are more language and/or literacy appropriate), and new local resources that are developed and used (e.g., at project completion, and provisions made to sustain and update them). Projects whose primary focus is 'Equipment Acquisition' will be asked to provide details on the type and quantity of specific hardware and software procured, and/or the utilization of outside technical consultants (e.g., who advise on equipment specifications, or establishment of Internet wi-fi connections) necessary to support efforts to improve access to or the sharing of AIDS-related information.

4.3 Extending the Benefits of Internet Access and Use

NLM was correct in 1994 to anticipate the emerging importance of access to the Internet and its use in HIV/AIDS prevention, treatment and research. Today, a significant proportion of PLWH - 45% - turn to the Internet for health information [19]. For these individuals, Internet health information seeking is associated with greater disease knowledge and social support, and active coping strategies [11,12]; higher rates of adherence to antiretroviral therapy [12,19]; and better overall health [12]. The Internet also serves as a valuable source of HIV health information for secondary populations such as HIV negative individuals concerned about sexually transmitted infections [16] and caregivers [9].

Yet research also points to disparities in health information seeking behaviors among PLWH. As is the case with the general population, higher instances of online health information seeking are associated with younger age, higher educational attainment, and greater self-efficacy [12,13,19]. A 2006 Pew Internet and American Life Project survey with the general population suggests that despite many programs and initiatives aiming to reduce the digital divide, low-income individuals are still less likely to access and use online heath information [15]. It is reasonable to expect that the divide similarly persists for PLWH. While the scope of the ACIOP is such that it cannot solve all aspects of the digital divide in HIV/AIDS, its initial focus on targeting community-based and community-engaged organizations serving high risk populations as key beneficiaries of ACIOP funding is well placed and achieving success; this focus will continue. Support of projects focusing on improving access to electronic HIV/AIDS information; providing training to develop users' information access skills; developing specific educational materials targeting local needs; and assisting community groups with ICT equipment acquisition requirements, all speak to the continued relevance of the ACIOP. They can have real payoff and will continue within the limits of available NLM resources.

4.4 Potential Limitations of the Evaluation

The evaluation findings must be considered in view of potential limitations. First, the sample for the evaluation included only 47 projects from 44 awardees and 17 awardee interviews. Although projects and related interviewees were carefully selected to create a purposive sample representative of geographical regions, type of awardee organization, and type of award (Standard vs. Express), the findings may not be representative of the population of ACIOP awardees. Second, ACIOP reporting requirements have evolved over time and the Awardee Final Report abstraction form was based upon the most recent Awardee Final Report Template. Moreover, based upon the RE-AIM Framework and literature on successful CBOs, the abstraction form also included some data elements explicitly to examine the feasibility of collecting such data. Thus, it was not surprising that some data elements were not present in the majority of reports. Despite these potential limitations, the attendees at the December 2012 ACIOP stakeholder meeting indicated that the evaluation was sufficiently comprehensive and that the findings were representative of their experiences as awardees. Moreover, their consensus was that there was no need for a larger evaluation at this point in time.

5. CONCLUSIONS

NLM created the ACIOP in 1994, consistent with its outreach mission and in direct response to an urgent request from the affected community for assistance in utilizing HIV/AIDS information resources that increasingly were becoming available via the Internet. Over the course of 19 years, nearly 300 projects were supported by means of relatively small contracts awarded competitively to mostly community-based organizations working alone or in partnership with health science libraries, or other entities.

The evaluation on which this paper is based was undertaken for the purpose of determining the performance and impact of the ACIOP, its continued relevance as presently structured, and to identify any needed modifications. A mixed-methods approach using quantitative analyses of a purposive sample of final project reports, along with qualitative analyses of telephone interviews with selected project leaders, produced the following major findings: 1) awardees' project objectives were mostly met; 2) high-risk populations were reached; 3) low resource organizations were funded; 4) community involvement and partnerships were significant; 5) projects built on existing efforts; 6) information resources were tailored to local needs; and 7) most projects successfully overcame barriers experienced.

The RE-AIM evaluation framework was selected because of its public health emphasis and utility for looking beyond efficacy/ effectiveness to identify other significant findings of relevance to the purpose of the evaluation. Ironically, it was with respect to assessing impact that the model fell somewhat short, not due to inherent limitations of the framework, but rather for a lack of uniformly robust measurable data that the individual projects were able to gather and report. In this respect, NLM determined that two key modifications of the ACIOP were in order: 1) enhance evaluation capacity at the individual project level by means of providing awardees with additional tools in the form of guided instruction and one-on-one expert consultation; and 2) revise project reporting requirements to require explicitly both qualitative and quantitative measures of project performance and accomplishment. These programmatic changes were implemented by NLM in 2013, and are expected to help ensure that the ACIOP continues to remain relevant and meet its intended purpose of enabling access to up-to-date and authoritative HIV/AIDS information as the epidemic evolves in the U.S. and abroad, new treatment and prevention measures emerge, new information resources are developed and disseminated, and individuals and organizations seek to remain current with advances in health information technology and communications infrastructure.