Laboratory System Improvement Program: First in the Nation— New Hampshire Reassessment

Abstract

The New Hampshire Public Health Laboratories (NH PHL) conducted an initial Laboratory System Improvement Program (L-SIP) assessment in March 2007 and a reassessment in May 2011. New Hampshire was a pilot state for the initial L-SIP assessment in 2007 and was the first laboratory system in the United States to conduct an L-SIP reassessment. The New Hampshire reassessment was also used as a pilot for revising the assessment tool. The NH PHL performed a high-level comparison benchmarking the work done between the two assessments. This comparison revealed areas of improvement and other areas that needed continued focus to align with model standards of the 10 Essential Public Health Services. This article outlines achievements, improvements, and outcomes made since 2007, as well as participants, activities, plans, resources, and other factors that contributed to the change in scores between assessments.

In 2002, the Centers for Disease Control and Prevention (CDC) established the National Public Health Performance Standards Program (NPHPSP)¹ to measure components of public health systems and local public health governance against a gold standard and to identify areas for improvement. Based on the 11 Core Functions of State Public Health Laboratories² and designed within the framework of the 10 Essential Public Health Services (hereafter, Essential Services)³ (Figure 1), the Association of Public Health Laboratories (APHL), in conjunction with CDC, developed a similar assessment program for public health laboratory (PHL) systems. Called the Laboratory System Improvement Program (L-SIP),⁴ this performance measurement project is used to determine state and local PHL systems' capabilities and capacity to provide adequate and appropriate laboratory system functions and services.

Figure 1.

The 10 Essential Public Health Services^a

^aSource: Centers for Disease Control and Prevention (US). National Public Health Performance Standards Program (NPHPSP): 10 essential public health services [cited 2013 May 7]. Available from: URL: http://www.cdc.gov/nphpsp/essentialservices.html

A state public health laboratory (SPHL) system is defined by the APHL as a partnership between PHLs and other state agencies, private laboratories, and other organizations and health-care providers to assure laboratory services essential to the health of the public.⁵ The performance assessment process was created to engage and leverage SPHL system partnerships to build a stronger foundation for public health, promote continuous quality improvement, and strengthen the scientific basis of public health practice improvements. Within the L-SIP, APHL developed an assessment tool to evaluate systems.⁶ Created in 2006, the assessment tool was used for the New Hampshire (NH) 2007 assessment, as well as in many other states from 2007 through 2009. In May 2011, NH was the first laboratory system to conduct a reassessment. Additionally, NH also piloted a newly revised assessment tool. Because this reassessment was an improvement project, APHL representatives were present during the NH reassessment and captured the efficacy of the newly revised tool. In August 2011, the tool was adopted and became the final assessment tool to be used for future laboratory system assessments throughout the U.S.

The intent of a reassessment is to determine the strengths and weaknesses of the system, benchmark rates of performance for each Essential Service, and recognize the improvements made since the initial assessment. After an initial assessment, APHL recommends that a reassessment take place periodically, about every three to five years, to benchmark system performance.

METHODS

As one of nine pilot state systems, the NH Public Health Laboratories (NH PHL) conducted an initial L-SIP assessment on March 26, 2007, with 89 participants. NH then held the first L-SIP reassessment in the U.S. on May 4, 2011, with 51 participants. The following were the objectives of the NH L-SIP assessment and reassessment:

• Inform participants about the NH PHL and build an appreciation of the interdependence of system partners.

• Improve communications among system partners.

• Expand collaboration with system partners.

• Recognize system strengths.

• Identify opportunities for improvement.

• Articulate the resources needed for optimal system functionality.

• Compare improvement rates between the initial assessment and the reassessment.

To complete the L-SIP assessment in one day, all participants were assigned to join in a plenary session and then organized into three workgroups to review and evaluate three of the Essential Services. Participants invited to the NH reassessment were those individuals and agencies (i.e., stakeholders) who use the laboratory system in some capacity or contribute to it (Figure 2).

Figure 2.

Suggested participant list of key partners and stakeholders for APHL L-SIP assessments, as used for the 2011 NH PHL L-SIP reassessment^a

^aSource: Association of Public Health Laboratories. Laboratory System Improvement Program: assessment tool (May 2011). Revised August 2011 [cited 2013 Apr 15]. Available from: URL: http://www.aphl.org/AboutAPHL/publications/Documents/LSS_2012March_Laboratory-System-Improvement-Program-LSIP-Flyer-Fact-Sheet.pdf

APHL = Association of Public Health Laboratories

L-SIP = Laboratory System Improvement Program

NH PHL = New Hampshire Public Health Laboratories

To maintain and standardize each workgroup, a core participant group of key stakeholders was created, including a PHL manager, a public health administrator, a hospital laboratory director, a Laboratory Response Network (LRN)⁷ representative, a public health nurse, an NH Bureau of Disease Control representative, a PHL technical supervisor, and a PHL representative with either technical or administrative skills within a PHL. Each group also had a core-facilitated cohort that included a professional facilitator, a system theme-taker, and an APHL theme-taker. Theme-takers assisted the facilitator and captured information conveyed and discussed throughout the assessment.

The L-SIP assessment process simulates the NPHPSP assessment process used to evaluate local and state public health systems but is adapted to laboratory services. Each Essential Service represents a major system component, activity, or practice, and is assessed individually. The L-SIP assessment tool consists of the breakdown of each Essential Service into model standards that describe high-level performance aspects, key ideas, and points of discussion. As components of a model standard, one or more key ideas are used to measure the L-SIP performance. Points of discussion are not measured but are used to trigger and facilitate participant dialogue of each key idea. Upon discussion, participants are asked to rate the performance of the system in achieving the key idea against the model standard. Scoring of the Essential Service takes place when consensus among the group is achieved for each key idea and is voted upon as a group. The key idea scores are then tabulated to provide an overall performance rate for each Essential Service. Prior to consensus, any discussion among the participants serves as a platform to close the gap between scores and improve the overall performance of the system. Issues are noted as either “next steps” or “parking lot” issues. The parking lot issues are used later as a basis for system improvement.

OUTCOMES

The scoring definition by which the NH system was graded in 2007 is different from the 2011 evaluation. The wording of the scoring system changed to more descriptive terms in determining the system's achievement in fulfilling Essential Service activities. The new 2011 scoring system defines the percentage of the rate of performance rather than activity of the Essential Service met by the system. A word score is chosen that directly correlates to a weighted score within the scoring tool. The previous word scorings—no activity, no, no partially, yes partially, yes, and does not apply (Figure 3)—were replaced with new word scorings—none, minimal, moderate, significant, and optimal (Figure 4)—in the revised assessment. The definitions of the word scores were changed to reflect tangible activities such as meetings, project coordination, and deliverable items, as well as to include system relationships, team-building, and more conceptual work.

Figure 3.

Scale for rating activity of the 10 Essential Public Health Services:^a NH PHL L-SIP assessment, 2007

^aSource: Association of Public Health Laboratories. Laboratory System Improvement Program: assessment tool (May 2011). Revised August 2011 [cited 2013 Apr 15]. Available from: URL: http://www.aphl.org/AboutAPHL/publications/Documents/LSS_2012March_Laboratory-System-Improvement-Program-LSIP-Flyer-Fact-Sheet.pdf

NH PHL = New Hampshire Public Health Laboratories

L-SIP = Laboratory System Improvement Program

SPHL = state public health laboratory

Figure 4.

Scale for rating performance of the 10 Essential Public Health Services:^a NH PHL L-SIP reassessment, 2011

^aSource: Association of Public Health Laboratories. Laboratory System Improvement Program: assessment tool (May 2011). Revised August 2011 [cited 2013 Apr 15]. Available from: URL: http://www.aphl.org/AboutAPHL/publications/Documents/LSS_2012March_Laboratory-System-Improvement-Program-LSIP-Flyer-Fact-Sheet.pdf

NH PHL = New Hampshire Public Health Laboratories

L-SIP = Laboratory System Improvement Program

PHL = public health laboratory

The score of each key idea was entered into a Microsoft® Excel spreadsheet provided by the APHL L-SIP assessment kit, and a cumulative score for the Essential Service was calculated. Each key idea has an assigned weighted value that, when added to all of the key idea scores in an Essential Service, yields an overall score. All overall scores are then added to create a final summary score result. During the final session of the NH reassessment, the 2007 initial assessment scores were compared with the 2011 reassessment scores and displayed in a modified trend chart (Figure 5), which was shown to the entire group of participants to visualize the progress of each Essential Service and where to channel resources for improvement. Two of the Essential Service scores remained the same (Essential Services 2 and 8), three decreased (Essential Services 1, 5, and 7), and five increased (Essential Services 3, 4, 6, 9, and 10).

Figure 5.

Comparison of the 2007 vs. 2011 Laboratory System Improvement Program scores of the 10 Essential Public Health Services^a in New Hampshire

^aCenters for Disease Control and Prevention (US). National Public Health Performance Standards Program (NPHPSP): 10 essential public health services [cited 2013 Apr 15]. Available from: URL: http://www.cdc.gov/nphpsp/essentialservices.html

ES = Essential Service

Observations of those Essential Services with scores that remained unchanged for both assessments (-Essential Services 2 and 8) revealed that work conducted since the initial assessment had not resulted in any dramatic improvements, but no worsening had occurred. Essential Service 2 received a score at the optimal level for both assessments. In 2007, Essential Service 7 was the subject of the plenary session; however, for 2011, Essential Service 2 was selected for the plenary session, which may have contributed to an overall high score due to a broader range of participants involved. The larger group's rating of Essential Service 2 at the highest level indicates that the system has continued to maintain high-quality services through federal and state regulations. This Essential Service review also indicated that rapid response to emergencies has been effective in assisting in the diagnosis of health problems in the system by offering ongoing training for specimen and sample collection, transport, and communication. Plans for continuity of operations have had a significant impact on the system's ability to continue with the LRN and other emergency responders during public health events. Weaknesses discussed included the lack of published turnaround times for testing and the assurance of communication between partners when tests were outsourced. Purchasing procedures from partners were questioned as group members observed that resources were quickly made available during critical events, such as the 2009 H1N1 influenza pandemic, yet are not routinely available.

Essential Service 8 remained unchanged at the significant level, indicating that workforce development will continue to be an issue for employers in the system, especially during a downward slope in the economy. One possible reason the score did not change is that all laboratories within the system have continued to satisfy the key ideas in this Essential Service. Since 2007, members of the system have taken part in many activities that assure a competent public health and personal health-care workforce. All clinical laboratory representatives in attendance were from institutions accredited by the Clinical Laboratory Improvement Amendments (CLIA)⁸ to perform human testing. CLIA requires defined job requirements and qualifications, as well as competency assessment for testing personnel. Laboratory staff performing nonhuman testing are not within the scope of CLIA but are overseen by other regulatory agencies such as the International Organization for Standardization, U.S. Environmental Protection Agency, and U.S. Food and Drug Administration. These agencies also require a level of competency within the workforce. Another reason why the Essential Service 8 score did not change was that the system offers continual support for staff development through training, education, and mentoring. To assure best laboratory practices in a safe work environment, NH PHL created and provided physician office laboratory training at three locations in the state, and other stakeholders have hosted similar workshops. Although the system has performed highly to achieve Essential Service 8, there is room for improvement. One reason for the less-than-optimal rating may be the inability to offer competitive salaries and benefits, which is important for workforce recruitment.

Review of the five Essential Service scores that increased from 2007 to 2011 (Essential Services 3, 4, 6, 9, and 10) showed that improved communication tools were a common factor. System members worked with the Health Alert Network (HAN),⁹ CDC's primary communication method, to assure that the system received rapid updates and advisories during public health events. The NH PHL's biannual newsletter, Extracts from the Lab, was also redesigned to disseminate information about laboratory events, testing updates, and other relevant system news, nationally and internationally. Increasing the use of the HAN and revising the newsletter proved direct correlations to improvement in Essential Service 3 (minimal to optimal).

Improvements in Essential Service 4 (significant to optimal) reflected new emergency communication procedures established among public health nurses, the NH PHL, and the LRN. A “call tree” (i.e., a telecommunications chain for notifying specific individuals of an event) was implemented for after-hours contact. Additionally, in federal grant proposals, system partners incorporated resource sharing (e.g., equipment and training) among stakeholders to assist in covering system needs when resources are scarce. To sustain this optimal level of performance, the strong relationships among system partners, such as the NH LRN and the NH PHL, should be maintained. Collaboration among the partners in response to community health issues proved successful during the NH public health incident of patients potentially exposed to the hepatitis C virus during a hospital stay.¹⁰

In the discussion of Essential Service 6 improvements (significant to optimal), those who claimed improvements were experienced in laboratory-related laws and regulations that protect health and assure safety. Many of these stakeholders were not present at the 2007 assessment; so by being present in 2011, they helped increase the rating by sharing their experiences. An NH Division of Public Health Services spokesperson described how laws and regulations are created and monitored. System members were aware of laboratory-related regulations but unsure of state statutes regarding individual entities. Improvements since 2007 reflected partner relationships that were responsible for supporting regulatory enforcement functions; however, the lack of funding for enforcement efforts, due to the loss of incoming fees, was identified as a problem, especially assigning designated personnel to oversee compliance issues. Future planning will include efforts to improve compliance and enforcement.

Understanding the voice of the customer drives change and assists in improving the quality of services. System partners revealed that many of their organizations are using feedback from customer satisfaction surveys to improve the effectiveness of laboratory test results and how they are used within their systems. This feedback has led to the improvement in scores for Essential Service 9 (none to minimal). Since 2007, many organizations have developed individual mission and vision statements for their own entities, yet none exist for the NH system. Plans to improve the rate of performance in this Essential Service include bringing partners together to develop a system mission statement that will define the system, set goals, and promote improvement activities by the next reassessment. Improvements in Essential Service 10 (minimal to moderate) revealed an increased use of electronic surveillance programs to gather data for research activities along with ongoing encouragement of staff to further their education by working with stakeholders to develop thesis projects. The development of quality improvement teams among stakeholders and the establishment of publishing groups to assist in writing research findings also helped to raise the score.

For those Essential Services with decreased ratings (Essential Services 1, 5, and 7), no action steps had been taken for some key ideas since 2007. The decrease in score for Essential Service 1 (significant to moderate) was attributed to a lack of information technology among the systems on a regular and user-friendly basis. As a system, communication worked well among partners, but there was no mechanism to offer a single electronic information system to assist with surveillance activities. Since 2007, the NH PHL implemented a Laboratory Information Management System (LIMS), which facilitated and improved the systems' capacity to exchange limited information. The LIMS provides an efficient method to track samples from receiving time until the report has been submitted to the provider. Real-time reporting of laboratory results allows for timely action among users and will be monitored for improvement.

The decreased rating for Essential Service 5 (optimal to significant) may be attributed to some participants lacking an accurate understanding of how policies are developed using laboratory data. Overall, a common theme was that the system is not robust in identifying ways it can capture public health data to incorporate into policies to support individual and community health. PHL staff typically do not have the expertise to create policies and usually are not allowed to advocate for state policy unless instructed to do so. To be successful in creating policies, the participants felt the need to be more proactive rather than reactive. In 2007, legislative activity was considered a weakness in the system, and this weakness was also reflected in the 2011 reassessment. While they were invited, no legislative or rule-making partners were present for the initial assessment or the reassessment. Partnership development improved among public health cohorts as the strength of planning for critical incidences proved successful in handling the 2009 H1N1 influenza pandemic.

A decrease in the score for Essential Service 7 (significant to moderate) was likely due to the 2010 cancellation of a state-funded courier to transport specimens. Many of the participants who had been directly affected by the loss of the courier were present, yet more input would have been helpful in discussing the impact this budget cut had on turnaround times, specimen integrity, and scope of services for private laboratories, physicians, and the general public. This Essential Service had an additional key idea added to it since 2007, so the scoring may have been lower due to the additional factor in calculating the overall score.

LESSONS LEARNED

The revised assessment tool consolidated, eliminated, or updated key ideas, which may have changed the scoring rates of each key idea as well as the overall score of each Essential Service. Scoring descriptions in the revised assessment are more detailed, reflecting performance rather than activities.

In NH, the 2007 and 2011 final scores were almost identical. The NH system participants believe this outcome does not truly reflect improvements made within each Essential Service; therefore, individual Essential Service scores are NH's target value for system improvement.

The L-SIP tool recommends assessments using a core group of participants and a suggested group of subject-matter experts and partners who are main contributors to an Essential Service. The decrease in attendees in 2011 seemed to elicit more sharing of information, contributions, and experiences from participants than it did in 2007. The selection of participants whose expertise and job functions align within the domain of an Essential Service contributed to increased scores, as participants were to share their knowledge and applicable facts and data during the discussion period. Some of the same attendees were present for both the 2007 assessment and the 2011 reassessment. Their experience with the process likely enhanced their participation and the formation of the scoring.

Since the reassessment, the NH PHL has collated the results, the “parking lot” issues, and participant feedback for each Essential Service. A voluntary core group has been tasked to evaluate and identify next steps for improvement. As the NH PHL moves toward optimal performance, the group will continue to monitor improvement projects as well as relate any new activities to an Essential Service. In NH, formal meetings such as forums or advisory boards have not been instituted, but future plans include reviewing and celebrating improvement activities by others within the system to promote the sustainability of high-quality laboratory services.

CONCLUSIONS

L-SIP assessments are not mandatory and require dedicated resources such as staff, time, and money. In 2007, the initial L-SIP assessment provided a baseline analysis of the system, and the reassessment in 2011 served as a benchmark that identified best practices used to substantiate the hard work and commitment of resources in achieving improvements within the system. Many system partners have interacted during the four years between the assessments and realize how important it is to have a system in place. Because several agencies have had to reorganize or downsize some of their programs, resource sharing has become critical. Networking and positive system relationships can foster collaborations and allegiances, encourage the sharing of assets and ideas, and help improve communications, especially during crucial times. Bringing partners together on a regular basis with a common goal strengthens relationships and ultimately helps improve the system.

Voluntary assessments of continual evaluations produce standardizations and help offset the propensity of system relationships to become stagnant. Proactively conducting a self-reflection and review of the system demonstrates caring about the system's customers, clients, and the public. With each new project, team meeting, or management meeting, awareness of implementing Essential Services as guiding principles will make an impact in changing the culture of continual system improvement in NH. To maintain that culture and be effective, the NH PHL system is committed to continuing evaluation and will perform another reassessment in the coming years.