Competency Assessment in the Clinical Microbiology Laboratory

Abstract

The laboratory comprises an invaluable part of the total health care provided to patients. Competency assessment is one method by which we can verify that our employees are competent to perform laboratory testing and report accurate and timely results. To derive the greatest benefit from the inclusion of competency assessment in the laboratory, we must be sure that we are addressing areas where our efforts can be best utilized to optimize patient care. To be competent, an employee must know how to perform a test, must have the ability to perform the test, must be able to perform the test properly without supervision, and know when there is a problem with the test that must be solved. In some cases, competency assessment protocols may demonstrate areas of competence but can fail to disclose incompetence. For example, challenges of low-complexity tasks (such as reading the technical procedure manual) are inferior to challenges that measure understanding and execution of a protocol, and poorly designed competency challenges will probably not detect substandard laboratory performance. Thus, if we are to receive the greatest benefit from our competency assessment programs, which may be time-consuming for the supervisors and the staff as well, we must not only meet the letter of the law but also find a way to make these assessments meaningful, instructive, and able to detect areas of concern. As we address competency assessment in our laboratories, we must understand that when done properly, competency assessment will reward our organizations and assist us in providing the best possible care to our patients.

INTRODUCTION: HISTORY AND OVERVIEW OF CLIA ’67 AND ’88

Few regulations for laboratory testing existed before the late 1960s. However, soon after the introduction of Medicare and Medicaid in the mid-1960s, a decades-long and continuing effort by the U.S. Government to regulate costs and ensure a high quality of health care ensued. To see that the system was not abused financially and that the quality of laboratory results was high, in 1967 Congress passed the federal Clinical Laboratory Improvement Act (CLIA '67) (1). The Health Care Finance Administration, now the Center for Medicare and Medicaid Services, was created as part of the Department of Health and Human Services to oversee the enforcement of the CLIA '67 regulations as well as to oversee the Medicare and Medicaid programs. However, CLIA '67 required only hospitals and large clinical laboratories to adhere to strict quality control, proficiency testing, test performance, and personnel standards. Each testing facility had to have a certificate and was subject to a compliance inspection every other year. CLIA '67 affected only laboratories engaged in interstate commerce and covered approximately 12,000 laboratories (mainly commercial and hospital). With the exception of a few states, this left laboratories located in physicians' offices or other small health care facilities largely unregulated.

Prior to 1988, fewer than 10% of all clinical laboratories were required by the government to meet minimum quality standards, and a significant percentage of patient testing performed in laboratories was not subject to minimum quality standards (8). Concerns raised by the media about the quality of cytology testing services, especially Pap smears, were a major catalyst behind passage of the Clinical Laboratory Improvement Amendments of 1988 (CLIA '88). A series of articles that appeared in the Wall Street Journal in the 1980s reported on the deaths of women from uterine and ovarian cancer whose Pap smears had been misread, exposed “PAP mills,” and called into question the quality of laboratories in general (3, 5, 19).

Congress held hearings at which people who had been harmed by laboratory errors testified. These hearings revealed serious deficiencies in the quality of work from physician office laboratories and in Pap smear testing results (R. D. Feld, M. Schwabbauer, and J. D. Olson, 2001, The Clinical Laboratory Improvement Act [CLIA] and the physician's office laboratory; Virtual Hospital, University of Iowa College of Medi-cine [www.vh.org/adult/provider/pathology/CLIA/CLIAHP.html]). In 1988, Congress once again responded to public concerns about the quality of laboratory testing by passing CLIA '88. CLIA '88 expanded the laboratory standards set by CLIA '67 and extended them to include any facility performing a clinical test. Currently, under CLIA '88, all ∼170,000 clinical laboratories, including physician office laboratories, are regulated.

CLIA '88 greatly broadened the definition of a laboratory. CLIA '88 defines a laboratory as “a place where materials derived from the human body are examined for the purpose of providing information for the diagnosis, prevention or treatment of any disease or impairment of, or assessment of the health of human beings. Laboratories may be located in hospitals, freestanding facilities or physician offices” (11). For the first time, federal laboratory regulation was site neutral. The level of regulation was determined by the complexity of the tests performed by the laboratory rather than where the laboratory was located. Physician office laboratories, dialysis units, health fairs, and nursing homes were all covered under the new law, along with other previously exempt and nonexempt laboratories. The CLIA '88 regulation unified and replaced past standards with a single set of requirements that applied to all laboratory testing of human specimens. Standards for laboratory personnel, quality control (QC), and quality assurance were established based on test complexity and potential harm to the patient. The regulations also established application procedures and fees for CLIA registration as well as enforcement procedures and sanctions applicable when laboratories fail to meet standards.

The purpose of CLIA '88 is to ensure that all laboratory testing, wherever performed, is done accurately and according to good scientific practices and to provide assurance to the public that access to safe, accurate laboratory testing is available. The ability to make this assurance has become even more urgent as knowledge of the impact of medical errors has reached both the medical and public arenas (13). One of the essential components identified as necessary to ensure high-quality test results for patients was employee training and competency. Thus, CLIA '88 set forth requirements for performance and documentation of initial personnel training and ongoing assessment of competency (11).

The following section outlines the sections of CLIA '88 that pertain to personnel training and competency assessment. As stated above, current governmental mandates make it necessary to assess the competency of all laboratory workers who handle patient specimens. The mandates are specific in what must be assessed; however, they do allow for considerable discretion on how to implement some of these specific assessments in a laboratory setting.

CLIA '88 outlines six areas that must be included as part of a laboratory competency assessment program; these are (i) direct observation of routine patient test performance; (ii) monitoring the recording and reporting of test results; (iii) review of intermediate test results, QC records, proficiency testing results, and preventive maintenance records; (iv) direct observation of performance of instrument maintenance and function checks; (v) assessment of test performance through testing previously analyzed specimens, internal blind testing samples, or external proficiency testing samples; and (vi) assessment of problem-solving skills (11).

To measure compliance with the CLIA '88 regulations, the College of American Pathologists (CAP) conducted a study in 1996 (CAP QProbes program) to survey employee competence assessment practices in departments of pathology and laboratory medicine (12). The goals of the study were to measure institutional competency assessment practices, to assess the compliance of each institution with its own practices, and to determine the competency of specimen-processing personnel. This three-part study consisted of a questionnaire concerning current competency assessment practices, evaluation of compliance with these practices using personnel records, and a written appraisal of the competence of five specimen-processing staff members per institution. The study surveyed a total of 552 institutions that participated in the CAP 1996 QProbes program (12). Their results showed that 89.2% of institutions had a written competency plan and that of those, 90.3% used their plan for microbiology. Approximately 98% of institutions reported reviewing employee competence at least annually; this consisted of direct observation in 87.5% of laboratories surveyed, review of test or QC results in 77.4%, review of instrument preventive maintenance in 60%, written testing in 52.2%, and other methods of assessment in 20.8%. When measuring adherence to the laboratory's own competence plan, it was found that the percentage of laboratory employees who complied was 89.7% when assessed using direct observation, 85.8% when assessed by reviewing QC and patient test results, 78% when assessed by reviewing instrument records, and 74% when assessed using written testing; 90.4% of new employees were assessed as indicated per policy, and 90% of employees were found to have responded satisfactorily to a written competency assessment regarding specimen processing. Failure to comply with the laboratory's own competence plan ranged from ca. 1 to 6.4%, and employees who failed competency assessment were not allow to continue their usual work in 8.6% of institutions.

This study concluded that opportunities for improvement in employee competency assessments were numerous. Toward these improvements, the CAP provided several suggestions which included the suggestion that direct observation can be used for assessing technical skills (as can patient and QC specimens), judgment and analytical decision-making processes, and teaching and training of personnel. The CAP also noted that communication, judgment, and analytical decision making are essential skills that are rarely evaluated but that when they are evaluated, written testing should be used since interpretation of these skills using direct observation is highly subjective. In addition, the CAP recommended that laboratory employees who fail an assessment should not be allowed to perform these tasks if the competency assessment is a valid test of their skills, knowledge, and abilities. The CAP also concluded that written testing was the one method of evaluation with the poorest compliance; thus, it did not recommend that written testing be used as an element of a competency assessment plan unless it can be performed consistently or is used as part of an assessment of communication and judgement skills.

The CAP QProbe suggested that “opportunities for improvement in employee competency assessment are numerous” (12), and our own experiences in presenting workshops on this topic at the American Society for Microbiology general meetings confirm that many laboratories continue to struggle with the design of a competency assessment program. The following is intended to provide guidance to supervisory personnel in clinical microbiology laboratories in the development and implementation of an effective competency assessment program and is taken, in part, from the 2003 Cumitech entitled Competency Assessment in the Clinical Microbiology Laboratory (4).

Competency assessment in the clinical laboratory, as mandated in U.S. law since 1988 as part of CLIA '88, is published in the Federal Register as part of the Code of Federal Regulations (CFR). The CFR defines the requirements for initial training verification, initial competency assessment, and ongoing competency assessments of laboratory personnel (11). As a brief explanation of the regulation titles, the number “42” indicates “Public Health,” CFR stands for “Code of Federal Regulations,” “493” indicates “Laboratory Requirements,” and the numbers “1445” or “1451” are the section standards. These standards were enacted on 28 February 1992, amended on 19 January 1993, and revised on 1 October 2002. They can be accessed online at www.gpoaccess.giv/cfr/Index/html. Included below are the pertinent CFRs relating to competency assessments in the clinical laboratory.

Code of Federal Regulations—42CFR493.1445. Standard: Laboratory Director Responsibilities

“Ensure that prior to testing patient's specimens, all personnel have the appropriate education and experience, receive the appropriate training for the type and complexity of the services offered, and have demonstrated that they can perform all testing operations reliably to provide and report accurate results.

“Ensure that policies and procedures are established for monitoring individuals who conduct pre-analytical, analytical, and post-analytical phases of testing to assure that they are competent and maintain their competency to process specimens, perform test procedures and report test results promptly and proficiently, and whenever necessary, identify needs for remedial training or continuing education to improve skills.

“Specify, in writing, the responsibilities and duties of each consultant and each supervisor, as well as each person engaged in the performance of the pre-analytical, analytical, and post-analytical phases of testing. This should identify which examinations and procedures each individual is authorized to perform, whether supervision is required for specimen processing, test performance or result reporting and whether supervisory or director review is required prior to reporting patient test results.”

Code of Federal Regulations—42CFR493.1451. Standard: Technical Supervisor Responsibilities

“The technical supervisor is responsible for identifying training needs and assuring that each individual performing tests receives regular in-service training and education appropriate for the type and complexity of the laboratory services performed.

“The technical supervisor is responsible for evaluating the competency of all testing personnel and assuring that the staff maintain their competency to perform test procedures and report test results promptly, accurately and proficiently. The procedures for evaluation of the staff must include, but are not limited to—

“1. Direct observation of routine patient test performance, including patient preparation, if applicable, specimen handling, processing and testing.

2. Monitoring the recording and reporting of test results.

3. Review of intermediate test results or worksheets, quality control records, proficiency testing results, and preventive maintenance records.

4. Direct observation of performance of instrument maintenance and function checks.

5. Assessment of test performance through testing previously analyzed specimens, internal blind testing samples or external proficiency testing samples.

6. Assessment of problem solving skills.”

ACCREDITATION

The three most widely used CMS-approved accreditation programs are the Laboratory Accreditation Program from the CAP, the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), and COLA, formerly known as the Commission on Office Laboratory Accreditation. Although each organization's testing requirements are at least equivalent to those of CLIA '88, they have somewhat different testing standards and philosophies in reaching the goal of quality laboratory testing. The CAP and the JCAHO have guidelines that include several items dealing with initial training and competency assessment of laboratory personnel as a requirement for laboratory certification or accreditation. The requirements for competency assessment by each of these organizations are discussed below.

College of American Pathologists

The CAP survey checklists currently include questions pertaining to CLIA '88 and assessment of competency for laboratory personnel (CAP 2003, Commission on Laboratory Accreditation, Laboratory Accreditation Program, Laboratory General Checklist: http://www.cap.org/apps/docs/laboratory_accreditation/checklists/checklistftp.html). These questions are included in the GENERAL area of the laboratory checklists in the PERSONNEL section. Specific questions, as well as “Notes” and “Commentary,” contained in the 2003 CAP checklists are indicated below. As a point of explanation, CAP guidelines are divided into “Phase I” and “Phase II” deficiencies. These deficiencies are defined by CAP as follows: “Deficiencies to Phase I questions do not seriously affect the quality of patient care or significantly endanger the welfare of a laboratory worker. If a laboratory is cited with a Phase I deficiency, a written response to the CAP is required, but supportive documentation of deficiency correction is not needed. Deficiencies to Phase II questions may seriously affect the quality of patient care or the health and safety of hospital or laboratory personnel. All Phase II deficiencies must be corrected before accreditation is granted by the CLA. Correction requires both a plan of action and supporting documentation that the plan has been implemented.” The CAP guidelines that address competency assessment are included in Table 1. CAP guidelines can be accessed at www.cap.org.

TABLE 1.

CAP guidelines addressing competency assessment

CAP no.	Phase deficiency	Question	Note	Commentary
GEN.54750	II	For laboratories subject to U.S. federal regulations, do all testing personnel meet CLIA '88 requirements?	There must be evidence in personnel records that all testing personnel have been evaluated against CLIA '88 requirements, and that all individuals qualify.	All testing personnel in the laboratory must meet the requirements specified in CLIA '88. There must be an indication in personnel records that testing personnel's qualifications have been evaluated and met.
GEN.55200	II	Are there annual reviews of the performance of existing employees and an initial review of new employees within the first 6 months?		The laboratory must conduct an annual performance review of all employees. New employees must be reviewed within 6 months of employment, and annually thereafter.
GEN.55500	II	Has the competency of each person to perform his/her assigned duties been assessed?	The manual that describes training activities and evaluations must be specific for each job description. Activities requiring judgment or interpretive skills must be included. The records must make it possible for the inspector to determine what skills were assessed and how those skills were measured. The competency of each person to perform duties assigned must be assessed following training, and periodically thereafter. Some elements of competency assessment include, but are not limited to, direct observations of routine patient test performance, including patient preparation (if applicable), specimen handling, processing and testing; monitoring the recording and reporting of test results; review of intermediate test results or worksheets, QC records, proficiency testing results, and preventive maintenance records; direct observation of performance of instrument maintenance and function checks; assessment of test performance through testing previously analyzed specimens, internal blind testing samples, or external proficiency testing samples; and evaluation of problem-solving skills.	The competency of each person to perform the duties assigned must be assessed following training and periodically thereafter. Retraining and reassessment of employee competency must be done when problems are identified with employee performance. The training and assessment program must be documented and should be specific for each job description. Activities requiring judgment or interpretive skills must be included. The records must make it possible for the inspector to be able to determine which skills were assessed and how those skills were measured. Some elements of competency assessment include, but are not limited to, direct observations of routine patient test performance, including patient preparation, if applicable, specimen handling, processing and testing; monitoring the recording and reporting of test results; review of intermediate test results or worksheets, QC records, proficiency testing results, and preventive maintenance records; direct observation of performance of instrument maintenance and function checks; assessment of test performance through testing previously analyzed specimens, internal blind testing samples, or external proficiency testing samples; and evaluation of problem-solving skills.
GEN.57000	I	If an employee fails to demonstrate satisfactory performance on the competency assessment, does the laboratory have a plan of corrective action to retrain and reassess the employee's competency?	If it is determined that there are gaps in the individual's knowledge, the employee should be reeducated and allowed to retake the portions of the assessment that fell below the laboratory's guidelines. If, after reeducation and training, the employee is unable to satisfactorily pass the assessment, then further action should be taken, which may include supervisory review of work, reassignment of duties, or other actions deemed appropriate by the Laboratory Director.	The laboratory should have a documented corrective-action plan to retrain and reassess employee competency when problems are identified with employee performance. If, after reeducation and training, the employee is unable to satisfactorily pass the assessment, then further action should be taken, which, may include supervisory review of work, reassignment of duties, or other actions deemed appropriate by the Laboratory Director.
GEN.58500	I	Is there documentation of retraining and reassessment for employees who initially fail to demonstrate satisfactory performance on competency assessment?		Documentation of retraining and reassessment of employees who initially fail competency assessment should be available.

The Joint Commission on Accreditation of Health Care Organizations

The JCAHO began evaluating hospital laboratory services in 1979. Since 1995, clinical laboratories surveyed using JCAHO standards have been deemed to be certifiable under CLIA '88 requirements. The current JCAHO laboratory standards include competency assessment of personnel under the Human Resources requirements and mandate that the organization provide for competent staff either through traditional employer-employee arrangements or through contractual arrangements with other entities or persons (Joint Commission of Accreditation of Health Care Organizations, 2003, 2004 Laboratory Standards: http://www.jcaho.org). JCAHO requires an initial review of credentials and qualifications of employees; it also requires that experience, education, and abilities be confirmed during orientation. JCAHO also mandates that the organization provide ongoing in-service and other education and training to increase staff knowledge of specific work-related issues and perform ongoing, periodic competence assessment to evaluate the continuing abilities of staff members to perform throughout their association with the organization (http://www.jcaho.org). The specific JCAHO standards involving competency assessment are indicated in Table 2.

TABLE 2.

JACHO standards regarding competency assessment

Standard no.	Standard	Explanation
HR.2.10	Orientation provides initial job training and information.	As appropriate, each staff member, student, and volunteer is oriented and then assessed to the following:
The organization assesses and documents each person's ability to carry out assigned responsibilities safely, competently, and in a timely manner on completion of orientation.
		The organization documents that each person has completed orientation and has been evaluated for competency in performing required laboratory tasks as well as other parameters defined in his or her job descriptions.
		Documentation of orientation participation includes written approval by the laboratory director or appropriate supervisor noting that the individual is capable of performing laboratory duties and confirmation by the employee that he or she feels qualified after orientation to perform the tasks required.
HR.2.30	Ongoing education, including in-services, training, and other activities, maintains and improves competence.	The following occurs for staff, students, and volunteers who work in the same capacity as staff providing care, treatment, and services:
Training occurs when job responsibilities or duties change.
Participation in ongoing in-services, training, or other activities occurs to increase staff, student, or volunteer knowledge of work-related issues.
		Ongoing in-services and other education and training are appropriate to the needs of the population(s) served and comply with law and regulation.
		Ongoing in-services, training, or other activities emphasize specific job-related aspects of safety and infection prevention and control.
		Ongoing in-services, training, or other education incorporate methods of team training, when appropriate.
		Ongoing in-services, training, or other education reinforce the need and ways to report unanticipated adverse events.
		Ongoing in-services or other education is offered in response to learning needs identified through performance improvement findings and other data analysis (that is, data from staff surveys, performance evaluations, or other needs assessments).
		Ongoing education is documented.
		At a minimum, for supervisory staff, attendance at outside workshops, institutes, and local, regional, or national society meetings occurs as feasible.
Standard     HR.3.10	Competence to perform job responsibilities is assessed, demonstrated, and maintained.	Competency assessment is systematic and allows for a measurable assessment of the person's ability to perform required activities. Information used as part of competency assessment may include data from performance evaluations, performance improvement, and aggregate data on competency, as well as the assessment of learning needs. This standard encompasses the following:
		The laboratory director or appropriate laboratory supervisor regularly assesses the continued competency of staff on all laboratory work shifts through performance evaluations.
		Staff members are evaluated for competency in performing required laboratory tasks as applicable, as well as for all other parameters defined in their job descriptions.
		Supervisory staff are evaluated for performance of their job responsibilities, as defined in their job descriptions.
		A job description and a completed competency assessment, an evaluation, or an appraisal tool are on file for each contracted or employed individual.
		Each staff member's performance is evaluated and documented after orientation and annually thereafter.
		An individual qualified to provide technical judgments about performance evaluates technical staff.
		The procedures to assess and document annually the competency of technical staff include but are not limited to the following:
		Routine patient test performance, including patient preparation, if applicable, and specimen collection, handling, processing, and testing.
		The recording and reporting of test results.
		QC, proficiency testing, and preventive maintenance performance.
		Instrument function checks and calibration performance.
		Test performance assessment as defined by laboratory policy (e.g., testing previously analyzed specimens, internal blind testing samples, and external proficiency or testing samples).
		Assessment of problem-solving skills as appropriate to the job.
		If a test method or instrumentation changes or the individual's duties change, his or her performance is reevaluated to include skills in the areas of change.
		Each laboratory employee performing such tests participates in the program.
		Acceptable performance criteria are established.
		Performance levels are documented.
		When indicated, remedial action is taken and documented.
Standard     LD.2.90	The laboratory director is responsible for determining the qualifications and competence of laboratory staff.	The director determines the procedures and tests that staff members are qualified and authorized to perform and is responsible for determining the competence and qualifications of laboratory staff. The director ensures that the level of supervision provided and the level of testing complexity is commensurate with the education, training, and experience of staff. The director must also require that staff demonstrate the ability to perform all duties before actually testing patient specimens and that staff maintain competencies to perform required tasks.

ELEMENTS OF A COMPETENCY ASSESSMENT PROGRAM

For a laboratory to comply with federal regulations and national accrediting agencies guidelines, a system must be in place that will allow verification of the initial training of staff and assessment of competence twice in the first year of employment and annually thereafter. Although CLIA '88 defines what must be tested in order to assess competence in laboratory employees, it does not specifically spell out how to do this assessment. Reflecting this was a study performed by Christian et al., who interviewed a sample of 20 laboratories including hospital, blood bank, commercial reference, physician office, and independent laboratories from 12 states (2). They found that assessing the competence of laboratory personnel was a complex issue reflecting the dynamics and environment of each unique laboratory. Their research found no consistent method of implementation of competency assessment. This is because there are many approaches and tools that can be utilized to meet the federal regulations. Four additional articles, specifically targeting competency assessment in clinical microbiology, have been published and can be reviewed prior to designing a competency assessment program for a microbiology laboratory (4, 14, 15, 18). In addition, tools and programs for use in laboratory competency assessment have also been included in publications concerning laboratory disciplines other than microbiology (6, 7, 9, 10, 20). One must also keep in mind that parts of a competency assessment program may be intimidating to some employees, some of whom may feel that it could jeopardize their relationship with coworkers. Care must be taken to assure the staff that the purpose of these programs, although required to meet governmental and accreditation agency requirements, is to identify areas where improvements can be made to ensure quality patient care.

As stated above, there are six areas that must be included as part of a competency assessment program: (i) direct observation of routine patient test performance; (ii) monitoring the recording and reporting of test results; (iii) review of intermediate test results, QC records, proficiency testing results, and preventive maintenance records; (iv) direct observation of performance of instrument maintenance and function checks; (v) assessment of test performance through testing previously analyzed specimens, internal blind testing samples, or external proficiency testing samples; and (vi) assessment of problem-solving skills (11). Ways to include each of the above six areas in a competency assessment program is discussed in greater detail in the following sections and summarized in Table 3. An example of a competency assessment form for bacteriology is reprinted from Cumitech 39 (4) and included in Fig. 1; a partially completed form is included in Fig. 2 as an example of how this form can be used. The reader is referred to Cumitech 39 for additional examples of competency assessment forms (4).

TABLE 3.

Summary of competency assessment^a

Items that must be included in a competency assessment program	Description of each item	Examples of each item
Direct observation of routine patient test performance	This is the actual observation of work as it is being performed by the laboratory staff. These observations are not limited to test performance but include all processes in which the employee is involved, including specimen collection and preparation, as well as the actual testing of the specimen.	Direct observation is used for areas involving a higher degree of decision making or which have a significant impact on patient care (e.g., new positive blood cultures, positive cerebrospinal fluid specimens, susceptibility testing, accurate interpretation of test reactions, following appropriate work instructions).
Monitoring the recording and reporting of test results	Review of patient results for the proper and correct recording and reporting.	This can be accomplished by the documentation of observation of an employee writing or entering patient test results on report forms or into the computer or by review of worksheets with report forms or computer entries.
Review of intermediate test results, QC records, proficiency testing results, and preventive maintenance records	This is as it is implied: one must review intermediate patient results, QC records, proficiency testing results and preventive maintenance records.	This can be accomplished by review of worksheets or computer entries for accurate recording of patient results, review of QC worksheets or printouts for acceptable results (within QC parameters) and for review of preventive maintenance records for the appropriate and timely checks and documentation.
Direct observation of performance of instrument maintenance and function checks	Direct observation must be used when employees are performing maintenance procedures and check of instruments.	One must directly observe an employee when performing maintenance procedures and function checks on instruments in the laboratory, such as the automated identification/susceptibility testing instrument, molecular diagnostic instrumentation, and blood culture instrumentation.
Assessment of test performance through testing previously analyzed specimens, internal blind testing samples, or external proficiency testing samples	One must assess employee competence by giving them unknown samples to evaluate as they would evaluate patient samples in the laboratory.	This can be accomplished by split-sample analysis, previously analyzed specimens, blind internal proficiency testing, or external proficiency testing such as CAP surveys, etc.
Assessment of problem-solving skills	One must assess the ability of employees to solve problems that arise during their practice.	This can be accomplished by (i) asking the employees to write up a situation where they had to solve a problem that related to an investigation they performed or (ii) giving a fictitious (or real) example of a problem encountered in the laboratory and asking the employee how he or she would handle the situation.

^aThis table summarizes the information included in the “Elements of a competency assessment program” section of this paper, to include the six areas of CLIA required assessment, a description of each requirement, and examples of how each could be accomplished.

FIG. 1.

Example of how the six areas of required CLIA competency assessment can be addressed and documented. FQ, fluoroquinolones. Reprinted from reference 4 with permission.

FIG. 2.

Example of how the assessment form can be used for documentation of competency.

The six areas that must be included as part of a competency assessment program are discussed in detail in the following sections.

Direct Observation of Routine Patient Test Performance

Direct observation is the actual observation of work as it is being performed by laboratory staff. These observations are not limited to test performance but include all processes in which the employee is involved, including specimen collection, preparation of the specimen for laboratory testing, and the actual testing of the specimen. Direct observation can be the most time-consuming way to monitor employee competency (particularly when the laboratory is large), and the areas to monitor should be carefully selected to maximize gains from the time spent in the process. For example, areas which involve a higher-than-average degree of decision making, which may have a major impact on patient care if performed incorrectly, or which have been found over time to have a greater degree of employee variability might all be good prospects for direct observation. Smaller laboratories with only a few staff members may find direct observation to be less onerous, and these laboratories can be more inclusive in the areas chosen for observation. Elder and Sharp provide an example that utilizes a statement included in the laboratory's competency assessment program indicating that a certain percentage of routine work is observed through direct visual evaluation. This can be followed by either a specific listing of tests to be observed or a general listing of tests that may be included in the direct-observation portion of competency assessment (4). McCaskey and LaRocco utilized direct observation in employee competency assessment of processing and reporting of new positive blood cultures, reading and reporting of positive routine cultures, automated identification procedures, susceptibility testing, rapid antigen testing, direct smears and fluorescent smears, as well as a large variety of biochemical testing performed (15). They also included a variety of checks while performing direct observation, including adherence to written protocols, accurate interpretation of test reactions, and appropriate notification of results, as well as many others. McCarter and Robinson utilized direct observation to assess safety and specimen-processing procedures in mycobacteriology and QC (14). One must keep in mind that CLIA '88 mandates that “at least” routine patient test performance, as discussed here, and performance of instrument maintenance and function checks (see below) be assessed by direct observation.

Monitoring the Recording and Reporting of Test Results

Elder and Sharp indicate that monitoring the recording and reporting of test results requires a review of results for the proper and correct recording and reporting of patient testing (4). This is most easily accomplished either by documentation of observation of an employee writing or entering patient test results on report forms or into the computer or by a review of worksheets with computer entries for appropriate recording of patient results. This review can be done at the time a final report is verified (before the results have been released) or after verification through comparison of worksheets and computer printouts. McCarter and Robinson reviewed worksheets and patient records in bacteriology to assess blood culture competency. This method was also applied in their laboratory to selected areas of the mycobacteriology, mycology, virology and serology sections (14).

Review of Intermediate Test Results or Worksheets, QC Records, Proficiency Testing Results, and Preventive Maintenance Records

Review of results and records may also be accomplished by directly observing an employee when writing or entering preliminary patient test results onto report forms or into the computer or by reviewing worksheets or computer entries for appropriate recording of preliminary patient results (4). Unless all worksheets or reports are going to be reviewed, effort should again be taken to ensure that the time spent reviewing test recording and reporting provides the best assessment of competency (e.g., review of positive cultures, review of results from critical specimens, and review of worksheets from culture types with complicated workups). Supervisor (or designee) review of QC records, proficiency testing results, and preventative maintenance records is most easily performed as a documented review of previous data entries, as is already routinely performed in laboratories to meet the QC requirements for accreditation (4).

Direct Observation of Performance of Instrument Maintenance and Function Checks

Direct observation must be done when employees are performing maintenance procedures and checks of instruments. Documentation of these observations is necessary for competency assessment and cannot be performed by an alternative method (4, 11). This should be assessed for each piece of equipment that the person being assessed is trained to operate (4). McCaskey and LaRocco utilized direct observation in all activities related to instrument monitoring, maintenance, and function checks, while McCarter and Robinson utilized direct observation for instrument function checks for RPR (Rapid Plasma Reagin) testing in the serology section (14, 15).

Assessment of Test Performance through Testing Previously Analyzed Specimens, Internal Blind Testing Samples, or External Proficiency Testing Samples

Blind retesting of previously analyzed specimens can be used as an assessment in a number of different areas of the laboratory, such as appropriate setup based on the source of the unknown organisms, correct identification of unknown organisms, appropriate titers of infectious-diseases serologies, testing and reporting of antimicrobial susceptibility results, and many more (4). In addition to using previously analyzed specimens, performing testing on unknown samples or split samples as part of a proficiency testing program or as part of an internal quality assurance program can serve to meet this requirement (4). Optimally, each employee is assigned at least one proficiency testing sample that applies to each area included in his or her scope of responsibility per competency evaluation period (15). Utilizing internal blind unknown samples prepared by the supervisory staff from known organisms, seeded specimens, or previously analyzed samples can accomplish this goal. As another example, McCarter and Robinson utilized previously analyzed specimens to assess competency for agglutination and enzyme immunoassay testing in the serology section. Employees were expected to retrieve specimens from coded samples maintained at −70°C and incorporate them into their daily testing (14).

Assessment of Problem-Solving Skills

Assessment of problem-solving skills may be accomplished in several ways (4). Examples include (i) asking employees to respond orally or in writing to simulated technical or procedural problems (perhaps in the form of case studies) and (ii) asking employees to document actual problem-solving issues that they have handled in the laboratory within the last year.

A specific example of a problem-solving skill as utilized by a microbiology technologist is outlined as follows. An occasion developed where cultures from two patients that were processed for mycobacteria on the same day both grew Mycobacterium tuberculosis. One of the patients (patient A) was smear positive with numerous acid-fast bacilli, while the other patient (patient B) was smear negative for acid-fast bacilli. The culture from patient A was positive after 10 days of incubation, while the culture from patient B was positive after 18 days of incubation. The technologist (Tech 1) noticed this situation and questioned whether patient B's sample may have been contaminated by the smear-positive sample from patient A. It was decided, after consultation with the supervisor, that both M. tuberculosis isolates would be sent for molecular testing to determine if they were in fact the same organism. Tech 1 discussed the situation with the less experienced technologist (Tech 2) who initially processed the specimens, in order to determine how this might have happened. No obvious reason was identified. Tech 1 and the supervisor decided that competency assessment might shed some light on the situation, and Tech 1 was assigned to carry out direct observation of Tech 2 as she processed specimens for mycobacterial smear and culture. While carrying out this observation, Tech 1 found that Tech 2 was not capping specimen transfer tubes after adding a patient's sample prior to transferring specimen from the next patient. Tech 1 discussed this with the supervisor, and both believed that this break in protocol may have led to the suspected contamination (which was subsequently confirmed by molecular testing). Due to this deviation from the standard protocol by Tech 2, the supervisor decided that direct observations were warranted for all the Mycobacterium-processing technologists to ensure that proper techniques were being adhered to by everyone. In this instance, the problem-solving skills of Tech 1 led to competency assessment by direct observation of Tech 2, which solved the issue at hand and assisted the laboratory in improving the quality of future results from the mycobacteriology laboratory.

The above example was taken, in part, from the American Society for Microbiology's Division C web site on Competency Assessment (www.asm.org/Division/c/competency.htm; accessed 21 December 2003; reprinted with permission). This site also includes other examples of problem solving as well as other issues dealing with competency assessment in the clinical microbiology laboratory.

Laboratory employees solve problems very often but are frequently not aware that they are doing so. Encouraging the employees to document problem-solving situations as they occur during the year (rather than once a year when summarizing competency assessments) will facilitate this portion of the assessment process. McCarter and Robinson required at least three problem-solving examples per year per employee (14), while McCaskey and LaRocco required five separate examples in writing of problem-solving skills per competency evaluation period (15). Further, they required an employee to include four areas in their problem-solving examples, which were to (i) identify the problem, (ii) perform and document steps taken to correct the problem, (iii) resolve the problem by adhering to and correctly applying hospital and departmental procedures, and (iv) if resolution is not possible, document the reason why a resolution could not be reached and indicate suggestions for further action that may contribute to resolution of the problem (15).

Both McCaskey and LaRocco (15) and McCarter and Robinson (14) utilized written tests to assess the individual's scope of knowledge in a specific area. However, the use of examinations (written or practical), although aiding the process of competency assessment, will not completely satisfy the regulatory requirements or provide a complete look at an employee's competence (14, 15; Virtual Hospital [www.vh.org/adult/provider/pathology/CLIA/CLIAHP.html]). Written examinations can be particularly useful in providing problem-solving scenarios but are generally unable to comprehensively reflect the many different facets of knowledge and judgement that must be used by employees in job performance. Written testing is not highly recommended by the CAP since it was the method of evaluation with the poorest compliance. The CAP recommends that written testing not be used as an element of a competency assessment plan unless it can be performed consistently (12).

DEVELOPMENT OF A COMPETENCY PROGRAM

The initial task of developing a competency assessment program can seem daunting, but it can be approached in a number of different ways. The steps taken to define the program should be included as part of the laboratory's competency program procedure. The steps commonly performed during the development of a program are discussed in the following sections.

Define Areas Requiring Competency Assessment

One of the most time-consuming portions of program development is identification of the areas requiring competency assessment, and this necessitates analysis of like tasks and skills (4). For example, identification of an isolate from a blood agar plate will be done in a similar fashion regardless of the source (e.g., urine, blood, or tissue) of the specimen. Therefore, it is not necessary to assess individual competency in each work area or division in the laboratory. On the other hand, the ability to assess whether an organism needs to be identified will vary from source to source. Similarly, the performance, recording, and QC of simple latex tests will not vary considerably from kit to kit and may be adequately assessed through evaluation of the employee's performance with any one of several different kits. Performance of this first step in program development must be done in sufficient detail that it is clear (to the person performing the assessment as well as to an inspector) what will be assessed, but with consideration of the similarity between many laboratory tasks. Organization of the areas to be assessed may be performed by bench assignment (respiratory specimens, stool specimens, etc.) or by test type (biochemical test, serologic test, etc.). As an example, areas requiring competency assessment for the anaerobe bench might include culture setup, selection of appropriate organisms for identification, identification of organisms, utilization of the anaerobic chamber, reporting of test results, and notification of critical values. One approach is to emphasize areas or methods in the design of the competency assessment program that are either problem prone or at high risk for error. Data from the laboratory's quality improvement processes (reviews of amended reports, incident reports, etc.) may be helpful in making this determination.

McCaskey and LaRocco used a team-based approach to define ongoing activities in QC and quality assurance that could easily be included in the competency program (15). They drafted lists for all tests and procedures for each subspecialty and developed a program where an employee participated in the process by selecting items from the test lists and scheduling the exercise to take place with an observer at a mutually convenient time. They felt that this participation created a more cooperative spirit between the observer and the person being evaluated and helped to eliminate negative associations with the competency assessment exercises. Care must be taken with this approach that employees are not always calling on their friends to act as observer for competency assessment, which may sway the impact of the program. These authors also stratified their activities into categories (category 1, 2, 3, or 4). Category 1 items were competencies that were deemed most critical for patient care, and employees were to be evaluated in all category 1 items during each evaluation period. In other categories, the employee could choose from several items for inclusion in their evaluation process (15). Similar to McCaskey and LaRocco, McCarter and Robinson created forms based on procedure-oriented tasks for each specialty area to be used in their competency assessment program (14). Competency assessment must also be specific for each job description; this must be taken into account when defining areas required for competency assessment, and competencies specific for each position within the microbiology laboratory must be included (12).

One of the challenges for any laboratory in establishing a competency assessment program is defining the extent of assessment that will be performed in each area once training is completed (4). Is it adequate to observe an employee work up one blood culture, or do 5 or 10 blood culture workups need to be observed? Should an employee be asked to demonstrate his or her ability to solve problems in each area of the laboratory, or is it sufficient to document problem-solving skills in only two or three key areas? Each laboratory will need to determine the extent of assessment in a way that best fits its size and complexity. For example, performing five anaerobic cultures for a successful competency assessment might prove quite impossible for a small laboratory where only one or two anaerobic cultures are performed per week or equally difficult for a large laboratory where multiple technologists perform anaerobic cultures. In this situation, instead of observing individuals performing anaerobic culture workup, direct observation of competency might be achieved through the use of a practical examination. Plates with important anaerobes and mixed organisms could be prepared and used to observe the employee's subsequent workup. If all employees performing anaerobic cultures were tested at the same time, the setup time would be reduced (4).

A helpful approach to solving the problem of how much to include in a competency assessment is to incorporate this goal into the integral part of other activities already occurring routinely in the laboratory (4). For example, performing competency assessments during routine review of QC records, review of positive-culture worksheets by a supervisor or designee, and review of results of proficiency testing surveys in which employees have participated are ways to incorporate the competency assessment program into the daily activities of the laboratory and lessen the workload associated with mandated competency assessments (4).

Identify Methods of Competency Assessment

The methods used in competency assessment should initially be driven by what is required by CLIA '88 as listed in the CFR for routine patient test performances (observation, review, proficiency testing, etc.).

Each type of assessment does not need to be performed for each area being assessed, and the type of assessment tool selected for use should be based on whether it will provide an accurate reflection of employee competency (4). As part of this process, it is very helpful to define what will be considered a successful demonstration of competency. This may be considerably different when an employee is being trained in a new area and is demonstrating competency for the first time and when an employee is demonstrating ongoing competency. For example, criteria established for an employee being evaluated following initial training in the anaerobe area will primarily utilize direct observation to assess the employee's ability to correctly follow the laboratory procedure while inoculating and incubating specimens for anaerobic culture; to identify inappropriate specimens for anaerobic culture; to demonstrate or describe the procedure followed when inappropriate specimens are received in the laboratory; to appropriately follow laboratory procedures while interpreting, working up, and reporting the results of anaerobic cultures; and to perform all required maintenance of the anaerobic chamber. In contrast, an evaluation for ongoing competency in the area of anaerobes for an experienced employee could be performed by a combination of several of the following: direct observation of the employee's workup of several cultures, indicating no deviations from written procedures; daily supervisor review of employee worksheets of positive cultures, indicating that the employee correctly selects appropriate identification and susceptibility tests and has followed the critical-value policy correctly; demonstration by the employee of the required maintenance for the anaerobic chamber; demonstration (through documentation from actual examples or through a practical examination) of the employee's ability to correctly identify and resolve problem situations with anaerobic cultures; or the use of proficiency testing samples to assess the ability of the employee to correctly identify anaerobic bacterial pathogens (4).

Problem solving, as already mentioned, could also be documented by employees throughout the year. One suggestion is to provide employees with a notebook, which will fit in a lab coat pocket, that can be used for documentation as situations occur and that will then be turned in to the manager at a scheduled time (4). This booklet could also include a schedule of other required elements of annual competency assessment (e.g., observed instrument maintenance) that the employee would be responsible for scheduling with a supervisor or designee. Use of such a booklet also helps place responsibility for part of the competency assessment with the employee. CLIA '88 does not make clear the number of assessments that must be performed in this area, only that it must be done. Each individual laboratory will have to determine the number of competency assessments in this area that it will require or the areas in which problem solving must occur.

Determine Who Will Perform Competency Assessment

Part of the written procedure for a competency program should include how competency assessment is determined and who will be allowed to perform the assessment. Although CLIA '88 states that the supervisor is responsible for competency assessment, it does not state that all assessments must be performed by the supervisor. Supervisors may choose to designate certain employees (e.g., lead technologists or employees with several years of documented successful competency) to assist with assessments (14, 15). These employees may be authorized to perform assessment in only a few tests or in multiple laboratory areas. In addition, these employees can perform competency assessment of supervisory personnel who also perform patient testing (4). The ability of certain staff members to serve as assessors of competency of other employees should be documented on their own competency assessment, e.g., “This employee has demonstrated competency in the area of {…} within the laboratory and is capable of assessing the competency of others in this area” (4). In this way, is it obvious to an inspector that a qualified employee performed the competency assessment.

Define the Documentation of Competency Assessment

A variety of manual and computerized tools are available for documentation of competency assessment, and examples of these are included in selected references (4, 6, 7, 9, 14-18; ASM Division C website [www.asm.org/Division/c/competency.html]; and in Antimicrobial Susceptibility Testing—a self-study program, Department of Health and Human Services and the CDC Foundation, 2002 [www.aphl.org/ast.cfm]). There are also a variety of commercially available manual guides (4, 16) and web-based or software systems (SoftETC [www.soft-etc.com]; Comptec-ASCP [www.asco.org]; Media Lab, Inc. [www.medialabinc.net]; GramStain-Tutor [medical.software-directory.com]; and ExamManager [www.exammanager.com]) available to assist in the development of a laboratory competency assessment program. Unless a decision is made to utilize one of these systems, easily used forms will have to be developed for documentation and to provide evidence of who was evaluated, what was evaluated, how it was evaluated, when it was evaluated, who performed the evaluation, what was done if problems were identified, whether the employee is authorized to perform and release results independently or whether review of the work is required before results are released, and whether the employee can serve as a competency assessor for other employees. Since the medical director is ultimately responsible under CLIA '88 for determining who will be allowed to work in the laboratory and what testing they can perform with or without supervision, it is prudent for the medical director to either review and sign the employee competency documentation or to delegate this task in writing to the supervisor or other appropriate personnel. Competency assessment records and forms should be retained for the entire time an individual is employed at the laboratory. Once an individual is no longer employed, discussions with Human Resources personnel can determine the appropriate length of time that competency records should be maintained for that facility.

REMEDIATION

The goal of competency assessment is to identify potential problems with employee performance and to address these issues before they affect patient care. Thus, performance and documentation of remediation is a critical component of the competency assessment process and is required by both CAP and JCAHO. Unless an employee has been deliberately negligent in the performance of his or her work, remediation should not be punitive but should, instead, be educational, and it should always be directed at improving performance (4). Employees who recognize that their mistakes will be addressed with the aim of performance improvement will be far more likely to seek assistance and admit problems than those who fear embarrassment, disciplinary action, or termination. (D. Marx, 2001, Patient safety and the “just culture”: a primer for health care executives; Columbia University. [http://www.mers.tm.net/support/Marx_Primer.pdf]).

A number of approaches can be taken to remedy problems identified through the competency assessment process, and some of these are outlined below (4). Since problems may develop because of the system rather than the employee, the first step is to analyze the problem so that the proper remediation can be identified and implemented. Analysis of the problem starts with looking at the protocols used for laboratory practice. The protocols should be clear and concise; if they are inadequate or confusing, this may account for the failure of competency of the employee. In proficiency testing, it should be ensured that the sample used as an unknown is adequate and that a problem with the sample itself is not what caused the competency failure. Also, the tools used for evaluation of competency should be clear, so that a consistent standard is applied to all employees.

If the above protocols are deemed sufficient and are not the cause of the competency failure, then one needs to identify the problem the employee is having. Is it a methodology problem, did the employee not perform the test correctly (i.e., did he or she not follow procedure), did the employee not understand the purpose or background of test (i.e., is he or she unable to solve problems or relate the test to the clinical situation), did the employee not understand the components of the test or instrument being used, was the employee unable to resolve QC problems, or did the employee perform correctly but made an error in documentation?

If necessary, an appropriate remedial action should be selected (4). First, discussion of the procedure with the employee is warranted to assess if further action is necessary based on the employee's verbal response. This step may be all that is necessary to identify the reason for the competency failure. Discussion of the procedure in a quality assurance-QC meeting with all employees could help everybody to understand how this type of error can be avoided. Additional actions that can be taken with an employee who fails competency include having the employee reread the procedure and discuss it with the supervisor to clarify any misinterpretations, having the employee produce a flow chart to assist him or her in properly performing a procedure, having the employee observe another trained and competent employee, having the employee practice the failed procedure with known specimens, or having the employee correctly retest the same specimen with the procedure that originally failed. Reinstitution of formal training may be necessary if the above opportunities fail to show that the employee is competent. Regardless of the method selected for remediation, it is necessary to repeat the competency assessment once remediation has been completed in order to document successful attainment of competency. As a last resort, it may be necessary to permanently remove an employee from selected duties and reassign him or her to another work area.

When an error or failure of competency was noted by McCaskey and LaRocco, corrective action was necessary within 30 days of the finding at their institution (15). If their corrective action did not resolve the failure, the employee was not allowed to perform patient testing in that area until he or she had completed further remedial action and had his or her competency reevaluated and was determined to be acceptable. Similarly, McCarter and Robinson did not permit employees who failed competency assessment to perform testing in that area until corrective action was determined (14). Following corrective action, the employee was reevaluated, and if the corrective action had been effective, the employee was considered to be competent. If the corrective action was not effective, the individual was not permitted to perform testing in the affected area until remedial training was successfully completed. In general, remediation should be instituted as quickly as possible after identification of a potential problem with employee competency. Each situation can be assessed initially to determine the extent of the problem and to determine if the employee understands the situation that has occurred, as well as the way it should have been handled. Based on this initial assessment, a decision can be made at that time about whether the employee should be allowed to continue to work independently in the area while further remediation or competency assessment (for example, direct observation) is carried out or whether the employee's work should be restricted until remediation and competence are fully documented.

QUALITY RESULTS

A formal defined competency program provides the laboratory with a valuable tool for identifying and correcting issues of employee competency. Just as valuable is the use of competency assessment as an ongoing part of the laboratory's quality assurance program to assist managers and supervisors in ensuring that high-quality results are reported. Competency assessment is an integral part of problem analysis and becomes a key tool in ensuring that errors identified through the quality assurance processes are prevented from recurring. Competency assessment procedures can help to identify problems occurring in the technical aspects of laboratory practice and assess performance deficiencies before they develop into major problems (7, 9, 10, 18).

Competency assessment is also an opportunity to provide continuing education and performance feedback to employees and to document valuable objective information for performance evaluations (15). It should and can be used as a positive experience that helps to ensure that employees and employers can perform assigned tasks.