Assessment of Proficiency and Competency in Laboratory Animal Biomethodologies

Abstract

Personnel working with laboratory animals are required by laws and guidelines to be trained and qualified to perform biomethodologic procedures. The assessment of competency and proficiency is a vital component of a laboratory animal training program, because this process confirms that the trainees have met the learning objectives for a particular procedure. The approach toward qualification assessment differs between organizations because laws and guidelines do not outline how the assessment should be performed or which methods and tools should be used. Assessment of clinical and surgical medicine has received considerable attention over the last few decades and has progressed from simple subjective methods to well-defined and objective methods of assessing competency. Although biomethodology competency and proficiency assessment is discussed in the literature, a standard and objective assessment method has not yet been developed. The development and implementation of an objective and standardized biomethodologic assessment program can serve as a tool to improve standards, ensure consistent training, and decrease research variables yet ensure animal welfare. Here we review the definition and goals of training and assessment, review assessment methods, and propose a method to develop a standard and objective assessment program for the laboratory animal science field, particularly training departments and IACUC.

The laboratory animal science (LAS) field has a long-standing tradition of high standards in animal care. Training for personnel working with animals is required by various national and international guidelines and laws.^1,2,21,30,32 In addition, proficient personnel minimize variables in experiments and improve animal welfare by reducing mistakes while working with animals^4,38 and limit stress to animals and its potential effects on the physiologic state.^9,11 Developing and implementing assessment methods is challenging and time-consuming. Adequate training should include assessment of competency to ensure proper techniques were learned and assessment of proficiency to ensure consistent and appropriate application of learned techniques.^15,20,25 Assessment programs provide a “framework for accurate, reliable, and valid assessment of the applicable competency standards.”³ In addition, high-quality assessment programs ensure that all assessment methods are complete, reliable, and valid. This attention helps to provide consistency among qualifications assessments.³ Assessment of biomethodologic competency has been a topic of discussion within the LAS field; however a standard and objective method has yet to be developed for these techniques. Assessment of biomethodologies is challenging for the LAS community for 3 primary reasons. First, the pool of trainees is diverse^20,28 and thus complicates the design of a single type of training and assessment program. Second, the number and diversity of techniques is vast, and third, the student's ability to acquire skills typically falls into 1 of 3 challenging learning domains.³⁷ These domains include concept learning (also known as category learning), cognitive skills (learning theory), and psychomotor (physical) skills.²⁴ Concept learning requires the gathering and organizing of information. Cognitive skills require critical thinking and problem solving. Psychomotor skills are complex because they are acquired through the cognitive, associative, and autonomous stages.²⁴ During the cognitive stage, the student learns theory and concepts. During the associative stage, the student practices the skill, and by the autonomous phase, the skill is performed with confidence and decreased mental effort.³⁷ Therefore, biomethodologic training assessment must consider, and allow for, assessment of all 3 learning domains.

The assessment methods we describe here provide a framework for both competency and proficiency assessment. In particular, we clarify the definition and goals of training and assessment, review methods for assessment currently in the industry, including those in the human clinical and surgical settings, and describe how to develop and implement a standardized and objective biomethodologic competency and proficiency assessment program.

The development and implementation of an objective and standardized assessment program can serve as one tool to improve standards, ensure consistent training, and decrease research variables while ensuring animal welfare.

Definitions and Goals of Biomethodologic Training and Assessment

Biomethodologic techniques.

Biomethodologic techniques have been defined as “those that permit one to monitor and record experimental data.”⁴⁰ These techniques are tools used to perform studies and limit or eliminate confounding factors and include animal handling and restraint, sample collection, anesthesia and surgery.^7,13 For consistency, we refer to these techniques (or procedures) as biomethods throughout this overview. Biomethodologic training refers to the physical act of teaching and the learning of these techniques. For our purposes here, basic biomethodologic training refers to teaching the methods for procedures. The goal of biomethodology training is teaching correct techniques while ensuring the humane use of the animals, the safety of the handlers, and the quality of research data.

Biomethodologic competency.

Biomethodologic competency refers to the ability to perform biomethodologic procedures according to standards accepted by the global LAS community.¹⁶ The term ‘competence’ refers to a trainee's potential knowledge and skill whereas ‘competency’ refers to a trainee's ability to perform and apply knowledge and skills.^5,34 Therefore, competency means that the trainee is able to demonstrate the knowledge and ability required to perform the biomethod and is aware of personal limitations. Competency assessment refers to a summative process that collects evidence about a trainee's progress; this evidence then is used to determine whether the trainee has achieved the ability to perform specific biomethods appropriately.^14,16,26

Competency assessment is performed during or immediately after the training session and is accomplished through observing a trainee perform a task in a controlled environment and evaluating whether the task was performed correctly.¹⁴

Biomethodologic proficiency.

Proficiency is defined as “advancement in knowledge or skill.”³⁰ The terms proficiency and competency are often used interchangeably in the published literature. However, we suggest that competency is achieved before proficiency. Biomethodologic proficiency means that the trainee can consistently and accurately perform the biomethod. Proficiency assessment is made after training (after the trainee has had the opportunity to practice and refine the newly learned skill) and may include measurements of speed and refined accuracy. The goal of proficiency assessment is to ensure that the trainee has not deviated from correct technique. Proficiency assessment should be performed by someone other than the person who provided the original training, to minimize bias and ensure objectivity.

Understanding the difference between competency and proficiency and knowing the goals for each will aid in using the tools provided herein to effectively develop and implement biomethodologic assessment programs for LAS personnel.

Methods of Training and Assessment in Laboratory Animal Science

Oral.

Oral assessments are beneficial for evaluating factual knowledge, especially for nonnative language speakers³⁹ and those with poor written test-taking skills. Oral assessment methods allow the assessor to learn the depth of trainee's knowledge, because questions can be adjusted during the time of assessment. These assessments also allow the instructor to support correct responses, to facilitate learning. However, it is difficult to make oral assessments objective and standardized.²⁹

Written.

Questionnaires or written exams may be used as part of the overall assessment. Questionnaires can be used before the training session, to assess the trainee's prior experience with specific biomethods, or after the training session, to assess the trainee's comfort level with the biomethods presented.

Written exams (or assessments) include a series of knowledge questions that are answered either on paper or on a computer. Questions can be in a variety of formats, including true–false, matching, multiple choice, matching, short answer, or essay. Each type of question evaluates factual knowledge in a different way, and each type has strengths and weaknesses. For example, true–false questions are good for testing knowledge about common misconceptions. A large amount of content can be tested quickly by using true–false questions; however, students have a 50–50 chance of answering correctly by guessing.³⁹ Alternatively, multiple-choice questions are very effective at evaluating knowledge.⁶ This format is the most commonly used in educational assessment even though it is very challenging to develop good multiple-choice questions. The additional question types begin to limit the amount of information that can be tested, due to the length of time it takes to answer each question. In general, written exams are more difficult for nonnative language speakers and for those with poor test-taking skills. The advantage of this format is the ability to standardize questions with objective grading. However, a disadvantage is the inability to assess technical competency or proficiency,⁸ given that written assessments do not directly measure performance.¹³

Training checklists and objective structured assessments of technical skills.

Training checklists can be used by either trainers or trainees. When used by a trainer, a checklist of procedures is used to document whether the trainee has demonstrated an understanding of the biomethod presented and is able to perform them. Used by the trainee, a training checklist may be a log of procedures that is maintained by the trainee. This log can provide a summary of the trainee's exposure to particular procedures. The challenge with this method is the inability to measure the actual level of technical competency without further assessment.^5,25

Methodology focused on objective structured assessments of technical skills has been validated in the human medical and surgical fields and involve the observation of trainees while they perform a standardized task. Observation should be performed by a minimum of 2 trainers. The trainee's performance is assessed by using 2 types of scoring sheets: a global scoring sheet and a training checklist. The global scoring sheet addresses predetermined tasks which are given ratings (for example, 1 [worst score] to 5 [best score]). The assessed items can range from animal handling to more advanced biomethods, such as the intravascular catheter placement technique. The score sheet incorporates core competencies and provides a consistent method to provide training and assess both competency and progress of training. The score sheet includes a list of correct–incorrect–not performed objectives that are specific to the procedure and that have been taught during a training session. This method can be standardized, and objective assessment can be implemented.^{8,12, 27,35}

Self-assessment.

Self-assessment is an evaluation and judgment of one's own abilities, actions, and attitudes.¹⁰ The ability to self-assess one's performance is an indispensable tool that should be used throughout a person's career.³³ However, this skill is not often addressed in the LAS field. Self-assessment is especially important within the LAS field because biomethods often are learned during workshops, and continued assessment feedback from trainers is rarely available. The ability to self-assess allows the identification of strengths but, perhaps more importantly, identifies weaknesses to allow for improvement in the suboptimal areas.³³ However, self-assessment is a difficult task to learn, and often self-assessment ratings are lower than those during an assessment performed by experts.^19,36 This issue could be due to novelty and unfamiliarity with self-assessment tools and techniques and can be addressed by providing and explaining the self-assessment tools to trainees. After trainees have learned self-assessment methods, they should continue to attend workshops during which feedback from experts is available to confirm their assessments. Periodic feedback confirms that a trainee has not deviated from the original competency.

AALAS certification.

AALAS certification offers competency assessment at 3 different levels for specific knowledge. Certified technicians are recognized by the industry as experienced practitioners with knowledge in specific areas. Although it confirms knowledge of how to perform species-specific procedures, AALAS certification does not offer proficiency assessment in the performance of actual techniques.

The Educational and Training Guide in the Care and Use of Laboratory Animals: A Guide for Developing Institutional Programs offers an example of self-assessment for species-specific training.⁹ Although this publication explains that these guidelines can be used both by trainers (to assess training) as well as learners (as self-assessment) to determine whether basic objectives for species-specific (or biomethodologic) training have been met, it does not actually provide an objective means to assess competency. For example, the following excerpt provides guidance on how to assess oneself on a particular procedure:

Assemble all instruments and material for performing a given procedure. Indicate the structures or landmarks that will guide performance of the technique, including any structures that must be avoided. State how you will know if the procedure is progressing as planned and how to respond to an error.⁹

Although this excerpt provides guidance on developing technical skill, it still results in a subjective assessment. Other resources suggest using computer- and web-based resources as self-assessment tools.³³ Although self-assessment is helpful when developing technical skills, it does not ensure competency or proficiency.

Methods for the Development of Standardized and Objective Biomethodologic Competency and Proficiency Assessment Program

A consistent, effective, biomethodology training program relies on a standardized, objective method of assessment. There are many obstacles that prevent the development of a standardized assessment program. One includes the scope of the procedures to include and the labor-intensive work that must be incorporated into developing a program.¹⁴ Identifying highly qualified trainers and assessors represents another obstacle. Another challenge revolves around determining how to perform competency assessment appropriately to ensure that the newly trained person knows how to perform the predetermined task. Currently, and in most cases, this assessment is subjective¹⁴ and can vary markedly between assessors. An additional challenge is how to assess proficiency, given that competency does not guarantee proficiency.¹⁴ A reminder—competency is the knowledge and understanding of the task, and proficiency assessment determines whether the task is being performed correctly and accurately on the job.

During the development of competency assessment programs, issues of consistency, reproducibility, and defensibility must be taken into consideration.⁸ This goal becomes more challenging as the tasks become more complex. For example, the assessment of mouse handling is not as challenging as is assessing mouse tail-vein injection. The more complex the procedure, the more steps involved, and those steps themselves become more complex as the level of difficulty of the procedure increases.The development phase of an assessment program should include deciding on the goal of the assessment; identifying the procedures that will be included in the assessment program; dissecting each procedure into core competencies; developing training score sheets, competency assessment checklists, or a verification-of-proficiency manual; and determining how assessment methods will be administered and documented.³¹

Deciding on the goal of competency and proficiency assessment.

The overall goal of an assessment program is determining that personnel have appropriate competency and proficiency in a given biomethodology while keeping in mind the welfare of the animals involved and the safety of trainees. This goal must be supported throughout all levels of the organization for the goal to be achieved successfully. Understanding the difference between competency and proficiency is crucial to develop a consistent program throughout the organization and to gain support from its end users.

Before development of the assessment program, the institution must decide on the objectives of the program. Will competency be assessed after mandatory training sessions? Will consistent training or documentation of progress be incorporated? Will proficiency only be assessed? Will animal users be assessed at several time points? These questions need to be considered and answered before beginning to develop an institutional program.

Methods to develop competency-assessment score sheets and checklists.

Because assessment during training sessions can be subjective, having a consistent approach to training is essential. A recommended approach to delivering consistent training is the use of competency assessment score sheets. When considering the development of a competency-assessment score sheet (Figure 1), a scoring method should be determined. If an average grade obtained by adding up the scores from each task will be used, it is important to set a minimal score for each task, so that substandard portions of the task are not ameliorated by tasks performed slightly above standard. There would be no way to ensure competency without a minimal score. In reality, trainees have to be able to perform each task of the biomethod competently for the entire biomethod to be successful.⁷ To develop the score sheet, consider a specific biomethod and the relevant assessment criteria or its competency assessment protocol (CAP). For example, by using the criteria decided for manual restraint in a mouse, a training score sheet can be developed that describes acceptable, suboptimal, and unacceptable methods. Figure 1 illustrates a score sheet developed to guide and assess for dorsal scruff restraint training.

Figure 1.

Competency assessment score sheet for dorsal scruff restraint technique. The first column lists the core competency. The second column addresses the ‘unacceptable’ method, scored as a 0. The third column lists the ‘needs improvement’ method, scored as a 1. The fourth column indicates the ‘acceptable’ method, scored as a 2. This score sheet is using a 3-score competency system (0, 1, 2), with a maximum score of 6, which is the passing score. It also includes instructions on scoring to ensure that trainees who perform unacceptable methods are not assessed as competent.

The score sheet allows for consistent, objective competency assessment and documentation of training progress. This tool helps the trainee and trainer determine whether the trainee is ready for proficiency assessment. Score sheets can be effective in evaluating a trainee's progress during the training period. However, this process differs from the evaluation of proficiency, for which there is no minimal score but only pass or fail. When proficiency is assessed, trainees either know how to perform the biomethod independently or they do not; if they do not know, more training is needed.

Another method that can be used to ensure consistent training is a competency-assessment checklist. This checklist simply lists the procedures that will be covered in the training session. These procedures are broken down into their core competencies. Again—doing so ensures that the trainer is assessing the trainee according to established guidelines and is providing the trainee with clear expectations on how to achieve competency. When competency is achieved, the trainer and trainee initial the box beside the specific competency. An example of a training checklist for mouse physical restraint is demonstrated in Figure 2.

Figure 2.

Competency assessment checklist for mouse physical restraint technique. Note that this form requires initials from both the trainee and the trainer, ensuring that both persons understand how the procedure is being assessed.

Methods for developing assessment criteria for proficiency.

After developing the assessment program goals, the topics that will be assessed need to be defined. For biomethodologies, all of the techniques used at the institution should be included in the program. An organization must determine how many different biomethods are being taught and used in the organization. Depending on the procedures performed, this number can vary between 60 and 1000 or more biomethods.¹⁶ Incorporating training and assessment into all procedures is a considerable challenge. According to the 2011 Guide “The IACUC is responsible for assessment and oversight of the institution's program components and facilities” and “It should have sufficient … resource[s] to fulfill this responsibility.”²¹ Therefore the IACUC will have to determine how it will abide by these guidelines, including determination of competency assessment and implementation of appropriate training. The Animal Welfare Act states that “…[the] agency conducting the research shall be responsible for … [the] training of scientists, animal technicians, and other personnel involved with animal care and treatment.”¹ Assessment and reassessment should involve experienced personnel such as veterinarians or veterinary technicians. It has been demonstrated that competency assessment of experienced surgeons identifies weaknesses and poor performance.¹⁷

Despite the challenge, all persons performing a technique must be evaluated for proficiency, regardless of their MD or PhD status. In some cases, veterinarians and licensed veterinary technicians may not be subject to the same levels of assessment for some procedures, as these are part of their professional training. Institutions with limited resources should address the development of an assessment program one procedure at a time. Doing so will ensure future achievement of a comprehensive assessment program.

The next step is to break down the biomethods into the core competencies of a proficient person. These are specific details that must be observed to consider performance of a task proficient. The Australian National Training Authority has developed extensive guidelines for assessment.³ They provide units of competency to cover a broad job task. Elements of competency provide specific tasks that are required to complete the unit of competency.

The process of defining core competencies and developing CAP can be overwhelming for a single person. Using a committee to tackle this task therefore is recommended. A committee is extremely useful, both to share this large task and because one person usually is not competent in all procedures that are performed within the organization. The committee should include personnel who are subject-matter experts in the target technique. If internal personnel lack the expertise, external personnel such as consultants should be considered. A veterinarian with LAS experience should be involved during this process to provide subject-matter expertise not only in biomethodologies but also in aspects of animal welfare.

During the final stage of development, all of the CAP can be combined into a manual and shared with institutional veterinarians and the IACUC. Manuals can be organized by species and then subdivided into individual techniques. Institutional training programs should not have to seek approval of the IACUC to initiate competency assessments. The IACUC should be informed and be allowed to comment on or require such competency assessment programs.

Proficiency criteria for training and assessment outline specific items that need to be completed to demonstrate that trainees can perform the stated task. The core competencies of these criteria are similar to this performance criterion. They differ in that the Australian guidelines are focused on assessing competency for job tasks, whereas the proposed method focuses on assessing proficiency in specific biomethodologic procedures.

To define the proficient core competencies, ask the question “What does a proficient skill look like while the biomethod is being performed?” Limit to 4 to 6 items the components that the person must perform to demonstrate competency and proficiency. To illustrate this point, consider an example of manual restraint of a mouse. What exactly should the assessor be looking at in a competent and proficient person who is manually restraining a mouse? What common mistakes should assessors watch for? For example, assessors need not be concerned with which hand trainees use to pick up a rodent. Some examples of assessing proficiency that can be used for manual restraint of a mouse include: the mouse is not chased aggressively around cage, causing stress; the trainee demonstrates gentle manipulation while using 1 or 2 hands to pick up a mouse either by the base of tail, scruff, or forceps; when used, forceps have appropriate padding and are used to grasp the scruff or tail gently; the trainee does not use excessive force to push a mouse against the cage top, and aggressive behaviors when performing manipulation are unacceptable; and the trainee ensures that restraint does not restrict the mouse's breathing. If all assessors used these or similar criteria as a guide for this procedure, then everyone would be evaluated consistently. Using a consistent set group of assessment criteria will ensure everyone is either meeting or exceeding expectations regarding that particular procedure.

A set of assessment criteria for competency and proficiency in the subcutaneous injection of mice includes: the correct needle size is used and the amount of substance to be injected does not exceed maximal allowable amounts; appropriate restraint is used to position mouse (which is shaved if needed), and needle insertion (bevel up) is between the skin and muscle layers (common sites are dorsal neck and flank); the trainee aspirates the syringe after insertion of needle to verify no entrance into blood vessels (needle should be repositioned if blood is withdrawn); trainee confirms that no resistance is met as the plunger is depressed to release material; needle is withdrawn slowly and discarded appropriately; and a compress is used when bleeding at a site occurs.

Notice that terms like ‘appropriate’ and ‘correct’ are used. These terms are intentionally generalized, because they are concepts that the trainee should have learned prior to being assessed for proficiency while giving injections. Core competencies should be developed for each biomethod, building on each other. These can be compiled into a manual and used to guide training, assess competency, and verify proficiency.

There are many different ways to competently perform the same technique, with the end result being the same. For example, mouse restraint can be achieved by scruffing the mouse or by using a restraining device. In both cases, the handler should be able to perform the technical manipulation required with the animal breathing freely and without struggling and biting. A standard must be set for each biomethod CAP but is uncharted territory for the LAS community, and specific standards for each institution can vary. However, development through harmonized collaborative efforts can make this process efficient and allow for the identification of best practices.

Verification of proficiency.

Once trainees have undergone consistent training according to the previously described methods, assessment or verification of proficiency should begin. An assessor assesses proficiency with the guidance of the CAP and approves a trainee on successful completion and proficient demonstration of the given procedure. The trainee is held to the same core competencies driven by the CAP during the assessment of competence, but the difference is that during proficiency verification, the outcome is only either successful or unsuccessful. Once successful demonstration of proficiency has been assessed, the trainee can be allowed to work independently on the specific biomethod. If the outcome of the proficiency assessment is unsuccessful, additional training should be suggested. Topics such as comfort level, training method, and alternate trainers should be discussed with the trainee to determine whether a problem other than the trainee's ability needs to be considered.

Proficiency assessment not only ensures that the trainee has the competency and proficiency to act independently, but it also evaluates how well trainers are training according to the outcome of the assessment. An important point is that prior to proficiency verification, both trainee and trainer should feel comfortable with the competency taught and learned. It is the responsibility of trainees to understand and feel comfortable with the technique and species they are using. Trainees should never go through proficiency verification if they are uncomfortable with a technique that is being asked of them.

Administering the assessment program.

Establishing how assessment methods should be administered includes determining when, how often, by whom, and how assessment will be documented. Whether assessment will occur during or immediately after training (to assess competency) or at a time point away from training (to assess proficiency) or both must be determined. The frequency of assessment must also be established.

Those doing the actual assessment—the assessors—need to be identified. Essentially, anyone who has been determined to be proficient in a procedure can train persons in that procedure. However, not everyone is an effective trainer. In addition, receiving training does not automatically ensure that a person can act independently regarding the target procedure—rather, he or she is learning the procedure. Once trainees are confident with their competence and training, they then should contact one of the designated assessors to be evaluated for proficiency.

Documentation.

Records should be kept by assessors or a training group and provided to trainees. This process allows trainees to review and concentrate on tasks that might need improvement and to include in job applications or to demonstrate to employers that they are capable of performing certain biomethodologies. Documentation is especially important for organizations that follow 21 CFR Part 58 Good Laboratory Practices regulations or that are ISO9000-certified, because they require that persons performing a task must be trained and that the training must be documented.^18,22,23 This type of documentation could be useful during the AAALAC accreditation process and animal welfare audits of contract research organizations. Documentation of proficiency may vary among organizations but usually includes the biomethod being verified, a date, and the signatures of the assessor and trainee.

Conclusion

The LAS community should introduce competency and proficiency assessment programs into their animal care and use training programs. The administration of a comprehensive assessment program often poses a challenge for the LAS community because the ratio of trainees to trainers usually is high. This high ratio can complicate direct observation of performance¹³ and is one reason why assessment methods such as multiple-choice exams and training checklists are used instead of more appropriate methods, such as CAP. To minimize this obstacle, observation of trainees for proficiency can and should be performed during day-to-day activities.

We believe that a biomethodology assessment program can be successfully developed by modifying objective structured assessments of technical skills to develop score sheets to guide training and assess competency. These methodologies are objective, can be standardized, can include a wide range of items to be assessed, and have been validated in the human resource and medical fields. Core competencies identified for specific biomethods can be used throughout the training program. Furthermore, CAP, which address core competencies, can be compiled as a verification manual and used to assess proficiency consistently.

The advantage for the LAS community is that many assessment methods have been validated in the human medical and surgical fields and can be adopted by the LAS community. A high-quality assessment program must be objective, standardized, and well defined prior to its implementation. Implementation should be performed by integrating the assessments into the biomethodology training program. Educating scientists and assessors can make the implementation of assessment less challenging.

As these methodologies are implemented, data should be collected to measure the effectiveness of these objective assessments. This validation could help overcome challenges presented by institutions reacting negatively to standard assessment. The human resource and medical fields were in the same position years ago, and through validation of the implemented assessment methods, they are leading the way with competency assessment methodologies.

A standard biomethodology assessment program can help boost a training program beyond the limits of regulations, increasing the quality of training and research while improving animal welfare.