Credentialing Imaging Informatics Professionals: Creation of Items for the CIIP Examination

Many people are under the mistaken belief that test creation is a simple endeavor. Few actually consider the steps involved. Just write a question down and then make up a few incorrect answers and you are done, right? In reality, nothing could be further from the truth. There is an entire profession focused on the process of creating tests to assess knowledge. The term “psychometrics” has been coined to describe this process. It turns out that question (item) writing is a very developed science. The procedure for item acquisition is outlined in this article.

In 2005, the Society for Imaging Informatics in Medicine (SIIM) launched the program to develop a certification credential for imaging informatics professionals, the Certified Imaging Informatics Professional (CIIP). In preparing the program, SIIM has dedicated substantial resources and support to establish an examination that could be considered as a standard for professionals who want to be certified in imaging informatics. Credentialing individuals in a profession establishes criteria for fairness, quality, competence, and safety. The CIIP program is a knowledge-based certification that intends to measure competence as well as establish standards for professional knowledge, skills, and practice.

The American Registry of Radiologic Technologists (ARRT) has also provided psychometric and training services to many of the individuals who have been tasked with the responsibility of writing questions for the CIIP exam. SIIM is pleased to be working in conjunction with such a well-respected and experienced organization.

SIIM endeavors to meet the standards of the National Organization for Competency Assurance in developing its program. These include the creation of a job task analysis,¹ followed by a test content outline that incorporates the tasks and skills central to the profession in the exam instrument.²

The first step in certification program development is to identify the goal. Primarily, we want to assess the knowledge and skills of the individual taking the exam.³ The creation of items must be carefully crafted to meet that goal. Although some of the questions should simply test whether the candidate can recall facts, others go a little deeper. Some questions on a professional-level exam should take the next step by assessing the individual's ability to understand the why and how of some concepts. It is also crucial that the questions asked are actually related to the practice being tested.² Although it would be easy to include nuclear reaction physics on an imaging informatics exam, it would be discriminatory against individuals who had not received specific training in that discipline and has little or nothing to do with the actual job of an imaging informatics professional.⁴

The next question is what format the test will take. Should the exam be paper-based or computer-based? Will the exam be multiple choice or essay? Should the exam have a practical portion requiring onsite review? Each of these questions has to be carefully considered against the goals of the program. Although each format has its advantages and disadvantages, in most cases the multiple-choice exam is the most cost-effective and easily standardized format. Multiple-choice exams also provide the most flexible format for continued exam development. In an industry such as imaging informatics, where change is a constant, the ability to adapt the exam over time is crucial. The decision to make the CIIP exam computer-based was also an easy choice—in an industry that is based on the process of moving from printed media to digital, it would be a contradiction to offer the exam in a paper-based format.

Several major tasks must take place before the first question is finalized for the test. To get an exam certified by the National Commission for Certifying Agencies, a strict set of guidelines must be followed.¹ First, a job task analysis must be completed to define the tasks that are actually performed by professionals in the field. Based on the results of that task analysis, a test content outline must be defined and published. The next step is the item writing. And this is just the beginning. A brief outline of the process is shown in Figure 1. For the sake of simplicity, let us just focus on the item development and validation process. There has actually been a great deal of work performed to teach individuals to take tests. In many cases, it is possible to train an individual to pass an exam with no real practical knowledge of the topics covered. Take the following question, for example:

• The flub extension of the triangular que is also sometimes referred to as an:

  1. Oxen
  2. Bustom
  3. Chainor
  4. Doonan

  If you chose a as the correct answer, you were correct. Simple grammatical cluing such as shown in the example are dead giveaways. Another example:

• What is the primary purpose of the frang in coniging?

  1. To remove frang-clangs
  2. To pare chamalls
  3. To upon rognor
  4. To fix barnets

Fig 1.

Steps required to create a certification examination.

Did you choose a again? You would not be alone if you were able to guess the answer based on the fact it was the only answer that included terms from within the question.

Other types of cluing are also easy to miss by question writers, such as giving the answer to a question in the form of a previous item on the exam, use of plurality, and including the answer phrase within the question. Nonsense answers also have been shown to greatly improve the examinee's chance of guessing the correct answer. Because of the risk of certifying an individual who may not completely understand the subject matter, the SIIM Certification Committee has gone to great lengths to provide training from top-quality psychometricians who have worked with such organizations as the ARRT and the American College Testing Assessment.

The committee has recruited some of the best and brightest from our field to assemble the questions. Given the diverse background of individuals who work in our profession, the goal was to have as many of those professional career paths as possible included. On the Item Acquisition Subcommittee, we have radiologists, radiologic technologists, physicists, computer science majors, network engineers PhDs, and others who now work within the field of imaging informatics. The items developed and approved for the exam will cover the entire domain of knowledge needed by an imaging informatics professional.

Questions submitted for the exam must go through a multistep process before approval. Each question must be submitted by an approved item writer through an electronic submission program developed in conjunction with ARRT. Each question has to be linked to an item on the test content outline discussed previously. For the question to make it through the submission process, it must also include the reference material from which it was developed. A full listing of these references will be made available to those wanting to study for the exam. Based on the test content outline number assigned, the question will be referred to expert reviewers who have been assigned the responsibility of that particular content area. This process takes the question from development through the technical validation of the material.

After the approval of the question by the expert reviewer, one final step is required. The Certification Committee felt it worthwhile to expand the process to include a final review by a committee of individuals who have experience and training in item review, and who will look at each question for cluing and/or grammatical errors. It is this additional check that finalizes the validation process and allows the item to be entered into the item bank.

Items that make it into the item bank are then available for the test assembly committee, which will use the items in the bank to build the exam. This is not the end of the review process, however. The real test of an item is how it performs on the exam. Within the item bank itself, there are tools to evaluate how difficult the item is, as well as whether it is correctly answered by individuals who demonstrate a thorough grasp of the subject based on similar subject items. Every exam administered will be reviewed for questions that may need to be removed from active status within the bank. This review is carried out to ensure that subsequent exams are relevant and complete.

In summary, the process of writing items for the SIIM CIIP exam is lengthy and difficult. From validating that the question is applicable to providing excellent training opportunities to the individuals involved, the volunteers in this process have worked very hard. It is only through their dedication and support that an endeavor such as this is possible. The goal of the certification committee is to produce a quality exam and certification credential that will make the individual achieving that credential a highly sought professional. We hope that the day will come that the bearer of these credentials will be able to do so with a great sense of pride and accomplishment.

SIIM Certification Committee

Imaging Informatics Administration Certification Committee

Chair

J. Anthony Seibert, PhD

Professor, Imaging Research Center

UC Davis Medical Center

4860 Y St, Ste 3100

Sacramento, CA 95817

Phone: 916-734-0493

Fax: 916-734-6548

Email: jaseibert@ucdavis.edu

Term: 2008

Member

D. Scott Griffin, BSRT(R)

PACS Administrator, Department of Imaging Services

Southeast Alabama Medical Center

1108 Ross Clark Circle

Dothan, AL 36301

Phone: 334-699-4430

Fax: 334-678-2883

Email: dsgriffin@samc.org

Term: 2008

Member

Shaun A. Hill, RT (R)

PACS Administrator, Department of Radiology

High Point Regional Hospital

601 North Elm Street

High Point, NC 27262

Phone: 336-878-6037

Fax: 336-878-6194

Email: shill@hprhs.com

Term: 2008

Member

Janice Honeyman-Buck, PhD

Associate Professor, Department of Radiology

University of Florida

P.O. Box 100374

Gainesville, FL 32610-0374

Phone: 352-265-0291

Fax: 866-634-4821

Email: honeyman@ufl.edu

Term: 2008

Member

Steven C. Horii, MD

Professor, Department of Radiology

University of Penn Health System

3400 Spruce Street

Philadelphia, PA 19104

Phone: 215-349-5183

Fax: 215-349-5627

Email: Steve.Horii@uphs.upenn.edu

Term: 2008

Member

Richard L. Morin, PhD

Professor, Department of Diagnostic Radiology

Mayo Clinic-Jacksonville

4500 San Pablo Road

Jacksonville, FL 32224

Phone: 904-953-8752

Fax: 904-953-2894

Email: morin.richard@mayo.edu

Term: 2008

Member

Paul G. Nagy, PhD

Assistant Professor, Department of Radiology

University of Maryland School of Medicine

22 South Greene Street

Baltimore, MD 21201

Phone: 410-328-6301

Fax: 410-328-0641

Email: pnagy@umm.edu

Term: 2008

Member

Keith L. Skaer

Imaging Manager, Clinical Engineering

BJC Health System

3165 McKelvey Road, Ste 110

St Louis, MO 63044

Phone: 314-996-8920

Fax: 314-996-8928

Email: kls7124@bjc.org

Term: 2008

Member

Charles W. Socia, RT (R)(CT)(QM)

VP of Operations

Arkansas Medical Imaging

9601 Lile Drive, Ste 1100

Little Rock, AR 72205

Phone: 501-223-3392

Fax: 501-223-4970

Email: csocia@arkansasmedicalimaging.com

Term: 2008

Member

David Wild

Director of Systems Integration, Radiology Dept

Albert Einstein Healthcare Network

11 Almond Court

Lafayette Hill, PA 19444-2500

Phone: 215-456-7117

Fax: 215-456-6227

Email: wildd@einstein.edu

Term: 2008

Member

Byrn Williamson, Jr., MD

612 9th Avenue SW

Rochester, MN 55902

Phone: 507-288-1339 (H)

Fax: 507-284-8996

Email: williamson.byrn@mayo.edu

Term: 2008