Training and experience of coding with the World Health Organization’s International Classification of Diseases, Eleventh Revision

Abstract

Background:

The new International Classification of Diseases, Eleventh Revision for Mortality and Morbidity Statistics (ICD-11) was developed and released by the World Health Organization (WHO) in June 2018. Because ICD-11 incorporates new codes and features, training materials for coding with ICD-11 are urgently needed prior to its implementation.

Objective:

This study outlines the development of ICD-11 training materials, training processes and experiences of clinical coders while learning to code using ICD-11.

Method:

Six certified clinical coders were recruited to code inpatient charts using ICD-11. Training materials were developed with input from experts from the Canadian Institute for Health Information and the WHO, and the clinical coders were trained to use the new classification. Monthly team meetings were conducted to enable discussions on coding issues and to select the correct ICD-11 codes. The training experience was evaluated using qualitative interviews, a questionnaire and a coding quiz.

Results:

total of 3011 charts were coded using ICD-11. In general, clinical coders provided positive feedback regarding the training program. The average score for the coding quiz (multiple choice, True/False) was 84%, suggesting that the training program was effective. Feedback from the coders enabled the ICD-11 code content, electronic tooling and terminologies to be updated.

Conclusion:

This study provides a detailed account of the processes involved with training clinical coders to use ICD-11. Important findings from the interviews were reported at the annual WHO conferences, and these findings helped improve the ICD-11 browser and reference guide.

Introduction

The International Classification of Diseases (ICD), developed by the World Health Organization’s (WHO) Family of International Classifications (FIC) division, is a multipurpose and coherent classification that enables international comparison of hospital data (WHO, 2020). This international and multilingual taxonomy is used for collection and reporting of data on health and health services (e.g. morbidity and mortality statistics, quality and safety, healthcare costs and clinical research) by multiple professionals, in different healthcare aspects and sectors (WHO, 2020). The Tenth Revision of ICD (ICD-10) has been in use around the world since 1992. Typically, ICD revisions have occurred every 10 years, but for ICD-10, the WHO has published major updates every 3 years. While functioning well for mortality data internationally, country comparison of morbidity data has been affected by national modifications. National clinical modifications used in Australia – ICD-10-AM, USA – ICD-10-CM, Canada – ICD-10-CA were made to improve specificity and detailed clinical coding options (Doyle and Dimitropoulos, 2009), while other changes have been undertaken to simplify ICD-10 for use in developing countries (Paoin et al., 2018). Given that a full revision was overdue, WHO developed the new International Classification of Diseases, Eleventh Revision for Mortality and Morbidity Statistics (ICD-11), released for international use in June 2018. New features of ICD-11 include enhanced usability and searching (tabular list and linear organisation), increased comprehensiveness, the ability to code more clinical detail (extension codes), updated scientific content (e.g. new terminologies) and readiness for electronic environments (WHO, 2018, 2021). In addition, multilingualism and better linkage to other relevant classifications and terminologies have been enhanced. As a result of the recent release of ICD-11, there was a WHO call for education, awareness and coding training curricula prior to the implementation of ICD-11 in each country.

Coded data quality is key and is impacted by physician documentation and clinical coders’ training and performance. To ensure consistency in data quality, ICD coding manuals are published by WHO, but, even with training, a learning curve post-implementation of ICD-11 is expected (Innes et al., 2000). Studies were conducted during the ICD-9 to ICD-10 transition, which demonstrated that ICD-10 codes were more specific and detailed compared to ICD-9 codes (Hazelwood, 2003; Murphy et al., 2004; Rubenstein, 2014). Training during these Australian, Cambodian and Canadian studies ranged from 16 hours to 37 hours plus practice hours for Canada (Hazelwood, 2003; Paoin et al., 2018). Studies indicated approximately 6 months required for coding proficiency to develop (Hazelwood, 2003; Innes et al., 2000). As such, Rubenstein concluded that even the 7 days of dedicated time to formal training was insufficient for acute care coders to learn a new classification. Thus, training is a crucial step for clinical coder performance. For ICD-10, coding guidelines, which include practical case examples, were developed by each country. For Canada, the Canadian Institute for Health Information (CIHI) published the ICD-10-CA coding guidelines during the ICD-9 to ICD-10-CA transition. In June 2018, the ICD-11 online browser for coding was launched with new codes, pre- and post-coordination features and online coding assistance tools. Hence, previous ICD-10-CA training content and examples must be adapted for ICD-11 training. To maintain production and coding accuracy, there is a need for developing high-quality training materials and training processes.

As a WHO Collaborating Centre for Classification, Terminology, and Standards, substantial input was contributed by our CIHI (DC, LM) and academic team members for developing the code content as well as training materials during an ICD-11 field trial. The purpose of this article is to share the training materials and processes for clinical coders using ICD-11 and to describe their experience of coding using ICD-11. The objectives were to (i) develop ICD-11 training materials; (ii) train clinical coders to use ICD-11 to code full inpatient hospital discharge records; and (iii) determine clinical coders’ perception of the training process; and (iv) evaluate the clinical coders’ knowledge, comprehension and application of the ICD-11 classification.

Method

The data used for the field trial were 3011 randomly selected adult discharges (>18 years) that occurred between 1 January 2015 and 30 June 2015 from three Calgary, Alberta hospitals. Obstetric cases were excluded to focus on coding multiple chronic conditions. We compared the original coding using ICD-10-CA and the new coding using ICD-11. A full description of the study methods is described in Eastwood et al. (2021).

Participants

In June of 2017, a multidisciplinary team of experts was assembled to develop the materials and define the process. The team consisted of clinician researchers, analysts, faculty members from the Cumming School of Medicine at the University of Calgary, as well as classification specialists from CIHI and the WHO. Six clinical coders were hired on an hourly basis to learn ICD-11 principles and code diseases and healthcare-related harms evident in inpatient visits. Advertisements were placed on the University Careers online page, circulated to a manager of the Health Information Management (HIM) department at Alberta Health Services (AHS), and to a CHIMA representative for sharing. Those hired expressed interest in research and demonstrated strong communication and team skills. These participants were Canadian College of Health Information Management-certified clinical coders, with an average of 2.75 years coding experience using ICD-10-CA (range of 1 to 6 years of experience). Most of the clinical coders were also full-time employees of AHS in HIM positions; one was employed by CIHI. Authors CAE (researcher) and DC (Classification and Terminologies Specialist with CIHI) conducted the training. Other team members who contributed to the training process included the Principal Investigators of the primary study (HQ, WG), a senior analyst on the project (DAS), a second classification specialist from CIHI (MP) and a senior classification specialist who consulted for the WHO (LM).

Material development

At the time of this study, coding standards for ICD-11 were unavailable and the ICD-11 Reference Guide was under development. Thus, training materials needed to be developed de novo. All available ICD-11 resources are web-based. WHO resources for this study included the draft Reference Guide (WHO, 2019), the ICD-11 online tabular list and the ICD-11 Coding Tool (WHO, 2021). The Coding Tool is an electronic index to search for codes using diagnosis or symptom terms. The live revision of the browser was used to ensure inclusion of ongoing updates and improvements during the field trial. Early revisions of brief educational video animations developed by the WHO on the ICD-11 information page were included. CIHI personnel’s early knowledge of ICD-11 was incorporated into the development of the training materials. Lastly, documents (Quick Reference charts, ICD-11 User Guide slide set) from the WHO Education and Implementation Committee (EIC) for ICD-11 were used as resources.

The training materials developed by the research team included class presentation slide sets, line coding, case scenarios, a knowledge assessment quiz and respective answer keys. The Bloom’s Taxonomy framework was used to guide development of training materials. The framework addresses remembering (knowledge), understanding (comprehension) and using ICD-11 concepts to code full inpatient records (application) (Armstrong, 2010; Krathwohl David, 2002). A series of PowerPoint presentations included foundational concepts of ICD-11 such as new codes, pre-coordination, post-coordination, code clustering and coding hospital-related harms (NHS Digital, 2020). The team was instructed to apply coding rules from the Canadian Coding Standards for Version 2015 ICD-10-CA created by CIHI, given that no ICD-11 coding standards otherwise existed.

Training process

Training took place at the University of Calgary, Community Health Sciences Department in the Cumming School of Medicine. Training occurred between July 2017 and October 2017, which included 20 hours of classroom learning and discussion. During the training period, the clinical coders completed an average of 44.5 hours of homework, as well as coding practice scenarios (ranging between 20 hours and 80 hours). The clinical coders worked through line coding exercises and scenarios for practice coding (simple and more complex).

Inter-rater reliability testing

To establish consistent coding of charts and ensure quality of coded data, randomly selected hospital charts were coded by two coders to establish inter-rater reliability (IRR) of the ICD-11 main condition code choices. Specifically, the six clinical coders were divided into teams of two. Sets of 10 records were coded by the two clinical coders then main condition code agreement was compared (Box 1). Re-training occurred after each set, then coding continued in sets of 10 charts until IRR of 0.8–1.0 was achieved (κ > 0.8 is considered high agreement). One dyad coded seven charts due to limited time hence, 57 charts were coded in total.

Box 1.

Inter-rater reliability testing.

	ICD-11 Dataset
Number of coders	6
Number of charts coded by two people	57
Kappa	0.88
Average time to code charts	15.9 min

After IRR testing and completion of the quiz in October 2017, coding progressed from training to coding of the study sample hospital discharges. Coding took place in the health records departments of three Calgary hospitals between November 2017 and June 2018. The coding team met monthly to ask questions and highlight functional issues with the new codes or the Coding Tool. While coding each chart, the clinical coders documented comments in the data collection tool. These comments were discussed at ongoing monthly team meetings after classroom training was complete. These team meetings included CIHI members (DC and MP) and the WHO senior classification specialist (LM). Some comments were forwarded to the Topic Advisory Groups (Quality and Safety, Psychiatric Conditions, Morbidity Reference Group) for advice. Upon resolving coding issues, the coding team’s comments prompted many changes and improvements to the content (e.g. code definitions, index terms) and tooling (e.g. code search function, post-coordination links) that were incorporated by the WHO and inserted into the ICD-11 Browser and Reference Guide during the course of the study.

Measurement tools for training evaluation and data collection procedures

To evaluate the experience of being trained to code using ICD-11, measurement was undertaken in three ways: a questionnaire, qualitative interviews and a quiz.

Questionnaire

A questionnaire was developed using program evaluation literature, such as Bloom’s cognitive domain taxonomy of learning (Armstrong, 2010). Questions addressed knowledge, comprehension and application of coding content and skills gained from the training sessions. The 19 questions included 9 questions using a 5-point Likert scale and 10 open-ended questions to obtain information about the strengths and limitations of the training program. The questionnaire was distributed 4 months after initial training began, via SurveyMonkey.com®. All coders responded within 1 week of distribution.

Interviews

To determine the six clinical coders’ perception of the training materials and the training process, six one-on-one interviews were conducted (5 months post start of program). Interviews included 10 questions about the ICD-11 training program that were not otherwise captured in the questionnaire. Questions addressed specific areas of skill, delivery and effectiveness of the training process and materials, perceptions of performance improvement and areas for improvement.

Telephone interviews were conducted by MM taking 20–30 min per participant. Verbal informed consent from each participant was obtained before proceeding to conduct and audio-record each interview. Interviews were transcribed verbatim into an electronic file and audio-recordings were erased. Interviews were de-identified to ensure participants anonymity.

Coding quiz

Quiz questions were derived from the training materials and the ICD-11 Reference Guide (developed in September of 2017). A set of 20 questions consisted of a mixture of multiple choice, true or false, matching, short answer questions, as well as a set of short and long coding scenarios. The quiz was used to further determine the knowledge, comprehension and application of ICD-11 coding concepts, before coding full charts for the field trial data collection. The quiz was distributed electronically to the five members of the coding team. The sixth coding specialist helped develop the coding scenarios and the answer key. The five clinical coders completed the quiz during a 2-hour supervised session, using the online ICD-11 Browser and Coding Tool as resources.

Data analysis

Quantitative questionnaire responses were obtained from SurveyMonkey.com®. Descriptive statistics were used for frequency counts. Quiz responses were counted manually for true and false, multiple choice and matching question scores. Coding case scenario answers were compared against the answer key and marked accordingly. IRR analysis was performed with SAS® software 9.4. Qualitative interview transcripts were reviewed by two researchers; words or phrases used frequently were selected and major themes identified.

Ethics approval

The rights of all participants were protected under the Conjoint Health Research Ethics Board (CHREB) approval for testing the WHO’s ICD-11 (Ethics certificate number REB15-0790).

Results

Training program and materials

The training materials are listed in Table 1. A supplementary package of training materials (i.e. slides, line coding, scenario coding, quiz and answer keys) is available on the Centre for Health Informatics Website. ¹ Training materials and clinical coder feedback were sent to WHO-FIC EIC. The materials were then intended to be shared with international members of WHO-FIC EIC to refine and train other clinical coders. We recommend training a small group of clinical coders at a time to facilitate interaction. We found that 10 sessions, 2 hours in length each, were sufficient for training. Clinical coders reported their time spent using the ICD-11 and hours spent in class to train on the new ICD-11 (Figure 1). Clinical coders were responsible for attending classes and completing practice materials. As a result, there was variability in the hours committed to learning ICD-11. As a metric of successful completion of ICD-11 coding training, clinical coders must have achieved a score of at least 80% on their quiz and achieved 80% or higher in IRR on main condition codes.

Table 1.

ICD-11 training materials.

Resource	Description
ICD-11: A User Guide	Slides including foundational concepts of ICD-11 and coding examples.
University of Calgary Training cases 1	24 general medical-surgical cases. Blank worksheet.
University of Calgary Training cases 1 - Answer Key	24 general medical-surgical cases with coding answers.
Line Codes 1	54 general diagnoses statements. Blank worksheet.
Line Codes 1 – Answer Key	54 general diagnoses statements. Includes coding answers.
Healthcare-Related Harms	Introduction to the Healthcare-related harms (3-part Quality & Safety model) of ICD-11 with coding examples.
Harms Training Cases	26 healthcare-related harms cases using the 3-part Quality & Safety model. Blank worksheet.
Harms Training Cases – Answer Key	26 healthcare-related harms cases using the 3-part Quality & Safety model. Includes coding answers.
Medical-surgical cases	21 cases provided by the Ottawa Hospital Research Institute. Blank worksheet.
Medical surgical cases – Answer key	21 cases provided by the Ottawa Hospital Research Institute with coding answers.
ICD-11 Coding Quiz	20 mixed questions to assess knowledge, comprehension, application of ICD-11
ICD-11 Coding Quiz Part 1 – Answer Key	20 answers, and grading scheme for quiz questions
ICD-11 Coding Quiz Part 2 – Answer Key	4-page document with codes corresponding to scenarios

Figure 1.

Clinical coder time spent on ICD-11 mortality and morbidity statistics training.

Coders’ perception of the ICD-11

Interviews with the clinical coders indicated that they possessed a strong understanding of why the ICD-11th Revision was being developed, how it differs in structure from ICD-10 and its potential benefits. Trainees were of the opinion that there is opportunity for ICD-11 to be used in more clinical settings in the future and adopting ICD-11 worldwide may lead to better international comparability of data. The coders found ICD-11 to be a comprehensive reporting tool which reflects advances in health sciences and medical practice. They also liked that the inclusion of stem and extension codes allows for greater specificity.

Content within ICD-11 that needed more clarification included post-coordination coding, code detail, injury and harms, complications, adverse reactions and extension codes. The coding team pointed out that there were some overlapping codes and some missing codes: “There is some redundancy in extension codes. For example, there is a code for a fall, and then the extension codes have a section on falls as well. It is not always clear which one you should use.” Coders also noted that looking up conditions presented unique challenges with this beta version of ICD-11: “The terminology isn’t always the same or the term I do put in brings up too many hits.” Comments like these were addressed in real time at team meetings with updates being made to the browser by the WHO consultant (LM). A noticeable difference in opinion was in rating the use of the ICD-11 browser, ranging from easy to very difficult (easy n = 1, moderate n = 2, somewhat difficult n = 2, very difficult n = 1).

Coders’ perception of the training process

Results of the questionnaire revealed that clinical coders were generally satisfied with the amount of training time provided, the pace of the training program,and the number of face-to-face meetings provided (Figure 2). However, clinical coders noted difficulty of learning ICD-11 with the lack of available reference materials, such as a more detailed reference guide, reducing their confidence in using ICD-11. When asked which training materials were most effective in increasing their confidence with using the ICD-11, all six of the clinical coders reported that coding practice with short scenarios was helpful (Figure 3). The majority of clinical coders (five/six) also found that coding practice with full charts, interacting with other clinical coders, and classroom training sessions were beneficial. Those coders that spent more time training and practicing ICD-11 tended to express greater satisfaction with the training process versus those who spent less time learning ICD-11.

Figure 2.

Perception of clinical coders regarding the training program.

Figure 3.

Clinical coder feedback on training materials.

Qualitative interviews revealed facilitators and barriers to learning ICD-11 during this training period (Box 2). All the participants appreciated that their input was sought and that their comments prompted changes to enhance the browser and coding tool. The clinical coders also found that independent practice time was beneficial. Clinical coders in our training program appreciated coding with real case scenarios, as these scenarios enabled real-world coding challenges to be addressed. Clinical coders considered peer-to-peer discussions and interactive learning in teams more helpful than slide presentations. Everyone emphasised the value of discussion with experts during training discussion sessions, including experts from WHO and CIHI. In general, clinical coders and team members considered answer keys with gold standard codes to be necessary and helpful in determining the best code choices and facilitating learning.

Box 2.

Strengths and weaknesses of the ICD-11 coder training program.

Strengths: Coding practice with short scenarios and original charts was a helpful method of training Group discussions with fellow coders and experts from WHO and CIHI was effective for learning coding procedures Most coders reported that using the ICD-11 draft reference guide, Canadian coding standards, answers from training case scenarios and emailed queries with answers from CIHI personnel, were effective resources Clinical coders agreed that they will be able to share the knowledge gained through the training program Coding consistency and confidence was gained from working in close proximity to the other coders Discussions with CIHI and WHO classifications experts were extremely helpful

Weaknesses: PowerPoint slides were useful for introducing ICD-11, but not very beneficial for learning about coding practices for ICD-11 Clinical coders reported difficulty coding complex cases and hospital harms, due to minimal reference materials available Clinical coders mentioned that it was challenging to use a 3-part model for hospital-related harms

Clinical coders stated that access to more experienced users of ICD-11 while coding would have improved their ability to learn. The clinical coders provided feedback on ways that could make the sessions more effective. For example, they suggested “Build a stronger foundation with easier cases for practice before proceeding to the hospital harms,” “include more rules regarding post-coordination,” “have a visual presentation of the system browser.” Box 3 displays recommendations by coders to further refine the training program.

Box 3.

Coder recommendations for strengthening the training program.

Recommendations: Start with simple terms, build to more complex scenarios Introduction and training on ICD-11 using line codes and simpler case scenarios would enable more effective learning Topics such as post-coordination, extension codes, injuries and harms, and adverse reactions require further clarification for future training programs Having a clear guideline on when to use the 3-part model and how to sequence the codes would be helpful Trainees suggested including an experienced coder to consult Having training competencies and timeline available would guide expected pace of learning Having the charts available for reference would result in more effective discussions Clear answer keys to coding scenarios is helpful Provision of guidelines, updated coding standards, and representative coding scenarios would support the learning experience Practice scenarios should cover varied health conditions and modules could be created Tooling is needed to transfer codes from the browser to data collection software Schedule face to face meetings with training participants in close succession, to ensure continuity, focus, involvement, and motivation Provide live or video demonstrations of the browser and coding tool versus screenshots Develop more visual training materials, such as diagrams, flowcharts, and decision trees Design electronic interactive training materials with instant feedback mechanisms for coding practice Designate the participants to research and present a coding topic, for example the main differences and findings in each chapter

Evaluation of knowledge and understanding: quiz results

Of the six clinical coders who participated in the ICD-11 training program, five completed the coding quiz (at 3 months post start of program). The sixth coder was involved in developing the quiz and was therefore exempted from completing the quiz. The coding quiz average score was 84%. The most difficult section involved achieving consistent main diagnosis codes for the brief scenarios provided. The quiz served as a useful incentive to complete the practice scenarios and review the coding rule content.

Discussion

The ICD-11 field trial at the University of Calgary WHO Collaborating Centre presented a unique opportunity to train clinical coders on ICD-11, using real-world hospital discharge records. Our training program included 60 hours of training (in class and homework combined) for coding of 3011 complete records by six clinical coders and evaluation of the experience by quiz (by five of the six clinical coders), questionnaire, interview and IRR testing. In general, clinical coders described the materials generated for the training session to be very useful and relevant. The training process was developed with the help of resources created for educational programs from previous studies on coder training, alongside experts at WHO and CIHI. Overall, target quiz scores were achieved suggesting that the training program was effective in educating clinical coders on ICD-11.

Feedback from the clinical coders enabled the research team to make recommendations to the WHO to improve the codes, the reference guide and coding tools. Salient findings were presented at several annual conferences for WHO’s FIC Network (the WHOFIC conferences in 2017 and 2018), the Quality and Safety Topic Advisory Group (conferences held in 2017 and 2018) and to the WHO Morbidity Reference Group (conferences held in 2018 and 2019). The clinical coders provided feedback on better tools for guiding coding decisions and helped develop a flow chart for the three-part model for coding healthcare-related harms in the ICD-11 Reference Guide (section 2.25.5) (WHO, 2019). This formative evaluation process helped to identify areas of improvement, which ultimately shaped both the training methods and the ICD-11 browser content.

The objectives of the training process were to develop preliminary training materials and gain coder feedback on their experience, as well as share the materials with the EIC of the WHO. The training was posited to be a self-learning process, with guidance from the trainers and discussion among coders. With the continued development of online and in-person training materials, coders were expected to become adept at ICD-11 coding in a reasonable time. However, depending on the learning style and experience level of the clinical coder, a longer training period and more guidance may be warranted. As such, we further incorporated multiple approaches during our field trial to accommodate different learning styles.

To date, this is the most detailed reporting on training methods and coder experiences for using ICD-11. In April, June and November of 2015, CIHI provided a three-part webinar to introduce ICD-11, along with high-level training information (Canadian Institute for Health Information, 2019). Another study tested IRR of line coding (coding of sets of terms) but did not describe training or how the classification in general might be received by clinical coders (Stanfill et al., 2014).

The ICD training process for clinical coders varies internationally. After the implementation of ICD-10, some studies described the training programs available for clinical coders in their country. Workshops offered in Ireland were aimed at three different skill levels (basic, intermediate and experienced) (Bramley and Reid, 2005). In the United States, Stanfill et al. (2014) investigated the impact of the transition from ICD-9 to ICD-10 on coder productivity by recruiting experienced coders who had varying amounts of ICD-10 training (Stanfill et al., 2014). Although an initial productivity loss was expected, those with the greatest amount of training resulted in the lowest average productivity loss, thus indicating a significant return on investment for staff training time (Stanfill et al., 2014). As we prepare for another transition in the upcoming years, this highlights the need for robust ICD-11 training. Further, the WHO will provide ICD-11 training materials (as well as the browser and coding tool) in various languages to assist with the transition.

Limitations

The current study used both quantitative and qualitative approaches to the evaluation of training materials and processes, offering a comprehensive and systematic approach to ICD-11 training procedures. However, limitations of the study include a small sample (n = 6) of clinical coders. Most challenges with ICD-11 coding stemmed from the lack of standards and guidelines needed for consistent coding. Using the ICD-10-CA coding standards was sufficient for the study purposes for consistency across clinical coders, in coding mandatory conditions, for example. Further, ICD-11 itself was a work in progress and evolved during the study. Codes and descriptions were in the development stage and were frequently changing. Teaching materials and answer keys needed several updates to ensure the most recent version of codes were used from the evolving ICD-11. The training tools generated from the current study offered a foundation for others (including the WHO) to build upon. EIC ICD-11 education and training resources such as the ICD-11 Education Tool (Beta version) can be found on the committee’s website (NHS Digital, 2020). ICD-11 reference materials have since become more available and the ICD-11 Browser and Coding Tool are more complete and more stable (WHO, 2020).

Conclusion

As ICD-11 is adopted, studies will need to examine the effectiveness of training programs in other settings and how coding reliability changes over time. We recommend that future work focus on the examination of the training program’s effectiveness in other countries, where different coding personnel, healthcare practices, languages and translation issues may become a factor.

Development of ICD-11 is a recent global initiative coordinated by the WHO. This large field trial at the University of Calgary (a collaborating centre for the WHO) is valuable, as it provides direction on training methods for national and potentially international implementation of ICD-11. The experience of working through issues with coding, as well as the training expertise provided by WHO and CIHI experts significantly contributed to the quality of the training program. Further, clinical coder training on ICD-11 is a sparsely researched area, and participation in this project provides a valuable experience with real-world challenges that can be encountered with a classification transition.