Improving Clinical Documentation with Artificial Intelligence: A Systematic Review

Abstract

Clinicians dedicate significant time to clinical documentation, incurring opportunity cost. Artificial Intelligence (AI) tools promise to improve documentation quality and efficiency. This systematic review overviews peer-reviewed AI tools to understand how AI may reduce opportunity cost. PubMed, Embase, Scopus, and Web of Science databases were queried for original, English language research studies published during or before July 2024 that report a new development, application, and validation of an AI tool for improving clinical documentation. 129 studies were extracted from 673 candidate studies. AI tools improve documentation by structuring data, annotating notes, evaluating quality, identifying trends, and detecting errors. Other AI-enabled tools assist clinicians in real-time during office visits, but moderate accuracy precludes broad implementation. While a highly accurate end-to-end AI documentation assistant is not currently reported in peer-reviewed literature, existing techniques such as structuring data offer targeted improvements to clinical documentation workflows.

Introduction

Robust clinical documentation is critical for efficiency and quality of care, diagnosis related group (DRG) coding and reimbursement, and is required to be in compliance with the Joint Commission on Accreditations of Healthcare Organizations (JCAHO).¹, ², ³ Physicians spend 34 percent to 55 percent of their work day creating and reviewing clinical documentation in electronic health records (EHRs), translating to an opportunity cost of $90 to $140 billion annually in the United States — money spent on documentation time which could otherwise be spent on patient care.¹, ³, ⁴ This clerical burden reduces time spent with patients, decreasing quality of care and contributing to clinician dissatisfaction and burnout.³, ⁵ Clinical documentation improvement (CDI) initiatives have sought to reduce this burden and improve documentation qualtiy.⁶

Background

Despite the need for increased documentation efficiency and quality, CDI initiatives are not always successful.⁷ Artificial intelligence (AI) tools have been proposed as a means of improving the efficiency and quality of clinical documentation,⁸, ⁹ and could reduce opportunity cost while producing JCAHO-compliant documentation and assisting coding and billing ventures.¹⁰ This study seeks to summarize available literature and describe how AI tools could be implemented more broadly to improve documentation efficiency, reduce documentation burden, increase reimbursement, and improve quality of care.

Methods

Best practices established in the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines were used to search, organize, and review papers (Figure 1). As no patient information was involved, no Institutional Review Board was undertaken. The review was registered on the Open Science Framework registry. The PubMed, Embase, Scopus, and Web of Science databases were queried using the search strategies found in Appendix 1. Articles were screened by three independent graders for the following inclusion criteria: full-text research article, written in English, describing novel development and application of AI tools to improve clinical documentation, and published between the earliest searchable year of each database and July 2024. Covidence software was used to organize the review (Veritas Innovation Ltd, Melbourne, Australia). The search results were last accessed on August 1, 2024. Exclusion criteria included studies which did not involve a new method or application of a tool, those which did not use an AI technique, and those which proposed methodology but did not validate an applicable tool. Disagreement between graders was resolved by discussion. Data extracted from studies include clinical data types, AI methods, tasks, reported effectiveness, and publication dates. Funding was not received for this review.

Figure 1.

PRISMA diagram of the study selection process[KM3].

Results

Six hundred and seventy studies were extracted after querying PubMed, Embase, Scopus, and Web of Science and three additional studies were found in the references of related literature. After screening articles for relevance and eligibility according to inclusion and exclusion criteria, 129 studies were included in the narrative systematic review. A complete overview of studies may be found in Table 1. Twenty-three were excluded due to reporting a non-novel tool or application,¹¹, ¹², ¹³, ¹⁴, ¹⁵, ¹⁶, ¹⁷, ¹⁸, ¹⁹, ²⁰, ²¹, ²², ²³, ²⁴, ²⁵, ²⁶, ²⁷, ²⁸, ²⁹, ³⁰, ³¹, ³², ³³ ten did not use an AI approach,³⁴, ³⁵, ³⁶, ³⁷, ³⁸, ³⁹, ⁴⁰, ⁴¹, ⁴², ⁴³ and two proposed but did not evaluate new methodology.⁴⁴, ⁴⁵

Table 1.

Comparison of CDI goals and domains by study and AI strategy

Goal	Domain	Study	Data	Task	Strategy	Outcome
Aid Clinicians	Structuring Free Text Data	Moen et al., 2020	Nursing note sentences	Organize into paragraphs with headings	ANN	69% of paragraphs were coherent
		Wang et al., 2020	Allergic reaction picklist	Display most probable reactions	Statistical	Recall = 0.822 for top k = 15
		Sholle et al., 2019	Clinical notes	Classify patient race/ethnicity	Rule-Based	Black: F-score = 0.911; Hispanic: F-score = 0.984
		Lindvall et al., 2022	Outpatient and inpatient notes	Identify advance care planning	ClinicalRegex	F-score = 0.84-0.97 across domains
		Afshar et al., 2019	Clinical notes	Identify alcohol misuse in trauma patients	Text Analysis and Knowledge Extraction system	ROC-AUC = 0.78
		Afzal et al., 2013	EHR notes	Identify hepatobiliary disease and acute renal failure cases	Various machine learning algorithms	Sensitivity = 0.89 − 0.95, specificity = 0.56 − 0.59
		Annapragada et al., 2021	Pediatric EHR notes	Identify child abuse	Multiple neural network architectures	ROC-AUC = 0.93
		Blanes-Selva et al., 2020	EHR notes	Identify obesity	Semi-supervised positive-mixture learning	Sensitivity = 0.98
		Brandt C.A. et al., 2021	Veterans Health Administration EHR notes	Identify firearm access	Random forest	Accuracy = 0.819
		Burnett et al., 2022	Clinical notes	Classify self-harm behaviors	Support vector machine and logistic regression	Sensitivity = 0.831, specificity = 0.893
		Caccamisi et al., 2020	EHR notes	Classify smoking status	Support vector machine	F-score = 0.981
		Carson et al., 2019	EHR notes	Identify suicidal behavior among psychiatrically hospitalized adolescents	Random forest	ROC-AUC = 0.68
		Chase et al., 2017	EHR notes	Identify multiple sclerosis symptoms	Naïve Bayes	ROC-AUC = 0.90
		T Chen et al., 2019	EHR notes	Identify geriatric syndromes	Conditional random fields	F-score= 0.155 − 0.858 for various syndromes
		Chilman et al., 2021	EHR notes	Identify occupation	General architecture for text engineering	Precision = 0.79, recall = 0.77
		Cohen et al., 2020	EHR notes	Identify acute hepatic porphyra	Support vector machine	AUC = 0.775
		Corey et al., 2018	EHR notes	Identify high-risk surgical patients	Logistic regression, random forest, and decision trees	AUC = 0.747 − 0.924
		Denny et al., 2012	EHR notes	Identify colorectal cancer testing	Modified KnowledgeMap	Recall = 0.93, precision = 0.94
		Diao et al., 2021	EHR notes	Identify etiological diagnosis of secondary hypertension	Gradient boosted decision trees	AUC = 0.924
		Egleston et al., 2021	EHR notes	Identify type 2 diabetes	Pointwise mutual information statistics and word2vec	AUC = 0.93
		Elkin et al., 2021	EHR notes	Identify nonvalvular atrial fibrillation	Semi-supervised learning	F-score = 0.964
		L Chen et al., 2019	Various Chinese clinical documents	Detect various clinical features of hepatocellular carcinoma	HanLP rule-based and hybrid rule/ML methods	F-score > 0.80 for most clinical features
		Wang SY et al., 2022	Ophthalmology clinical notes	Identify 24 examination entities	Fine-tuned transformers	Recall > 0.93, precision = 0.815-0.843
		Yu Z et al., 2022	Clinical notes	Identify social determinants of health in lung cancer patients	Transformer-based NLP	Average F-score for all categories = 0.92
		Ge W et al., 2022	EHR notes	Identify delirium	Support vector machine, recurrent neural network, and transformer	Best F-score = 0.978, positive AUC = 0.984, negative AUC = 0.992
		Mashima Y et al., 2022	Oncology progress notes	Identify adverse events due to chemotherapy	Rule-based	Gold standard concordance = 0.835 and 0.977 for two different test sets
		Wu DW et al., 2022	EMR notes	Identify patients with monogenic conditions	Rule-based NLP	Precision = 0.88, recall = 0.48. Outperformed ICD-10 search.
		Masukawa et al., 2022	Narrative clinical records	Identify social distress, spiritual pain, and severe physical psychological symptoms in terminally ill cancer patients	Various machine learning methods	AUC-ROC > 0.8 for all measures
		Yuan et al., 2021	EHR notes	Identify lung cancer patients	Phenotyping common automated pipeline with feature extraction and penalized regression	Positive predictive value = 0.94
		Parikh et al., 2021	Administrative claims and EHR notes	Identify patients at risk for hospitalization or death	Dimensionality reduction and clustering	30 distinct subgroups identified
		Yang et al., 2022	EHR notes	Determine disability status scale score for patients with multiple sclerosis	Combined keyword/neural network model	F-score = 0.83
		Zhou et al., 2016	Primary care EHR notes	Identify rheumatoid arthritis	Machine learning decision trees	Sensitivity = 0.94, specificity = 0.99
		Hatef et al., 2022	EHR notes from three healthcare systems	Identify residential instability	Rule-based NLP	Precision = 0.45-1.0, sensitivity = 0.68-0.96, specificity = 0.69-1.0
		Leiter et al., 2020	Clinical notes of patients with congestive heart failure receiving cardiac resynchronization therapy	Identify present, absent, and context-dependent symptoms	Deep NLP	F-score = 71.8
		Montoto et al., 2022	Spanish EHR notes	Identify Chron disease-related variables	EHRead rule-based system	F-score = 0.80 − 0.90 for various characteristics
		Landsman et al., 2021	EHR notes	Identify tuberculosis diagnosis and medication	Rule-based	Recall, precision, and F-score > 0.93
		Van Vleck et al., 2019	Clinical notes	Identify patients with non-alcoholic fatty liver disease	CLiX stochastic parser	Sensitivity = 0.93, F2 = 0.92. NLP outperformed ICD code search.
		Ogunyemi et al., 2021	EHR notes	Detect diabetic retinopathy in diabetic patients	Deep neural network	AUC = 0.8
		Moehring et al., 2021	EHR records	Identify inpatient antibiotic use	Random forest	AUC = 0.85
		Liu et al., 2017	EHR notes	Identify severe hand, foot, and mouth disease	Random forest	AUC = 0.916
		Maarseveen et al., 2020	EHR notes	Identify rheumatoid arthritis	Various machine learning and word-matching algorithms	AUC = 0.98
		Wang et al., 2012	Primary care records	Detect coronary angiogram results and ovarian cancer diagnoses	Custom semi-supervised set covering machine learning algorithm	Coronary angiogram: F-score = 0.73, ovarian cancer: F-score = 0.79
		Zhong et al., 2019	EHR notes	Identify pregnant women with suicidal behavior	Rule-based NLP	Sensitivity = 0.58, specificity = 0.90
		Ramesh et al., 2021	Clinical records	Identify obstructive sleep apnea	Support vector machine	F-score = 0.75
		Zheng et al., 2017	EHR notes	Identify type 2 diabetes mellitus	Various machine learning methods	Average AUC = 0.98
		Gustafson et al., 2017	EHR notes	Identify atopic dermatitis	Lasso logistic regression	Positive predictive value = 0.83, sensitivity = 0.73
		Rouillard et al., 2022	EHR notes	Identify social determinants of health	SpaCy statistical machine learning	Sensitivity = 0.77, specificity = 0.69
		Sada et al., 2016	Pathology and radiology reports	Identify cases of hepatocellular cancer	Custom case-finding algorithm combining ICD-9 codes with automated console retrieval NLP	Pathology: sensitivity = 0.96, specificity = 0.97; Radiology: sensitivity = 0.94, specificity = 0.68
		Hardjojo et al., 2018	Clinical records	Identify infections disease symptoms	Keyword-based NLP	Overall precision = 0.967, recall = 0.976
		Fernandez-Gutierrez et al., 2021	EHR notes	Identify rheumatoid arthritis and ankylosing spondylitis	Custom representation, feature selection, and model selection framework	Rheumatoid arthritis: accuracy = 0.86, positive predictive value = 0.88. Ankylosing spondylitis: accuracy = 0.99, positive predictive value = 0.97
		Hazlehurst et al., 2019	EHR notes	Identify opioid-related overdoses	Rule tree	Sensitivity > 0.72, specificity > 93 for various opioid and overdose-related characteristics
		Zhang et al., 2018	Chinese EHR notes	Identify diagnosis, test, symptom, body part, and treatment	Conditional random fields and bidirectional long short-term memory	Best overall F-score = 0.89
		Kim et al., 2020	Pathology reports	Extract specimen, procedure, and pathology type	Various deep learning methods	Precision = 0.99, recall = 0.99
		Wheater et al., 2019	Text brain imaging reports of stroke/transient ischemic attack patients	Phenotype reports as ischemic stroke, hemorrhagic stroke, brain tumor, or cerebral small vessel disease and cerebral atrophy	Rule-based NLP	Sensitivity and specificity > 0.96 for all categories
		Kogan et al., 2020	Clinical notes with diagnosis of ischemic stroke, hemorrhagic stroke, or transient ischemic attack	Impute National Institutes of Health Stroke Scale score	Random forest	NLP scores vs. ground truth: R-squared = 0.57, R = 0.76, root-mean-squared error = 4.5
		Zeng et al., 2019	Breast cancer patient EHR notes	Identify distant breast cancer recurrences	MetaMap NLP with support vector machine	F-score = 0.78, 0.74; AUC = 0.95, 0.93 for two test sets
		Marella et al., 2017	EHR notes	Identify patient safety reports	Naive Bayes kernel	F score = 0.877
		Kim et al., 2022	EHR notes	Identify delirium	Logistic regression	AUC = 0.87, positive predictive value = 0.80
		Moon et al., 2022	EHR notes	Identify gynecological surgery history	Rule-based NLP	F-score = 0.76
		Okamoto et al., 2020	Clinical notes	Identify medical accidents	Linear kernel SVM	Precision = 0.28
		Han et al., 2022	Social-related sentences and clinical notes	Classify social determinants of health	Various deep learning architectures	Best overall AUC = 0.854
		Wang et al., 2009	Narrative discharge summaries	Identify adverse drug events	Rule-based NLP	Recall = 0.75, precision = 0.31
		Zheng et al., 2016	EHR notes	Identify patients with diabetes mellitus	Custom prospective case-finding algorithm	Sensitivity = 0.68, specificity = 0.98. NLP enabled 8.97% increase in true case detection compared with ICD code alone.
		Murff et al., 2011	Surgical notes	Identify postoperative complications	Concept-based indexing	Sensitivity = 0.59 − 0.91, specificity = 0.91 − 0.95 for various complications
		Thompson et al., 2019	EHR notes	Classify hormone receptor status of breast cancer patients	Various ML NLP methods with relevant word order vectorization	F-score = 0.40 − 0.96 for various methods and receptors
		Rybinski et al., 2021	EHR records	Extract family history information	Rule-based NLP	F-score = 0.81
		Kormilitzin et al., 2021	Free-text patient records	Identify drug name, route of administration, frequency, dosage, strength, form, and duration	Deep learning with self-supervised pre-training	F-score = 0.957
		Zhao et al., 2021	EHR notes of patients with atrial fibrillation	Identify incident stroke	Logistic regression and random forest	Best positive predictive value = 0.86, negative predictive value = 0.96
		Escudie J.-B. et al., 2015	Various clinical documents	Identify 15 auto-immune diseases	Regex automated pruning with human supervision	741 patients selected from 6340 in 2 hours
		Osborne J.D. et al., 2016	Clinical notes	Identify nationally mandated reportable cancer cases	Custom architecture of rule-based and machine learning annotators	Precision = 0.84, recall = 0.84
		Forsyth A.W. et al., 2018	EHR notes	Identify breast cancer symptoms as positive, negative, or absent	Conditional random field	Precision = 0.82-0.99, recall = 0.56-1.00
		Penrod N.M. et al., 2019	EHR notes	Identify mentions of yoga	Naive Bayes classifier	Increasing trends of yoga mentions observed over time
		Sagheb E. et al., 2021	Knee arthroplasty notes	Detect category of surgery, laterality, constraint, and patellar resurfacing	Rule-based NLP	Accuracy > 0.98 for all measures
		Zhu Y. et al., 2022	Last hospital visit discharge records of deceased persons	Identify cause of death	Transformer model	Accuracy = 0.81
		Rabbani et al., 2023	EHR notes	Identify confidential content in adolescent clinical notes	Rule-based NLP	AUC = 0.88
		Visser J. et al., 2022	Radiology reports	Detect actionable findings and document nonroutine communication	Random forest	AUC = 0.876, sensitivity = 0.841, specificity = 0.990
		Lybarger et al., 2023	EHR notes	Extract social determinants of health from EHR	Deep learning	F-score = 0.86
		Gao Y. et al., 2022	EHR notes	Automatically summarize patients’ main problems from daily progress notes	Adaptive NLP	Significant performance gains compared to rule based system (+0.45 - +8.72 BERTScore)
		Kiser A. et al., 2022	EHR notes	Group EHR data to improve transfer between institutions	Random Forest, SVM, XGBoost	AUC difference in difference (DiD) from 0.005 to 0.248 between baseline and grouped models
		Ozonoff et al., 2023	EHR notes	Extract patient safety events	NLP, random forest	Accuracy > 0.9
		Allen et al., 2023	EHR notes	Extract social factors from EMR	NLP state machine	PPVs from 0.95-0.97
		Litake et al., 2024	EHR notes	Classify acute renal failure	Large language models	AUC as high as 0.84
		Gray et al., 2023	Medical notes	Identify social needs	Scalable rule-based model	F-score 0.79-0.92
		Yang et al., 2024	Clinical notes	Identify stroke	Relational rule-based NLP	AUC = 0.67
		Chiang et al., 2024	Clinical notes	Extract headache frequency	Generative NLP framework	Accuracy = 0.92
		Leory et al., 2024	Clinical notes	Identify autism spectrum disorders	Transparent deep learning	F-score = 0.91
		Hua et al., 2024	Psychiatric admission notes	Identify psychosis episodes	Rule-based, decision-tree, and deep learning methods	Decision tree and deep learning outperformed rule-based, F-score = 0.88
		Yoshida et al., 2024	EHR notes	Detect gout flare	NLP methods with Medicare claims data	AUC = 0.73
	Increasing Patient Understanding	Chen et al., 2018	Clinical notes	Link medical terms to lay definitions	Statistical	Improved lay understanding21
		Moramarco F et al., 2021	Clinical notes sentences	Convert sentence in medical language to simplified sentence in lay terms	Custom candidate search and ranking algorithm	System demonstrated as effective
	Speech Recognition and Error Detection	Lybarger et al., 2018	Clinical notes generated with SR	Detect errors	LR and CRF	Detected 67% of sentence-level edits, 45% of word-level edits. False-detection rate = 15%
		Minn et al., 2015	Radiology reports	Detect sex and laterality errors	Rule-Based	Significant improvement in proportion of errors corrected.
		Lee Y. et al., 2023	Clinical notes	Detect missing vitrectomy codes	XGBoost, Random Forest	AUC of 0.87, detected 66.5% of missed codes
		Voll et al., 2008	Clinical notes generated with SR	Detect errors	Statistical	Recall = 0.83, precision = 0.26 for all notes. For subgroups of notes, recall as high as 96%
	Integrative Documentation Assistant	Wang et al., 2022	Patient encounter audio and physician notes	Speech recognition, speaker diarization, medical term extraction, abbreviation disambiguation, clinical decision support	Kaldi, ASpIRE chain model, long short-term memory, neural concept recognizer, PhenoTips methods	Speech recognition word error rate = 0.53; Phenotype recognizer precision = 0.83, recall = 0.51
		Kaufman et al., 2016	Prospective study of clinical note dictation protocols	Reduce average time to complete a clinical note	Standard, custom, and hybrid dictation-based protocols	Custom NLP-NLP protocol decreased documentation time, but quality also decreased slightly.
		Xia X. et al., 2022	EHR notes	Develop speech-recognition based EMR	Recurrent Neural-Network based Language Model	Accuracy = 0.97, decreased documentation time by 56%
		Mairittha T. et al., 2019	Prospective study of spoken dialogue system	Increase documentation speed, accuracy, and user satisfaction	Frame-based dialogue system	Average documentation speed increased by 15%, 96% accuracy observed, average satisfaction rating increased
		Owens et al., 2024	Observational study of primary care providers with ambient documentation assistant	Decrease documentation time, disengagement, and burnout scores	Ambient voice natural language processing	Documentation time and provider disengagement decreased, but burnout score did not.
		Hartman et al., 2023	Inpatient neurology notes	Automate discharge summary hospital course generation	Transformer-based methods	62% of summaries met the standard of care
Aid CDI Initiatives	Assessing Clinical Note Quality	Kshatriya et al., 2020	Clinical notes	Identify discussion of inhaler technique	Modified BERT	Precision = 1.0, recall = 0.82
		Bozkurt et al., 2018	Digital rectal exam note sentences	Classify exam temporality	Rule-Based	Historical DRE: precision = 0.84 and recall = 0.67; Other: precision ≥ 0.95 and recall ≥ 0.90
		Stemerman et al., 2021	Clinical notes	Identify social determinants of health domains	BLSTM	AUC-ROC = 0.939, average precision-recall = 0.76
		Tseng et al., 2021	Clinical notes	Identify discussion of prediabetes	LR and bi-directional recurrent neural networks	Sensitivity, specificity, positive predictive value, and negative predictive value > 0.9100
		Zhang et al., 2014	Clinical note sentences	Classify as new or redundant information	Statistical	Recall = 0.83, precision = 0.74
		Gabriel et al., 2018	Clinical notes	Identify near-to-exact duplicates	Minhashing and clustering	Showed large number of near-to-exact duplicate notes
		Deng et al., 2019	Contrast allergy notes	Classify quality as high, intermediate, or low	MTERMS	Identified need for improved quality
		Agaronnik et al., 2020	Colorectal cancer notes	Assess functional status documentation frequency	Rule-Based	Identified widespread lack of functional status documentation
		Denny et al., 2015	Medical student clinical notes	Identify competencies	KMST	Advanced directives competency: precision = 1.0, recall = 0.69 Altered mental status competency: precision = 0.93, recall = 1.0
		Cruz et al., 2019	EHR notes	Deliver real-time feedback regarding clinical pathway adherence	Savana NLP system	Adherence rate improved in 8 out of 18 practices
		Schmeelk S et al., 2022	Clinical notes	Classify clinical note cyber-risk of patient health information content	Support vector machine	F-score = 0.792
		Schaye et al., 2022	Resident and fellow admission notes	Classify clinical reasoning documentation quality as high or low	Logistic regression	AUC = 0.88
		Uyeda et al., 2022	Inpatient EHR notes	Identify goals-of-care discussion	Bag-of-words, rule-based, and hybrid approaches with regularized logistic regression	AUC = 0.90-0.96 for various approaches
		Razjouyan et al., 2021	EHR notes	Identify patient priorities care language	Statistical machine learning	Precision = 0.84, recall = 0.84
		Schwartz et al., 2022	Clinical notes	Identify prediabetes discussion	Rule-based and various machine learning techniques	F-score = 0.731-0.984 for various methods
		Hazlehurst et al., 2005	EMR records of known smokers	Assess adherence to tobacco treatment guidelines	Rule-based NLP	NLP performance appeared adequate to replace human evaluation of guideline adherence
		Lee et al., 2021	EHR notes	Identify goals of care conversations	Regularized logistic regression	AUC = 0.94
		Steiner J.M. et al., 2020	EHR notes	Identify goals-of-care documentation for adults with congenital heart disease at end of life	Probabilistic NLP/ML	Sensitivity = 0.13, specificity = 0.93
		Marshall T. et al., 2023	EHR notes	Detect diagnostic uncertainty	Rule-based NLP	Sensitivity = 0.894, specificity = 0.967
		Li et al., 2014	Care events from EHR notes	Analyze adherence to a congestive heart failure care pathway	Hidden Markov model	Deviations from care pathway identified
		Seinen et al., 2024	EHR notes	Refine unspecific condition codes	Semi-supervised models	Improved specificity of over half of unspecific codes
		Barcelona et al., 2024	Labor and birth notes	Detect stigmatizing language	Natural language encoding, machine learning classification	F-score = 0.73-0.91
		Zuo et al., 2023	Clinical notes	Standardize format	Transformer large language models	Accuracy = 0.90
	Identifying Documentation Trends	Young-Wolff et al., 2017	Clinical notes	Determine ENDS documentation frequency	Rule-Based	Showed increase over time, informed need for structured documentation
		Modre-Osprian et al., 2015	Collaborative health network notes	Analyze topic trends	Word Bagging	Led to identification of ways the network could be improved
		Gong JJ et al., 2021	Clinical note metadata	Analyze production patterns	Unsupervised learning	Identified relationship between note production style and clinical work hours
		Dugas et al., 2013	Clinical documentation forms	Compare data elements	Semantic enrichment, rule-based comparison	Compared and visualized form similarity with grid images and dendrograms
		Chen et al., 2021	EHR audit logs	Mine task characteristics	Unsupervised machine learning	Identified trends in clinician workflows
		Rajkomar et al., 2016	EHR notes	Characterize utilization patterns of primary care patients and create weighted panel sizes for providers based on patient patterns	Decision rules and k-means clustering	7 utilization phenotypes identified. Future utilization predicted with r-squared = 0.394
Abbreviations: artificial neural network (ANN), bidirectional encoder representations from transformers (BERT), conditional random field (CRF), electronic nicotine delivery system (ENDS), KnowledgeMap and SecTag (KMST), logistic regression (LR), medical text extraction, reasoning, and mapping system (MTERMS)

The earliest included study was published in 2005, with the number of studies increasing from 2005 to 2022 (Figure 2). Notably, while 25 studies (an average of 2.08 per month) were published in 2022, only 18 studies (an average of 0.95 per month) were published from January 2023 to July 2024. (Figure 2) This 46 percent decrease in peer-reviewed studies per month was noted to coincide with the release of ChatGPT on November 30, 2022. Current AI tools improved clinical documentation by aiding clinicians or CDI initiatives in six domains expanded on below: tools aided clinicians by structuring data, annotating notes, detecting errors, or serving as AI documentation assistants; or aided CDI initiatives by evaluating documentation quality or identifying trends. Seventy-seven percent of studies aided clinicians, while 23 percent aided CDI initiatives (Figure 3). Most studies concerned data structuring algorithms (68 percent), followed by evaluating quality (18 percent), identifying trends (5 percent), detecting errors (3 percent), AI-enabled assistants (5 percent), and annotating notes (1 percent) (Figure 3). While the prevalence of studies in each domain varies, each has the potential to improve clinical documentation as discussed below.

Figure 2.

Distribution of studies by time and domain[KM4].

Figure 3.

Distribution of studies by domain[KM5].

Structuring Free-Text Data

Once the standard in documentation, free-text notes are flexible and allow clinicians to dictate or type. In contrast, structured data consists of pre-populated fields offering a less flexible but organized, searchable, and easily analyzed note format.⁴⁶, ⁴⁷ AI tools have the potential bridge this gap, saving clinicians time by organizing text into paragraphs, presenting only the most relevant options in picklists, and automatically placing important information in structured data fields.

By necessity, clinic notes contain headings and an inherent organization on which an AI system can be applied. Rule-based approaches have been effective for various data structuring tasks, including classifying race with F-score = 0.911-0.984 (higher F-score indicating better performance),⁴⁸, ⁴⁹ identifying confidential content in adolescent clinical notes,⁵⁰ extracting social needs,⁵¹ and identifying stroke.⁵² Wang et al. developed an AI-guided dynamic picklist to display the most probable allergic reactions once an allergen is entered, resulting in 82.2 percent of the top 15 picklist choices being relevant to a given note.⁵³ Beyond rule-based methods, Gao et al. developed an adaptive unsupervised algorithm which automatically summarizes patients’ main problems from daily progress notes with significant performance gains compared to alternative rule-based systems.⁵⁴ To further enable data structuring, natural language processing (NLP) models were built by Ozonoff et al. and Allen et al. to extract patient safety events and social factors from EMR, with accuracy > 0.9 and positive predictive value from 0.95-0.97, respectively.⁵⁵, ⁵⁶ Yoshida et al. improved the accuracy of automated gout flare ascertainment using least absolute shrinkage and selection operator methods in addition to a Medicare claims-only model.⁵⁷

In recent years, neural networks have been increasingly used for AI CDI. Moen et al. used a neural network to organize free-text sentences from nursing notes into paragraphs and assign subject headings with 69 percent coherence.⁵⁸ Deep learning and generative models have been applied to extract social determinants of health,⁵⁹ classify acute renal failure with AUC = 0.84 (AUC = 1.0 indicates perfect classification),⁶⁰ extract headache frequency,⁶¹ and identify autism spectrum disorders.⁶² Hua et al. identified psychosis episodes in psychiatric admission notes, showing that decision-tree and deep-learning methods outperformed rule-based classification.⁶³

Algorithms were also developed to structure data for inter-department communication and even communication between institutions. For example, Visser et al.'s random forest model detected actionable findings and nonroutine communication in radiology reports with an AUC of 0.876.⁶⁴ Kiser et al. developed models to group EHR transfers to improve transfer between institutions, yielding AUC difference-in-difference ranging from 0.005 to 0.248.⁶⁵ Other studies have structured a wide variety of data, often with high accuracy; in total, 88 studies in the domain of structuring free text data were identified (Table 1). While promising, the above methods were not compared against the accuracy and efficiency of unassisted physicians, limiting external applicability.

Increasing Patient Understanding

As patient access to clinical documentation is increasingly mediated through online portals, medical terminology remains difficult for patients to understand.⁶⁶, ⁶⁷ Chen et al. developed a system with rule-based and hybrid methods to link medical terms in clinical notes to lay definitions, which improved note comprehension among laypeople.⁶⁸, ²¹ Also towards this goal, Moramarco et al. used an ontology-based algorithm to convert sentences in medical language to simplified sentences in lay terms.⁶⁹ In the future, these two methods of increasing patient understanding could increase patient adherence to treatment and decrease costs associated with nonadherence (Table 1).²¹, ⁶⁸, ⁷⁰

Speech Recognition and Error Detection

AI-based speech recognition (AI-SR) programs promise to create the possibility of a “digital scribe” to decrease documentation burden. Such programs evaluated in the peer-reviewed literature are limited by an increased error rate while newer commercial programs have not been well-studied.⁷¹ Five studies reported a 19.0 to92.0 percent decrease in mean documentation time with AI-SR, four studies reported increases of 13.4 to 50.0 percent, and three studies reported no significant difference.⁷², ⁷³, ⁷⁴, ⁷⁵, ⁷⁶, ⁷⁷, ⁷⁸, ⁷⁹, ⁸⁰, ⁸¹, ⁸², ⁸³ The ability of NLP tools to identify features of grammatically correct versus incorrect language shows promise for improving SR error rates by detecting errors in SR-generated notes.⁸⁴, ⁸⁵, ⁸⁶ Lybarger et al. detected 67.0 percent of sentence-level edits and 45.0 percent of word-level edits in clinical notes generated with SR,⁸⁴ while Voll et al. detected errors with 83.0 percent recall and 26.0 percent precision.⁸⁶ Lee et al. developed a model which was able to detect missing vitrectomy codes with an AUC of 0.87 while detecting 66.5 percent of missed codes.⁸⁷ Since high error rate could be rectified in the future, clinicians already appear interested in adopting AI-SR: a survey of 1731 clinicians at two academic medical centers reported high levels of satisfaction with AI-SR and the belief that SR improves efficiency.⁸⁸ In total, four studies were identified which addressed the domain of speech recognition and error detection (Table 1).

Integrative Documentation Assistance

AI has also been proposed as a real-time assistant to improve documentation during patient encounters by recording patient encounter audio, supporting physician decisions, calculating risk scores, and suggesting clinical encounter codes.⁸, ⁸⁹ Wang et al. developed an AI-enabled tablet computer-based program that transcribes conversations from patient encounter audio, generates text from handwritten notes, automatically suggests clinical phenotypes and diagnoses, and incorporates the desired images, photographs, and drawings into clinical notes with 53.0 percent word error rate for the SR component and 83.0 percent precision and 51.0 percent recall for the phenotype recognizer.⁹⁰ Mairittha et al. developed a dialogue-based system that increased average documentation speed by 15 percent with 96 percent accuracy.⁹¹ Kaufman et al. developed an AI tool to transcribe data from speech and convert the resulting text into a structured format, decreasing documentation time but causing a slight decrease in documentation quality.⁹² Xia et al. developed a speech-recognition based EHR with accuracy of 0.97 that reduced documentation time by 56 percent.⁹³ Owens et al. reported that use of an ambient voice documentation assistant was associated with significantly decreased documentation time and provider disengagement, but not total provider burnout.⁹⁴ Hartman et al. developed an integrated documentation assistant for automated generation of summary hospital course texts for neurology patients, 62 percent of which were judged to meet the standard of care by board-certified physicians.⁹⁵ The increased efficiency of these studies is promising, but clinical implementation may be precluded by time spent correcting errors resulting from decreased documentation quality (Table 1).

Assessing Clinical Note Quality

A component of CDI initiatives is often manual chart review to assess clinical notes for timeliness, completeness, precision, and clarity.⁶, ⁹⁶ AI tools can assist in that end by recognizing the presence or absence of knowledge domains, social determinants of health, performance status, and topic discussion, prompting clinicians to make additional notes relating to a domain when needed (Table 1).^97–101 In addition to these domains, note unclarity and redundant information comprise major problems in clinical documentation.¹⁰² Deng et al. used an NLP system to evaluate the quality (classified as high, intermediate, or low) of contrast allergy records based on ambiguous or specific concepts, finding that 69.1 percent were of low quality.¹⁰³ Zhang et al. developed a method with conditional random fields and long-short term memory to classify information in clinical notes as new or redundant at the sentence level, achieving 83.0 percent recall and 74.0 percent precision.³³, ¹⁰⁴ Similarly, Gabriel et al. developed an algorithm to classify pairs of notes as similar, exact copies, or common output (automatically generated notes such as electrocardiogram, laboratory results, etc.).¹⁰⁵ Seinen et al. both detected and improved note quality, using semi-supervised models to refine unspecific condition codes in EHR notes.¹⁰⁶ Zuo et al. also improved quality by standardizing clinical note format using a transformer-based approach.¹⁰⁷ Regarding specific content standards, Barcelona et al. used natural language processing to identify stigmatizing language in labor and birth clinical notes.¹⁰⁸

The concepts of content domains, note clarity, and redundancy do not account for changes in these domains over time. This meta-data element can be harnessed to improve clinical documentation as demonstrated by Bozkurt et al., who used an NLP pipeline to evaluate documented digital rectal examinations (DRE) by insurance provider and classify them as current, historical, hypothetical, deferred, or refused.¹⁵, ¹⁰⁹ Other studies identified time-sensitive documentation concerns including goals-of-care discussions at the end of life, patient priorities language, and adherence to care pathways in heart failure.¹¹⁰, ¹¹¹, ¹¹² Another model developed by Marshall et al. detected diagnostic uncertainty from EHR notes using a rule-based NLP with a sensitivity of 0.894 and specificity of 0.967.¹¹³ In total, 20 studies were identified which addressed the domain of clinical note quality (Table 1). Such algorithms could be used to prompt clinicians if protocols and procedures are not correctly documented within a given time after disease diagnosis.

Identifying documentation trends

While recognizing and tracking meta-data trends in documentation may improve documentation, it also has a role in intelligent modification of EHR systems and documentation policies. Young-Wolff et al. used an iterative rules-based NLP algorithm to demonstrate that electronic nicotine delivery system (ENDS) documentation increased over nine years; the team recommend an ENDS structured field be added to the EHR.¹¹⁴ Since clinicians vary in their documentation styles, Gong et al. extracted a “gold standard” style by evaluating note-level and user-level production patterns from clinical note metadata with unsupervised learning. Their results implied uninterrupted, morning charting could improve efficiency.¹¹⁵

Besides individual styles, documentation is complicated by health system factors such as heterogeneous medical forms and many compartmentalized specialties.²⁰, ¹¹⁶ AI may collimate these trends leading to intelligently standardized forms and a more efficient system. Dugas et al. automatically compared and visualized trends in medical form similarity using semantic enrichment, rule-based comparison, grid images, and dendrograms.¹¹⁶ Modre-Osprian et al. analyzed topic trends in notes from a collaborative health information network, yielding insights about wireless device usage that improved network functioning.¹¹⁷ Further studies used AI to find trends in EHR audit logs and utilization patterns of notes, allowing efficiency trends to be identified.¹¹⁸, ¹¹⁹ In total, 6 studies were identified which addressed the domain of identifying documentation trends (Table 1). By overviewing meta-data trends in both individual clinical documentation patterns and rapidly changing health systems, AI tools could aid system optimization as medical infrastructure changes and care is delivered in new, increasingly specialized ways.

Discussion

As reviewed above, current AI tools improved clinical documentation by structuring data, annotating notes, and providing real-time assistance. Other features of AI CDI tools include assessing clinical note quality based on concept domains and providing insight into hospital systems and provider practices. A truly practical comprehensive clinical AI assistant has not yet been reported in the peer-reviewed literature to our knowledge, but current AI tools could confer specific improvements to documentation workflows.

To overcome limitations in generalizability, future work should involve larger datasets and broader training data availability. AI processing of large amounts of data would require large computational processing power, a requirement which may become feasible as computational power continues to increase in the future.⁹, ¹²⁰ This necessitates carefully regulated and secure computing systems which must also account for documentation variations between geographic regions, institutions, and EHRs.¹²¹ AI-based systems that promote documentation inter-operability could play a role in overcoming these challenges by creating larger unified training datasets.⁶⁵, ¹⁰⁷ While a widely generalizable AI system could possibly be trained, such data is often proprietary and not readily shared. Transfer learning techniques, which apply previously learned information to a new situation or context, may bridge this gap, enabled by collaboration and data sharing between health systems.¹²² Lexical variations can be overcome either by semantic similarity in rule-based NLP or implementing machine learning techniques.¹²¹

Legal and ethical concerns relating to encounter recording and AI processing must also be addressed simultaneously with these changes for these systems to be successful long-term.¹²³ Patients may have privacy concerns regarding the automatic collection, storage, and processing of encounter data, and the liability implications of AI-assisted clinical documentation, such as where blame falls when a documentation error occurs, are currently unclear. Ethical concerns raised in the literature include the nature of informed consent, algorithmic fairness and biases, data privacy, safety, and transparency.¹²⁴ Legal challenges include safety, effectiveness, liability, data protection, privacy, cybersecurity, and intellectual property.¹²⁴

For AI CDI systems to implemented clinically, they must increase efficiency without sacrificing accuracy.⁷¹ In some cases, time spent fixing errors produced by AI outweighs time saved using the AI tool.⁷¹ The accuracy of AI-assisted versus clinician-generated notes has not been widely compared,⁸⁴ and there is also a lack of studies investigating clinical outcomes and patient care which must be assessed before widespread AI CDI implementation.⁹⁰, ⁹¹, ⁹², ⁹³, ⁹⁴, ⁹⁵

While further studies of AI CDI tools are needed, this systematic review is the first to our knowledge to highlight a decrease in peer-reviewed AI CDI studies published following the release of ChatGPT on November 30, 2022.¹²⁵ Reasons for this trend are not entirely clear, but may be due to researchers publishing on preprint servers amidst rapidly advancing techniques or developing proprietary models without publishing. The advent of large transformer language models shows promise, but rigorous peer-reviewed evaluation of proprietary models for improving clinical documentation is lacking.¹²⁶, ¹²⁷

Limitations and Future Studies

Strengths of this narrative systematic review include that it presents AI tools for clinical documentation improvement in the context of medical practice and health systems, and that it is the first study to do so comprehensively. This study is subject to several limitations: the relevance of studies was determined by the authors, the efficacy of all tools was not objectively compared, commercial programs not studied in the peer-reviewed literature could not be evaluated, and studies may exist outside of the queried databases. Concerns of cost, physician and hospital system acceptance, and potential job loss regarding AI CDI tools are not negligible. However, this is beyond the scope of this review which set out to report solely on methods and formats of improving documentation.

Conclusion

While current AI tools offer targeted improvements to clinical documentation processes, moderately high error rates preclude the broad use of a comprehensive AI documentation assistant. While large language models have the potential to greatly reduce error rates, many of these models are proprietary and not well-studied in the peer-reviewed literature. In the future, this hurdle may be overcome with further rigorous tool evaluation and development in direct consultation with physicians, as well as robust discussion of the legal and ethical ramifications of AI CDI tools.

Biographies

Cleveland Clinic Lerner College of Medicine of Case Western Reserve University.

Cleveland Clinic Cole Eye Institute.

Case Western Reserve University School of Medicine.

Vice President and Chief Medical Officer of Martin North and South Hospitals and Professor of Ophthalmology at Cleveland Clinic Lerner College of Medicine.

Appendix 1. Search strategies used.

PubMed

(“artificial intelligence” OR “machine learning” OR “natural language processing” OR “deep learning” OR “automatic categori*” OR “automatic encoding” OR “Artificial Intelligence”[Majr]) AND (“electronic medical record*” OR “electronic health record*” OR “EHR” OR “EMR” OR “computerized physician” OR “computerised physician” OR “computerized provider order entry” OR “computerised provider order entry” OR “Electronic Health Records”[Majr] OR “Medical Order Entry Systems”[Majr]) AND (“note”[Ti] OR “notes”[Ti] OR notation[Ti] OR documentation[Ti] OR (“code”[Ti] OR “codes”[Ti])) NOT (predict* OR extract*)

Embase

• 1 exp electronic health record/ 31290

• 2 exp artificial intelligence/ or exp ambient intelligence/ or exp artificial general intelligence/ or exp automated reasoning/ or exp computer heuristics/ or exp multicriteria decision analysis/ 63699

• 3 exp machine learning/ or exp artificial neural network/ or exp automated pattern recognition/ or exp automatic speech recognition/ or exp back propagation/ or exp bayesian learning/ or exp classification algorithm/ or exp classifier/ or exp computer heuristics/ or exp cross validation/ or exp data mining/ or exp feature detection/ or exp feature extraction/ or exp “feature learning (machine learning)”/ or exp feature ranking/ or exp feature selection/ or exp fuzzy system/ or exp hidden markov model/ or exp iterative closest point/ or exp k nearest neighbor/ or exp kernel method/ or exp knowledge discovery/ or exp learning algorithm/ or exp “least absolute shrinkage and selection operator”/ or exp markov state model/ or exp memristor/ or exp molecular docking/ or exp multicriteria decision analysis/ or exp multifactor dimensionality reduction/ or exp network learning/ or exp online analytical processing/ or exp outlier detection/ or exp perceptron/ or exp radial basis function/ or exp random forest/ or exp recursive feature elimination/ or exp recursive partitioning/ or exp relevance vector machine/ or exp rough set/ or exp semi supervised machine learning/ or exp supervised machine learning/ or exp support vector machine/ or exp unsupervised machine learning/ 3218494 exp computer interface/ 34754

• 5 2 or 3 or 4 372966

• 6 1 and 5 3590

• 7 exp documentation/ 575335

• 8 6 and 7 3590

• 9 exp medical documentation/ 29023

• 10 6 and 9 52

• 11 or/2-3 342397

• 12 1 and 4 and 11 48

• 13 10 or 12 99

Scopus

((TITLE-ABS-KEY (electronic AND health AND record) OR TITLE-ABS-KEY (clinical AND documentation)) OR (TITLE-ABS-KEY (electronic AND medical AND record))) AND (TITLE-ABS-KEY (ai OR artificial AND intelligence OR machine AND learning OR natural AND language AND processing OR speech AND recognition OR fuzzy AND logic))

Web of Science

(TI=(Electronic health record OR electronic medical record OR clinical documentation) AND TI=(artificial intelligence OR AI OR machine learning OR natural language processing OR fuzzy logic OR speech recognition)) AND (LA==(“ENGLISH”) AND TASCA==(“MULTIDISCIPLINARY SCIENCES” OR “PUBLIC ENVIRONMENTAL OCCUPATIONAL HEALTH” OR “MATHEMATICAL COMPUTATIONAL BIOLOGY” OR “INFORMATION SCIENCE LIBRARY SCIENCE” OR “COMPUTER SCIENCE ARTIFICIAL INTELLIGENCE” OR “MEDICINE GENERAL INTERNAL” OR “COMPUTER SCIENCE INTERDISCIPLINARY APPLICATIONS” OR “COMPUTER SCIENCE INFORMATION SYSTEMS” OR “HEALTH CARE SCIENCES SERVICES” OR “MEDICAL INFORMATICS”))