Explainability Enhanced Machine Learning Model for Classifying Intellectual Disability and Attention-Deficit/Hyperactivity Disorder With Psychological Test Reports

Abstract

Background

Psychological test reports are essential in assessing intellectual functioning, aiding in diagnosing and treating intellectual disability (ID) and attention-deficit/hyperactivity disorder (ADHD). However, these reports can have several problems because they are diverse, unstructured, subjective, and involve human errors. Additionally, physicians often do not read the entire report, and the number of reports is lower than that of diagnoses.

Methods

We developed explainable predictive models for classifying IDs and ADHDs based on written reports to address these issues. The reports of 1,475 patients with IDs and ADHDs who underwent intelligence tests were used for the models. These models were developed by analyzing reports using natural language processing (NLP) and incorporating the physician’s diagnosis for each report. We selected n-gram features from the models’ results by extracting important features using SHapley Additive exPlanations and permutation importance to make the models explainable. Developing the n-gram feature-based original text search system compensated for the lack of human readability caused by NLP and enabled the reconstruction of human-readable texts from the selected n-gram features.

Results

The maximum model accuracy was 0.92, and the 80 human-readable texts were restored from four models.

Conclusion

The results showed that the models could accurately classify IDs and ADHDs, even with a few reports. The models were also able to explain their predictions. The explainability-enhanced model can help physicians understand the classification process of IDs and ADHDs and provide evidence-based insights.

Graphical Abstract

INTRODUCTION

Neurodevelopmental disorders are defined as conditions that begin during the developmental period and result in deficits that cause impairment in functioning.1 Among them, intellectual disability (ID) and attention-deficit/hyperactivity disorder (ADHD) are mostly prevalent.2 According to the Diagnostic and Statistical Manual of Mental Disorders,3 patients are diagnosed with an ID if they have deficiencies in both intellectual functioning and adaptive behaviors that originate during developmental stages. Individuals with IDs often experience challenges in learning, reasoning, problem-solving, and adjusting to daily life. Such disabilities can impact cognitive abilities and the capacity to learn and adapt to unfamiliar scenarios.3 Therefore, treatment and support for IDs should be individualized and may involve educational interventions, speech therapy, occupational therapy, and social skills training. The emphasis is on helping individuals obtain adaptive skills and reach their maximum potential despite their cognitive limitations.4,5

On the other hand, ADHD patients are diagnosed with specific symptoms, which fall into two main categories: inattention and hyperactivity-impulsivity. The primary symptoms of ADHD are inattention (difficulty sustaining attention, careless mistakes), hyperactivity (excessive restlessness or fidgeting), and impulsivity (difficulty waiting one’s turn, blurting out answers). It does not inherently affect an individual’s intellectual abilities. Unlike ID patients, ADHD patients do not have deficiencies in intellectual functioning and adaptive behaviors but have average or above-average intellectual functionality.3 Therefore, their difficulties are mainly related to execution function and attention control rather than the cognitive impairment suffered by ID patients.3 Treatment for ADHD generally involves behavioral interventions, psychoeducation, and medication, such as stimulant or non-stimulant medications. These aims to improve attention, control impulses, and reduce hyperactivity.6

To summarize, ID and ADHD differ significantly regarding cognitive functioning when comorbidities are not considered. ID is characterized by below-average intellectual functioning, whereas ADHD is characterized by average or above-average intellectual functioning.3 Therefore, the treatments and interventions for these two conditions should be prescribed differently based on intellectual functioning.7,8,9,10

Psychologists select and administer various tests in clinical settings, recording observations and scores in a psychological test report, which will be stored in the hospital’s electronic medical record (EMR) system.11,12 Physicians then diagnose comprehensively based on these reports and other clinical data. However, these reports have limitations, including potential human error in scoring and subjective analysis.13,14 Variations in writing styles, language use, and potential typographical errors further complicate consistency.15,16 Also, physicians often do not review entire reports, and with an 84% match rate between psychologists’ suggestions and final diagnoses (Table 1), understanding the complete report is crucial. Additionally, psychological tests are conducted infrequently, leading to fewer reports relative to physician diagnoses.

Table 1. Demographic characteristics of the ID and ADHD groups.

Characteristics		No. (%) of participants			P valuea
		ID (n = 1,014, 68.7%)	ADHD (n = 461, 31.3%)	Total (N = 1,475, 100%)
Sex					< 0.001
	Male	641 (43.5)	350 (23.7)	484 (32.8)
	Female	373 (25.3)	111 (7.5)	991 (67.2)
Age, yr					< 0.001
	< 10	239 (16.2)	233 (15.8)	472 (32.0)
	10–19	360 (24.4)	217 (14.7)	577 (39.1)
	20–29	153 (10.4)	11 (0.7)	164 (11.1)
	30–39	73 (4.9)	0 (0.0)	73 (4.9)
	40–49	79 (5.4)	0 (0.0)	79 (5.4)
	50–59	88 (6.0)	0 (0.0)	88 (6.0)
	60–69	19 (1.3)	0 (0.0)	19 (1.3)
	70–79	3 (0.2)	0 (0.0)	3 (0.2)
Department					< 0.001
	Pediatrics	48 (3.3)	91 (6.2)	139 (9.4)
	Psychiatry	797 (54.0)	364 (24.7)	1,161 (78.7)
	Rehabilitation Medicine	168 (11.4)	6 (0.4)	174 (11.8)
	Plastic and Reconstructive Surgery	1 (0.1)	0 (0.0)	1 (0.1)
Diagnostic concordance rate					< 0.001
	Consistent	913 (61.9)	327 (22.2)	1,240 (84.1)
	Inconsistent	38 (2.6)	90 (6.1)	128 (8.7)
	Both	63 (4.3)	44 (3.0)	107 (7.3)

ID = intellectual disability, ADHD = attention-deficit/hyperactivity disorder.

^aThe difference in the distribution of each variable between ID and ADHD was tested using the χ² test.

To address challenges, this study aimed to develop a system with high classification accuracy for ID and ADHD using few reports. It focused on minimizing human error by converting subjective texts into objective factors and ensuring report standardization. Sensitive data handling necessitated the use of explainable artificial intelligence17,18 to provide evidence-based insights.19

We utilized Natural Language Processing (NLP) to transform unstructured data into structured formats, facilitating text data use in EMRs.20 Four classification models—Naïve Bayes (NB),21 random forest (RF),22 eXtreme Gradient Boosting (XGB),23 and Light Gradient Boosting Machine (LGBM)24—were selected and optimized via Bayesian Optimization (BO).25 Explainability was achieved by extracting features using SHapley Additive exPlanations (SHAP)26 and permutation importance (PI).27 An n-gram feature-based original text search (NOTS) system was developed to present NLP-derived features in a human-readable format to enhance interpretability.

METHODS

Participants and study design

The study was conducted on a cohort of patients (n = 1,816) with IDs and ADHDs between January 1, 2011, and May 31, 2022. The selection criteria were age 0–90 years, a confirmed diagnosis of ID or ADHD, and having undergone an intelligence test before the diagnosis. The final number of reports was 1,475, as 341 reports were excluded based on the following exclusion criteria: 1) identical contents (n = 202), 2) a diagnosis of both ID and ADHD (n = 136), and 3) missing contents (n = 3).

Fig. 1 shows the flowchart of developing explainable predictive models for classifying IDs and ADHDs in this study. We first went through pre-processing (Fig. 1A) and NLP (Fig. 1B) to transform unstructured data into model-optimized structured data. Then, we used the generated structured data to develop classification models (Fig. 1C). Finally, we provided the explainability of the models’ results (Fig. 1D).

Fig. 1. Flowchart of developing explainable predictive models for classifying intellectual disabilities and attention-deficit/hyperactivity disorders. (A) Data pre-processing, (B) natural language processing, (C) classification model development, (D) explainable model development.

POS = part of speech, KoNLPy = Korean natural language process in Python, BoW = Bag-of-Words, TF = text frequency, IDF = inverse document frequency, SHAP = SHapley Additive exPlanations, PI = permutation importance, NOTS = n-gram feature-based original text search.

Data pre-processing

The input data used in this study were the contents of reports written by the psychologists who examined the patients. The output data were the diagnoses of ID or ADHD by a physician, integrating the report with other EMR data. Fig. 1A shows how the data of 1,475 reports were distributed. The reports were then divided in an 8:2 ratio by stratified sampling into a training and a test set (n = 1,180 and 295, respectively). All digits and special characters were deleted from all reports to measure the impact of the natural languages used by psychologists. Deleting the digits allows the models to classify IDs and ADHDs by focusing on natural languages rather than being influenced by various test scores (Fig. 1A).

We also compared the diagnostic concordance rates between the physicians’ diagnoses and the psychologists’ diagnostic suggestions to compare the accuracy of the models developed in this study. After comparing the physicians’ and psychologists’ diagnoses, we found three prominent categories: cases where the psychologist diagnosed both ID and ADHD as comorbid conditions, cases where the diagnoses of the physician and psychologist were consistent, and cases where their diagnoses were inconsistent (‘Diagnostic concordance rate’ of Table 1).

NLP

The psychologists’ reports were mainly written in Korean. Fig. 2 shows an example of the English translation of one such report. Because NLP performs best when applied appropriately to a language,28 we used NLP optimized for Korean. First, we analyzed the reports based on morpheme analysis or phoneme separation29,30 using the MeCab Korean natural language process in Python (KoNLPy) library.31 Based on the analysis results, we tokenized all words in the reports and selected principal parts of speech (POSes): verbs, adjectives, adverbs, nouns, and foreign languages.32

Fig. 2. Example of an English-translated version of a report.

Before the classification models could be trained using the tokenized and selected words from the reports, these words needed to be feature-engineered and transformed into numeric representations. We used the text frequency-inverse document frequency (TF-IDF) method33 for feature engineering. The TF-IDF is a frequency-based numeric representation of each word and is calculated as the product of TF and IDF. TF is the word count in each observation, and IDF is calculated as the log of document counts divided by the count of documents containing the word in question.34 We first fit the TF-IDF vectorizer with the tokenized and selected training data. Then, we applied the fitted TF-IDF vectorizer to all data. We calculated the TF-IDF weight scores from unigram to heptagram (n-gram = 1 to 7) because a continuous word series can create new meanings (Fig. 1B).

Classification model development and evaluation

We used four classification models for explainability and enhanced the model: NB, RF, XGB, and LGBM. Deep learning models such as LSTM and Transformer can provide explicability through technologies such as SHAP and Attention Mechanisms,35 but we excluded them because they cannot be fully converted to human-readable text using the NOTS system described later.

All models were fitted on the vectorized training split using stratified 5-fold cross-validation to mitigate potential bias in diagnosing ID and ADHD. This method ensures that each subset of the data used during the training and evaluation phases is representative of the overall distribution of key demographic and diagnostic variables. By maintaining a proportional representation of these variables within each fold of the cross-validation process, we aim to enhance the generalizability and reliability of our diagnostic model outcomes, thereby reducing the likelihood of biased results. This stratified approach is particularly crucial in medical and psychological assessments where discrepancies in sample representation can lead to skewed interpretations and ultimately affect clinical validity. Hyperparameter tuning through trial and error can be tedious and may lead to unsatisfactory outcomes.36

For this reason, robust tuning methods are crucial, particularly when aiming to find the maximum value at a sampling point for an unknown function.37 Therefore, we used BO25 to tune hyperparameters of NB, RF, XGB, and LGBM models using the Gaussian process as a posterior distribution. BO is a more efficient algorithm for hyperparameter optimization than commonly used Grid Search and Random Search techniques.38 After training the models, we applied the fitted model and a combination of hyperparameters to all the data with the best accuracy (Fig. 1C).

The evaluation metrics used in this study were accuracy, Area Under the Receiver Operating Characteristic Curve (ROC_AUC), positive predictive value (PPV), sensitivity, and F1 score. A comprehensive list of metrics was used because a single metric cannot judge models’ performance. For example, accuracy and ROC_AUC are sensitive to imbalanced data, such as the data used in this study, and using these measures alone would give clinicians inaccurate results. Therefore, we also measured the PPV, sensitivity, and F1 scores to compensate for the problem caused by the data imbalance. PPV, sensitivity, and F1 scores had different values depending on whether ID or ADHD was labeled positive or negative. Therefore, the PPV, sensitivity, and F1 scores were calculated considering cases where the two diagnoses were labeled as positive and negative or negative and positive, respectively.39,40

Explainable model development

After predicting the test data using the four models developed, we made the prediction results explainable by the NOTS system (Fig. 1D). Fig. 3 shows the detailed explainable model development flowchart. The explainable model consisted of two main processes: selecting n-gram features (Fig. 3A-C) and applying the NOTS system (Fig. 3C-G).

Fig. 3. Flowchart of the explainable model development. (A) Extract the top 30 important features from each method. (B) Select ten n-gram features from the top 30 important features. (C) Insert one of the ten n-gram features in the n-gram feature-based original text search system. (D) Check if the words of the n-gram feature are in the report. (E) Check if the words in the n-gram feature are in order. (F) Search texts with no other principal POS between the words of the n-gram feature. (G) Restore the original human-readable text.

NB = Naïve Bayes, RF = random forest, XGB = eXtreme Gradient Boosting, LGBM = Light Gradient Boosting Machine, SHAP = SHapley Additive exPlanations, PI = permutation importance, IF = important features, POS = part of speech.

For selecting n-gram features, we first applied the SHAP26 and PI27 methods to the four models to extract 240 important features, the top 30 for each method (Fig. 3A, Supplementary Tables 1 and 2). The important features were printed as a list of words up to seven principal POSes because we created the training data only with principal POS (verbs, adjectives, adverbs, nouns, and foreign languages) and set the TF-IDF weight score from unigram to heptagram (n-gram = 1 to 7). However, there are cases where duplicated features appear consecutively due to the use of n-grams, and we have defined these features as ‘duplicated features.’ In addition, we have defined some features as ‘irrelevant features’ because they have very different meanings than other features. As a result, we removed 160 duplicate and irrelevant features out of 240 important features, leaving a total of 80 features (10 for each method), which we defined as ‘n-gram features’ (Fig. 3B). Finally, we inserted n-gram features into the NOTS system to restore them as human-readable texts, because these n-gram features were not in sentence form, they had to be restored to human-readable sentences or phrases (Fig. 3C). In summary, the process of selecting n-gram features resulted in the selection of 80 n-gram features out of 240 important features.

We developed a NOTS system that restores human-readable texts from the n-gram features (Fig. 3C-G). The system receives the inserted n-gram features (Fig. 3C). Then, the system checks if the words of the n-gram feature are in the report (Fig. 3D). Next, the system checks if the words in the n-gram feature are present and if the words are in the same order as the inserted n-gram feature (Fig. 3E). The system then searches texts with no other principal POS between the words of the n-gram feature (Fig. 3F). Because the n-gram features are already made up of only principal POS, there should be no principal POS other than n-gram features. Finally, the system restores the original human-readable text (Fig. 3G). Based on the results of the NOTS system, we restored 80 human-readable texts from 80 n-gram features (Supplementary Tables 3 and 4).

Consequently, the NOTS system enhances the transparency and comprehensibility of machine learning model decisions for clinicians by converting important features that significantly influence the model’s outcomes into a more human-readable format.

Ethics statement

This study was approved by the Institutional Review Board of The Catholic University of Korea, Catholic Medical Center (XC22WIDI0062). The ethics panel determined that the study does not involve Human Subjects. Informed consent was waived because of the study’s retrospective nature, and the analysis used anonymous clinical data.

RESULTS

Data characteristics

Table 1 summarizes the report characteristics of the ID and ADHD groups. There were more ID diagnoses (n = 1,014, 68.7%) than ADHD diagnoses (n = 461, 31.3%) across all sexes, and there were more diagnoses in males (n = 991, 67.2%) than in females (n = 484, 32.8%). The < 10 years (n = 472, 32.0%) and 10–19 years (n = 577, 39.1%) age groups accounted for the most significant proportion of the study subjects (n = 1,049, 71.1%). Among them, ID (n = 239, 16.2%) and ADHD (n = 233, 15.8%) were similar in the < 10 years age group. There was no ADHD in the patients aged 30–79 (n = 0, 0.0%). Most departments diagnosed ID more commonly (n = 966, 65.5%) than ADHD (n = 370, 25.1%); however, only the pediatric department diagnosed ADHD more commonly (n = 91, 6.2%) than ID (n = 48, 3.3%). The diagnostic concordance between physicians’ diagnoses and psychologists’ diagnostic suggestions was only 84.1%.

Word counts from NLP

Table 2 lists the top 20 words and their counts for each ID and ADHD when a common noun, proper noun, general adverb, verb, adjective, and foreign language were extracted using POS analysis. Eleven of the 20 words appeared equally frequently in both diagnoses, among which the words ‘ability,’ ‘level,’ and ‘test’ had the same frequency ranking. The number of words with the same frequency ranking was twice as high for ID as for ADHD. ID diagnoses had several words related to intelligence (e.g., intellectual, disorder, development, retardation), whereas ADHD diagnoses had many words related to behavior (e.g., task, understanding, processing, demonstrate).

Table 2. Top 20 word counts for each ID and ADHD.

ID	Count	ADHD	Count
Ability	11,026	Ability	4,557
Level	8,812	Level	4,524
Society	5,428	Performance	2,638
Test	5,126	Test	2,415
Language	4,204	Task	2,315
Intellectual	3,776	Assessment	2,259
Intelligence	3,732	According	2,192
Assessment	3,649	Language	2,148
Age	3,638	Range	1,976
Performance	3,526	Understanding	1,918
According	3,428	Indicator	1,848
Disorder	3,380	Average	1,749
Development	3,099	Ordinary	1,722
Possibility	3,004	Society	1,711
Case	2,971	Processing	1,533
Indicator	2,863	Reasoning	1,524
Status	2,861	Results	1,452
Very	2,827	Demonstrate	1,396
Retardation	2,571	Intellectual	1,381
Reasoning	2,502	Behavior	1,354

ID = intellectual disability, ADHD = attention-deficit/hyperactivity disorder.

Classification models

Table 3 compares the performances of the four models. The first five columns show each model’s accuracy, ROC_AUC, PPV, sensitivity, and F1 score values with the test data. The accuracies of the NB, RF, XGB, and LGBM models were 0.92, 0.91, 0.87, and 0.89, respectively. The NB model exhibited the highest accuracy (0.92) and ROC_AUC (0.91). All scores were higher when the ID was set to positive than when ADHD was set to positive.

Table 3. Comparison of the performance of the four models.

Models	Accuracy	ROC_AUC	PPV (ID, ADHD)	Sensitivity (ID, ADHD)	F1 score (ID, ADHD)
NB	0.92	0.91	0.95, 0.86	0.94, 0.89	0.94, 0.88
RF	0.91	0.87	0.91, 0.90	0.96, 0.78	0.93, 0.84
XGB	0.87	0.83	0.88, 0.85	0.94, 0.73	0.91, 0.78
LGBM	0.89	0.86	0.90, 0.89	0.96, 0.75	0.93, 0.82

ROC_AUC = Area Under the Receiver Operating Characteristic Curve, PPV = positive predictive value, ID = intellectual disability, ADHD = attention-deficit/hyperactivity disorder, NB = Naïve Bayes, RF = random forest, XGB = eXtreme Gradient Boosting, LGBM = Light Gradient Boosting Machine.

Explaining reports

Table 4 shows the counts of duplicated, irrelevant, n-gram, and total features from the SHAP and PI results. The XGB and LGBM models did not have duplicated feature counts (0, 0%), whereas the RF (31, 12.9%) and NB (29, 12,1%) models had many duplicated features.

Table 4. Counts of duplicated, irrelevant, n-gram, and total features from the SHAP and PI results.

No. (%) of features	NB		RF		XGB		LGBM		Total count
	SHAP	PI	SHAP	PI	SHAP	PI	SHAP	PI
Duplicated	12 (5.0)	17 (7.1)	16 (6.7)	15 (6.3)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	60 (25.0)
Irrelevant	8 (3.3)	3 (1.3)	4 (1.7)	5 (2.1)	20 (8,3)	20 (8,3)	20 (8,3)	20 (8,3)	100 (41.7)
N-gram	10 (4.2)	10 (4.2)	10 (4.2)	10 (4.2)	10 (4.2)	10 (4.2)	10 (4.2)	10 (4.2)	80 (33.3)
Total	30 (12.5)	30 (12.5)	30 (12.5)	30 (12.5)	30 (12.5)	30 (12.5)	30 (12.5)	30 (12.5)	240 (100)

NB = Naïve Bayes, RF = random forest, XGB = eXtreme Gradient Boosting, LGBM = Light Gradient Boosting Machine, SHAP = SHapley Additive exPlanations, PI = permutation importance.

Supplementary Tables 1 and 2 show the English version of the 240 important features extracted, the top 30 for each method. These important features were originally extracted in Korean since they were extracted using the PI and SHAP methods applied in Korean reports. All four models found common occurrences of the words ‘borderline intellectual,’ ‘comprehensive ability,’ ‘intellectual,’ etc. Supplementary Tables 3 and 4 list the selected 80 human-readable texts. In all four models, ID- and ADHD-related words were evenly selected. However, in the case of the XGB and LGBM models, words related to ID (BGT, KFD, TMT) tended to appear more frequently, whereas terms related to ADHD (adjustment disorder, borderline intellectual functioning) appeared more often in the RF and NB models.

DISCUSSION

In this study, we developed explainable predictive models for classifying IDs and ADHDs from free text in reports. We transformed unstructured text into structured data optimized for classification models using appropriate pre-processing and NLP techniques. Four models were developed and tailored to the report data, with optimization achieved through BO and k-fold cross-validation. We selected 80 n-gram features from the classification results to enhance model explainability using SHAP and PI, highlighting important features. The NOTS system was developed to address the readability issues introduced by NLP, restoring 80 human-readable texts from the selected n-gram features. Thus, the NOTS system improves the transparency and understandability of decisions made by machine learning models for clinicians by translating important features, which substantially impact the model’s outcomes, into a format that is more accessible to human interpretation.

Analysis of the diagnostic concordance rates between the physician’s diagnoses and the psychologist’s diagnostic suggestions showed that the four models developed in this study were more accurate than the psychologist’s diagnostic suggestions. The maximum accuracy of our models was 92%, and the diagnostic consistency between the psychologists and doctors was 84%; therefore, the model’s accuracy was approximately 8% higher (Tables 1 and 3).

The words listed in Table 2 have the same rank, but their counts are more than twice as many in the ID group as those in the ADHD group because the amount of ID data was twice as much as the ADHD data. Table 2 also shows that more than half of the word counts in the ID and ADHD groups were similar. Therefore, it is impossible to classify ID and ADHD based on counts, and high accuracy cannot be achieved. These results suggest that AI methodologies are needed to classify ID and ADHD with high accuracy.

Table 3 shows that the accuracy and ROC_AUC of the NB model were higher than those of the XGB, LGBM, and RF models. Since the number of ID data was more significant than that of the ADHD data, PPV, sensitivity, and F1 scores increased when the ID was set to positive. When considering accuracy, ROC_AUC, PPV, sensitivity, and F1 score, it can be concluded that the NB model is better than the XGB, LGBM, and RF models. The NB model is more accurate than others because it tends to show high accuracy in sparse matrix data, like those in the document-term matrix we created.41

Duplicated and irrelevant feature results are shown in Table 4. The NB and RF models had many duplicate features, whereas the XGB and LGBM models did not have duplicate features. This is because duplicated features can be extracted using NB and RF, as NB assigns high weights to features with specific n-gram features42 and RF samples data points or features by restoration extraction.43

The four models and two feature extraction methods have different calculation formulas, resulting in slightly different important feature extractions, as demonstrated in Supplementary Tables 1, 2, 3, 4. Nevertheless, the four models have commonalities as words related to ID or ADHD, such as ‘intellectual,’ ‘ability,’ ‘cognitive,’ ‘perceptual,’ etc., frequently appear. The findings also reveal that the four models rarely chose words associated with the physical appearance of patients, such as ‘clean,’ ‘unsanitary,’ ‘chubby,’ etc. Instead, all four models generated inventories of subtests, subtest outcomes, and behavioral descriptions during the examination.

This study has several limitations. It is necessary to verify the model’s robustness by obtaining data from other institutions and conducting external validation. With an increased data volume from additional institutions, a multi-class classification model can be developed by further subdividing labels according to severity. In the pre-processing stage, a more diverse application of NLP methods, including stop words, is required. Additionally, a comparison of the performance of each morpheme analyzer should be conducted. Finally, we employed conventional machine learning models as classification tools to utilize techniques for reconstructing human-readable sentences based on SHAP and PI results. In future work, we plan to apply advanced models, such as large language models, to compare their performance with conventional machine learning models.

This study offers several key contributions. Firstly, we developed four explainable predictive models that accurately classify IDs and ADHDs using solely physician reports, eliminating numerical scores and focusing on textual information. Despite the limited quantity of reports, the models demonstrated high accuracy, ROC_AUC, PPV, sensitivity, and F1 scores, underscoring their efficacy even with a small dataset. This was achieved through meticulous pre-processing and applying NLP techniques to convert unstructured texts into structured data. Additionally, incorporating the physician’s diagnosis into each report further refined the models’ PPV. To enhance the explainability of the models, we employed SHAP and PI methods to identify relevant n-gram features, providing insights into the models’ decision-making processes. To mitigate any loss of readability due to pre-processing, we developed a NOTS system, which clarifies the extracted n-gram features, thereby improving human understanding. Consequently, these models not only facilitate accurate ID and ADHD classification but also aid physicians in comprehending the classification process, offering evidence-based insights.

SUPPLEMENTARY MATERIALS

Supplementary Table 1

Top 30 SHAP-based important features extracted from each model

Supplementary Table 2

Top 30 PI-based important features extracted from each model

Supplementary Table 3

Selected 10 SHAP-based human-readable texts extracted from each model

Supplementary Table 4

Selected 10 PI-based human-readable texts extracted from each model