Table 4. Study stages.

Features of the stages	Stage 1	Stage 2	Stage 3
Object of assessment	CXR	Labeling of nodules on CXR by AI-based software	Labeling and classification of nodules on CXR by AI-based software
Who or what performed the assessment	AI-based software	3 radiologists	3 radiologists
Assessment process	AI-based software assessed the images and returned a Kafka message with a probability of unspecified pathology	Radiologists checked for nodules in the exact places where the AI-based software put the markups	Radiologists checked for nodules in the exact places where the AI-based software put the markups, and checked whether the AI-based software classified the marked-up opacity as a nodule and not otherwise
Assessment result of one CXR image	Probability of pathology provided by AI-based software	Binary estimate of a region with a lung nodule markup made by AI-based software	Binary estimate of a lung nodule markup and labeling made by AI-based software
Assessment result presentation scale	Probability of pathology presented as an integer number (from 0 to 100)	Categorical values:	Categorical values:
		• For 50 CXR with nodules: FN—nodule is absent (the actual nodule on CXR is not marked-up); TP—nodule is present (the actual nodule on CXR is marked-up)	• For 50 CXR with nodules: FN—nodule is absent (the actual nodule on CXR is not marked-up or marked-up but not labeled as a nodule); TP—nodule is present (the actual nodule on CXR is marked-up and labeled as a nodule)
		• For 50 CXR without pathological changes: TN—nodule is absent (no markup on CXR); FP—nodule is present (the markup is present on CXR)	• For 50 CXR without pathological changes: TN—nodule is absent (no markup and label on CXR); FP—nodule is present (markup and label of nodule are present on CXR)
Approach to choosing a cut-off value	Modified. The cut-off value corresponded to the maximum value of the Youden index	Unchanged. The cut-off value was pre-set by the developer, since the AI-based software marked-up only the nodules whose probability exceeded the pre-set cut-off value	Unchanged. The cut-off value was pre-set by the developer, since the AI-based software marked-up and labeled only the nodules whose probability exceeded the pre-set cut-off value
Ground truth	Each study was initially assigned one of the two classes: 1—a nodule on CXR with a diameter of 9–30 mm, confirmed on CT; 0—a CXR image without pathology, with the corresponding CT containing no findings

CXR, chest X-ray; AI, artificial intelligence; FN, false negative; TP, true positive; TN, true negative; FP, false positive; CT, computed tomography.