Table 4.

Examples of false negatives and false positive from final query

False negatives (6 total)		False positives (21 total)
Report text	Explanation	Report text	Explanation
There is a 3 cm benign adrenal myolipoma.	Spelling error: Algorithm searches for “myelolipoma”, and this term was misspelled within the original report text	Metastatic pheochromocytoma of the right adrenal post adrenalectomy.	Error due to missing “History”: heading for the history section of the report. The report was labeled negative on manual review as no positive findings were reported in the body of the report
No adrenal masses are seen probable tiny right adrenal myelolipoma.	Missing punctuation leading to a sentence-based negation error (i.e., negated finding “no adrenal masses” is in the same sentence as the positive terms of “adrenal myelolipoma”)	The adrenal glands appear thickened but discrete mass.	Radiologist dictation error: There is a missing negation term “no” before “discrete mass”. Manual review interpreted this as negative
Up to five small bilateral pulmonary nodules are unchanged without evidence of new lung nodule including: 6 mm left lower lobe nodule image 17 3 mm left upper lobe nodule image 20 5 mm right lower lobe nodule image 26 3 mm left lower lobe nodule image 29 4 mm right lower lobe nodule image 34 Limited cuts into the upper abdomen demonstrate a stable 15 mm fat containing left adrenal nodule suspected adenoma which is partially imaged.	Missing punctuation leading to a sentence-based negation error. Sentence contains a positive finding (in bold), but this was negated by the negation algorithm due to italicized phrase, which contains “without” and “nodule”. A period after “image 34” would have allowed the algorithm to correctly identify this report as having a positive adrenal finding	The high attenuation masslike abnormality in the left adrenal probably represents an adrenal hematoma which is resolving.	Adrenal hematoma was considered as a negative finding for the purpose of this paper. However, this sentence was marked positive secondary to presence of “masslike” in association with “adrenal”
There is nodularity of the right adrenal gland and some thickening in the left. IMPRESSION: 2. Bilateral nodularity in the adrenal glands likely adenomatous/hyperplastic changes.	Positive finding detection error. Algorithm searches for “adenomatous changes” but was unable to identify “adenomatous/hyperplastic changes” due to insertion of “/hyperplastic”	The spleen pancreas both adrenal glands and both kidneys are unremarkable aside from a 7 mm low-density lesion in the lower pole of the left kidney which is too small to characterize but probably benign.	Complex sentence error. Multiple organs were described within a same sentence, and the algorithm failed to associate the positive term “lesion” with the appropriate organ (left kidney). Also, the term “unremarkable” was not detected by the algorithm for negation. A total of 18 cases had similar error. Some examples are shown below
Nodular thickening of the right adrenal gland measuring 2.2 × 1.7 cm on image 98 stable in size. The CT attenuation characteristics of this lesion are indeterminate (with an average density of 25 HU). The prior MRI tissue characteristics suggested an adenoma as the most likely diagnosis.	Positive finding detection error. Because the terms “adrenal,” “lesion,” and “adenoma” were in separate sentences, this was missed by the sentence-based detection algorithm	The liver spleen gallbladder biliary tree pancreas adrenal glands and kidneys are unremarkable with the exception of the low attenuating lesion in the interpolar aspect of the right kidney IV representing a small renal cyst.	Complex sentence error
There is continued nodular thickening of the left adrenal gland with a maximum diameter of 8 mm unchanged since prior study. The appearance is benign and may represent small adenoma or nodular hyperplasia.	Positive finding detection error. Similar to above where “adrenal” and “adenoma” were in separate sentences	The adrenal glands and kidneys are unremarkable except for bilateral low attenuation renal lesions which are subcentimeter in size likely small cysts.	Complex sentence error
		FINDINGS: The liver spleen pancreas adrenal glands and kidneys are normal except for a subcentimeter cystic lesion in the right kidney which is too small to characterize.	Complex sentence error
		There is a 10 × 17 mm low attenuation soft tissue mass adjacent to the lateral limb of the left adrenal gland that probably represents a lymph node that is increased in size from the prior exam.	Complex sentence error. The spatial relationship was not accounted for with our algorithm to determine that the “mass” is “adjacent” and not part of the adrenal gland