A Method for Efficient De-identification of DICOM Metadata and Burned-in Pixel Text

Abstract

De-identification of DICOM images is an essential component of medical image research. While many established methods exist for the safe removal of protected health information (PHI) in DICOM metadata, approaches for the removal of PHI “burned-in” to image pixel data are typically manual, and automated high-throughput approaches are not well validated. Emerging optical character recognition (OCR) models can potentially detect and remove PHI-bearing text from medical images but are very time-consuming to run on the high volume of images found in typical research studies. We present a data processing method that performs metadata de-identification for all images combined with a targeted approach to only apply OCR to images with a high likelihood of burned-in text. The method was validated on a dataset of 415,182 images across ten modalities representative of the de-identification requests submitted at our institution over a 20-year span. Of the 12,578 images in this dataset with burned-in text of any kind, only 10 passed undetected with the method. OCR was only required for 6050 images (1.5% of the dataset).

Background

The Digital Imaging and Communications in Medicine (DICOM) standard is widely adopted for acquiring medical images in clinical settings [1, 2]. DICOM images contain not only visual data, but also substantial non-image data, including protected health information (PHI), used to identify the patient and the examination and its associated parameters. Therefore, for research use, DICOM data should be de-identified whenever possible [3]. Given the sensitive nature of medical data, the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule [4] outlines a safe harbor method of de-identification, in which 18 identifiers must be removed from data. Upon completion of this process, the data are no longer considered human subject data and can more easily be shared or moved to other platforms, thereby significantly enhancing the data’s suitability for research applications.

For medical images in the DICOM format, many of these identifiers are stored in pre-defined attributes in the DICOM header, such as “(0010,0010) Patient’s Name” or “(0010,0030) Patient’s Birth Date” [2]. The well-defined structure of these DICOM attributes should allow for automated software to remove corresponding PHI with relative reliability, and a variety of free software packages are available with this aim. Aryanto et al. [5] validated some of these tools on a set of 50 DICOM elements with the potential to contain identifying patient information. They found the RSNA Clinical Trial Processor (CTP) [6] and DICOM Library [7] showed good performance out-of-the box, while other tools required customized settings (if supported) to reach acceptable performance. Still, outside of straightforward DICOM attributes, challenges can still arise in metadata de-identification, due to the broad interpretation of the 18th HIPAA identifier (“Any other unique identifying number, characteristic, or code”), DICOM attributes that allow free text that could unexpectedly contain PHI, and private vendor tags whose contents are not rigidly defined.

De-identification efforts focused solely on the DICOM header may fall short of meeting safe harbor requirements, as some DICOM images also contain PHI “burned-in” to the pixel data from either the original image acquisition or postprocessing in image analysis software [8]. Unlike the attributes in the DICOM header, there are no standards for placing such information in an image, which may vary between vendors, scanner models, and individual technologists or radiologists. As such, it is not a trivial matter to perform automated detection of burned-in PHI. The “(0028,0301) Burned In Annotation” DICOM attribute is available with such applications in mind, but it is an optional field in the DICOM standard and thus cannot be relied upon to identify all burned-in PHI. Optical character recognition (OCR) models can provide an alternative solution [9], but early attempts at such an approach have yielded mixed results. Monteiro et al. developed a pipeline to detect all burned-in text in ultrasound images and only remove PHI, but almost 11% of images were not anonymized, limiting utility [10]. In a small sample of 128 ultrasound images, Zhu et al. demonstrated a model that detected all text with recall = 0.99, but required approximately 1.5 s of computation time per image [11]. Given that large clinical studies and modern data science projects often include hundreds of thousands of images that require de-identification, such an approach would take days to weeks to complete OCR. Thus, there is a need for a rapid, reliable method for filtering out DICOM images with burned-in pixel data that may contain PHI to allow for more targeted pixel de-identification. Here we present our initial experiences with a method for full image de-identification, including a targeted OCR module for images with high probability of burned-in PHI as determined from attributes in the DICOM header.

Methods

The system reported in this manuscript is the result of a quality improvement project, rather than formal research, and did not require ethical approval. The method for de-identification consisted of three main components: (1) metadata de-identification, (2) secondary capture filtration, and (3) OCR pixel de-identification. For metadata de-identification, we used a custom Python program utilizing the Pydicom library (v2.2.2) [12]. All DICOM attributes were removed from the image metadata unless the attribute was contained in a white-list dictionary of metadata tags that had been previously vetted to never hold a PHI risk. DICOM attributes that supported free-text entry were always excluded, even if the intended use was not to hold PHI, due to the possibility of unintended abuse. For attributes with high PHI risk that were required by the DICOM standard to be present in the metadata, such as “(0008,0050) Accession Number” or “(0010,0030) Patient Birth Date,” a blank string was assigned to the attribute. All exam or series-specific UIDs were remapped to new, randomized values (but kept consistent within the exam or series in question). Uniform reference values were set for “(0008,0020) Study Date” and “(0008,0030) Study Time.” All other attributes related to dates and times during the study were shifted relative to these reference values, allowing any information about relative timing during the exam (such as series timing or bolus timing) to be preserved. Private vendor tags were subject to the same rules for inclusion and modification as standard tags, with the additional requirement that the manufacturer tag of the DICOM image in question match the expected vendor for any private tags that were included in the white list.

In the second stage of this method, images labeled as secondary captures, deemed more likely to contain burned-in pixel PHI, were filtered out. Images were defined as secondary captures if the attribute “(0008,0016) SOP Class UID” held a value of 1.2.840.10008.5.1.4.1.1.7. An alternative approach was required for one vendor due to improper labeling of the SOP Class UID. For the CT vendor CurveBeam, secondary captures were identified as images where the attribute “(0008,0008) Image Type” did not include “PRIMARY” as one of the values.

The final stage of the method involved applying a custom-trained OCR model to images identified as high-risk for burned-in pixel PHI. Our method was designed to apply such a model to all primary ultrasound images, as they commonly contain burned-in PHI. A custom OCR model was trained using the keras-ocr library (commit: e8d34a4) [13]. This model, based off the CRAFT detector [14], was selected because it had the best performance “out-of-the-box” on a small validation set of medical images. In addition, it featured robust, built-in tools to generate custom training images. With this functionality, a new training dataset was generated where randomized text was placed on ultrasound image backgrounds. Transfer learning was performed by retraining the model with this new training dataset using the default model weights as the starting point. To preserve potentially useful burned-in text, any text detected by the OCR model was checked to see if it matched an approved text pattern (example: a number immediately followed by a recognized unit of measure) or was listed in a dictionary of approved words (examples: kidney, liver, lateral, inferior). If the text matched neither, it was blacked out to create the final de-identified image. A comprehensive visualization of the entire method is presented in Fig. 1.

Fig. 1.

De-identification method. Red blocks represent PHI, while green blocks indicate de-identified data. The red and green bars at the top of each “image” are a symbolic representation of image metadata

To validate the de-identification method, it was applied to a dataset representative of de-identification requests at our institution. A faculty radiologist with 13 years of post-fellowship experience reviewed: (1) all images processed by the method to assess if any PHI was missed and (2) all images that were removed by the method to assess how many images with PHI were successfully removed and how many images without PHI were unnecessarily removed. True positive, true negative, false positive, and false negative counts were tabulated for PHI detection.

To create the validation dataset representative of medical images requiring de-identification, we systematically sampled a database of all previous DICOM de-identification requests that were handled by an internal group at our institution. This database contained 37,866 imaging exams between 2001 and 2021. Ten imaging modalities and fourteen imaging device manufacturers were represented in the dataset. The included imaging modalities were computed radiography (CR), computed tomography (CT), digital radiography (DX), mammography (MG), magnetic resonance (MR), nuclear medicine (NM), positron emission tomography (PT), radio fluoroscopy (RF), ultrasound (US), and X-ray angiography (XA). For our sample, we included any combination of modality and manufacturer that had at least 10 unique exams. If a particular combination had more than 50 exams, we selected 50 exams evenly spaced across the entire date range for that manufacturer and modality to make the sample more representative of different scanner models and software versions. If a combination had less than 50 exams, all exams were included regardless of date. The initial resulting dataset included 1391 unique exams. While no specific anatomical regions were targeted, the dataset included exams for cranial, abdominal, breast, pelvic, and extremity imaging, among others. Table 1 provides a breakdown of the number of exams for each manufacturer and modality in the resulting dataset.

Table 1.

Number of exams for each combination of imaging device manufacturer and imaging modality in the representative sample of de-identification requests at our institution

	CR	CT	DX	MG	MR	NM	PT	RF	US	XA	Total
Carestream Health	50		50								100
Canon Inc	50										50
CurveBeam		36									36
EOS Imaging			50								50
FUJIFILM Corporation	39										39
GE Medical Systems		50	50	50	50	50	50	50	50		400
HOLOGIC, Inc.				40							40
KODAK	50										50
Mobius Imaging, LLC		50									50
NeuroLogica		19									19
Philips Medical Systems	50	23	50		17	50		50	50	50	340
SIEMENS		50			50				47		147
Samsung Electronics	50										50
TOSHIBA		20									20
Total	289	248	200	90	117	100	50	100	147	50	1391

CR computed radiography, CT computed tomography, DX digital radiography, MG mammography, MR magnetic resonance, NM nuclear medicine, PT positron emission tomography, RF radio fluoroscopy, US ultrasound, XA X-ray angiography

Results

The final representative dataset for de-identification included 1045 CR images (1% with burned-in text), 218,798 CT images (3%), 438 DX images (1%), 1229 MG images (6%), 98,044 MR images (0.3%), 724 NM images (41%), 84,540 PT images (6%), 1658 RF images (7%), 6271 US images (100%), and 1889 XA images (0.7%). Metadata de-identification was successfully performed on all images. All PHI was removed from the DICOM attributes, while preserving the required attributes to remain compliant with the DICOM standard. This ensured the de-identified images remained compatible with DICOM viewers and relevant software. Secondary capture filtration identified and removed 68,111 (16%) images as potentially text-bearing. These included 76 (7%) CR images, 61,408 (28%) CT images, 66 (15%) DX images, 275 (22%) MG images, 423 (0.4%) MR images, 295 (41%) NM images, 5224 (6%) PT images, 110 (7%) RF images, 221 (4%) US images, and 13 (1%) XA images. The remaining 6050 US images (96%) were processed using the custom OCR model, and the model ran successfully for all of those images. Figure 2 presents a representative example of an US image post-OCR.

Fig. 2.

Representative ultrasound image following application of an optical character recognition (OCR) model. Special logic was applied to preserve non-PHI text containing information relevant to the scan

Table 2 provides summary statistics for detection of burned-in text in the images for each modality for both images that passed through the entire method and those that were filtered out as secondary captures.

Table 2.

Summary of de-identification method performance for each modality

Modality	Images assessed	Images with burned-in text	Images filtered by workflow	Images preserved with text (FN)	Images preserved without text (TN)	Images filtered with text (TP)	Images filtered without text (FP)	False positive rate [%]	False negative rate [%]
CR	1045	12	76	0	969	12	64	6.2	0
CT	218,798	6352	61,408	0	157,390	6352	55,056	25.9	0
DXinitial	438	6	66	3	360	3	63	14.9	50
DXadd	546	7	136	7	403	0	136	25.2	100
MG	1229	72	275	0	954	72	203	17.5	0
MR	98,044	282	423	0	97,621	282	141	0.1	0
NM	724	294	295	0	429	294	1	0.2	0
PT	84,540	5209	5224	0	79,316	5209	15	0	0
RF	1658	110	110	0	1548	110	0	0	0
USa	6271	221	221	0	6050	221	0	0	0
XA	1899	13	13	0	1886	13	0	0	0
All	415,182	12,578	68,247	10	346,926	12,568	55,679	13.8	0.08

FN false negative, TN true negative, TP true positive, FP false positive, CR computed radiography, CT computed tomography, DX_initial initial selection of digital radiography exams, DX_add additional digital radiography exams inspected when burned-in text was found in the DX_initial, MG mammography, MR magnetic resonance, NM nuclear medicine, PT positron emission tomography, RF radio fluoroscopy, US ultrasound, XA X-ray angiography

^aUS statistics describe text not automatically blacked out by the applied OCR model. “Filtered” images were removed from the examinations as potentially text-bearing secondary captures

For all modalities except DX, no images with burned-in text passed through the de-identification method. In the DX images, only 3 images from a single platform managed to pass through the method with burned-in text, specifically describing the date and time of the scanner use. Given this, further validation was performed on 50 additional exams for each of the four DX manufacturers (an additional 546 images). It was found that 7/546 images (this time spread across all manufacturers) had a burned-in date and time, bringing the total to 10/984 for the entire DX dataset. Notably, all of these images were from portable examinations. These ten cases represented only 0.08% of images with text in the dataset. CT, DX, and MG had high false positive rates of secondary captures with burned-in text (15–26%), driven by the large number of secondary captures inherent to these modalities, such as multiplanar reconstructions and volume renderings.

Discussion

In this work, we describe the implementation and validation of a method for full DICOM de-identification, including PHI in the DICOM header and text burned-in to the image pixels. The method was validated on a set of 415,182 images spanning ten modalities drawn from de-identification requests submitted at our institution over a 20-year span. Of the 12,578 images with burned-in text, only 10 passed through the method undetected. These 10 cases were exclusive to portable DX units, and the text corresponded to the scan date and time of imaging. Given the performance of portable radiographic examinations in areas of the hospital in which patients may need to receive multiple examinations in a day, it is likely this burned-in text was included by the technologist operating the unit rather than automatic processing included by the manufacturer. To mitigate this risk, an additional OCR module could be added for DX images from portable units. In our process, for this small subset of images corresponding to that combination of modality and model, a manual review is conducted as an additional layer of security.

The custom metadata de-identification we implemented was comparable to the top performing tools tested by Aryanto et al. [5] By their validation metrics, our tool had a 98% success rate. The only tag from their validation set that was not de-identified was “(0010,0040) Patient Sex.” This was an intentional inclusion in our tool, as this parameter is not one of the HIPAA Identifiers and is often of significant value in research applications.

Identification of secondary captures was chosen as the primary mechanism to filter out images with a high likelihood of burned-in text since primary images for most imaging modalities do not contain any burned-in text (with US being the notable exception). Such text is typically added by postprocessing software which will relabel the newly modified images as secondary captures. The attribute “(0008,0016) SOP Class UID” was used to detect secondary captures instead of the attribute “(0008,0008) Image Type” since the former was more likely than the latter to be correctly assigned by postprocessing software. Filtering out images based only on metadata attributes rather than more time-consuming pixel-based approaches carried the risk of removing images with no text present in the pixel data. This was observed for CR, CT, DX, and MG. We deemed this an acceptable trade-off for the low false negative rate with this method because a conservative approach for identifying PHI-free images is essential to protect patient data. In addition, many research applications for very large data sets are primarily interested in the original imaging data from primary images (for example, radiomics and deep learning projects), and any necessary postprocessing can be repeated with the de-identified primary images. Alternatively, as referenced in Fig. 1, if it becomes desirable for certain types of secondary captures to be retained, the method can be modified to properly handle those images based on meta-data fields and/or additional OCR modules.

In the de-identification method presented here, a custom-trained OCR model was applied to all primary US images by default, as primary US images commonly contain PHI such as medical record numbers and patient names. Though most PHI is found in the upper banner of US images, in our experience, sonographers sometimes add clinical information as free text elsewhere. Our workflow was effective at removing all PHI on US images, while maintaining some burned-in text that contained useful scan information. The process we employed for choosing which burned-in text to keep in an image was more conservative than the approach employed by Monteiro et al. [10], resulting in all PHI being de-identified but a loss of some non-PHI text on images. We deemed this tradeoff to be acceptable as PHI going unredacted is significantly worse than some peripheral scan information being removed. Furthermore, as the volume of de-identification work performed continues to grow, we hope to iterate on the pattern matching and white list dictionary of approved words, reducing the amount of non-PHI removed with this approach. The time to perform OCR on an image was highly dependent on image resolution and number of frames, but a conservative estimate of the typical processing time for a single image with our model was approximately 1–2 s, comparable to that reported by Zhu et al. [11]. While this would not be an issue for small datasets, OCR times may be prohibitively long for larger datasets with hundreds of thousands of images. By using secondary capture filtration to remove images with burned-in text for other modalities, the method only had to run on 1.5% of all images in the dataset, significantly saving on processing time. OCR processing was performed on a single NVIDIA GeForce RTX 3090 GPU, and processing times could be significantly improved with distributed processing across multiple GPUs.

While the method described here used a custom metadata de-identification tool and OCR model with custom-trained weights, the general framework is modular and the main components can be replaced with a variety of readily available “off-the-shelf” options. For metadata de-identification, such options include the RSNA Clinical Trial Processor (CTP), DICOM Library, and others [5]. However, we note that even with these software tools, the desired configuration for metadata field handling may depend on the research application and institutional preferences regarding privacy requirements. For OCR, there is a wide range of open-source Python libraries including keras-ocr [13], EasyOCR [15], and pytesseract [16].

There are a few notable limitations to this de-identification method. By default, all secondary captures are removed. If the removed secondary captures hold scientific value, and an appropriate OCR model has not been trained for that modality, they must be recreated from the de-identified primary images, which may be time-consuming. In addition, secondary capture removal to account for burned-in PHI has only been validated for the modalities, manufacturers, and date ranges presented in this manuscript. This validation set is intended to continually grow and evolve over time as more de-identification requests are handled with this approach, but it may result in limited applicability in the short term. Institutions may also have different standard operating procedures for their technologists that may result in different levels of occurrence for burned-in text in primary images than reported here. It is advised that any who wish to adopt this approach perform validation at least on their institution’s own historical data.

Overall, we believe the described de-identification method represents a balance of efficiency and efficacy: metadata de-identification is performed for all images, while slower pixel-based models for burned-in text are only employed for a targeted subset of images, and some images are discarded outright based on the assumption of non-trivial risk and low value for the research application. We hope sharing this method can enable the deployment of faster and safer de-identification routines in the research community.

Conclusions

In this work, we present our experience with a method for full DICOM image de-identification, including handling images with burned-in pixel PHI. Alongside metadata de-identification, we utilized an approach of filtering out secondary captures and applying a custom OCR model to primary images with high likelihood of containing PHI. When tested on a dataset representative of de-identification requests at our institution, all metadata were successfully de-identified, and all but ten images with burned-in text were successfully handled. The rate-limiting factor in our method was the OCR model, and this only had to run on 1.5% of images in the dataset, suggesting a process that is both efficient and effective. The method is compatible with many out-of-the-box metadata de-identification and OCR modules, allowing ease of adoption in other settings.

Author Contribution

MM, CE, JYL, MS, DCS, and MRB contributed to study conception and design. JAM, KRM, BK, and KA contributed to the de-id method design and implementation. Validation was performed by JAM, KRM, and MRB. The first draft of the manuscript was written by JAM and reviewed by all other authors. All authors read and approved the final manuscript.

Funding

Research reported in this publication was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002553. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Data Availability

The data used for validation in this manuscript was approved for use as an internal quality improvement project and is not approved for distribution outside of Duke University.

Declarations

Ethics Approval

The system reported in this manuscript is the result of a quality improvement project, rather than formal research, and did not require ethical approval.

Consent to Participate

The system reported in this manuscript is the result of a quality improvement project and did not require written informed consent.

Consent to Publish

The system reported in this manuscript is the result of a quality improvement project and did not require written informed consent. Any included images were fully de-identified.

Competing Interests

DCS has received research support from the National Institutes of Health. MRB has received research support from Siemens Healthineers, Bayer Healthcare, Madrigal Pharmaceuticals, NGM Biopharmaceuticals, Carmot Therapeutics, and Corcept Therapeautics. All other authors have no financial interests to disclose.