Use of Natural Language Processing to Identify Sexual and Reproductive Health Information in Clinical Text

Abstract

Objectives

This study aimed to enable clinical researchers without expertise in natural language processing (NLP) to extract and analyze information about sexual and reproductive health (SRH), or other sensitive health topics, from large sets of clinical notes.

Methods

(1) We retrieved text from the electronic health record as individual notes. (2) We segmented notes into sentences using one of scispaCy’s NLP toolkits. (3) We exported sentences to the labeling application Watchful and annotated subsets of these as relevant or irrelevant to various SRH categories by applying a combination of regular expressions and manual annotation. (4) The labeled sentences served as training data to create machine learning models for classifying text; specifically, we used spaCy’s default text classification ensemble, comprising a bag-of-words model and a neural network with attention. (5) We applied each model to unlabeled sentences to identify additional references to SRH with novel relevant vocabulary. We used this information and repeated steps 3 to 5 iteratively until the models identified no new relevant sentences for each topic. Finally, we aggregated the labeled data for analysis.

Results

This methodology was applied to 3,663 Child Neurology notes for 971 female patients. Our search focused on six SRH categories. We validated the approach using two subject matter experts, who independently labeled a sample of 400 sentences. Cohen’s kappa values were calculated for each category between the reviewers (menstruation: 1, sexual activity: 0.9499, contraception: 0.9887, folic acid: 1, teratogens: 0.8864, pregnancy: 0.9499). After removing the sentences on which reviewers did not agree, we compared the reviewers’ labels to those produced via our methodology, again using Cohen’s kappa (menstruation: 1, sexual activity: 1, contraception: 0.9885, folic acid: 1, teratogens: 0.9841, pregnancy: 0.9871).

Conclusion

Our methodology is reproducible, enables analysis of large amounts of text, and has produced results that are highly comparable to subject matter expert manual review.

Introduction

While research regarding the unique sexual and reproductive health (SRH) needs of adolescents and young adults is unquestionably important, barriers to conducting this research range from the ethical to the practical.^1–3 One practical barrier to large-scale retrospective analysis of relevant SRH care provision is that—similar to mental health data—information regarding SRH is frequently only recorded as natural language text rather than as structured data, which are highly organized and typically stored in a spreadsheet format.⁴ Using traditional approaches such as manual chart review to analyze these unstructured data can be time-intensive, especially if the data sample is large. Thus, a methodology using natural language processing (NLP) and machine learning to extract and transform these data into a form suitable for conventional analysis could help facilitate large-scale studies of SRH and other sensitive health care topics.

Whereas SRH is routinely discussed during Obstetrical/Gynecological and Adolescent Health visits, SRH counseling and information gathering is not as common during other kinds of subspecialty or primary care visits. For example, multiple recent qualitative studies have shown that both providers and patients report the frequent omission of SRH counseling during routine Child Neurology visits.^5–7 Consequently, the presence of SRH-related text in subspecialty notes could be considered a “rare event,” making an efficient extraction mechanism particularly important for this information.

In applying NLP to clinical text, it is important to acknowledge that biomedical data comprise a set of specific lexicons, in which the connotations of certain words and phrases are different from those same words and phrases when used outside of a clinical context. While this vocabulary can be frustrating for nonclinicians to understand, it is helpful from a research standpoint, because it can be leveraged to help identify specific health care entities, including procedures, medications, and diagnoses.⁸ There are now multiple models for processing biomedical text, which have been trained on bodies of text such as the CRAFT corpus—a set of 67 manually annotated biomedical journal articles.⁹

In a recent systematic review of clinical studies using NLP, the “annotation bottleneck”—or the need to manually label data—was identified as a key obstacle to effective data analysis. This problem leads to use of smaller datasets and slows the pace of research.¹⁰ Even when using biomedical-specific packages for processing text, the preparation of training data frequently involves some amount of manual annotation. Fortunately, new tools are now available that help streamline this process, some of which are highlighted in this paper.

Objectives

The purpose of this methodology is to enable clinical researchers without expertise in NLP to extract and analyze information about SRH, or other sensitive health topics, from large sets of clinical notes.

Methods

Our proposed method for SRH text identification comprises seven steps: (1) text retrieval, (2) text segmentation and cleaning, (3) transfer of clean text to a data labeling application, (4) labeling of text, (5) training of a machine learning model for each SRH topic, (6) application of the machine learning models to discover additional mentions of each SRH topic in the remaining unlabeled text, and (7) repeated iteration of steps 3 to 6, as illustrated in ►Fig. 1. This methodology is an example of active learning, a well-established data science paradigm in which machine learning models help to identify which data points need manual annotation.¹¹ The tools used throughout our methodology are detailed in ►Table 1.

Fig. 1.

An illustration of our methodology in seven steps, including iteration, as well as the relevant data science tools used for each step.

Table 1.

Tools used throughout our methodology

Tool	Description	Reference
Python	An object-oriented, high-level programming language	https://www.python.org 12
Jupyter Notebook	An interactive computing environment for data analysis and software development	https://jupyter.org 13
Pandas	A data-analysis framework for Python	https://pandas.pydata.org 14
Pandas DataFrame	A two-dimensional table with rows and columns of data	https://pandas.pydata.org/docs/ref-erence/api/pandas.DataFrame.html
spaCy	A natural language processing toolkit for Python	https://spacy.io/ 15
scispaCy	A Python package that provides spaCy with specialized models for processing biomedical, scientific, and clinical text	https://allenai.github.io/scispacy/ 16
Watchful	A specialized application for labeling training data for machine learning development	https://www.watchful.io/ 17
Scikit-learn	A library of machine learning algorithms for Python	https://scikit-learn.org/stable/ 18

The University of Pittsburgh Institutional Review Board deemed this research exempt from review.

Text Retrieval

Our “raw data” took the form ofa set of whole clinical notes in a plain-text format, which was loaded into Python as a Pandas DataFrame with one note per row. The specific method used to retrieve this set of notes from the electronic health record system is outside the scope of this paper.

Text Segmentation and Cleaning

In preprocessing the clinical note data, we first segmented the text of each note into sentences with spaCy and scispaCy. Specifically, we used one of scispaCy’s models—“en_core_sci_lg”—to segment the full text of each note into its constituent sentences. We then collected the sentences in a new Data-Frame, with a new row for each individual sentence. Once the text had been segmented into individual sentences, it was easier to remove the unwanted components.

Clinical notes tend to share unique structural characteristics that are relevant to the text cleaning task. As detailed in Bate’s Guide to Physical Examination and History Taking, most comprehensive clinical notes comprise four major sections: subjective information, objective information, an assessment, and a plan.¹⁹ The most relevant sections to an investigation of SRH topics will typically include the history of present illness and the social history (which exist within the subjective information section). Many other components, including headers, will be irrelevant. Consequently, we began text cleaning by removing all sentences that ended in certain types of punctuation, such as a colon, which resulted in the removal of all section headers. We also removed sentence fragments that included fewer than two words.

Another unique aspect of clinical text is verbatim repetition of certain phrases. This is due to providers using dot phrases and templates, or saved text that can be entered quickly into electronic documentation.²⁰ To account for this, we removed all duplicated text that appeared within the same encounter. To ensure the removal of all duplicated phrases, we first stripped the text of punctuation and extra spaces.

Transfer of Text to a Labeling Application

Once the text had been segmented into sentences and cleaned, we exported the DataFrame from Python in comma-separated value format. We then imported this new file into Watchful, which helped to facilitate and accelerate the process of exploring and labeling the data prior to creating the machine learning models.¹⁷

Text Labeling

To search for text relevant to SRH, we identified multiple specific subtopics, including menstruation, sexual activity, contraception, teratogens, folic acid use, and pregnancy counseling. Within Watchful, we then created regular expressions to search for sentences containing words or phrases potentially related to each of these topics. A regular expression is a sequence of characters that describes a set of strings and is used to search for very specific patterns of text.²¹ For example, the regular expression ^man will match any text that begins with the characters m a n (such as “manly”). Likewise, the regular expression man$ will match any text that ends with the characters m a n (such as “human”).

For each regular expression, the application produced a set of matching sentences. Consequently, we quickly reviewed and manually annotated sentences as relevant or irrelevant to the topic in question (referred to as “positive” or “negative” examples, respectively). The application then analyzed these manual annotations and produced suggestions for new regular expressions to identify additional relevant sentences. These automated suggestions were occasionally useful and accelerated the labeling process. We also used regular expressions to exclude sentences containing certain words or phrases we identified as unlikely to be relevant to the given topic. Including all six SRH categories, we created over 100 regular expressions to assist with annotation.

Once a reasonable sample of sentences had been labeled with the application (~1% of all sentences in the dataset with the majority of labels being “positive”), we exported the text from Watchful as a new comma separated value file and reimported the file into Python as a Pandas DataFrame.

As an example, the following terms were initially identified as references to menstruation: menstruation, menstrual, menses, menstruating, period, LMP, menorrhagia, metrorrhagia, and menometrorrhagia. In Watchful, each of these terms was formatted as a regular expression in the following manner: /\bword\b/i. This syntax indicates that the text should be case-insensitive and that the word is bounded (when the text is part of a longer word, it will not be identified in the search). The majority of searches using these regular expression terms revealed sentences that were relevant to SRH care. However, the use of “period” identified a mix of relevant and irrelevant sentences due to the use of “period” to denote time in addition to menstruation. We addressed this issue by creating “negative” regular expressions such as /\bperiod of time\b/i, /\bneonatal period \b/i, and /\bweeks period\b/i. Once enough of these were created, all irrelevant instances could be easily labeled as “negative” examples. Examples of some “positive” and “negative” regular expressions used, as well as the concomitant sentences identified, are illustrated in ►Table 2.

Table 2.

Examples of regular expressions and matched sentences

Regular expression	Type of sentence identified	Number of sentences identified in dataset	Matched sentence example
/\bher periods\b/i	Positive	124	“Her periods are irregular, although she is on the birth control”
/\bmenarche\b/i	Positive	147	“Since she reached menarche, feels that they are seeing normal seizures”
/menorrhagia/i	Positive	180	“She is also having significant menorrhagia and intermittent epistaxis that has yet to be evaluated by Hematology” “She has metromenorrhagia”
/\bmenses\b/i	Positive	534	“Her parents did not want to take any chances that these seizures could have been triggered by her menses”
/menstrual/i	Positive	698	“They were interested in trying oral contraceptive medications as her mother believed that her seizures were increased during her menstrual cycles” “She does have some perimenstrual headaches, but now she has had more frequent headaches, and she has been off of magnesium until recently”
/\bmenstruation\b/i	Positive	32	“Since they feel her seizures are worse around menstruation, birth control pills such as progesterone pills have been recommended and to discuss this with Adolescent Medicine”
/\bneonatal period\b/i	Negative	72	“No complications during pregnancy or neonatal period”
/\bperiod\b/i	Positive	1,088	“She also notices that those episodes start to happen after she started having her period and right now she is on Depo shot and mother noticed some improvement of her symptoms, so that could be probably due to dysmenorrhea”
/\bperiod of time\b/i	Negative	182	“The patient was treated with Trileptal, unclear for what period of time, and then lost for follow-up”
/\byear period\b/i	Negative	20	“Should she have an abnormal video EEG, we would then continue Trileptal for an additional 1- to 2-year period, and at that time, to start supplementation with folic acid 1 mg daily and vitamin D 400 units with calcium 600 mg daily”

Training Machine Learning Models

To prepare the DataFrame of labeled sentences as training data for each machine learning model, we added new columns for each identified SRH topic with binary labels (0 or 1), indicating each sentence as relevant (1) or irrelevant (0) to the topic. We then filled these binary values by applying functions to the DataFrame that interpreted the manual annotations made in the Watchful application as either a “1” for a relevant/“positive” sentence or “0” for an irrelevant/“negative” sentence.

Then, we separated the labeled data for each particular topic from the unlabeled data. This labeled data served as a training set to create a machine learning model for each SRH topic. Once trained, we used these machine learning models to generate predictions indicating whether unlabeled sentences referred to the given SRH topic.

We created each of our machine learning models using tools provided by spaCy. We started with an empty spaCy text processing pipeline and added one of spaCy’s components—a machine learning architecture for classifying text called “textcat”—to the pipeline. Specifically, we used the default text classification component provided by the spaCy library, spacy.TextCatEnsemble.v2. This machine learning architecture is an ensemble consisting of a traditional, linear bag-of-words model and a neural network model. The neural network model consists of an embedding layer encoding information about a variety of lexical attributes of each token in an input sentence, a convolutional layer enriching the token embeddings with additional information from the surrounding context, an attention layer enabling the model to “pay attention” to the most important tokens, and a softmax output layer. The output layer provides a prediction score indicating the likelihood that the input sentence represents the SRH topic of interest. The training process uses the cross-entropy loss function and the Adam optimizer. We used all of the default hyperparameter configurations provided by the spaCy library. Detailed information about the machine learning architecture and optimization process underlying the spacy.TextCatEnsemble.v2 pipeline component is available in the spaCy documentation.

We prepared the training text and labels in spaCy’s preferred format, and trained the models using small batches of training data. We used a minimum of approximately 100 to a maximum of approximately 1,500 labeled sentences in the initial training sets, with the size of each training dataset being dependent on the relative rarity of each topic being identified throughout the texts. We trained each model from scratch for 10 epochs over all the labeled data for each SRH topic.

Using the Models to Help Classify Unlabeled Sentences

Next, we applied the trained machine learning models to each sentence in the unlabeled sentence data to predict the likelihood that sentence would be a positive example (i.e., would be relevant to the given SRH topic).

Each time, we sorted these predictions in descending order, such that the sentences a model predicted as most likely to represent positive examples were listed at the top.

For the first run of each model, the training data did not include a large, representative sample of “negative” examples (i.e., irrelevant sentences). Consequently, the models consistently produced poor initial predictions (based on their rankings of sentences); the vast majority of the sentences each ranked as having a high likelihood of being “positive” were truly “negative” examples. However, we could easily identify the true “positive” sentences and manual annotation of the first several hundred predictions produced by each model took only 5 to 10 minutes. To be clear, we did not use thresholds or cutoffs in the machine learning models for class assignments. Instead, we manually assigned each of the sentences we had reviewed with a label of “0” or “1” and combined them with the initial training data for the given SRH topic (adding hundreds of “negative” examples, which helped to balance the training data for future runs). We then retrained each model using the new, larger training set. After this process had been repeated a few times, the machine learning models began to identify novel relevant sentences in the remaining unlabeled data (and after ranking the machine learning output, these were typically found in the first hundred sentences).

Iterative Approach

As each model identified new relevant examples in the unlabeled data, we assigned these sentences appropriate labels and added them to the training set. Perhaps more importantly, manually reviewing these sentences occasionally revealed relevant words and phrases that had been overlooked or omitted when creating regular expressions in Watchful. After several runs with each machine learning model, we drafted a list of new relevant vocabulary. We used this to identify new sentences via additional regular expressions in Watchful.

Again, as an example, our machine learning model for menstruation identified the following as most likely to be relevant: (1) “She has recently started her menarche and mom has noticed changes in her blood pressure related to that,” (2) “Since [NAME] has not yet gone through menarche, we will go ahead and continue maintaining the medication,” (3) “I suspect her migraine headaches are perimenstrual,” (4) “[NAME] follows with Adolescent Medicine due to catamenial seizures and has switched from the oral contraceptive pill to Depo-Provera,” (5) “[NAME] has a past history of reported sexual and physical abuse at her group home,” and (6) “[NAME] will continue to be followed by her physician for her birth control pills.” Although all of these sentences are relevant to SRH, we considered only the first four as representative of discussions about menstruation for the purposes of our project. The new terms the model identified included “perimenstrual,” “menarche,” and “catamenial.” On returning to Watchful, we added “menarche” and “catamenial” as new search terms. Additionally, given that the term “peri-menstrual” contained a previously used term with the prefix “peri,” we adjusted the regular expression syntax of “menstrual” to /menstrual/i so that our search would include all words containing this term (ex. perimenstrual, premenstrual, postmenstrual), as demonstrated in ►Table 2.

We repeated steps 3 to 6 iteratively until the machine learning models identified no new relevant sentences. During each run, we trained a new machine learning model for each SRH topic from scratch.

Data Recombination

Finally, we recombined all labeled sentences to reconstitute the original full text of whole notes and then further aggregated as necessary for analysis.

Results

We successfully used this methodology to evaluate a set of 3,663 notes from clinical encounters with 971 adolescent and young adult people with epilepsy of childbearing potential for references to SRH across six categories (menstruation, sexual activity, contraception, folic acid, pregnancy, and teratogens).²² This dataset was chosen given the researchers’ subject matter expertise in SRH within Child Neurology, as well as the availability of the data through the researchers’ home institution.

By cleaning the segmented text of unwanted or duplicate components, we were able to decrease the total number of sentences for analysis by over 20%, from 493,818 to 388,019. After exporting labeled sentences from Watchful, our initial training dataset comprised 4,438 sentences. The distribution of “positive” and “negative” labeled sentences for each SRH category are outlined in ►Table 3. We considered any sentence that mentioned a particular SRH topic as rele-vant/“positive”—even when the sentence specified sexual abstinence, premenstrual status, lack of contraceptive use, no supplemental folic acid, and/or teratogenic medication avoidance—as this knowledge is equally important with regards to clinical decision-making.

Table 3.

Distribution of labeled sentences for each sexual and reproductive health category in the initial training data sets

SRH category	Number of “positive” sentences	Number of “negative” sentences
Menstruation	1,230	420
Contraception	659	95
Folic acid	1,079	72
Pregnancy	102	517
Teratogens	84	87
Sexual activity	60	33

Abbreviation: SRH, sexual and reproductive health.

With the first run of each machine learning model, we identified up to 15 additional “positive” sentences and approximately 500 “negative” sentences for each SRH category. Subsequent runs identified fewer and fewer truly “positive” examples, though the machine learning models would produce predictions that were more and more relevant to SRH overall. When we added the terms identified with our models to Watchful, however, we identified many more relevant sentences, particularly for menstruation and contraception. The distribution of labels for each SRH category in the final fully labeled dataset is detailed in ►Table 4. Of note, sexual activity was the only category for which we identified all relevant sentences using regular expressions and manual annotation in Watchful, without the assistance of the machine learning component of our methodology.

Table 4.

Distribution of labeled sentences for each sexual and reproductive health category in the final labeled data set

SRH category	Number of “positive” sentences	Number of “negative” sentences
Menstruation	1,547	386,475
Contraception	1,176	386,843
Folic acid	1,112	386,907
Pregnancy	130	387,889
Teratogens	95	387,924
Sexual activity	60	387,959

Abbreviation: SRH, sexual and reproductive health.

Comparison to Manual Annotation

To objectively evaluate the potential benefits of our methodology over the traditional approach of manual annotation, we completed and timed a manual annotation of 36 clinical notes in the dataset. This random sample of notes comprised 3,382 sentences. Of these, 21 sentences mentioned menstruation, 18 sentences mentioned contraception, 8 sentences mentioned folic acid, 2 sentences mentioned sexual activity, 1 sentence mentioned pregnancy, and 1 sentence mentioned teratogens. These equate to the following frequencies of “positive” sentences in the cleaned text for each category: menstruation 0.62% (95% confidence interval [CI] 0.36%, 0.89%), contraception 0.53% (95% CI 0.29%, 0.78%), folic acid 0.24% (95% CI 0.07%, 0.40%), sexual activity 0.06% (95% CI 0.00%, 0.14%), pregnancy 0.03% (0.00%, 0.09%), and teratogens 0.03% (95% CI 0.00%, 0.09%). In our final fully labeled set of 388,019 sentences, the frequency of “positive” sentences for each category consistently fell within the 95% CIs noted above: menstruation 0.40%, contraception 0.30%, folic acid 0.29%, sexual activity 0.02%, pregnancy 0.03%, and teratogens 0.02%.

A single subject matter expert reading the 36 clinical notes in a Word document and manually highlighting the sentences relevant to SRH care took exactly 1 hour 10 minutes 2 seconds. Given that this set of 36 notes represents approximately 1% of the total data, one could extrapolate that manual annotation of all 3,660 Child Neurology notes (comprising a total of 388,019 sentences after cleaning), would take at least 116 hours (not taking into account reader fatigue, need for breaks, or time spent entering label results into a program for analysis). Once our methodology had been developed, we produced the final set of labels using our methodology in two full work days (or ~16 h), equating to 100 hours time saved when compared with manual annotation.

Validation

We used external subject matter experts to validate our results, employing Cohen’s kappa to compare the labels produced using our methodology to reviewer sentence classification on a sample of sentences. We chose to use Cohen’s kappa because our goal was to evaluate the validity of our final labels (as a proxy for the entire iterative methodology) rather than the performance of the machine learning models (which were just one component of our process and were primarily used to identify “positive” sentences that had been missed with the combined use of regular expressions and manual annotation). Using random sampling with specific constraints to ensure a balanced set of “positive” and “negative” examples for each category, we selected out 400 sentences for our validation set. This set included between 37 and 53 “positive” examples for each SRH category, with 74 sentences that were relevant to more than one category and 233 sentences that were not relevant to any of the categories. We chose to use 400 sentences for our sample, with an average of approximately 40 “positive” sentences (10% of the sample) for each category, to ensure variety in the “positive” examples while minimizing redundancy (given that many of the “positive” examples for each category used similar verbiage).

Two reviewers independently and blindly labeled each sentence for all relevant categories. We calculated Cohen’s kappa between reviewers for each category (human vs. human). Cohen’s kappa values between these reviewers were 1 (menstruation), 0.9499 (sexual activity), 0.9887 (contraception), 1 (folic acid), 0.9499 (pregnancy), and 0.8864 (teratogens), averaging 0.9625 across all six SRH categories. Then, for each category, we removed all sentences on which the reviewers did not agree (n = 13). We did this to ensure that our methodology’s performance would be evaluated only on cases for which human experts could reach a consensus (i.e., nonambiguous cases). Finally, we compared the labels in this subset of human-labeled sentences to the labels produced using our methodology and reported a separate Cohen’s kappa for each category (humans vs. methodology). Cohen’s kappa values between the methodology and reviewers were 1 (menstruation), 1 (sexual activity), 0.9885 (contraception), 1 (folic acid), 0.9871 (pregnancy), and 0.9841 (teratogens), averaging 0.9933 across all six SRH categories.

Discussion

In this paper, our team offers a reproducible approach to identifying clinical text relevant to SRH care.

SRH information is frequently key to clinical decisionmaking, including for pediatric subspecialty practices that are primarily unrelated to SRH. For example, it is important to know whether a patient with epilepsy is sexually active to initiate contraception, adjust teratogenic medications, initiate folic acid, or provide counseling regarding changes in seizure frequency during pregnancy. To improve outcomes related to SRH in subspecialty practices, more research in this area is needed. However, in recent surveys, pediatric subspecialty providers have reported concerns for patient, parental, or personal discomfort when discussing SRH, indicating that they perceive this information as particularly sensitive.⁶ It is, therefore, more likely for this information to be documented solely in the block text of a clinic note (and not coded elsewhere in the electronic health record). As unstructured textual data, this information has traditionally required manual review for extraction and analysis. However, NLP techniques can help accelerate this process.⁴

We believe our methodology to be superior to manual labeling or regular expressions alone due to its strength in accurately and exhaustively evaluating a large dataset for a rare event (in this case, sentences relevant to SRH counseling) in a relatively short period of time. While regular expressions alone might have identified a large proportion of relevant sentences, the use of machine learning models allowed us to identify additional relevant sentences that were semantically similar but did not include the specific phrases we knew to search for. As an example, an additional 517 sentences regarding contraception (43% of the final “positive” labels in this category) were identified after use of our machine learning models when compared with the training sets, which were formed primarily using regular expressions. That said, our proposed methodology includes multiple steps that could have been approached in a variety of ways, each with arguable benefits.

While we chose to use scispaCy and its “en_core_sci_lg” pipeline, which has a large vocabulary and 600,000 word vectors, it would have been equally valid to prepare the data by creating a pipeline based on concepts in the Unified Medical Language System, as demonstrated by Yetisgen et al.²³ Future studies might consider combining these methods, and this is an area rich for future methodologic exploration. We would also recommend that researchers considering the use our of methodology for future studies try medscpaCy in place of scispaCy, as this toolkit (which was released after the completion of this project) is specifically designed for clinical NLP.

Alternatives to our approach for cleaning the text include the use of tools or methods to sectionize each note prior to further segmentation. Recent biomedical informatics research has focused on identifying defined sections within narrative content extracted from electronic health records (ex. the physical exam portion of a note). A systematic review of these efforts notes that 59% of studies examined used rulebased methods, 22% used machine learning methods, and 19% used a combination of both.²⁴ Each appeared to have strengths and weaknesses with regards to granularity, scope, and accuracy of results. While segmenting notes into sections could help filter out irrelevant content more efficiently, this could result in accidental deletion of relevant content.

Our methodology makes extensive use of regular expressions to accelerate the process of finding and manually labeling sentences that mention relevant SRH topics. There are a variety of possible alternative (or complementary) approaches to accelerate this step. For example, pretrained transformer language models or other neural network methods could be used to generate vector embeddings for sentences. These models typically assign semantically similar sentences’ vectors that are close to each other in the embedding space. If given a small set of SRH sentences that had been hand-labeled by a subject matter expert, these sentence embedding models could potentially be used to find additional SRH sentences by looking up nearby sentence vectors in the embedding space. Chen et al demonstrate one neural network method for generating such sentence embeddings in biomedical and clinical text.²⁵ Again, this is an area full of potential for future research.

Another aspect of our methodology that could be approached differently is the use of Watchful for data labeling. Watchful expedited the creation of our training sets in part by providing a user-friendly interface for visualizing and labeling textual data, as well as by providing suggestions for additional regular expressions during the labeling process. It would be possible to create an initial training set without a labeling application by using regular expressions in Python, and there are numerous alternative applications that provide a similar service to Watchful. A future direction for research would be to quantify performance in speed and efficiency with using one of these labeling applications compared with regular expressions alone.

For our machine learning text classification models, we found success using spaCy’s default (spacy.TextCatEnsemble.v2) and did not test alternative models for superiority. However, this would be an interesting opportunity for additional research.

Finally, in validating our results, we removed all sentences for which the reviewers did not agree prior to comparing the human-labeled sentences to the sentences labeled using our methodology. Although we could have adjudicated disagreements with an additional reviewer or via majority agreement, many of these cases were felt to be ambiguous by all reviewers and we wanted to evaluate our methodology on only clear-cut examples.

Limitations of this methodology include the persistence of an “annotation bottleneck,” which is only partially addressed with the use of a labeling application. The need for manual review during several steps of our approach could preclude its application to much larger datasets, or may simply necessitate adaptation. Further, this methodology is best suited for relatively simple annotation tasks; complicated annotation tasks that involve identification of very specific scenarios would take a longer time and might not be optimized using our approach. Finally, a likely limitation is our method’s generalizability beyond clinical notes. The use of our methodology to identify narrative content related to SRH that has been extracted from other rich sources, including online forums and social media data, would require substantial modification.

Conclusion

Despite its limitations, our approach enables the analysis of large quantities of unstructured, natural language data and— with iteration—is effective even when the text of interest is rare. Our validation results demonstrate that this method’s identification of SRH topics is essentially equal to the manual identification provided by independent subject matter experts. Further, our methodology is flexible enough to be adapted for a variety of sensitive health topics, in both pediatric and adult medical realms. Possible future applications include topics such as suicidal ideation, substance use, child abuse, elder abuse, interpartner violence, sexual orientation, gender identity, and sexually transmitted infections.