Raman Spectroscopy in Open-World Learning Settings Using the Objectosphere Approach

Abstract

Raman spectroscopy, combined with machine learning techniques, holds great promise for many applications as a rapid, sensitive, and label-free identification method. Such approaches perform well when classifying spectra of chemical species that were encountered during the training phase. That is, species that are known to the neural network. However, in real-world settings, such as in clinical applications, there will always be substances whose spectra have not yet been taken. When the neural network encounters these new species during the testing phase, the number of false positives becomes uncontrollable, limiting the usefulness of these techniques, especially in public safety applications. To overcome these barriers, we implemented the recently introduced Entropic Open Set and Objectosphere loss functions. To demonstrate the efficacy and efficiency of this approach, we compiled a database of hyperspectral Raman images of 40 chemical species separating them into three class categorizations. The known class consisted of 20 biologically relevant species comprising amino acids, the ignored class was 10 “irrelevant” species comprising bio-related chemicals, and the never seen before class was 10 various chemical species that the neural network had not seen before. We show that this approach not only enables the network to effectively separate the unknown species while preserving high accuracy on the known ones and reducing false positives but also performs better than the current gold standards in machine learning techniques. This opens the door to using Raman spectroscopy, combined with our novel machine learning algorithm, in a variety of practical applications. Availability and implementation: freely available on the web at https://github.com/BalytskyiJaroslaw/RamanOpenSet.git.

Graphical Abstract

INTRODUCTION

Raman spectroscopy was independently developed in 1928 by Raman¹ and Landsberg,² and further supplemented by the usage of laser equipped spectrometers.^3,4 This method is based on light scattering and its interaction with the chemical bonds of the material of interest and provides fingerprint-like spectrum of a given sample. The properties of Raman spectroscopy, combined with its sensitive and non-destructive nature, make it a reliable and universal tool for material analysis suited for both the laboratory as well as field experiments.⁵ There are a number of applications of Raman spectroscopy in different fields, see⁶ for a comprehensive review.

In brief, Raman spectroscopy enabled a detailed characterization of carbonaceous materials^7,8 and has a number of applications in bioanalytics⁹ and in the diagnostics of the cultural heritage.¹⁰ Raman analysis already has large-scale industrial applications including in the food¹¹ and textile industries.¹² Raman spectroscopy also serves as a powerful tool for planet exploration. In particular, the Perseverance rover has two Raman devices installed, SHERLOC (Scanning Environments with Raman and Luminescence for Organics and Chemicals), and SuperCam.¹³

Raman spectroscopy is an especially promising candidate for applications where public safety is a concern such as testing water quality,^14,15 identifying agricultural contaminants,^16,17 and in clinical settings for bacterial pathogen identification.¹⁸ Growing concern regarding contamination from chemicals such as per- and polyfluoroalkyl substances (PFAS)¹⁹ and lead²⁰ in food and water as well as the rise of antibiotic-resistant infections²¹ in our society requires efficient interventions in order to help mitigate public health crises. In these applications, there is a large amount of data or chemical species that need to be identified, which can be a challenge in the case of unknowns or samples with multiple chemicals present.

Raman spectroscopy has significant promise to meet this challenge as it is a sensitive and non-destructive probe providing a unique, fingerprint-like spectrum of a sample,^22,23 but the development of robust and reliable quantitative methods of the data analysis is needed.²⁴ Not only this, but there are often large amounts of data which are acquired, making it difficult and impractical to hard-code all of the rules of classification and analysis. Therefore, some form of machine learning (ML) is needed to ensure this process is practical and efficient.

As a note, we adopt the definition of ML from²⁵ as “the effort to automate intellectual tasks normally performed by humans”. Rather than being hard-coded, the ML system finds appropriate representations of the data which allows determining the rules of classification. “Deep learning (DL) is a specific subfield of machine learning: a new take on learning representations from data that puts an emphasis on learning successive layers of increasingly meaningful representations”.²⁵ For our purposes, these layered representations are learned by the models called neural networks (NNs).

In the definitions of,²⁵ in contrast to the DL, shallow learning techniques only learn one or two consecutive representations of the data. One model of such kind, the principal component analysis (PCA), in combination with linear discriminant analysis (LDA) was applied to Raman spectra analysis.^26–30 Older DL methods have been shown to successfully distinguish between the spectra of different molecules^31–36 and do so more effectively than the typical linear regression methods for data analysis of Raman spectra.³⁷ However, attempts to boost the accuracy by naively stacking extra layers and making the models deeper are limited by the vanishing gradient problem.³⁸ The ResNet architecture overcomes this problem and boosts the accuracy even further by introducing skip connections between the layers.³⁹ Current state of the art deep learning algorithms for Raman spectroscopy favor the ResNet architecture because it can maintain high performance at a lower complexity than its competitors. Using these advances, Raman spectroscopy combined with ML has shown significant promise for real-world applications as a rapid, label-free identification method.¹⁸

However, a significant limitation of the aforementioned NNs is that they operate in “closed-world” settings, where they are only tested on the species on which they were initially trained. As a result, if the inputs for the NN include new, never seen before species, the NN’s behavior becomes ill-defined and will lead to errors and misclassifications. Simply put, if the NN is trained to identify either “cats” or “dogs” but sees a new sample of “fish” during testing, the false-positive rate becomes uncontrollable and the NN will misclassify “fish” as either “cat” or “dog”.

When we consider real-world conditions and practical applications, it is impossible to collect data on all possible chemical or pathogenic species, so there will always be unknowns. To avoid false-positives and misclassifications, the ideal NN should be able to not only accurately identify the samples of the known classes, but should also be able to handle any unknowns without misclassification. That is, it must have the ability to operate in an “open-world” setting and be able to manage new, never-before-seen species. Overcoming the limitation of the “closed-world” setting is the key to the adaptation of NN architectures into routine practices in different applications.

In this article, we implement ideas from Entropic Open-Set and Objectosphere approaches⁴⁰ and combine them with the ResNet26 architecture to show, for the first time, that the ResNet architecture can be modified to accurately and reliably reject samples it has not seen before while remaining highly accurate in its identification of known classes. This new, combined architecture significantly outperforms the naive approaches such as the thresholding softmax score^41–43 and background class methods.⁴¹ The general idea of implementing these approaches is to teach the NN to ignore the features of the irrelevant classes and only focus on the features of the relevant classes (i.e., identify classes that it knows while saying “I don’t know” when presented with unknowns).

To demonstrate the efficiency of our proof-of-concept approach, we compiled a database of spectra of 40 chemical species and separated them into distinct categories. Twenty of them are amino acids, the building blocks of proteins, and we test the classification accuracy of the NN on them. The remaining 20 species were divided into two groups of 10 each, and with their help, the rate of false positives for species not previously encountered in the “open-world” was tested.

The spectra of the 20 amino acids were categorized as a “known” (K) class as they are the class of interest, 10 species comprising biologically relevant chemicals were categorized as an “ignored” (I) class as they are known, but not of interest, and the last 10 species are other various chemicals that are classified as a “never seen before” (N) class. The latter classes are unique in that the NN was not previously trained on them and sees them for the first time during the testing stage. During the training stage, the NN is taught to be highly accurate on the K class and separate them from the I class. During the test stage, we test its accuracy on K as well as determine the rate number of false positives upon the introduction of N.

METHODS

The 20 amino acids were purchased from Carolina Biological Supply Company (Burlington, NC). All other chemicals were purchased from either Alfa Aesar or Sigma-Aldrich. All samples were measured in their crystalline form on glass-bottomed petri dishes. Raman maps were collected on a WITec alpha300 microscope in the confocal Raman mode, schematically shown in Figure 1. The excitation source was a 75 mW, 532 nm green laser, focused with a Zeiss EC-Epiplan 20 × 0.4 NA objective. A WITec UHTS 300 spectrometer with a 600 groove/mm grating was used to record the spectra. The detector was an Andor DV401-BV spectroscopic CCD. This spectrometer has a resolution of 3 cm⁻¹ or 0.009 nm. Raman maps were collected over an area of 50 μm by 50 μm. This area was divided into 100 × 100 pixels, each containing a Raman spectrum collected with an integration time of 0.1 s.

Figure 1.

Schematic drawing of our experimental setup. The incident 532 nm green laser interacts with the sample on a glass slide and the scattered light (red) is collected by the detector.

DATA PREPARATION AND MACHINE LEARNING

Of the collected spectra, 5/6 were used for training (2500 spectra per chemical species) and 1/6 were used for testing (500 spectra per chemical species). Training and testing data were collected separately from one another to ensure appropriate variation in the spectra. Before feeding the data into the NN, the spectra were processed such that the intensity was normalized between 0 and 1.0 and the Rayleigh peak from the laser was removed, as shown in a representative image in Figure 2. Once the spectra were recorded, they were categorized as described above into either “known”, “ignored”, or “never seen before”, further detailed in Table 1 below.

Figure 2.

Representative spectra for (a) raw and (b) normalized lysine data.

Table 1.

Table of Chemical Names for the Known (K), Ignored (I), and Never Seen Before (N) Class Identifications

class	chemical Name
K	dl-alanine, dl-aspartic acid, dl-isoleucine, dl-leucine, dl-methionine, dl-phenylalanine, dl-serine, dl-threonine, dl-tyrosine, dl-valine, glycine, l-arginine, l-asparagine, l-cystine, l-glutamic acid, l-histidine, l-lysine, l(+)-cysteine, l-proline, l-tryptophan
I	anthrone, beta estradiol, chloroquine, fluconazole, laminarin, lauric acid, MOPS, methyl viologen, progesterone, uridine
N	ampicillin, CHAPS, D(+) maltose monohydrate, forskolin, 1-Decanesulfonic acid, polyvinyl pyrrolidone, potato starch, sodium deoxycholate, sodium dodecylsulfate, silver nitrate

In the “open-world” settings, the N class is not seen during the training stage and instead appears for the first time during the testing stage. However, to ensure that our NN performed well in the “closed-world” setting, before going to the “open world”, we first exposed it to all classes, K, I, and N. The conventional ResNet26 architecture shown in Figure 3 was run five times to help boost the accuracy of the network. While a few misclassifications may occur during each separate run, we can average the predictions (e.g., majority voting), and boost the accuracy to 99.99%, as shown in Figure 4. The accuracy outputs for each of the five runs as well as the majority vote accuracy for the “closed-world” ResNet26 settings are as follows

$$\begin{gathered} \{\begin{array}{l}\text{accuracy\hspace{0.17em}}\left[1\right]=98.72\%\hfill \\ \text{accuracy\hspace{0.17em}}\left[2\right]=99.28\%\hfill \\ \text{accuracy\hspace{0.17em}}\left[3\right]=99.06\%\hfill \\ \text{accuracy\hspace{0.17em}}\left[4\right]=99.98\%\hfill \\ \text{accuracy\hspace{0.17em}}\left[5\right]=97.49\%\hfill \end{array} \\ \Rightarrow \underset{\text{prediction\hspace{0.17em}ensemble}=1/5\sum _{i=1}^5\text{prediction}\left[i\right]}{\underbrace{\text{final\hspace{0.17em}accuracy}=99.99\%}} \end{gathered}$$

Figure 3.

Schematic representation of the ResNet26 architecture. The processed spectra are fed into the NN in order to output the classification probabilities.

Figure 4.

Correlation table after the majority vote of five runs is taken. Accuracy = 99.99%.

Next, before delving into variations on the ResNet architecture, we tested the accuracy of a more conventional method of spectra identification, PCA-LDA as used in.^26–30 PCA-LDA is a shallow method (as opposed to the deep methods we’ll use later), meaning that it does not examine deep and abstract features of the data. Moreover, this method only works in “closed-world” settings. On our data set, the accuracy, on average, of PCA-LDA was

$${\text{accuracy}}_{\text{PCA}-\text{LDA}}=84.88\pm 3.30\%$$

When comparing the accuracy of this “closed-world” method to the accuracy of our “closed-world” ResNet architecture, we note that ResNet performs much better. It should also be noted that PCA-LDA will only work in “closed-world” settings and cannot be adapted to the “open-world”. Therefore, in our further considerations, we focus solely on the ResNet architecture.

As previously discussed, there will always be the possibility of encountering unknowns in real-world settings. Though the NN performs well and with high accuracy in “closed-world” settings, it is not entirely realistic and limits its applications in clinical settings. Here, we explore various “open-world” approaches to demonstrate how well the NN performs when presented with new, never before seen chemical species.

We focused on K, chose to ignore I, and presented N only during the testing phase, effectively forcing the NN to expect the unexpected while maintaining high accuracy on the known classes. In the following sections, we investigate four different “open-world” approaches: (1) background class, (2) thresholding of the softmax score,^41–43 (3) Entropic Open Set, and (4) Objectosphere,⁴⁰ and demonstrate the advantages and limitations of each approach. We show that while all these approaches are highly accurate on the known class, with an average accuracy = 99.97 ± 0.02%, the Objectosphere approach is much more efficient in its handling of the N class and is thus better prepared for any unexpected inputs the NN may encounter.

RESULTS

Background Class Approaches and Thresholding the Softmax Score.

The NN has an initial input, x, which, in our case, corresponds to the values of the intensities found in the normalized spectra. The output corresponding to each chemical species is defined as, l_c(x). The deep feature of the NN, F(x), which is the output of the second to last layer of the NN, consists of the outputs of the convolution’s blocks, and the final output is obtained by multiplication by the weights W to the last layer

$$l_c(x)=W\cdot F(x)$$ (1)

The final probability of which chemical species, c, that a particular input will belong to is calculated by the “softmaxing” procedure

$$S_c(x)=\frac{e^{l_c(x)}}{{\sum }_c{e}^{l_c(x)}}$$ (2)

which lies in the range S_c(x)∈[0,1], and ∑_cS_c(x) = 1 and therefore is interpreted as a probability. In the case where the input sample belongs to the K class, it should be classified as whichever chemical species has the highest softmax score in eq 2. Cases in which the input corresponds to the I and N are described further in the text.

The output of the NN is high-dimensional, between 20 and 40 dimensions in our case, so to provide an illustration of the separability of the known and unknown classes, we use the Uniform Manifold Approximation and Projection (UMAP) dimensionality reduction technique.⁴⁴ This allows for the high-dimensional output of the NN to be visualized in a two-dimensional way, showing a clear overlap or separation between the classes. In all our runs, we used the default parameters of the algorithm (i.e., n_neighbors = 15, min_dist = 0.1). In this approach, n_neighbors limits the number of local neighbors that the algorithm considers while trying to determine the manifold structure of the data, and the min_dist parameter is a minimum distance between points that is allowed in the low-dimensional representation of the data. In a 2D plane, the different species correspond to different colors and if the NN is highly accurate, we should see distinct groupings of colors with no overlap between species (i.e., if there is overlap, there are mistakes and misclassifications). As an example, we show the UMAP output of the 40 species in the “closed-world” setting in Figure 5. Here, each of the 40 species corresponds to a color in the gradient shown in the legend to the right of the UMAP and the separation between the colors as well as the fact that each color is concentrated in a particular area of a 2D space rather than being scattered over all the plane is a good indication that the NN is classifying correctly. As we shift to the various “open-world” settings, we can use the UMAP technique to quickly visualize the efficiency of the NN. For these tests, the K class corresponds to the red to blue gradient and new, never seen before class corresponds to the violet dots on the UMAP diagrams.

Figure 5.

UMAP of 40 species in the closed-world setting.

When unknown, never seen before species are present, one naive approach is to include a trash or background class which corresponds to the background class method. Here, rather than having the network try to classify data that are far from the known class or it has not seen before, they will be categorized as “background” or “trash” class. For this approach, we implemented the following condition. The amino acid number i∈[0,C-1], and the {I,N} classes are encoded as

$$\left(\text{amino\hspace{0.17em}acid}\right)[i]=\underset{\text{length}=C+1}{\underbrace{[0,\dots ,\underset{i\text{th\hspace{0.17em}position}}{\underbrace{1}},\dots ,0]}}$$ (3)

$$\{I,N\}\to \underset{\text{length}=C+1}{\underbrace{[0,\dots ,\underset{\text{20th position}}{\underbrace{1}}]}}$$ (4)

where C = 20—number of chemical species of interest (amino acids).

In this case, the output of the NN is the probability of the chemical species being identified as either a “known” or “ignored”/“never seen before”. Therefore, if the output is in the 20th position (as shown in eq 4), then it will be classified as “trash” or “background” otherwise, it will be classified as “known”. Although this scheme may seem effective, putting 10 separate chemicals in the category of “background” results in the NN needing to identify a large number of features as “background” which interferes with its efficiency.

Another naive approach we investigated was to threshold the softmax score. The softmax score, obtained by eq 2, converts a vector of numbers into a vector of probabilities. By implementing a threshold, the NN will essentially give a percentage corresponding to how certain it is that the data belong to a certain chemical species. However, this technique assumes that the probabilities of the K and {I,N} classes are sufficiently separated in the feature space.

To put this another way, it is assumed that Shannon’s entropy⁴⁵ of the NN’s output on the {I,N} classes is close to the maximal value log₂ (C)

$$\{I,N\}\to \underset{\text{length}=C}{\underbrace{\left[\frac{1}{C},\dots ,\frac{1}{C}\right]}}$$ (5)

while the entropy of the output of K class is close to zero. Assuming that this condition is fulfilled, one can introduce a cutoff (Λ), and if the maximum value of the softmax score is lower than the cutoff value, the input is classified as the new, never seen before class, N.

Though both the background class and thresholding of the softmax score methods achieve high accuracy on the K alone, they work poorly upon the introduction of the N class and the accuracy drops significantly. In both cases, overlap and mixing occurs, signifying mistakes and misclassifications by the NN, as seen in Figure 6. Furthermore, as illustrated in Figure 7, even in a case of introducing a high cutoff value of Λ = 0.99 (corresponding to the NN being confident by 99%), the false positive (FP) rate is 10.0%. In fact, regarding the K class, in 7.0% of the cases, the NN produces an inconclusive output, that is, classifies K as part of the unknown, N class.

Figure 6.

UMAP diagrams of (a) the background class and (b) the thresholding the softmax score methods. Class N (violet) is scattered over a large part of the diagram, mixing the known and the unknown.

Figure 7.

Even at the large cutoff value Λ = 0.99, the FP = 10.0% and the number of inconclusive outcomes on K class is 7.0%.

Though the absolute values of the deep features ‖F‖ of the known class tends to be slightly higher than those of ignored and never before seen classes, as shown in Figure 8, the amount of overlap between the three classes in the UMAP diagram indicates that the NN has difficulty distinguishing between K and N. To overcome these limitations, we investigated the Entropic Open Set and Objectosphere approaches and demonstrate their efficiency in the next section. These methods implement tuning parameters and numerical approaches aimed at increasing the separation between the probability distributions of the K and N classes.

Figure 8.

Absolute value of the deep feature ‖F‖ of K class tends to be higher than those of N and I.

Entropic Open Set and Objectosphere Methods.

Unlike the naive approaches which assume that the NN already produces sufficiently high entropy on the I and N classes, Entropic Open Set and Objectosphere approaches implement loss functions as a means to increase entropy on the I (i.e., ignoring it) and concentrate on the K. That is, by numerically changing the coefficients in the loss functions, the user can achieve greater separation between the probabilities rather than assuming the classes are already sufficiently separated (such as the case of thresholding the softmax score method). The Entropic Open Set loss function⁴⁰ is defined as

$$V_{\text{E}}(x)=\{\begin{array}{l}-\text{log}\left(S_c(x)\right),\text{if }x\in K\hfill \\ -\frac{1}{C}\sum _{c=1}^C\text{log}\left(S_c(x)\right),\text{if }x\in I\hfill \end{array}$$ (6)

where C = 20 is the number of chemical species of interest (amino acids). Note, for the K class, it reduces to a regular categorical cross entropy loss function. The N class is not involved here because the NN is not aware of it until the testing stage. For the samples belonging to I class, V_E(x) aims to maximize the entropy and uniformly spread the NN’s output over the known class K.

The Objectosphere loss function, on the other hand, involves the addition of the deep feature F(x) parameter. This allows for minimization of its absolute value for the ignored class and maximization of its absolute value for the known one, resulting in sufficient separation of the K and {I,N} classes

$$V_{\text{O}}(x)=V_{\text{E}}(x)+\alpha \{\begin{array}{l}\text{max}\left(\beta -{\Vert F(x)\Vert }^2,0\right),\text{ if }x\in K\hfill \\ {\Vert F(x)\Vert }^2,\text{ if }x\in I\hfill \end{array}$$ (7)

where the ‖·‖ represents a regular Euclidean norm. The values of the α and β coefficients are adjusted numerically by minimizing the number of inconclusive outcomes on the K class. The result of this is that FP = 0% on the I class, and this property is generalized for the N class, even though the NN is not aware of it before the testing phase.

Both the Entropic Open Set and Objectosphere approaches provide a significant improvement over the naive approaches as shown in Figure 9. As shown in Figure 9a, for the Entropic Open Set loss function with the cutoff value Λ_entropic = 0.93 and FP = 0%, the NN has 0% wrong outcomes on the K class. Not only this, but the number of inconclusive outcomes on the known class is 2.37%. Compared to the naive and thresholding softmax approach, this approach reduced the inconclusive outcomes by about 66.1%. For the Objectosphere loss function, shown in Figure 9b, with the cutoff value Λ_{Objectosphere} = 0.91 and FP = 0%, we observe 0%, wrong outcomes on the K class. The number of inconclusive outcomes on the known classes is also drastically reduced to 0.26%, as shown in Figure 9. This shows that the Objectosphere reduces inconclusive outcomes by 89% from the Entropic Open Set loss and 96% from the naive approaches. This is a significant improvement and demonstrates that the Objectosphere approach successfully avoids misclassifications and false positives.

Figure 9.

Results of the (a) Entropic Open Set and (b) Objectosphere loss functions. The Objectosphere improves the treatment of the N class even more by reducing the number of inconclusive outcomes on the K class to 0.26%.

The corresponding UMAPs, shown in Figure 10, visually demonstrate the significant improvement in the treatment of the N class by the NN. The area taken by the N class is significantly reduced in comparison to the naive approaches and one can make a clear distinction between N and K classes. Finally, as demonstrated in Figure 11, the Objectosphere loss function further enhances the separation between the features of the N and K classes, minimizing the incorrect responses from the NN to the new, never seen before inputs.

Figure 10.

UMAPs for the (a) Entropic Open Set and (b) Objectosphere loss functions. For the Objectosphere case, the area over which the N classes are scattered shrinks significantly and a clear separation between K and N is observed.

Figure 11.

Distribution of the deep features of the known (K) and unknown/unexpected inputs (N) by the (a) Entropic Open Set and (b) Objectosphere loss functions. An improved separation of K/N classes for the case of Objectosphere is observed.

CONCLUSIONS AND FUTURE WORK

Raman spectroscopy, in combination with ML architectures, demonstrates significant promise as a rapid, label-free, chemical identification method. It has a strong potential to identify dangerous and toxic contaminants, mitigate public health crises, and save human lives. However, traditional ML architectures applied in this context only work in “closed-world” settings and become ill-defined when presented with unknowns, limiting their practical applications. Through this proof-of-concept work, we have demonstrated that we can overcome this challenge by modifying the ResNet26 architecture to operate in “open-world” settings in order to handle unknowns. Our application of the recently introduced Entropic Open Set and Objectosphere approaches enables the ResNet26 architecture to maintain a high accuracy on the known classes while successfully avoiding mistakes and misclassifications of new, never seen before classes. This method is an important step toward the translation of using Raman spectroscopy combined with ML methods in settings where accurate and efficient identification is of utmost importance. In future work, we plan to explore mixtures of chemical classes and spectra with low signal-to-noise ratios to better mimic real-world settings and further improve the performance of our NN.

CODE AVAILABILITY AND REPRODUCIBILITY OF OUR RESULTS

Our code is publicly available on GitHub.⁴⁶ In order to reproduce our results, one needs to upload these data and code into Google Colab,⁴⁷ connect to the TPU, and follow the instructions in the Jupyter notebooks provided.