Deep learning-assisted sensitive detection of fentanyl using a bubbling-microchip

Abstract

Deep learning-enabled smartphone-based image processing has significant advantages in the development of point-of-care diagnostics. Conventionally, most deep-learning applications require task specific large scale expertly annotated datasets. Therefore, these algorithms are oftentimes limited only to applications that have large retrospective datasets available for network development. Here, we report the possibility of utilizing adversarial neural networks to overcome this challenge by expanding the utility of non-specific data for the development deep learning models. As a clinical model, we report the detection of fentanyl, a small molecular weight drug that is a type of opioid, at the point-of-care using a deep-learning empowered smartphone assay. We used the catalytic property of platinum nanoparticles (PtNPs) in a smartphone-empowered microchip bubbling assay to achieve high analytical sensitivity (detecting fentanyl at concentrations as low as 0.23 ng/mL in phosphate buffered saline (PBS), 0.43 ng/mL in human serum and 0.64 ng/mL in artificial human urine). Image-based inferences were made by our adversarial-based SPyDERMAN network that was developed using a limited dataset of 104 smartphone images of microchips with bubble signals from tests performed with known fentanyl concentrations and using our retrospective library of 17,573 non-specific bubbling-microchip images. The accuracy (± standard error of mean) of the developed system in determining the presence of fentanyl, when using a cutoff concentration of 1 ng/mL, was 92.66±0.3% in human serum (n=100) and 94.66±1.2% in artificial human urine (n=100).

Graphical Abstract

Introduction

Deep learning holds great potential in the development of image-based analysis.^1–3 However, most supervised deep learning-enabled image processing architectures are dependent on large, expertly-annotated data collected from specific domains.^4–6 Therefore, current deep learning applications are oftentimes focused on problems that have access to large retrospective datasets available for network development. In general, the importance of problems is not well reflected by data availability and many such problems that can benefit from deep learning applications tend to possess very little retrospective digitized data. In fact, a recent survey suggests that data sourcing and preparation can make up as much as 60% of an artificial intelligence (AI) development project’s budget and be the most challenging stage of development.⁷ Particularly in the field of medicine and diagnostics, the cost of reference tests and samples for data collection and generation for AI development can quickly become exorbitant and hinder the newer artificial intelligence-empowered technologies and applications.

AI technologies that emphasize development of learning models while taking cost constraints into account are loosely called “budget-sensitive” learning models.^{8, 9} While all fields of medicine can benefit from such approaches, it can be argued that point-of-care diagnostics (POC), which highly values cost-effectiveness, tends to benefit the most. Furthermore, a budget sensitive model synergizes well with the more recent wave of POC diagnostics that have incorporated more “smart” elements in the assay workflow. In fact, we have previously shown that for viral diagnostics such budget-sensitive AI models combined with smartphone-based assays can offer a rapid development pathway. We used an adaptive adversarial learning network with domain adaptation to generate a dataset library of unlabeled, heterogeneous, and nonspecific images for deep learning model training, for the detection of 5 different viruses with classic sandwich immunoassays.¹⁰ However the applicability of this approach toward other non-virus analytical targets and assay formats have not been validated as the signal patterns could be significantly different. For example, competitive assays, which are widely used for the detection of target analytes with small molecular weights or limited availability of antibody pairs, are usually associated with inverse logarithmic dose to signal responses,¹¹ which is significantly different from the linear dose to signal response in classic sandwich assays. To explore the applicability of this deep learning approach in the detection of small molecules with competitive assays, we chose fentanyl as a model analyte.

Fentanyl (N-phenyl-N-[1-(2-phenylethyl)piperidin-4-yl]propionanilide), is an addictive potent synthetic opioid originally synthesized for chronic pain management and surgical anesthesia.¹² Now, it has increasingly been observed as an adulterant in various illicit substances such as heroin and cocaine, often in imprecise manners, which result in unintentional or unknowingly fentanyl abuse.¹³ A dose as low as 2 mg of fentanyl can result in death.¹⁴ It has been reported that fentanyl is the leading cause of drug fatalities in the United States in 2020.^{15, 16} To address this crisis, POC fentanyl testing, particularly in community settings or clinics, is needed to better control the use of this dangerous narcotic, as it allows for insights into its diffusion among the population.¹³ Gas chromatography-mass spectrometry (GC/MS) or liquid chromatography-mass spectrometry (LC/MS) are the current gold standard for fentanyl detection.¹² However, these techniques require well-trained technicians and laboratory-based instrumentation, as well as tedious and time-consuming sample preparation and operation, not suitable for POC use. There are POC fentanyl tests available, such as lateral flow assays (LFAs) with detection limits of 10–20 ng/mL.¹⁷ However, these assays would yield false-negative results in overdose cases where the fentanyl concentrations are typically in a lower range.¹⁸

To explore the applicability of our reported deep learning approach in the detection of small molecules with competitive assays and also to address the current challenges of POC fentanyl testing, we have developed, for the first time, a deep learning-enabled smartphone-based POC assay for sensitive, specific, and accurate fentanyl detection in biological samples using platinum nanoparticles (PtNPs) as labeling agents to generate large, visible bubble signals. Furthermore, the assay makes use of our previously developed deep learning approach to reduce the need for human-annotated image datasets by capitalizing on virus-generated and synthetically produced bubble images as retrospective data for the development of a model to detect the fentanyl biomolecule.¹⁰ We were able to detect fentanyl in samples with concentrations as low as 0.23 ng/mL in phosphate buffered saline (PBS), 0.43 ng/mL in human serum, and 0.64 ng/mL in artificial human urine samples. With our deep learning approach, we were able to rapidly augment and retrain our previously AI model for bubble signal detection (SPyDERMAN) using a retrospective collection of 17,573 microchip images with bubble signals, including 16,000 images synthetically generated through an adversarial neural network, and a limited set of microchip images (104) collected from samples spiked with the target drug with broad, clinically relevant concentrations. The success and ease of adapting our deep learning approach from whole virus detection to small molecule detection with minimal retraining or alteration of parameters indicate its potential to be applied in other AI-assisted microfluidic systems especially those involving smartphone-based imaging analysis.

Methods

Materials

Bovine serum albumin (BSA, A7906-50G), TWEEN^® 20 (Molecular Biology Grade, P9416-100ML), (±)-Amphetamine solution (A-007), Heroin solution (H-038), Cocaine solution (C-008), (±)-Methamphetamine solution (M-009), fentanyl solution (F-013), Norfentanyl oxalate solution (N-031), double-sided adhesive (DSA) sheets (3M, 8215, 125 μm), and were purchased from Sigma-Aldrich, Inc. (St. Louis, MO, USA). Fentanyl-BSA (80–1409) and mouse monoclonal fentanyl antibody (10–2446) were purchased from Fitzgerald (Fitzgerald Industries International, Acton, MA, USA). Zeba^™ spin desalting columns (89882), magnetic separator (A14179), hydrogen peroxide (30% in water, BP2633500), Pierce^™ premium grade Sulfo-NHS (PG82072), Pierce^™ premium grade EDC (PG82079), sodium citrate (78-101-KG), and glycerol (MFCD00004722) were purchased from ThermoFisher Scientific, Inc. (Rockford, IL, USA). LodeStars^® High Bind Carboxyl magnetic beads (trial pack) were purchased from Agilent Technologies, Inc. (Santa Clara, CA, USA). Phosphate-buffered saline (PBS) tablets (T9181), pH 7.4, magnetic stand (631964) were purchased from Clontech Laboratories, Inc. (Mountain View, CA, USA). MES Buffer (50 mM, pH 6.0, 21420006-1) was purchased from Spectrum Chemical Manufacturing Corp. (New Brunswick, NJ, USA). Platinum Nanoparticles (PTCN70-50M) were purchased from Nanocomposix, Inc. (San Diego, CA, USA). VWR glass slides (48311-703) were purchased from VWR International (Radnor, PA, USA). Neodymium Block Magnets (B001-2020-035N) were purchased from W.W. Grainger, Inc. (Alsip, IL, USA). Poly(methyl methacrylate) (PMMA) sheets (8560K239, 3.175 mm thickness) were purchased form McMaster-Carr (Elmhurst, IL, USA).

De-identified, heat inactivated human serum samples (from male AB clotted whole blood), USA origin, sterile-filtered, H5667) were purchased from Sigma-Aldrich, Inc. (St. Louis, MO, USA). Artificial Urine samples (pH 6.5, stabilized, 50-197-4234) were purchased from Thermo Fisher Scientific, Inc. (Rockford, IL, USA).

Bubble Microchip Fabrication

The channel dimensions of microfluidic chip were 40 × 5 × 0.125 mm (LxWxH). The ends of the channel opened out to an inlet and an outlet which were roughly 1 mm in radius. The PMMA sheets and the double-sided adhesive (DSA) sheets were both cut using a VLS 2.30 CO₂ laser cutter (Universal Laser Systems Inc., Scottsdale, AZ, USA). The PMMA and DSA sheets, as well as glass microslides, were cleaned with hydrogen peroxide, followed by ethanol, and water, and then dried with a nitrogen flush. The final microchip is assembled by stacking the parts on top of each other and binding them together using the DSA.

Functionalization of Platinum Nanoparticles

For the preparation of anti-fentanyl antibody conjugated platinum nanoparticles (Fen-PtNPs) and BSA conjugated platinum nanoparticles (BSA-PtNPs), free anti-fentanyl antibody or BSA was diluted into 4 mM sodium citrate buffer using a Zeba column with 7k MW cutoff. Then 100 μL of 2 mg/mL anti-fentanyl antibody or BSA was mixed with 1 mL of 0.05 mg/mL (1.2×10¹⁰/mL) 70 nm PtNPs in 4 mM sodium citrate buffer, and continuously mixed using a rotator (20 rpm) at 4 °C overnight. To block the PtNP surface, 400 μL of 10% BSA in 4 mM sodium citrate buffer was added and mixed with PtNPs on a rotator (20 rpm) at 4 °C overnight. Unconjugated antibody was removed by centrifugation at 3500 g for 9 min, 6 times. Finally, the prepared Fen-PtNPs were suspended and stored in 800 μL of PBS, pH 7.4, containing 1% BSA at 4 °C.

Functionalization of Superparamagnetic Microbeads.

LodeStars^® High Bind 2.7-μm diameter carboxyl-terminated superparamagnetic beads were functionalized with fentanyl-BSA or antibody using EDC coupling following the manufacturer’s instructions. Briefly, 50 μL of beads at concentration of ~2.9×10⁹/mL were first rinsed twice with 100 μL of 0.01 M sodium hydroxide to activate the carboxyl groups on the beads. The beads were rinsed 3 times with 500 μL deionized water followed by rinsing three times with MES buffer, pH 5.0. Then they were further reacted with 800 μL of a solution containing 50 mg/mL of Sulfo-NHS and 50 mg/mL of EDC in MES buffer, pH 5.0, on a roller (20 rpm) at 23 °C for 25 min. After 3 quick rinses with 800 μL MES buffer, pH 5.0, the beads were then reacted with 300 μL of 1 mg/mL fentanyl-BSA in MES buffer, pH 5.0, at 4 °C overnight. To quench the uncoupled NHS group on the surface, 100 μL of 100mM Tris-HCl was added and reacted at 4 °C for 2 h. Finally, the functionalized beads were rinsed 3 times with 800 μL of PBS buffer pH 7.4 containing 1% BSA, and then resuspended in 1 mL PBS buffer pH 7.4 containing 1% BSA and stored at 4 °C.

Fentanyl bubbling assay

For the fentanyl bubbling assay, 100 μL of sample was mixed with 5 μL of Ab-PtNP (1.2×10¹⁰/mL) and allowed to react at room temperature (23 °C) for 10 min. Then, 6 μL of magnetic fentanyl-BSA-microbeads (5mg/mL, 3.8×10⁹/mL) were added into the mixture, mixed and reacted for 40 min at room temperature. The beads were then separated using a magnetic separator and suspended in 40 μL of 24% hydrogen peroxide and 20% glycerol solution (Fuel). The magnetic bead slurries were then transferred into the chambers of a microchip. Then a neodymium block magnet was placed on top of the chamber of the bubbling microchips to pull the beads up against the PMMA ceiling of the microchips. Finally, within 10 min, microbubbles with various sizes could be observed in the chamber of the microchip, which was imaged using a smartphone (iPhone 10, iPhone 11, iPhone 12, or Motorola Moto X Pure).

Manual area calculation

The area calculation of bubbles in microchip images was done using the Weka Segmentation plugin in the Fiji image analysis software.^{19, 20} A classifier was used to identify pixels corresponding to bubbles in the images. This classifier model was created using the Fast Random Forest algorithm with the following parameters: membrane thickness of 1, membrane patch size 19, minimum sigma of 1, maximum sigma of 16, gaussian blur, Sobel filter, Hessian, difference of gaussians, and membrane projection. Pixels in the images were divided into 3 classes: chamber background pixels, bubble-associated pixels, and microchip side pixels. A classifier was created with these parameters using a dataset that contained 2125478 selected data instances that were selected in 6 manually segmented microchip images. For test images, the images were segmented using the classifier in the Weka Segmentation plugin, as shown in Figure S1. After segmentation, the images were transformed to analyze the bubble events. For bubble detection, the images were thresholded for the pixel value of the bubble class from the segmentation plugin. Then the images were analyzed using the “Analyze Particles” plugin in Fiji with the following parameters, size: 0-infinity and circularity: 0–1. For determining the area of the chamber, the segmented images were thresholded for the pixels corresponding to the chamber area. Then the thresholded pixels were selected and measured using the “Measure” command in Fiji. The performance of the random forest classifier was initially evaluated through cross validation using 10 folds and a 95% confidence interval. With cross validation on the training set of 2125478 data instances, the classifier correctly identified 98.03% of instances with 97% precision. The classifier was then reevaluated on an external test set of 2430912 instances. The classifier correctly identified 98.1674% of test instances with a root mean squared error of 0.0908%.

Image preprocessing for deep learning

Images were pre-processed prior to use for both training and testing with an adversarial network similar to our previous work, so that we may effectively capitalize on the data library developed in that work.¹⁰ Briefly, the microfluidic channels of the chips imaged with the camera were isolated by cropping the images. These channels were resized (250 × 2250 pixels), split into 3 equal parts (250 × 750 pixels) and tiled to obtain our final image (750 × 750 pixels).

Development of the deep-learning classifier for fentanyl detection

The deep learning model developed for fentanyl detection used the SPyDERMAN platform that we previously utilized in detecting viruses.¹⁰ Briefly, the network can sufficiently generalize by capitalizing on unannotated data from the data library. This is achieved by modeling the problem as a domain adaption challenge. SPyDERMAN’s adversarial network, based on DANN, CDAN, and MD-nets, tries to minimize domain discrepancies between feature representations of source and target images along with classifier error.^21–23 The network can be broken down into a feature extractor, classification layer and adversarial block. Annotated fentanyl data (source) and non-annotated data from the data library (target) are transformed into the respective feature representations by the feature extractor and these feature representations are utilized by the classifier and adversarial blocks during training to effectively classify between the different classes (positive and negative for fentanyl). SPyDERMAN, due to this design, focuses on the class-specific features that retain its form when the domain shifts from Source to Target.

Using the data library, it widens the data distribution though addition of image variabilities, such as bubble shapes, sizes, concentrations, and positions to improve network generalizability. The library contains 17,573 images, which includes 16,000 synthetically generated images. Therefore, the large dataset allows the network to cover a larger distribution of data although the actual annotated dataset may be small. In this study, SPyDERMAN used the 17,573 unannotated images from the library as target and 104 images from tests performed with known fentanyl concentrations as source. The network was defined as a binary classifier wherein positive and negative predictions represented samples with drug concentrations ≥1 ng/mL and <1 ng/mL concentrations, respectively. Images were resized to 224 × 224 pixels prior to use with the network. The network used the ResNet-50 architecture as its feature extractor. The losses generated by the classifier block and adversarial block are minimized together (L_total); min(ε(C) − λε(D)), where ε(C) is the classifier loss (L_c) and ε(D) is the adversarial (transfer) loss (L_adv) (Fig. 3A). ε(C) is given by ${\mathbb{E}}_{({\mathrm{X}}_{\mathrm{i}}^{\mathrm{s}},{\mathrm{Y}}_{\mathrm{i}}^{\mathrm{s}})~{\mathrm{D}}_s}\mathrm{L}\left(\mathrm{C}\right({\mathrm{X}}_{\mathrm{i}}^{\mathrm{s}}),{\mathrm{Y}}_{\mathrm{i}}^{\mathrm{s}})$, where ${\mathrm{X}}_{\mathrm{i}}^{\mathrm{s}}$ and ${\mathrm{Y}}_{\mathrm{i}}^{\mathrm{s}}$ represent the source images and their respective ground truth annotations. The loss L() represents cross-entropy loss and C() is the classifier network. $\mathrm{\epsilon }(\mathrm{D})={-\mathbb{E}}_{{\mathrm{x}}_{\mathrm{i}}^{\mathrm{s}}~{\mathrm{D}}_{\mathrm{s}}}\mathrm{w}(\mathrm{H}{(c}_i^s))\mathrm{l}\mathrm{o}\mathrm{g}[\mathrm{D}({\mathrm{h}}_{\mathrm{i}}^{\mathrm{s}})]{-\mathbb{E}}_{{\mathrm{x}}_{\mathrm{j}}^{\mathrm{t}}~{\mathrm{D}}_{\mathrm{t}}}\mathrm{w}(\mathrm{H}{(c}_j^t))\mathrm{l}\mathrm{o}\mathrm{g}[1-\mathrm{D}({\mathrm{h}}_{\mathrm{j}}^{\mathrm{t}})]$, likewise, provides the discriminator error, where w(H(c)) is the weighted uncertainty of predictions, and h is multilinear feature map that combines the class confidence and feature representations. In the L_total equation, λ is the trade-off, optimized during training, that allows prioritization of classifier or adaption performance. In this study λ was set to 0.5. Source data was loaded in batches of 16 by random sampling with data augmentations such as rotations and flips applied with a probability of 0.5. Batch sizes of 8 and 16 were used. Optimizer used was Adam with the initial learning rate set at 0.001 and a weight decay of 0.0005. L2 penalty was used to prevent overfitting. A gamma value of 0.001 and a power of 0.75 was modify the learning rate (lr) thus: lr1 = lr0 * (1 + gamma * i) ^ (−power), where i indicates training progress. A momentum of 0.9 was used to accelerate the learning process. The optimal batch size and learning rate was determined manually by the selection of the best model based on lowest validation loss. Validation was performed at intervals of 50 iterations. Early stoppage was adopted with a patience of 500 iterations to avoid overfitting. Three models were trained using three seed initializations.

Figure 3. Development and evaluation of the developed deep learning framework for fentanyl detection using spiked samples.

A) The schematic illustrates the general architecture and developmental pipeline of the adversarial network used in this study. f_s, f_t, L_c, L_adv, and L_total represent feature maps of source images (obtained from tests with known fentanyl concentrations), feature maps of target images (unannotated data obtained from the library), classifier loss, transfer loss, and total loss, respectively. B) The validation, classifier, and transfer loss curves of the network over the course of development. The dotted lines represent the iteration where the model was saved due to early stoppage conditions (lowest validation loss) in place. Validation loss was obtained at 50 iteration intervals when evaluating the validation set during training. The classifier and transfer loss represent the training losses C) The confusion matrix of the best model, when evaluating PBS samples spiked with Fentanyl (n=45) D) The bar graph illustrates the average performance metrics of the model when evaluating 45 PBS samples spiked with Fentanyl (n=3 seeds). The error bars represent standard error of mean.

The code can be accessed via https://github.com/shafieelab/SPyDERMAN and the data library is available at https://osf.io/et9s6/. The fentanyl dataset generated and analyzed in this study can be obtained via https://osf.io/saux3.

Results and Discussion

Fentanyl is a small biomolecule with a molecular weight of 336.5 g/mol and limited antibody binding sites.²⁴ Therefore, it is challenging to directly use the detection techniques commonly used for larger biotargets like the “sandwich” format immunoassay, which requires two separated antibody binding sites. Instead, we use a competitive strategy in the assay, which relies on the competitive binding between target analytes and synthetic analytes.^{25, 26} To maximize the assay sensitivity and simplify the assay process, we also introduce magnetic microbeads that have ultra-high surface-to-volume ratio and can be simply and quickly separated from the solution using magnets. Its working principle is shown in Figure 1. The pre-immobilized fentanyl molecules on the magnetic beads (magnetic beads coated with fentanyl conjugated bovine serum albumin, Fen-BSA-MB) compete with the target fentanyl molecules in the sample for binding to the anti-fentanyl antibody PtNP conjugates (FenAb-PtNP). FenAb-PtNPs that bind to the magnetic beads catalyze the formation of visible bubbles in bubbling microchips. As the fentanyl concentrations in the sample increases, the number and volume of bubbles in the bubbling chip decreases. The bubbles are finally imaged by a smartphone camera and the images are analyzed by a pre-trained AI algorithm to calculate the presence of fentanyl in the sample.

Figure 1. Schematic illustration of the deep learning-assisted fentanyl bubbling assay.

A) Human body fluid samples, such as blood or urine, are first mixed with platinum nanoparticles conjugated with anti-fentanyl antibodies (FenAb-PtNP). If fentanyl molecules exist, they bind to the fentanyl binding sites on the FenAb-PtNP. B) Then, the mixture is further mixed with magnetic beads coated with fentanyl conjugated bovine serum albumin (Fen-BSA-MB), which capture FenAb-PtNPs. If fentanyl molecules already occupy the binding sites of FenAb-PtNPs, less FenAb-PtNPs are captured. C) Then, all Fen-BSA-MB are pulled down with magnet, mixed with bubbling fuel (24% H₂O₂, 20% Glycerol) and transferred to the bubbling microchip. When fentanyl molecules exist, less bubbles show up in the bubbling microchip. D) Finally, the images of bubbles are captured by smartphone camera and quickly and accurately analyzed with a deep learning network. Not to scale.

For the detection probe of the assay, we conjugated anti-fentanyl antibody to PtNP via thiol-Pt chemical bonds as previously described.²⁷ To evaluate the function of FenAb conjugated to PtNPs for fentanyl capture, FenAb-PtNPs and BSA coated PtNPs (BSA-PtNPs) as a negative control were incubated with Fen-BSA immobilized on cellulose ester membranes. As shown in Figure S1, only FenAb-PtNPs bind to the Fen-BSA, indicating the successful functionalization of PtNP with active anti-fentanyl antibodies. We then coated magnetic microbeads with Fen-BSA via amine/carboxyl coupling through EDC/NHS as described previously.²⁸ Different concentrations of FenAb-PtNPs were incubated with Fen-BSA-MBs to find the optimal concentration of FenAbPtNPs for bubble formation. As shown in Figure S2, the bubble area ratios increased linearly with the concentration of FenAb-PtNPs, while area ratios generated by BSA-PtNPs were consistently low at different concentrations of BSA-PtNPs due to the lack of FenAb on the surface of PtNPs. In the presence of excess amount of fentanyl in the sample, all fentanyl binding sites on the FenAb-PtNPs will be occupied eliminating specific bindings between PtNPs and Fen-BSA-MBs. This is equivalent to replacing all FenAb-PtNPs with BSA-PtNPs. Therefore, the signals of FenAb-PtNP and BSA-PtNP at different concentrations represent the maximum and minimum signals. To achieve a balance between maximum dynamic range and minimum background signal, we chose 6×10⁸/mL as the optimal concentration of FenAb-PtNP for the fentanyl bubbling assay.

In order to initially quantitate the production of bubbles in the assay, we evaluated bubble volume as a bubble signal parameter that could be measured manually to evaluate the relationship between bubble signal and the concentration of drug in serially diluted samples. Considering that the generated microscale bubbles, which are visible with the naked eye, were restricted in a thin microchamber with a 125 μm channel height, the bubble area in the microchamber can linearly represent the bubble volume. Using Fiji image software, we manually calculated the area of the bubbles in each image, as shown in Figure S1.^{19, 20} To avoid variation caused during imaging, we normalized bubble area with respect to the corresponding microchamber area.

To determine the intrinsic analytical sensitivity of the fentanyl bubbling assay, we first tested different concentrations of fentanyl in buffer. As shown in Figure 2A, B and Figure S4, the bubble area ratios decreased as the concentration of fentanyl increased. The dose response curve fits well (R²=0.972) to a three-parameter logistic model equation for classic competitive immunoassay:¹¹

$$y=a-b\ln (x+c)$$ (1)

where y is the bubble area ratio, x is the concentration of fentanyl. The limit of detection (LOD) was 0.23 ng/ml, which was calculated by extrapolating the fentanyl concentration that produced bubble area ratio signal of 3 times the standard deviation of the zero-analyte signal below the zero-analyte signal. This analytical sensitivity is comparable with current central clinical laboratory gas chromatography–mass spectrometry or LC-MS/MS–based methods and over 20 times more sensitive than commercially available fentanyl lateral flow assays (LFAs),²⁹ as shown in Figure S5. An average coefficient of variation (CV) of 16% has been achieved for the fentanyl bubbling assay, as shown in Table S1. The CV of the fentanyl bubbling assay could be further decreased by integrating automated modules for mixings and washing steps.

Figure 2. Analytical performance of fentanyl bubbling assay.

A) Dose response curve of the bubble area ratio of fentanyl bubbling chips against different concentrations of fentanyl (0, 0.1, 0.5, 1, 2, 5, 10, and 100 ng/mL). l. B) Zoom in of the portion in A) inside of the square with x axis in linear scale. C) Dose response curve of the bubble area ratio of fentanyl microchips against different concentrations of norfentanyl (0, 10, 100, and 1000 ng/mL). D) Comparison of the bubble area ratios of fentanyl bubbling microchips with different drugs at the concentration of 100 ng/mL. ns, P > 0.05; *, P ≤ 0.05; **, P ≤ 0.01, ***, P ≤ 0.001; ****, P ≤ 0.0001. Mean ± standard deviation; n=3.

In addition to evaluating the assay response to fentanyl, we also tested for norfentanyl. Norfentanyl is the major metabolite of fentanyl that is metabolized through oxidative N-dealkylation in the liver, with an elimination half-life of 219 min.³⁰ It has been reported that norfentanyl can be detectable in human urine samples up to 72 hours after patients receiving a single 50- to 100-mg intravenous fentanyl dose.³¹ Therefore, detection of norfentanyl is important because it is widely used as an indicator to judge fentanyl abuse. To access the analytical coverage of the fentanyl bubbling assay for norfentanyl, we tested the developed fentanyl bubbling chips with norfentanyl at different concentrations. As shown in Figure 1C and Figure S6, the bubble area ratios decreased as the concentration of norfentanyl increased, fitting to the same three-parameter logistic model equation created with fentanyl (1), with a LOD of 21.6 ng/ml (calculated by extrapolating the norfentanyl concentration that produced signal of 3 times standard deviations of the zero-analyte signal below the zero-analyte signal), which is over 15 times more sensitive than commercially available LFA strips.¹⁸

To evaluate the analytical specificity of the fentanyl bubbling assay, we tested three common drugs of abuse, including amphetamine, cocaine, heroin, and methamphetamine. As shown in Figure 2D and Figure S7, the bubble area ratios of fentanyl and norfentanyl are significantly smaller than that of the blank buffer control, while the bubble area ratios of amphetamine, cocaine, heroin are not significantly different with the blank buffer control. Only methamphetamine (cross-reactivity 4.9%) cross-reacted in the fentanyl bubbling assay. The cross-reactivity with methamphetamine has also been reported for commercial LFA strips likely due to the structure similarity between the two molecules.³²

In order to make the assay as user friendly as possible, the augmented SPyDERMAN framework, as shown in Figure 3A, was used to automate a smartphone-based analysis of bubble outputs. The network was trained and optimized adversarially to capitalize on bubble data generated with other targets.¹⁰ During training, the classifier loss and adversarial (transfer) loss were minimized together prioritizing on classifier loss over adversarial loss. The loss curves confirmed adequate classifier performance and adaption of the saved model, with low classifier loss and high transfer loss (Figure 3B). Furthermore, we confirmed model adaption and performance through t-SNE. Negatives and positives were separated well, while target and source data strongly overlap (Figure S8). The adapted models performed well on our held-out test set of 45 fentanyl-spiked PBS samples as part of our initial evaluation. Figure 3C represents the confusion matrix of the best performing model. To verify the stability and repeatability of the adaption process, we trained models using the 3 different seed initializations and observed their performances. The classifiers on average (± standard error of mean) performed with 88.9±6.7% accuracy (n=45; 3 seeds) (Figure 3D). Their sensitivity and specificity were 89.6±8.3% and 88.5±5%, while the positive and negative predictive values were 89.9±3.8% and 87.9±9.9%, respectively (n=45; 3 seeds) (Figure 3D). The algorithm does not require higher than normally available pixel qualities found in most consumer smartphones. The algorithm, as part of its preprocessing step, resizes and restructures the image to best capture the bubble signal information for analysis. The only user-dependent quality criterion is the need to collect bubble signal data that discernable by naked eye, i.e., the image is in focus. To demonstrate this, in this study, we have used different smartphone models including iPhone 10, iPhone 11, iPhone 12, and Motorola Moto X Pure to collect microchips images for training, validation, and testing the reported deep learning framework.

After validating the neural network, we wanted to evaluate the combined assay/algorithm efficacy on fentanyl samples that mimic clinical conditions. Fentanyl can be found in a variety of bodily specimens such as urine, blood, hair, sweat, and saliva, among which urine and blood are the two major specimens tested in current clinical practice.^{33, 34} This is mainly because urine sample collection is noninvasive and concentrations of fentanyl and its metabolites have been reported higher than the serum equivalent. Blood sample can be obtained from anuric patients when it is challenging to obtain urine samples, such as intoxicated patients occasionally seen in the emergency department.³⁴ To validate the clinical performance of the fentanyl bubbling assay, we evaluated the developed assay using artificial urine samples and human serum samples spiked with fentanyl at various concentrations. As shown in Figure 4 and Figure S9–11, the trained deep learning classifier could predict the presence of fentanyl based on a cutoff concentration of 1 ng/mL in both artificial urine samples and human serum samples. We evaluated 100 fentanyl-spiked artificial urine samples using the three models used to evaluate the spiked PBS samples. Figure 4A represents the confusion matrix of the best performing model when evaluating fentanyl-spiked artificial urine samples. The classifiers on average (± standard error of mean) performed with 95.3±1.5% accuracy (n=100; 3 seeds) (Figure 4B). Their sensitivity and specificity were 97.6±1.2% and 92.6±1.8%, respectively, while the positive and negative predictive values were 94.2±1.5% and 96.9±1.6%, respectively (n=100; 3 seeds) (Figure 4B).

Figure 4. Validation with spiked fentanyl in artificial urine and human serum.

(A) and (B) represent the confusion matrix of the best model, when evaluating artificial urine samples spiked with Fentanyl (n=100) and the average performance metrics of the model when evaluating 100 Fentanyl-spiked urine samples (n=3 seeds). (C) and (D) represent the confusion matrix of the best model, when evaluating human serum samples spiked with Fentanyl (n=100) and the average performance metrics of the model when evaluating 100 Fentanyl-spiked human serum samples (n=3 seeds). The error bars represent standard error of mean.

Similarly, we evaluated 100 fentanyl-spiked human serum samples using the same three models. Figure 4C represents the confusion matrix of the best performing model when evaluating fentanyl-spiked human serum samples. The classifiers on average (± standard error of mean) performed with 93±0% accuracy (n=100; 3 seeds) (Figure 4D). Their sensitivity and specificity were 94±2.2% and 92.2±3.4%, respectively, while the positive and negative predictive values were 95.2±2.4% and 89.2±4.1% (n=100; 3 seeds), respectively (Figure 4D).

In both sample types, the bubble area ratios decreased as the concentration of fentanyl increased, both fitting to the same three-parameter logistic model equation from before (1), with LODs at 0.6 ng/ml in artificial urine and 0.4 ng/mL in human serum (calculated by extrapolating the fentanyl concentration that produced signal of 3 times the standard deviation of the zero-analyte signal below the zero-analyte signal).

Conclusion

Here, we reported the development and validation of a POC diagnostic assay for qualitative drug detection in complex biological samples using platinum nanoprobes for signal amplification and a deep learning-enabled smartphone application for qualitative assessment of microchip images with bubble-based optical signals. The competitive binding between target fentanyl and fentanyl conjugated on the surface of magnetic beads to FenAb-PtNPs results in bubble formation inversely proportional to the concentration of target fentanyl. This assay has high\ sensitivity and has promise to achieve deep learning assisted POC fentanyl detection. The fentanyl bubbling assay also has high analytical specificity, detecting fentanyl and its metabolite, norfentanyl, without cross-reactivity with most other common drugs of abuse. Further system integration is required for simple sample handling using microfluidic cartridges. With the successful application of our deep learning approach in the fentanyl assay, we demonstrated: (I). the first application of an adaptative adversarial learning network in a competitive immunoassay format with an inversely proportional signal to dose response; (II). detection of a small molecular drug using a relatively limited microchip image dataset of the target drug in serially diluted samples and retrospectively collected smartphone-taken microchip images of viral-infected samples as well as synthetically generated microchip bubble images; and (III). the first application of a deep learning-enabled assay for POC detection of fentanyl. Our deep learning approach can minimize the amount of expertly-annotated data needed for training supervised deep learning-enabled image processing architectures by leveraging a customized synthetic dataset library of unlabeled, heterogeneous, and nonspecific images generated through an adversarial neural network. We demonstrated that our deep learning approach was successfully and rapidly adapted from whole virus detection to small molecule detection with minimal retraining or alteration of parameters. We believe that this ease of adaption allows for the reported adversarial deep learning approach to be potentially applied to the AI-assisted image analysis of other entities in microfluidic systems, such as microparticles and cells, without a significant amount of reworking the algorithm.