Skip to main content
NCBI home page
PLOS One logoLink to PLOS One
. 2023 Jul 13;18(7):e0288656. doi: 10.1371/journal.pone.0288656

Development of a neural network model to predict the presence of fentanyl in community drug samples

Lianping Ti 1,2,*,#, Cameron J Grant 1,#, Samuel Tobias 1,3,#, Dennis K Hore 4,5,#, Richard Laing 6,#, Brandon D L Marshall 7,#
Editor: Muhammad Hanif8
PMCID: PMC10343080  PMID: 37440523

Abstract

Introduction

Increasingly, Fourier-transform infrared (FTIR) spectroscopy is being used as a harm reduction tool to provide people who use drugs real-time information about the contents of their substances. However, FTIR spectroscopy has been shown to have a high detection limit for fentanyl and interpretation of results by a technician can be subjective. This poses concern, given that some synthetic opioids can produce serious toxicity at sub-detectable levels. The objective of this study was to develop a neural network model to identify fentanyl and related analogues more accurately in drug samples compared to traditional analysis by technicians.

Methods

Data were drawn from samples analyzed point-of-care using combination FTIR spectroscopy and fentanyl immunoassay strips in British Columbia between August 2018 and January 2021. We developed neural network models to predict the presence of fentanyl based on FTIR data. The final model was validated against the results from immunoassay strips. Prediction performance was assessed using F1 score, accuracy, and area under the receiver-operating characteristic curve (AUROC), and was compared to results obtained from analysis by technicians.

Results

A total of 12,684 samples were included. The neural network model outperformed results from those analyzed by technicians, with an F1 score of 96.4% and an accuracy of 96.4%, compared to 78.4% and 82.4% with a technician, respectively. The AUROC of the model was 99.0%. Fentanyl positive samples correctly detected by the model but not by the technician were typically those with low fentanyl concentrations (median: 2.3% quantity by weight; quartile 1–3: 0.0%-4.6%).

Discussion

Neural network models can accurately predict the presence of fentanyl and related analogues using FTIR data, including samples with low fentanyl concentrations. Integrating this tool within drug checking services utilizing FTIR spectroscopy has the potential to improve decision making to reduce the risk of overdose and other negative health outcomes.

Introduction

Recent decades have seen a rapid shift in the unregulated drug market, with the emergence of novel synthetic opioids (e.g., fentanyl, carfentanil) and their associated harms (e.g., overdose, mortality) [1–3]. In response, drug checking services have been implemented as an important harm reduction intervention for people who use drugs to chemically analyze their substances and receive fact-based information and consultation regarding the compounds detected in their sample [4–6]. Not a new concept, the vast majority of drug checking services have previously focused on drugs used in party and festival settings (e.g., psychedelics, stimulants) [7, 8], so less is known about drug checking in the context of the synthetic opioid-driven overdose epidemic. While limited, recent studies have started to show a positive impact of drug checking, including increased engagement in overdose risk reduction practices following the use of the service in some communities [5, 9–11].

Drug checking can be conducted using a range of technologies, offering both benefits and challenges in terms of accuracy, timing, cost, portability, ability to detect a range of compounds, preparation, and required human resources [4, 6, 12, 13]. For example, gas chromatography/mass spectrometry (GC-MS) is highly sensitive, but is expensive, time-consuming, and requires a trained laboratory technician to operate the machine [6, 14]. In contrast, immunoassay strips are inexpensive, easy to use, portable, and can provide results in less than five minutes, but they are unable to detect a range of compounds and can only produce qualitative results (i.e., presence, absence) [15, 16]. Fourier-transform infrared (FTIR) spectroscopy is being implemented across North America for drug checking given the ease and speed of sample preparation and analysis, portability, and ability to detect a range of compounds [15, 16]. However, recent validation studies have indicated that FTIR spectroscopy can only detect compounds above a certain limit; for fentanyl and some fentanyl analogues, that limit is 3–10% quantity by weight, depending on the setting and the presence of specific cuts and buffs [16, 17]. Given that some analogues (e.g., carfentanil) can cause toxicity at very low concentrations [18, 19], failing to detect these compounds when present is a substantial concern when using FTIR spectroscopy on its own. Additionally, there is some subjectivity in interpreting FTIR spectra by drug checking technicians that could result in differing outcomes. Previous studies have shown that when compared to laboratory reference standards, analysis using FTIR spectroscopy can have false negative rates as high as 27.9% [15]. As a result of this limitation, many settings have adopted an approach that utilizes a combination of both FTIR spectroscopy and immunoassay strips (which have a lower detection limit) in tandem to offset the limitations of each technology when used alone [20–22]. However, this can come with its own set of logistical challenges, including additional costs, sample preparation, and timing.

Machine learning applications in the field of health research have been steadily growing due to advancements in processing power and data sharing [23–25]. In recent years, deep learning methods have gained popularity in predictive modelling for substance use research [26]. For example, recurrent neural networks utilizing electronic health records have been applied to classify people who use opioids as long-term patients, short-term patients, and opioid-dependent patients [27]. Other studies have demonstrated the ability of long short-term memory models, an application of deep learning, to accurately predict new onset and prevalence of opioid use disorder using electronic health records [28].

Despite this promising line of inquiry, little is known about whether there is utility in applying deep learning methods to improve the accuracy of FTIR spectroscopy in detecting fentanyl and its analogues, especially when fentanyl is present at low concentrations. The present study aimed to develop a machine learning model that could predict the presence of fentanyl and related analogues with greater accuracy compared to analysis of spectra by a drug checking technician, with a second aim of the model being able to detect fentanyl at low concentrations. Building tools that can be used to accurately detect fentanyl and its analogues is key to ensuring that point-of-care drug checking technologies are providing people who use drugs with the most accurate information to make decisions about their use.

Methods

Study setting

British Columbia (BC), the westernmost province of Canada, has been disproportionately affected by the opioid-driven overdose epidemic [3]. By the time a public health emergency was declared in 2016, a significant number of lives had already been lost to illicit drug toxicity, largely due to the widespread presence of fentanyl and other novel synthetic opioids in the unregulated drug supply [29]. In response, a diverse range of harm reduction interventions, including supervised consumption sites, take-home naloxone programs, and drug treatment have been scaled-up [30–32]. As a result, illicit drug toxicity death rates were decreasing between 2018 and 2019 from 31.2 per 100,000 to 19.4 per 100,000, respectively; however, the emergence of the COVID-19 pandemic and corresponding social distancing and self-isolation measures resulted in a significant rise in death rates from 34.4 per 100,000 in 2020 to 44.1 per 100,000 in 2021 [3, 33].

Data sources

The Drug Checking BC Database includes de-identified, point-of-care drug checking data collected utilizing Bruker (Billerica, Massachusetts, United States of America) ALPHA FTIR spectrometers and BTNX (Markham, Ontario, Canada) immunoassay strips from over a dozen participating community harm reduction sites in BC. Further details about the drug checking service and the use of these technologies, which have been operating since October 2017, have been described elsewhere in detail [15, 20]. Briefly, individuals in harm reduction sites anonymously provided an approximately 5 mg drug sample to a trained drug checking technician and information about what they believed the sample to contain [34]. The drug sample was then analyzed with FTIR and immunoassay strips, with results being immediately available to the person who provided the sample. Results from FTIR analysis include compounds detected in abundant amounts (typically those 5% by weight and above), including both active ingredients (e.g., fentanyl, heroin, methamphetamine) and inert cutting agents (e.g., mannitol, inositol, lactose) [6, 17, 20]. An illustrative example of what a drug checking technician analyzes (and attempts to determine if fentanyl is present) is displayed in Fig 1.

Fig 1. Sample drug checking Fourier-transform infrared spectra.

Fig 1

Open in a new tab

Drug checking samples (blue) are overlaid with the fentanyl hydrochloride spectrum (black). The drug sample in A is an illicit opioid mixture containing fentanyl, caffeine, and mannitol. The drug sample in B is methamphetamine with no fentanyl present. The red arrows indicate characteristic peaks of fentanyl hydrochloride that are recognizable when present in drug mixtures.

In addition to the results obtained from the point-of-care technologies, the database also includes self-reported information about drug expectation and appearance (i.e., color, texture) [35]. Additionally, a separate and validated quantitative model using Bruker Quantitative Analysis 2 (QUANT 2) software was applied retrospectively to fentanyl-positive samples (identified via immunoassay strips) to generate approximate fentanyl concentrations for each sample [34]. The present study was restricted to samples obtained between August 2018 and January 2021, as linked data were available during this time period.

Participants provided verbal consent by agreeing to have their drugs checked and no personal identifying information was ever collected. This study was conducted as part of a larger drug checking evaluation project and was approved by the University of British Columbia/Providence Health Care Research Ethics Board.

Data preparation

In terms of input data, Joint Committee on Atomic and Molecular Physical Data exchange (jcamp-dx) encoded FTIR data files were converted to data tables containing wavenumbers and the corresponding absorbance for each spectrum (i.e., drug sample). Sample data tables meeting the following conditions were kept for analysis: first, the data table had at least 1,626 sampling points and second, the range of wavenumbers covered the range 600–3,800 cm-1 including endpoints. A cubic spline was fitted to the samples by treating the absorbance as a function of the wavenumbers, which were then sampled from the cubic spline interpolated function at every second integer value of wavenumber starting from 600 to 3,798 cm-1 which resulted in 1,600 features. Each sample was then standardized by subtracting the mean and dividing by the standard deviation of the discrete function.

Output data (i.e., reference standard) were recorded as the binary encoded results of the fentanyl immunoassay strips, which can consistently detect fentanyl and a range of analogues [16, 17], including carfentanil, acetyl fentanyl, para-fluorofentanyl, and furanyl fentanyl [36]. We chose fentanyl test strips as the output data given the high accuracy for detecting fentanyl reported in previous validation studies and the lack of confirmatory analysis for a large proportion of the samples [15, 16]. Indeed, past research has shown that immunoassay strips can detect fentanyl with a sensitivity and specificity as high as 100% and 98%, respectively [16].

Feature selection

Absorbance data were used as the sole feature for predicting the presence of fentanyl in samples. As part of the drug checking service, only minimal information is collected on drug samples to ensure that the service was low-barrier and wide-reaching. While additional sample-level data exist for the included samples, we chose to not include features such as colors and textures given that these are based on subjective human observable features and previous studies have shown that visual appearance of drug samples is not accurate in detecting the presence of fentanyl [37].

Prediction methods and model selection

The goal of the study was to use anonymous, point-of-care drug checking data obtained from FTIR spectra to predict the presence of fentanyl and related analogues in drug samples. We implemented an artificial neural network-based model, which was composed of six layers in the following order (Fig 2):

Fig 2. A structural illustration of the final neural network model.

Fig 2

Open in a new tab

The network’s input is 1600-dimensional and the remaining layers are 1589, 794, 32-dimensional with a 1-dimensional output.

  1. an input layer

  2. a one-dimensional convolution layer with eight filters, a kernel size of 12, and a rectified linear unit (ReLU) activation function

  3. a dropout layer with a 50% dropout rate

  4. a one-dimensional max pooling layer with a pool size of two

  5. a dense layer with output dimension 32 and a ReLU activation function

  6. a dense output layer with output dimension one and a sigmoid activation function.

The model was trained on 80% of the data and 20% of the data was reserved for testing the model’s accuracy. At each epoch, the accuracy (i.e., number of correct predictions among total number of predictions) was measured on the test set and the model with the highest accuracy in the test set was kept.

We implemented the models and performed the study using the Python (version 3.9.6) programming language and the TensorFlow package (version 2.6.0).

Statistical analyses

In assessing our model’s performance, we used the prediction value which is a continuous variable ranging from 0 to 1. We applied a rounding procedure to transform this continuous output into a binary classification to align with our label values (i.e., the results from the corresponding fentanyl test strip for that sample). Specifically, prediction values greater than or equal to 0.5 were rounded up to 1, indicating a positive fentanyl result, while values less than 0.5 were rounded down to 0, indicating a negative result. These rounded prediction values were then compared against our label values, which represent the binary outcomes of the fentanyl test strip. Using these results, we calculated several diagnostic metrics, including F1 score, accuracy, precision, recall, and area under the receiver-operating characteristic curve (AUROC). An F1 score describes a model’s accuracy, taking the precision (correctly identified positives among test positives) and recall (correctly identified positives among true positives) into account. Given the imbalance in the dataset, calculating accuracy alone could be misleading; thus, the F1 score assessing the overall prediction performance using both precision and recall may be more appropriate. Bootstrapping methods based on 10,000 bootstrapped runs were used to create confidence intervals (CI) for the estimates. This methodology was chosen because of its ability to make strong inferences from the dataset, especially when the underlying distribution is unknown or not normal. As a secondary analysis, using the fentanyl concentration data available, we examined the descriptive statistics for fentanyl concentration among fentanyl positive samples that were identified by the neural network model but missed by the drug checking technician.

Results

In total, 12,684 samples were included in our study: 6,099 (48.1%) were identified as fentanyl positive by immunoassay strips. Shown in Table 1, the large majority of samples were checked in the Vancouver Coastal Health Authority region (11,092; 87.4%). Among fentanyl immunoassay strip positive samples, the most common color and texture were purple (1,300; 23.0%) and pebbles (3,010; 40.4%), respectively, whereas for fentanyl immunoassay strip negative samples the most common color and texture were white (3,552; 53.9%) and powder (2,483; 37.7%), respectively.

Table 1. Characteristics of the study sample, stratified by fentanyl immunoassay strip results (n = 12,684).

Fentanyl immunoassay strip result
Positive Negative
n = 6,099 n = 6,585
Year
    2018 519 (8.5) 394 (6.0)
    2019 2473 (40.5) 1718 (26.1)
    2020 2723 (44.6) 3960 (60.1)
    2021 384 (6.3) 513 (7.8)
Health Authority Region
    Fraser 533 (8.7) 194 (2.9)
    Interior 359 (5.9) 445 (6.8)
    Vancouver Coastal 5161 (84.6) 5931 (90.1)
    Vancouver Island 46 (0.8) 15 (0.2)
Color
    Black 72 (1.2) 28 (0.4)
    Blue 413 (6.8) 138 (2.1)
    Brown 1173 (19.2) 769 (11.7)
    Colourless 25 (0.4) 1009 (15.3)
    Green 1229 (20.2) 143 (2.2)
    Grey 292 (4.8) 195 (3.0)
    Orange 217 (3.6) 116 (1.8)
    Pink 368 (6.0) 175 (2.7)
    Purple 1400 (23.0) 167 (2.5)
    Red 140 (6.2) 34 (0.5)
    White 380 (5.8) 3552 (53.9)
    Yellow 351 (5.8) 227 (3.4)
    Other 5 (0.1) 0 (0.0)
    Not reported 34 (0.6) 32 (0.5)
Texture
    Chunk 1471 (24.1) 567 (8.6)
    Crystal 48 (0.8) 2264 (34.4)
    Flake 18 (0.3) 68 (1.0)
    Granules 533 (8.7) 151 (2.3)
    Liquid 20 (0.3) 110 (1.7)
    Paste 145 (2.4) 49 (0.7)
    Pebble 3010 (49.4) 271 (4.1)
    Powder 704 (11.5) 2483 (37.7)
    Pressed tablet 7 (0.1) 170 (2.6)
    Residue 65 (1.1) 19 (0.3)
    Tablet (pharmaceutical) 65 (0.4) 368 (5.6)
    Other 3 (0.05) 28 (0.4)
    Not reported 23 (0.9) 37 (0.6)
Open in a new tab

Note: Categories where cells were less than 5 were recategorized into ‘Other’

Using common diagnostic statistics to comprehensively evaluate the model’s performance, we found that the neural network model outperformed results from those analyzed by a drug checking technician in metrics overall. Shown in Table 2, the neural network model achieved an F1 score of 96.4% (95% CI: 95.6–97.1) compared to 78.4% (95% CI: 76.4–80.3) by a technician. Regarding accuracy, the model achieved a score of 96.4% (95% CI: 95.6–97.1) compared to 82.4% (95% CI: 80.9–83.9) by a technician. For precision, the model achieved a score of 95.7% (95% CI: 94.5–96.7) compared to 99.4% (95% CI: 98. 8–99.9) by a technician. For recall, the model achieved a score of 97.1% (95% CI: 96.1–98.0) compared to 64.7% (95% CI: 62.1–67.3) by a technician. Lastly, as shown in Fig 3, the AUROC for the model was 99.0% (95% CI: 98.7–99.4).

Table 2. Comparing the performance of different methods for detecting fentanyl.

F1 score Accuracy Precision Recall
Neural network model 96.4 96.4 95.7 97.1
% (95%CI) (95.6–97.1) (95.6–97.1) (94.5–96.7) (96.1–98.0)
Drug checking technician 78.4 82.4 99.4 64.7
% (95%CI) (76.4–80.3) (80.9–83.9) (98.8–99.9) (62.1–67.3)
Open in a new tab

CI: confidence interval

Fig 3. Receiver-operating characteristic curve for the final neural network model for detecting fentanyl.

Fig 3

Open in a new tab

In secondary analysis, there were a total of 2,117 samples that were correctly identified as fentanyl positive by the neural network model but were incorrectly labelled as fentanyl negative by the drug checking technician. Among those, the median fentanyl concentration was 2.3% (quartile 1–3: 0–4.6%).

Discussion

In summary, our findings demonstrate that a neural network model was able to achieve highly promising and accurate results in predicting the presence of fentanyl in drug samples analyzed using FTIR data files, with an F1 score of 96.4%. Interestingly, the model was typically able to detect very low concentrations of fentanyl and analogues, with the majority of samples falling below the detection limit of FTIR spectroscopy [16, 17]. Previous data in this setting have indicated the emergence of highly potent fentanyl analogues (e.g., carfentanil) [3, 38, 39], which may be present in drug samples at very low concentrations which are not captured by trained technicians using FTIR. However, fentanyl immunoassay strips, with their high sensitivity and specificity to potent fentanyl analogues, account for this limitation of FTIR, hence the widespread use of these technologies in tandem [15, 16]. Thus, this study sheds light on the utility of a deep learning model to augment existing point-of-care drug checking technologies.

As indicated previously, many settings have adopted the use of combination FTIR spectroscopy and immunoassay strips as a drug checking intervention. Our findings show that given the high predictive performance of a neural network model for detecting fentanyl at low concentrations, it may be possible to replace fentanyl immunoassay strips with an automated model that can be used in conjunction with FTIR spectroscopy for services that already operate a spectrometer. Nevertheless, given that we only applied one feature (i.e., FTIR absorbance data) to the model to avoid subjective noise, future research should seek to explore additional characteristics of drug checking samples (e.g., appearance) that can be used to optimize the model’s prediction performance. It may also be worthwhile to retrain the model on an ongoing basis as novel psychoactive substances, including novel synthetic opioids, are identified in the highly dynamic, unregulated drug supply. Future research should also seek to examine whether a machine learning model can accurately quantify fentanyl concentration in samples, as this information may be more valuable in a fentanyl-saturated unregulated drug market.

There are some limitations to consider. The neural network model was applied to drug samples checked in BC; as such, the model may not be generalizable to other settings where the unregulated drug supply may be vastly different. For example, in settings where different cutting agents are used, equivalent amounts of fentanyl appear different with FTIR analysis. Future research with additional sources of data, such as those from outside of BC, will determine if or how our results may differ from the unregulated drug supply internationally. Given that the point-of-care technologies were limited in their ability to differentiate between fentanyl and its analogues at low concentrations, we were unable to confirm whether the model captured fentanyl analogues or if the model only captured fentanyl. Lastly, recent studies have suggested that depending on the composition and concentration of drug samples, line faintness on immunoassay strips (which were used as the reference standard for our model) may lead to subjectivity in determining the presence of fentanyl [40, 41]. Nevertheless, our study employed trained drug checking technicians to interpret the immunoassay strips and other studies have shown the immunoassay strips can perform with high sensitivity and specificity [15, 16].

Conclusion

In sum, we developed a neural network model that can accurately predict the presence of fentanyl using absorbance spectrum data files obtained from FTIR spectroscopy. The model was able to correctly detect fentanyl-positive samples below the limit of detection of the FTIR spectrometer. Our findings point to the potential of integrating this tool within drug checking services utilizing FTIR spectroscopy to improve decision making and reduce harms associated with overdose and other negative health outcomes.

Acknowledgments

We offer thanks to those individuals who participated directly in the study by having their drugs analyzed, with the hopes that this involvement will contribute to utilizable public health information, improved harm reduction care, and potentially, decreased loss of life. We would also like to thank researchers and staff at various community organizations, health authorities, and laboratory services across the province for their work in this area. Health Canada Drug Analysis Service provided confirmatory testing services; however, the findings reported here should in no way be taken as an endorsement of the specific point-of-care technologies that were used for this study.

Data Availability

All data table files containing wavenumber and corresponding absorbance for each spectrum (i.e., drug sample) and corresponding immunoassay strip results are available from the following public repository on GitHub: https://github.com/DrugCheckingBC/nn-fentanyl-prediction.

Funding Statement

The study was supported by the Health Canada Substance Use and Addictions Program (1718-HQ-000024; https://www.canada.ca/en/health-canada/services/substance-use/canadian-drugs-substances-strategy/funding/substance-use-addictions-program.html), Vancouver Foundation (https://www.vancouverfoundation.ca), and the US National Institutes of Health-National Institute on Drug Abuse (R01DA052381; https://nida.nih.gov). The content is solely the responsibility of the authors and does not necessarily represent the official views of these funding agencies. LT is supported by a Michael Smith Health Research British Columbia (MSHRBC; https://healthresearchbc.ca) Scholar Award. The funding organizations had no role in the design, data collection, analysis, or preparation of the manuscript. The co-author (RL) who is affiliated with Health Canada is not a co-investigator of the grants that funded our study.

References

Decision Letter 0

Muhammad Hanif

11 Apr 2023

PONE-D-23-02432Development of a neural network model to predict the presence of fentanyl in community drug samplesPLOS ONE

Dear Dr. Ti,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by May 26 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Muhammad Hanif

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

3. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. 

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

4. Thank you for stating the following financial disclosure: 

"The study was supported by the Health Canada Substance Use and Addictions Program (1718-HQ-000024; https://www.canada.ca/en/health-canada/services/substance-use/canadian-drugs-substances-strategy/funding/substance-use-addictions-program.html), Vancouver Foundation (https://www.vancouverfoundation.ca), and the US National Institutes of Health-National Institute on Drug Abuse (R01DA052381; https://nida.nih.gov). The content is solely the responsibility of the authors and does not necessarily represent the official views of these funding agencies. LT is supported by a Michael Smith Health Research British Columbia (MSHRBC; https://healthresearchbc.ca) Scholar Award."

We note that one or more of the authors is affiliated with the funding organization, indicating the funder may have had some role in the design, data collection, analysis or preparation of your manuscript for publication; in other words, the funder played an indirect role through the participation of the co-authors. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please do the following:

(1) Review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. These amendments should be made in the online form.

(2) Confirm in your cover letter that you agree with the following statement, and we will change the online submission form on your behalf: 

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

5. Thank you for stating the following in your Competing Interests section:  "No authors have competing interests."

Please complete your Competing Interests on the online submission form to state any Competing Interests. If you have no competing interests, please state "The authors have declared that no competing interests exist.", as detailed online in our guide for authors at http://journals.plos.org/plosone/s/submit-now 

This information should be included in your cover letter; we will change the online submission form on your behalf.

6. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. 

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

Reviewer #4: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: 1.English grammar needs corrections

2.Statistical analysis needs more details

3.analyzed using FTIR spectra files ???

4.FTIR peaks which

4. Missing the main peaks indicated this space

5.References needs to be coorected for style specific

Reviewer #2: The manuscript is a technically sound piece of scientific research with data that supports the conclusions. Experiments were performed with appropriate sample sizes. The objective of this study was to develop a neural network model to identify fentanyl and related analogues more accurately in drug samples compared to traditional analysis by technicians. They discussed that Neural network models can accurately predict the presence of fentanyl and related analogues using FTIR data, including samples with low fentanyl concentrations. Integrating this tool within drug checking services utilizing FTIR spectroscopy has the potential to improve decision making to reduce the risk of overdose and other negative health outcomes and concluded that their findings point to the potential of integrating machine learning within drug checking services utilizing FTIR spectroscopy to improve decision making and reduce harms associated with overdose and other negative health outcomes.

FTIR is being used in developed economies for quantitative analysis and its susrquent integration with machine learning methods can be helpful

Though neural network modelling is a san-statistical approach, but authors have applied required stat while comparing supplementary groups.

Manuscript was presented in a precised and concised way with a good standard of english.

Reviewer #3: Manuscript Number: PONE-D-23-02432, Reviewers Comments:

The paper presented in this study is “Development of a neural network model to predict the presence of fentanyl in community drug samples”. The paper is clearly written and well organized. The introduction and background are reasonable given the premise of the paper. Figures and tables are comprehensive and helpful. The problem statement and objectives are clearly defined and explained with the help of different instrumentations. At the end of the introduction; it would be helpful to add more information of your current study and clear statements of the objective and results of this study. Such a statement would provide a transition to the main ideas being presented. Conclusion section could be improved to better reflect the large amount of information reviewed in relation to the title/objective of the paper. In my view, the conclusions should be expanded to better summarize the overall "feel" of the main review section to give the reader a strong message. Some of the references are outdated. However, before I can recommend its publication, the authors should address the following questions

Some questions to author

1. Abstract was too long of 325 words try to summarize to maximum 250 words

2. Keywords seemed to be quite unimportant like “harm reduction” etc and few so should be revised and add more to number of six.

3. The term drug checking in the manuscript is not appropriate. Is this term was from any scientific source please add reference.

4. Line 61 to 71 sentence is too long and did not understand what author wanted to say. Please clarify and rewrite such kind of sentence.

5. Line 122 (a significant number of lives had already been claimed) is not very clear.

6. Line 143-145 the author mentioned fentanyl positive samples but did not clear how these samples were obtained and what kind of samples it was.

7. The author described in manuscript FTIR analysis but it was quite difficult to understand that FTIR only describes presence of specific peak of compound but it does not give information about quantity of compound.

8. The manuscript needs to include more description about artificial neural network.

9. Figure 2 does not explain any information about characteristics

10. Conclusion should be separate from discussion

11. No graphical representation of full factorial design was seen in the manuscript.

General Assessment

1. The method presented here is not thrillingly novel and distinctly superior.

2. The material used here is not excitingly novel.

3. The article does not have wider scope and applicability for reader of PLOS ONE.

4. The method is not the selective enough over chemically related drugs.

Reviewer #4: Following are my comments for the article titled “Development of a neural network model to predict the presence of fentanyl in community drug samples”

1. There are other similar research work published in this area and for this compound like(Chen et al., 2022; Xu et al., 2020) therefore authors should discuss whats new compared to other similar work

Chen, H., Kim, S., Hardie, J.M., Thirumalaraju, P., Gharpure, S., Rostamian, S., Udayakumar, S., Lei, Q., Cho, G., Kanakasabapathy, M.K., Shafiee, H., 2022. Deep learning-assisted sensitive detection of fentanyl using a bubbling-microchip. Lab on a Chip 22, 4531-4540.

Xu, M., Wang, C.-H., Terracciano, A.C., Masunov, A.E., Vasu, S.S., 2020. High accuracy machine learning identification of fentanyl-relevant molecular compound classification via constituent functional group analysis. Scientific Reports 10, 13569.

2. There are more sensitive techniques like LC-MS/MS and GC-MS and considered as the gold standard techniques for estimating fentanyl in blood samples due to their high sensitivity and specificity. FTIR (Fourier Transform Infrared Spectroscopy) is not typically used in diagnostic labs for fentanyl toxicity testing. FTIR is a powerful technique that is commonly used in materials science, chemistry, and forensic analysis, but it is not well-suited for the detection and quantification of small molecules like fentanyl in biological samples. The techniques such as LC-MS/MS, GC-MS, ELISA, HPLC, and CE, are more commonly used for detecting and quantifying fentanyl in biological samples such as blood, urine, and hair.

3. Manuscript is poorly written, though the information is given that the model is developed but it not properly explained. More Tables and Figures should be added for the better understanding of the model. When writing a Research article it should be written in a way that it can be replicated.

4. There is no description about the drug structure, FTIR spectra etc .

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Dr. Asad Majeed Khan

Reviewer #3: No

Reviewer #4: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2023 Jul 13;18(7):e0288656. doi: 10.1371/journal.pone.0288656.r002

Author response to Decision Letter 0

1 Jun 2023

Response to Reviewer 1:

We thank Reviewer 1 for their succinct comments about our manuscript. We have responded to Reviewer 1’s individual comments using numbered bullets below.

1. We thank Reviewer 1 for the comment regarding the need for grammatical corrections. The updated version of the manuscript has received thorough proofreading and we feel it is free of grammatical errors.

2. We thank Reviewer 1 for their suggestion that the “statistical analysis needs more details.” In light of this comment, we have included the following information that further describes the statistical analysis portion of methods:

In assessing our model's performance, we utilized the prediction value which is a continuous variable ranging from 0 to 1. We applied a rounding procedure to transform this continuous output into a binary classification to align with our label values (the results from the corresponding fentanyl test strip for that sample). Specifically, prediction values greater than or equal to 0.5 were rounded up to 1, indicating a positive fentanyl result, while values less than 0.5 were rounded down to 0, indicating a negative result. These rounded prediction values were then compared against our label values, which represent the binary outcomes of the fentanyl test strip. Using these results, we calculated several diagnostic metrics, including F1 score, accuracy, precision, recall, and area under the receiver-operating characteristic curve (AUROC). An F1 score describes a model’s accuracy, taking the precision (correctly identified positives among test positives) and recall (correctly identified positives among true positives) into account.

3. We thank Reviewer 1 for raising the issue of ambiguity regarding how we speak of file types. We have removed reference to “FTIR spectra files” and replaced it with the following:

…(jcamp-dx) encoded FTIR data files were converted to data tables…

4. We apologize to Reviewer 1 and the Editor that we are unable to determine what Reviewer 1 meant by the comment, “4. FTIR peaks which 4. Missing the main peaks indicated this space.” We hope that with the revisions in the updated manuscript, this comment has been sufficiently addressed.

5. We thank Reviewer 1 for highlighting the need to ensure consistency in reference style. As a part of our proofreading process, we have ensured consistency in reference formatting.

Response to Reviewer 2:

We thank Reviewer 2 for their time and generous comments regarding our manuscript. Reviewer 2 commented that “the manuscript is a technically sound piece of scientific research” and raised no concerns or need for any edits.

Response to Reviewer 3:

We thank Reviewer 3 for their comments about our manuscript. We have responded to Reviewer 3’s numbered comments below:

1. We appreciate Reviewer 3’s comments about the abstract being too long. While the abstract length is within the limit outlined in the PLOS ONE submission guidelines, we are happy to shorten it if the Editor deems it necessary.

2. We thank Reviewer 3 for noting additional keywords are beneficial. We do, however, feel that the keyword “harm reduction” is relevant to this article. We have added the additional keywords of “deep learning” and “FTIR spectroscopy.”

3. We thank Reviewer 3 for their comment regarding the use of the term “drug checking” in the manuscript. Drug checking is the internationally recognized term to refer to the harm reduction intervention of analyzing illicit drugs to determine what they contain before consumption. We feel the use of this specific term is adequately defined with supporting references in the manuscript:

In response, drug checking services have been implemented as an important harm reduction intervention for people who use drugs to chemically analyze their substances and receive fact-based information and consultation regarding the compounds detected in their sample [4–6]. Not a new concept, the vast majority of drug checking services have previously focused on drugs used in party and festival settings (e.g., psychedelics, stimulants) [7,8], and less is known about drug checking in the context of the synthetic opioid-driven overdose epidemic. While limited, recent studies have started to show a positive impact of drug checking, including increased engagement in overdose risk reduction practices following the use of the service in some communities [5,9–11].

4. Unfortunately, we are unable to discern which specific sentence Reviewer 3 is referencing in their 4th comment, as the line numbers referenced do not contain sentences, but instead are the manuscript keywords. We apologize to Reviewer 3 and hope that during the revisions stage their concern has been adequately addressed otherwise.

5. We thank Reviewer 3 for their comment regarding the clarity of the use of the term “claimed” to refer to loss of life. We have amended the sentence to the following:

By the time a public health emergency was declared in 2016, a significant number of lives had already been lost to illicit drug toxicity, largely due to the widespread presence of fentanyl and other novel synthetic opioids in the unregulated drug supply [29].

6. We appreciate Reviewer 3’s comment about the clarification needed on how fentanyl-positive samples were obtained. The manuscript has been updated to provide further detail:

Further details about the drug checking service and the use of these technologies, which have been operating since October 2017, have been described elsewhere in detail [15,20]. Briefly, individuals in harm reduction sites anonymously provided an approximately 5 mg drug sample to a trained drug checking technician and information about what they believed the sample to contain [34]. The drug sample was then analyzed with FTIR and immunoassay strips, with results being immediately available to the person who provided the sample. Results from FTIR analysis include compounds detected in abundant amounts (typically those 5% by weight and above), including both active ingredients (e.g., fentanyl, heroin, methamphetamine) and inert cutting agents (e.g., mannitol, inositol, lactose) [6,17,20].

7. We thank Reviewer 3 for their comment about details of FTIR analysis and how it does not give information about quantity of specific compounds. To address this, we have added information to the methods section about the results generated by FTIR analysis at point of care:

The drug sample was then analyzed with FTIR and immunoassay strips, with results being immediately available to the person who provided the sample. Results from FTIR analysis include compounds detected in abundant amounts (typically those 5% by weight and above), including both active ingredients (e.g., fentanyl, heroin, methamphetamine) and inert cutting agents (e.g., mannitol, inositol, lactose) [6,17,20].

8. We thank Reviewer 3 for the comment about how the manuscript would benefit from further description about the artificial neural network. To address this comment, we have restructured and added additional text to the “Prediction methods and model selection” subsection of the methods section:

The goal of the study was to use anonymous, point-of-care drug checking data obtained from FTIR spectra to predict the presence of fentanyl and related analogues in drug samples. We implemented an artificial neural network-based model, which was composed of six layers in the following order (Fig 1):

1) an input layer

2) a one-dimensional convolution layer with eight filters, a kernel size of 12, and a rectified linear unit (ReLU) activation function

3) a dropout layer with a 50% dropout rate

4) a one-dimensional max pooling layer with a pool size of two

5) a dense layer with output dimension 32 and a ReLU activation function

6) a dense output layer with output dimension one and a sigmoid activation function.

The model was trained on 80% of the data and 20% of the data was reserved for testing the model's accuracy. At each epoch, the accuracy (i.e., number of correct predictions among total number of predictions) was measured on the test set, and the model with the highest accuracy in the test set was kept.

9. We thank Reviewer 3 for their suggestion about how to improve Fig 2 (now Fig 3). We feel that the ROC curve presented in Fig 2 sufficiently describes the performance classification characteristics that it seeks to depict. If the Editor feels that further description of characteristics is warranted, we can address this comment in another fashion.

10. We thank Reviewer 3 for suggesting how the conclusions should be separated from the discussion section. We have separated the two sections in the manuscript.

11. We thank Reviewer 3 for their comment regarding a graphical representation of a full factorial design. We have included a graphical representation of the model (Fig 2) and we hope that this graphic helps illustrate the final model developed over the course of the present study.

Response to Reviewer 4:

1. We thank Reviewer 4 for identifying previous research involving machine learning to identify fentanyl in various forms. Unfortunately, these articles are not related to the identification of fentanyl in illicit drug samples, but instead one is related to biological fluids and the other is related to chemical structure analysis. We feel these previous articles do not detract from the novel findings of our research, as it remains the first in published literature to describe the use of deep learning to detect fentanyl in illicit drug samples directly.

2. We thank Reviewer 4 for sharing their expertise regarding the use of different (and more sensitive) technologies to detect fentanyl in biologic samples. We have included new text that describes the advantages and disadvantages of using such technologies in a point-of-care drug checking setting to address the unregulated drug supply. We have also provided further information on the FTIR.

Drug checking can be conducted using a range of technologies, offering both benefits and challenges in terms of accuracy, timing, cost, portability, ability to detect a range of compounds, preparation, and required human resources [4,6,12,13]. For example, gas chromatography/mass spectrometry (GC-MS) is highly sensitive, but is expensive, time-consuming, and requires a trained laboratory technician to operate the machine [6,14]. In contrast, immunoassay strips are inexpensive, easy to use, portable, and can provide results in less than five minutes, but they are unable to detect a range of compounds and can only produce qualitative results (i.e., presence, absence) [15,16]. Fourier-transform infrared (FTIR) spectroscopy is being implemented across North America for drug checking given the ease and speed of sample preparation and analysis, portability, and ability to detect a range of compounds [15,16]. However, recent validation studies have indicated that FTIR spectroscopy can only detect compounds above a certain limit; for fentanyl and some fentanyl analogues, that limit is 3-10% quantity by weight, depending on the setting and the presence of specific cuts and buffs [16,17]. Given that some analogues (e.g., carfentanil) can cause toxicity at very low concentrations [18,19], failing to detect these compounds when present is a substantial concern when using FTIR spectroscopy on its own.

…

Further details about the drug checking service and the use of these technologies, which have been operating since October 2017, have been described elsewhere in detail [15,20]. Briefly, individuals in harm reduction sites anonymously provided an approximately 5 mg drug sample to a trained drug checking technician and information about what they believed the sample to contain [34]. The drug sample was then analyzed with FTIR and immunoassay strips, with results being immediately available to the person who provided the sample. Results from FTIR analysis include compounds detected in abundant amounts (typically those 5% by weight and above), including both active ingredients (e.g., fentanyl, heroin, methamphetamine) and inert cutting agents (e.g., mannitol, inositol, lactose) [6,17,20].

As well, we have described in more detail the use of FTIR spectroscopy more generally, in addition to including a new figure that depicts a typical FTIR spectrum (per Reviewer 4’s fourth comment).

3. We thank Reviewer 4 for their comment about providing further details on the methods to allow for replicability of the manuscript. We feel the manuscript, particularly with the new amendments in the course of the revisions process, have provided sufficient instruction to PLOS ONE readers should they wish to replicate the study, including new text and restructuring the “Prediction methods and model selection”. Upon further review, should the Editor feel further enhancements are necessary, we defer to their decision.

4. We agree with Reviewer 4’s comment that the manuscript lacks description of the structure of the fentanyl molecule and what an FTIR spectrum is and looks like. To address this issue, we have added an additional figure to the manuscript that depicts what a typical FTIR spectrum of a sample in our analysis looks like. The following text has also been added to the manuscript to more thoroughly describe FTIR spectra.

An illustrative example of what a drug checking technician analyzes (and attempts to determine if fentanyl is present) is displayed in Fig 1.

[Figure caption] Fig 1. Sample drug checking Fourier-transform infrared spectra. Drug checking samples (blue) are overlaid with the fentanyl hydrochloride spectrum (black). The drug sample in A is an illicit opioid mixture containing fentanyl, caffeine, and mannitol. The drug sample in B is methamphetamine with no fentanyl present. The red arrows indicate characteristic peaks of fentanyl hydrochloride that are recognizable when present in drug mixtures.

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file. (6.7MB, docx)

Decision Letter 1

Muhammad Hanif

3 Jul 2023

Development of a neural network model to predict the presence of fentanyl in community drug samples

PONE-D-23-02432R1

Dear Dr. Ti,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Muhammad Hanif

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

I have gone trough the revision submitted by the authors and found that all the points have been addressed

Reviewers' comments:

Acceptance letter

Muhammad Hanif

5 Jul 2023

PONE-D-23-02432R1

Development of a neural network model to predict the presence of fentanyl in community drug samples

Dear Dr. Ti:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Muhammad Hanif

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    Attachment

    Submitted filename: Response to Reviewers.docx

    Click here for additional data file. (6.7MB, docx)

    Data Availability Statement

    All data table files containing wavenumber and corresponding absorbance for each spectrum (i.e., drug sample) and corresponding immunoassay strip results are available from the following public repository on GitHub: https://github.com/DrugCheckingBC/nn-fentanyl-prediction.

    Articles from PLOS ONE are provided here courtesy of PLOS

    RESOURCES