Artificial intelligence reinforced upconversion nanoparticle-based lateral flow assay via transfer learning

Highlights

• •The first report introduces the concept of transfer learning into the quantitative detection of upconverted nanomaterials.
• •Designed an AI-based solution strategy (including unique feature engineering and transfer learning) for quantitative detection of UCNP-LFAs in small datasets without special preprocessing and requiring only a small amount of data.
• •Experiments on the accuracy of eight AI models trained by transfer learning in quantitative detection of methamphetamine (MET) and morphine (MOP), compared the accuracy of models without transfer learning with that of traditional classification algorithms, and an in-depth study of the effect of image noise on quantitative methods detection results.
• •An efficient, universal, portable commercial IoT device for upconversion luminescence quantitative detection is practically developed by deploying the trained transfer learning model in a local IoT device. The device is 100mm×120mm×74mm and weighs only 351.2g, and can infer highly accurate results in real-time in only 20s.

Abstract

The combination of upconverting nanoparticles (UCNPs) and immunochromatography has become a widely used and promising new detection technique for point-of-care testing (POCT). However, their low luminescence efficiency, non-specific adsorption, and image noise have always limited their progress toward practical applications. Recently, artificial intelligence (AI) has demonstrated powerful representational learning and generalization capabilities in computer vision. We report for the first time a combination of AI and upconversion nanoparticle-based lateral flow assays (UCNP-LFAs) for the quantitative detection of commercial internet of things (IoT) devices. This universal UCNPs quantitative detection strategy combines high accuracy, sensitivity, and applicability in the field detection environment. By using transfer learning to train AI models in a small self-built database, we not only significantly improved the accuracy and robustness of quantitative detection, but also efficiently solved the actual problems of data scarcity and low computing power of POCT equipment. Then, the trained AI model was deployed in IoT devices, whereby the detection process does not require detailed data preprocessing to achieve real-time inference of quantitative results. We validated the quantitative detection of two detectors using eight transfer learning models on a small dataset. The AI quickly provided ultra-high accuracy prediction results (some models could reach 100% accuracy) even when strong noise was added. Simultaneously, the high flexibility of this strategy promises to be a general quantitative detection method for optical biosensors. We believe that this strategy and device have a scientific significance in revolutionizing the existing POCT technology landscape and providing excellent commercial value in the in vitro diagnostics (IVD) industry.

Graphical Abstract

Graphical Abstract

.

1. Introduction

In recent years, there have been outbreaks of sudden global public health problems such as COVID-19 [1], [2], [3], Middle East respiratory syndrome (MERS) [4], and Ebola virus disease (EVD) [5]. Additionally, with the increasing global trade, urbanization, and environmental changes that continue to aggravate the risk of transmission, the development of instantaneous rapid detection of infectious diseases is of great significance for the prevention and control of global public health. At this stage, mainstream POCT is based on a fluorescence immunochromatography assay with high sensitivity and specificity. Quantitative detection is achieved by applying fluorescent markers (such as UCNPs [6], [7], [8], quantum dots (QDs) [9,10], fluorescent microspheres (FMSs) [11,12], and organic dyes [13]) to lateral flow assays (LFAs) using radiometric strategies. However, the QDs, FMSs, and organic dyes typically require ultraviolet (UV) and visible light excitation, a process that comes with the challenges of photobleaching and phototoxicity. UCNPs are new fluorescent probes that convert near-infrared excitation light into high-energy visible or ultraviolet light through the anti-Stokes process. Their near-infrared excitation properties can effectively avoid interference from background fluorescence and co-excitation. UCNPs also offer unique advantages, such as better optical and chemical stability, luminescence tunability, resistance to photobleaching, and low cytotoxicity. This makes the use of upconversion fluorescence resonance energy transfer (UC-FRET) technology with UCNPs an energy donor, highlighting promising applications in POCT, biosensing, and medical diagnostics [14].

In the early 21^st century, the first application of UCNPs in LFAs was reported by Niedbala et al. [15] and Hampl et al. [16], who successfully detected 103 org/mL Escherichia coli O157:H7 in a sample on a medium and 10 pg hCG in a 100-μl sample. This marked the beginning of the "Age of Discovery" for UCNPs. Quantitative assays can provide more accurate and important information than stereotypic analysis. For UCNP-LFAs, quantitative detection is achieved by measuring the fluorescence intensity on the control line (CL) and test line (TL). Qu et al. [17] developed fluorescent lateral flow immunochromatographic assays (FLFIAs) based on UCNPs to achieve rapid quantification of Brucella using changes in the fluorescence signal ratio of TL/CL, and obtained satisfactory detection limits in different spiked samples. The results demonstrate the use of UCNPs for quantitative assays with high specificity, reproducibility and stability. Hu et al. [18] utilized UCNP-LFAs for drug-field detection, a similar approach to our study. They also selected methamphetamine (MET) and morphine (MOP) in simulated saliva samples as targets for quantitative detection and achieved a sensitivity of 10 and 5 ng/mL, respectively, for MET and MOP detection under 15 min using the TL/CL ratio on LFAs. UCNP-LFAs showed faster detection efficiency and more accurate quantitative results than liquid chromatography-mass spectrometry (LC-MS) in simulated saliva samples. However, the low luminescence efficiency and noise interference still significantly hinder various UCNP-based detection techniques from the application of theoretical laboratory studies in real-world practical situations. In the existing research, the luminescence efficiency of UCNPs has been continuously improved from the material itself [19], [20], [21], [22], [23]; however, the design of quantitative detection platforms based on UCNPs has been steadily integrated, made more intelligent, and miniaturized [24], [25], [26], [27], [28], [29], [30], [31], [32].

In summary, the above detection methods have a strong identification specificity and high detection sensitivity. However, optical signals are inevitably subject to adverse factors, such as instrument parameters, changes in the field environment, and interference from background light scattering in complex sample matrices, resulting in enhanced image noise and reduced detection sensitivity. This causes devastating interference superimposed on the quantitative results of fluorescence detection. This interference cannot be decoupled using an optical analysis method. Although some techniques [30] can eliminate some of the impacts, quantitative detection accuracy, and instrument portability, there is an irreconcilable contradiction, which still does not meet the practical needs of rapid detection. How can the luminescence intensity and efficiency of the UCNPs be enhanced? How can the accuracy, detection limit, and detection time of optical probe signal detection in POCT be enhanced for use as portable equipment in a complex detection environment? These two critical issues are of great relevance and commercial value. In our previous study, we successfully enhanced the luminescence intensity and efficiency of UCNPs by constructing up-conversion nanomaterials based on mesoporous silica-encapsulated core-shell structures to reduce their quenching effect [33], [34], [35]. Furthermore, our research group developed a variety of small quantitative devices based on upconversion luminescence and 5G technology for different detectors [36], [37], [38]. Considering these factors, we believe that the best combination of UCNP-LFAs and powerful AI technology is available for POCT. However, little has been reported to date.

Transfer learning [39], [40], [41] is an essential approach in AI. The research challenge in the biomedical field, unlike in other areas, is the inability to obtain sufficient valid medical data. Although much research has been aimed at combining transfer learning with bio-detection sensors, it relies on a large amount of high-quality data labeling and robust computational power devices. The introduction of transfer learning into the biomedical field can solve the conflicts between large amounts of data and small amounts of annotation, between large amounts of data and low computational costs, and between personalized application scenarios and pervasive models. It improves its generalization performance by considering empirical parameters learned in a one-dimensional space and using them in another domain [42]. In recent years, there has been much literature [43], [44], [45], [46], [47], [48] on combining transfer learning with biological detection. Kermany et al. [49] developed AI systems based on transfer learning for diagnosing two basic classes of eye diseases and pneumonia, which is the first time in the world that massive amounts of well-labeled, high-quality data for transfer learning were used to achieve ultra-high accuracy diagnoses that can completely surpass the accuracy of human doctors.

We foresee that this research can be extended to any optical-probe-based biosensors. By combining UCNP-LFAs with transfer learning for use in optical biosensors, we completely get rid of the complicated preprocessing and image enhancement process, with a vast amount of worker-labeled data being through traditional methods. It simplifies the detection process and improves detection efficiency while reducing the hardware computing power requirements. However, owing to transfer learning, better AI models that are easier to deploy in local IoT devices are available. In terms of safety and sustainability, the device is less harmful to humans and the environment, and it can continuously deploy the latest training models through continuous updates to achieve device sustainability. These properties address the issues of real-time local response, reliable service, and data privacy raised by on-site sensor detection and POCT.

Specifically, 1. The first report introduced the concept of transfer learning for the quantitative detection of upconverted nanomaterials. 2. We designed an AI-based solution strategy (including unique feature engineering and transfer learning) for the quantitative detection of UCNP-LFAs in small datasets without special preprocessing and requiring only a small amount of data. 3. Experiments were performed to determine the accuracy of eight AI models trained by transfer learning in the quantitative detection of MET and MOP (the sensitivities of the MET and MOP were 1 ng/mL and 0.1 ng/mL), the accuracy of the models without transfer learning were then compared to those of traditional classification algorithms and an in-depth study on the effect of image noise on quantitative method detection results conducted. 4. An efficient, universal, portable commercial IoT device for upconversion luminescence quantitative detection was developed by deploying a trained transfer-learning model in a local IoT device (Fig. 5). The device was 100 mm $\times$ 120 mm $\times$ 74 mm and weighed only 351.2 g, with the capability of inferring highly accurate real-time results in only 20 s.

Fig. 5.

Actual development of a transfer learning-based quantitative detection system for UCNP-LFAs. (a) Mobile phone application matched with the device, which can control the self-test and operation of the device through communication with the device, and can display the detection results in real-time, (b) UCNP-LFAs quantitative detection device based on transfer learning, which is portable and highly accurate due to the introduction of transfer learning, and does not need to rely on computing units with high computing power, with data localization being able to solve data privacy problems, and (c) the actual internal structure of the assay instrument.

2. Experimental section

2.1. Data preparation for transfer learning

Image sequences containing CL and TL fluorescence excitations on the LFA sensor (Fig. 1c) were obtained by the luminescence capture sensor (Fig. 1a and b). Due to the camera fixation, we can easily obtain the region of interest (ROI) for CL and TL fluorescence excitation. Subsequently, we stitched the two-strip ROI together using a preset program (Fig. 1d). The above process was repeated using image sequences that corresponded frame by frame. Finally, MET, MOP, and two small datasets for proof-of-concept were constructed separately. The data set for quantitative detection of MET contains four standard concentration gradients of 1 ng/mL, 2.5 ng/mL, 10 ng/mL and 20 ng/mL; with 37 images (240 px $\times$ 240 px) available for each concentration making a total of 148 raw data, of which 7 images were selected for each concentration for a total of 28 as the MET test data set. Similarly, the data set for the quantitative detection of MOP contains four standard concentration gradients of 0.1 ng/mL, 1 ng/mL, 10 ng/mL and 100 ng/mL (0.1 ng/mL is the lower concentration), 29 images (240 px $\times$ 240 px) for each concentration, a total of 116 raw data, of which 9 images were selected for each concentration, for a total of 36 images as the MOP test data set (Fig. 2).

Fig. 1.

The proposed implementation of the flow of artificial intelligence-based quantitative upconversion luminescence detection under small samples. The method enables rapid upconversion luminescence quantitative detection with high accuracy, ultra-sensitivity, and strong noise tolerance. (a) The actual developed portable device for upconversion luminescence quantitative detection. (b) Diagram of the hardware structure of the device. (c) Schematic diagram of UCNP-LFAs. (d) Implementation scheme for constructing a training database using a small number of samples. (e) Implementation of the data augmentation process. (f) The workflow for implementing transfer learning into the pre-trained network. (g) Deployment of trained AI models to local devices. (h) Majority Voting strategy, which aims at absolute accuracy of the final prediction results. (i) Fast transfer of prediction results to PC or mobile interfaces through real-time inference.

Fig. 2.

Data composition of the small dataset constructed for transfer learning. Our proposed feature engineering was performed for fluorescence images of each concentration gradient in both MET and MOP datasets.

2.2. Feature engineering and data augmentation

Feature engineering plays a crucial role in determining AI accuracy. Good feature selection determines the upper limit of the accuracy of the AI model. Owing to the high cost of dataset label collection, we propose a feature-engineering method applicable to upconverted fluorescence detection through extensive preliminary research. Specifically, the raw data were transformed in the preprocessing stage using seven transformations: image stitching, polar coordinate conversion, 10 % Gaussian noise, rotation, Gaussian smoothing, horizontal flip, and image RGB channel extraction of the G channel. Meanwhile, to enhance the model generalization ability to avoid model overfitting, two data augmentation methods of random scaling (scaling factor: 0.9 to 1.1) and random cropping (cropping range: -30 to 30 px) will be used in the training process to avoid overfitting (not included in the training data). Finally, the training dataset for all four concentrations of quantitative detection of MET was expanded to 2,520 sheets, and the training dataset for quantitative detection of MOP was expanded to 1,920 sheets (Fig. 2).

2.3. Models for UCNP-LFAs using transfer learning

We selected ResNet50 [50], ResNet101 [51], VGG16 [52], VGG19, GoogleNet [53], MobileNet V2 [54], AlexNet [55], and DenseNet201 [56], making a total of eight classical AI models (Table 1 and Fig. 3). The pre-trained models that can be used for transfer learning and their related properties are listed in Table 1. It can be seen that each of these pre-trained models has its characteristics in terms of network depth, convolution method, and parameter size. For example, DenseNet201 has a network depth of up to 201 layers and VGG19 has a maximum parameter size of 144 million. Most of these pre-trained models were trained based on the ImageNet database with more than one million images. They can classify input images into up to 1,000 object classes, covering almost all common objects, plants, and animals in life. Transfer learning fine-tunes deeper layers in the network by training these pre-trained models based on new datasets. Fine-tuning the network is usually faster and easier than building and training a new network, and allows learning features specific to the new self-built dataset, where the network depth is defined as the maximum number of sequential convolutional or fully connected layers in the path from the input layer to the output layer. The inputs to all the models were RGB images.

Table 1.

Properties of the 8 pre-trained models selected for the experiments.

Model	Depth	Size	Parameters (Millions)	Image Input Size
ResNet50	50	96 MB	25.6	224 × 224
ResNet101	101	167 MB	44.6	224 × 224
VGG16	16	515 MB	138	224 × 224
VGG19	19	535 MB	144	224 × 224
GoogleNet	22	27 MB	7	224 × 224
MobileNet V2	53	13 MB	3.5	224 × 224
AlexNet	8	227 MB	61	227 × 227
DenseNet201	201	77 MB	20	224 × 224

Note: These 8 pre-trained networks have different network depths, network sizes, network parameter scales, and input image sizes, and they cover almost all forms of current deep learning models. The results obtained from the validation experiments on these 8 pre-trained models with small samples are generalizable.

Fig. 3.

Specific implementation process of transfer learning. The pre-trained models have consumed significant time and computational resources in building neural networks. Here, pre-trained models have already learned rich feature representations based on a large number of images [57,58] and transfer learning is able to transfer powerful skills that have been acquired to relevant problems. Compared to using randomly initialized training fine-tuned networks, transfer learning is faster and simpler than training models from scratch using randomly initialized weights.

2.4. Transfer learning model configuration and training

In Fig. 3, the implementation process is roughly divided into loading data, loading the pre-trained network, transfer learning (fixing the initial layer and replacing the final layer), training the network, and evaluating the network after transfer learning. The fixed initial layer of transfer learning means that the weight parameters of the shallow layer are fixed by setting the learning rate of the shallow network layer to 0. However, the parameters of the fixed network layers are not updated during the training process, and fixing the weights of multiple initial layers can significantly speed up network training because the gradients of the fixed network layers no longer need to be calculated (Table 3). However, fixing shallower network layers also prevents these layers from overfitting the dataset used for training because of the small size of the upconverted light-emitting dataset. The final replacement layer of transfer learning, the convolutional layer of the network, extracts the image features used by the last learnable layer and the final classification layer to classify the input image. In most models, the final layer with learnable weights is the fully connected layer. This fully connected layer was replaced with a new fully-connected layer, where the number of outputs was equal to the number of classes in the new dataset.

Table 3.

Comparison of the accuracy of the 3 traditional classification methods in the two test sets of MET/MOP respectively.

Model Database	SVM	KNN	Random Forest*
MET	53.57%	28.57%	32.14%
MOP	55.56%	33.33%	30.56%

*The number of random forests containing decision trees is 20.

Specifically, we trained the model using a single NVIDIA RTX 3080 graphics card, while setting the learning rate of the first 10 initial layers to zero. The Adam optimization algorithm was used to optimize the network parameters to minimize the loss function (the default parameters were set for all eight models). Ten batch samples were used along with 40 training rounds. Because the model using transfer learning converged on the original data, it was necessary to set a smaller learning rate (the initial learning rate was 3e-4) and use a learning rate decay strategy. The learning rate is reduced by a factor of 0.2 every five rounds, until the final 40th round.

2.5. Majority Voting strategy

The biomedical field has extremely stringent requirements for the accuracy and reliability of the results. To further improve robustness and accuracy, the chance error caused by a single detection picture should be reduced and the characteristics of optical sensors should be combined. In the actual curbside detection, for each detection strip inserted into the luminescence capture device, the device automatically captures images in a time series, and through the image preprocessing process, the CL-and TL-excited fluorescence images are stitched into five pictures to be detected according to the time series (Fig. 1). These five images were then fed into the trained network. Eventually, if a category receives more than half of the votes, it is predicted to belong to that category; otherwise, the prediction is rejected. Using this simple and effective judgment strategy, extremely accurate detection results can be obtained within 20 s in a practical and complex test environment (Fig. 1h). If continuous data (e.g., predicted concentration values between 1 and 10 ng/mL) need to be predicted, only the regression layer needs to be included at the end of the network to fit the regression model.

3. Results and discussion

3.1. Characterization of upconversion fluorescent probes

The preparation flow of the upconversion fluorescent probe is shown in Fig. 4a. After preparation of upconversion fluorescent probes (Supplementary Methods. A, Supplementary Methods. B), we performed transmission electron microscopy (TEM) (Fig. 4b and c), X-ray diffraction (XRD), and upconversion luminescence on the crystal structure of the UCNPs. The XRD (Fig. S2) and upconversion luminescence (UCL) spectrograms were obtained (Fig. S3). The particle size distribution (Table S1) and zeta potential of UCNPs@SiO₂, UCNPs@SiO₂-NH₂, and UCNPs@SiO₂-COOH were characterized using dynamic light scattering (DLS) (Fig. S4 and Table S2). Finally, we succeeded in preparing UCNPs with homogeneous size, good dispersion, and green fluorescence, and modified their surfaces to obtain biocompatible upconversion fluorescent probes on this basis.

Fig. 4.

(a) Schematic illustration of the synthesis process of UCNPs@SiO₂ labeled with MET/MOP-MAbs. TEM images of (b) NaYF₄:Yb, Er@NaYF₄(UCNPs) and (c) UCNPs@SiO₂. It can be seen that the prepared UCNPs have a hexagonal phase structure with homogeneous size, which is consistent with the results obtained by XRD. Moreover, the prepared UCNPs did not show agglomeration, which indicates that the synthesized UCNPs have good dispersion.

3.2. Design of a transfer-learning-based system for the quantitative detection of UCNP-LFAs

The upconversion fluorescence detector consists of a 980 nm laser, custom guide, filter, CMOS camera, stepper motor, microcontroller unit, AI acceleration module, and USB interface (Fig. 5c). After inserting the lateral flow chromatography strips into the custom guide, the stepper motor drags the lateral flow chromatography strips at a constant speed and direction (Fig. S1). Simultaneously, the 980 nm laser emits near-infrared light through the lens to form a rectangular focal line of 3 mm × 1 mm to excite the UCNPs captured on the TL and CL, which results in green fluorescence. After filtering out the impurity light, the CMOS camera captured a green fluorescent image. Following simple image pre-processing, the final concentration of the substance to be measured can be calculated using an AI model deployed in the device for transfer learning. Finally, the test results can be displayed on a screen or uploaded to a smartphone via a communication interface such as Bluetooth/USB to create electronic medical documents. In this study, we developed a smartphone application for use with this detection device (Fig. 5a). In conclusion, the entire system is less harmful to the environment and human body, simple to operate, and can be applied to home testing. Stabbing reduces the burden on large numbers of people visiting central hospitals and improves the efficiency of scarce medical resource utilization.

3.3. Experimental results using transfer learning

As shown in Fig. 6, the accuracy curves, loss curves, and test results during the training of 8 (ResNet50, ResNet101, VGG16, VGG19, GoogleNet, MobileNet V2, AlexNet, and DenseNet201) models based on transfer learning on both MET/MOP datasets show a confusion matrix. It can be seen from the figure that the accuracy rate of the training set gradually increases during the training process until it reaches a high accuracy; the loss curve also decreases steadily, and the accuracy and recall rate of the confusion matrix is maintained at a high level.

Fig. 6.

Experimental results of transfer learning for the 8 transfer-learning-based models on 2 small sample datasets of MET/MOP. (a) and (b) show the changes in the accuracy rate, change in the loss function and confusion matrix of the test results during training of the MET dataset and MOP dataset, respectively.

In the cross-sectional comparison of the performance of the eight models (Table 2), the transfer-learning-based DenseNet201 network achieved an impressive 100 % accuracy on both the MET and MOP test datasets, which is certainly an encouraging result. The accuracy of the VGG16 and VGG19 models was high in both datasets, especially in the MOP dataset, where the accuracy was 100 %. The accuracy was slightly lower than that of DenseNet201, VGG16, and VGG19, but they all remained high. At the same time, it can be seen that GoogleNet, MobileNet V2, and AlexNet are three lightweight models, which are not as good as the other five models in terms of the general characterization ability owing to the network depth and parameter scale, and the small difference between the classifications in the dataset based on the up-transferred luminescence, resulting in a prediction accuracy below 90 %.

Table 2.

Accuracy of 8 AI models in 4 comparative experiments on different test datasets.

	Model Database	ResNet50	ResNet101	VGG16	VGG19	GoogleNet	MobileNetV2	AlexNet	DenseNet201
Using Transfer Learning	MET	96.43%	92.86%	92.86%	96.43%	60.71%	85.71%	82.14%	100.00%
	MOP	94.44%	88.89%	100.00%	100.00%	86.11%	55.56%	75.00%	100.00%
Using Transfer Learning(10% Gaussian noise)	MET	96.43%	96.43%	96.43%	85.71%	64.29%	64.29%	71.43%	85.71%
	MOP	94.44%	86.11%	91.67%	94.44%	88.89%	75.00%	69.44%	83.33%
Using Transfer Learning(20% Gaussian noise)	MET	92.86%	82.14%	92.86%	92.86%	53.57%	57.14%	53.57%	75.00%
	MOP	88.89%	83.33%	88.89%	94.44%	77.78%	69.44%	52.78%	88.89%
Using Transfer Learning(30% Gaussian noise)	MET	82.14%	78.57%	89.29%	71.43%	53.57%	67.86%	57.14%	64.29%
	MOP	86.11%	80.56%	41.67%	86.11%	58.33%	61.11%	41.67%	80.56%
No Transfer Learning	MET	75.00%	82.14%	57.14%	53.57%	67.86%	75.00%	71.43%	78.57%
	MOP	77.78%	66.67%	58.33%	50.00%	72.22%	38.89%	63.89%	33.33%

The above experimental results show that the accuracy of the quantitative detection results based on upconversion luminescence can be well resolved using our proposed transfer learning solution combined with AI models, even in very small training samples. This method is also applicable to the quantitative detection of other optical biosensors. In addition, the classification results and prediction percentages for all test set images on both the MET/MOP datasets can be intuitively derived from this conclusion. (Figs. S5 and S6)

3.4. Experimental results without using transfer learning

The purpose of this experiment is to further verify that transfer learning is the main factor for the substantial increase in accuracy, and we took eight classical models as examples (Fig. 7 and Table 2). None of the eight models was pre-trained in any dataset, and all parameters were initialized with random assignments. The results showed that the accuracy of the training set of the models without transfer learning can only be maintained at a low level. The loss curve decreases slowly, particularly in VGG16 and VGG19, where the loss curve is almost horizontal during the entire training process. The accuracy and recall of the confusion matrix were low.

Fig. 7.

Experimental results of transfer learning for the 8 unused transfer learning models for 2 small sample datasets of MET/MOP. (a) and (b) show the changes in the accuracy rate and loss function, and the confusion matrix of test results during training of the MET dataset and MOP dataset.

In conclusion, AI models without transfer learning cannot solve the contradiction between the lack of training samples and striving for highly accurate results in the biomedical field under sample scarcity. In addition, the classification results and prediction percentages for all test set images on both MET/MOP datasets intuitively led to this conclusion. (Figs. S7 and S8)

3.5. Comparison with traditional classification algorithms

Three traditional classification algorithms were selected: SVM [59,60], KNN [61] and random forests [62]. The specific implementation scheme is as follows: A histogram of oriented gradient (HOG) features is extracted for each image in the training set, and the obtained HOG features are trained for multi-classification learning using the three classification algorithms. In the prediction session, HOG features were extracted from the predicted images and imported into the corresponding algorithms to obtain the classification results. As shown in Table 3, the three traditional methods were separately applied to the two MET/MOP datasets. In the MET dataset, SVM performed the best with 53.57% accuracy, and in the MOP dataset, SVM still performed the best with 55.56% accuracy, but compared poorly with the network applying transfer learning.

In general, the three classical classification algorithms, SVM, KNN, and random forest, have low prediction accuracy (Figs. S9 and S10), which cannot solve the contradiction between the lack of training samples and the high accuracy of results in the biomedical field. Second, the three classification algorithms could not cope with the noise interference generated by the actual complex field detection environment; and the experiments revealed their high sensitivity to noise, serious overfitting, and weak generalization ability.

3.6. Noise addition experiments

We further validated the robustness of transfer learning in realistic and complex roadside detection environments (Fig. 8 and Table 2). We further validated this by adding 10%, 20%, and 30% Gaussian noises to the training and test sets in the three controlled experiments (real extreme environments with noise levels close to 10%). The results are presented in Table 2. At noise levels close to approximately 10%, most of the models showed a slight decrease in accuracy, which is in line with the expectations. Among them, the ResNet50, ResNet101, and VGG16 models performed well with 96.43% accuracy in the MET dataset; ResNet50 and VGG19 performed best with 94.44% accuracy in the MOP dataset. At noise levels of 20% and 30%, all models exhibited a decrease in accuracy with increasing noise.

Fig. 8.

Various image data for the small dataset constructed after adding 10%, 20% and 30% Gaussian noises, respectively.

In conclusion, the results show that the transfer-learning-based model has a strong generalization ability and stability. Validating the literature, it was concluded [63] that models need larger parameter sizes to achieve higher accuracy when noise is present in the dataset. Specifically, models with larger parameter sizes, such as VGG16 and VGG19, had a stronger tolerance to environmental noise.

3.7. Training time comparison experiment

In the field of POCT, the use of a fast detection method with portable and stable equipment is important for detection. One of the difficulties in the medical field is the lack of access to sufficient valid medical data. As shown in Table 4, because transfer learning freezes most of the network layer parameters, these parameters do not need to be trained again, and their training time is greatly reduced. This also makes it possible to train the model using ordinary equipment, which will greatly reduce the threshold of combining AI with fast detection equipment. This "cliff-like" decrease in the training time is particularly evident in VGG16 and VGG19, with a maximum reduction of about 22 times. The experiments show that the introduction of transfer learning is a good solution for the conflict between large amounts of data and low computational cost, allowing for local miniaturization of the detection instrument with high accuracy while also addressing medical data privacy issues due to the local deployment of model inference.

Table 4.

Comparison of the training time for the AI models with and without transfer learning.

Database	Model	Training time (Transfer Learning)	Training time (without using Transfer Learning)
MET	ResNet50	30 min	119 min
	ResNet101	158 min	237 min
	VGG16	26 min	492 min
	VGG19	40 min	524 min
	GoogleNet	32 min	57 min
	MobileNetV2	107 min	145 min
	AlexNet	10 min	42 min
	DenseNet201	209 min	534 min
MOP	ResNet50	19 min	77 min
	ResNet101	47 min	157 min
	VGG16	14 min	305 min
	VGG19	16 min	326 min
	GoogleNet	13 min	25 min
	MobileNetV2	41 min	52 min
	AlexNet	3 min	15 min
	DenseNet201	135 min	283 min

3.8. Model evaluation standards

Model selection is of utmost importance for practical deployment. Ideally, models that have undergone transfer learning have extremely high prediction accuracy, strong noise tolerance, and fast inference data. However, a real situation often requires a careful trade-off between accuracy, stability, and portability, and this compromise depends on the actual application detection scenario and needs. The following conclusions were obtained from the analysis of the convergence speed, accuracy, and loss value changes during training (Fig. 6); the relationship between the three important characteristics of the model accuracy, parameter size, and training speed (Fig. 9); the relationship between different environmental noises and accuracy (Table 2); and different model training times (Table 4).

Fig. 9.

The comparison of the accuracy, training time and parameter size of the 8 AI models in MET detection. The bubble size in the figure is proportional to the parameter size. Note: The plot above only shows an indication of the relative speeds of the different models. The exact prediction and training iteration times depend on the hardware and mini-batch size that you use.

(1) In upconversion luminescence quantitative detection, the model depth is positively correlated with accuracy and negatively correlated with environmental noise tolerance. It is positively correlated with the training speed (the deeper the model, the slower the training speed).

(2) In upconversion luminescence quantification, the parameter size is positively correlated with the accuracy, ambient noise tolerance, and convergence speed (faster to reach a high accuracy).

Because the eight AI models cover various architectures of the current deep learning models, the above conclusions are universal. Finally, VGG19 was selected as the "best model" for the final deployment in IoT devices owing to its balanced performance in terms of accuracy, environmental noise tolerance, and convergence speed.

3.9. Stability and performance

The stability of the proposed system was verified. In the experimental evaluation phase, we performed six replicate experiments (Rounds 1-6 in Fig. 10) with four groups of MET solutions of different concentrations (1, 2.5, 10, and 20 ng/mL). According to the implementation flow (Fig. 1), each time the same test strip was inserted into the system, the system obtained the test results and predicted probabilities (representing the confidence of the results) for each of the five test images within 20 s. The final test results were obtained using a majority-voting strategy. The results showed that correct results were obtained for the final assay of concentrations in all four groups in these six replicated experiments (Fig. 10). Although there were incorrect predictions in rounds 3 and 4, the majority voting strategy ensured the absolute accuracy of the predicted results. It can also be observed that the incorrect prediction results have a low prediction probability, which means that the model is less confident in providing the prediction results. This phenomenon also illustrates the stability of the AI model after transfer learning. In general, these instabilities were acceptable.

Fig. 10.

The stability of the proposed system. Note: The results are obtained in MET detection with devices deployed using the VGG19 model after transfer learning.

Finally, the system has advantages in terms of detection sensitivity and detection time in comparison with reported related detection techniques (Table 5).

Table 5.

Comparison between other methods and the proposed technique.

Methods	Detection target	Limit of detection	Detection range	Detection time (min)
UCNP-LFAs (This work)	MET and MOP	1 ng/mL for MET;0.1 ng/mL for MOP	1-20 ng/mL for MET;0.1-100 ng/mL for MOP	0.33
UCNP-LFAs[18]	MET and MOP	10 ng/mL for MET;5 ng/mL for MOP	10-250 ng/mL for MET;5-100 ng/mL for MOP	2∼15
UCNP-LFAs[38]	MOP	0.1 ng/mL	0.1-10 ng/mL	0.5
LFAs[64]	MOP	1 ng/mL	1-100 ng/mL	5∼20
High-Performance Liquid Chromatography (HPLC)[65]	MET	1.7 ng/mL	10∼1000 ng/mL	A few Seconds
Gas Chromatography-Mass Spectrometry (GC-MS)[66]	MET	0.09 ng/mL	0.09∼0.81 ng/mL	12.7
LC-MS[67]	MET	0.2 ng/mL	4∼20 ng/mL	20
Capillary Electrophoresis (CE)[68]	MET	0.5 ng/mL	0.5∼50000 ng/mL	15
Quantum dot-based[69]	MET	6 ng/mL	-	1∼3

4. Conclusion

In this study, we report for the first time a novel transfer-learning-based full-flow system for the quantitative detection of UCNP-LFAs. Among them, the introduction of transfer learning into optical biosensors resolves the irreconcilable contradiction between the biomedical field and traditional machine learning, which requires a large amount of labeled data and substantial computational cost, making model training possible for common devices, which will strongly clarify the significant obstacle of combining AI with POCT neighborhoods. It provides extremely simple pre-processing, higher accuracy, and greater noise tolerance without increasing the complexity of the existing systems. Moreover, we propose a suitable feature engineering and decision strategy based on an actual quantitative fluorescence detection application scenario. Finally, using a large amount of experimental data, we demonstrate that a quantitative detection AI model with high accuracy can be built using transfer learning with only a small dataset. Most importantly, we developed a portable and highly accurate commercial platform based on transfer learning and quantitative detection of UCNP-LFAs. We believe that introducing transfer learning into quantitative upconversion fluorescence detection will make transfer learning widely applicable to upconversion fluorescence measurement studies and imaging analysis, and even expand into a general detection method for optical biosensors in the future.

Data availability

The device captures the upconverted luminescent image database in this study, and is available from the corresponding author upon reasonable request. The MET and MOP training and testing datasets can be accessed from https://dx.doi.org/10.17632/kbmb83ybhz.1.

Declaration of competing interest

The authors declare that they have no conflicts of interest in this work.

Biographies

Wei Wang is currently pursuing a degree in biomedical equipment, chemical sensors, and computer vision at the School of Information and Communication Engineering, University of Electronic Science and Technology of China.

Jinhong Guo received his bachelor's degree in electronic engineering from the University of Electronic Science and Technology of China, Chengdu, China in 2010 and Ph.D. degree in biomedical engineering from the Nanyang Technological University in 2014. His current research interests are micro-nano probes and sensing devices, machine learning, biological and medical data security, and digital therapeutics.