Skip to main content
NCBI home page
Springer Nature - PMC COVID-19 Collection logoLink to Springer Nature - PMC COVID-19 Collection
. 2023 Feb 21;20(2):147–179. doi: 10.1007/s11633-022-1382-8

Pre-training in Medical Data: A Survey

Yixuan Qiu 1, Feng Lin 1, Weitong Chen 2,✉, Miao Xu 1
PMCID: PMC9942039

Abstract

Medical data refers to health-related information associated with regular patient care or as part of a clinical trial program. There are many categories of such data, such as clinical imaging data, bio-signal data, electronic health records (EHR), and multi-modality medical data. With the development of deep neural networks in the last decade, the emerging pre-training paradigm has become dominant in that it has significantly improved machine learning methods′ performance in a data-limited scenario. In recent years, studies of pre-training in the medical domain have achieved significant progress. To summarize these technology advancements, this work provides a comprehensive survey of recent advances for pre-training on several major types of medical data. In this survey, we summarize a large number of related publications and the existing benchmarking in the medical domain. Especially, the survey briefly describes how some pre-training methods are applied to or developed for medical data. From a data-driven perspective, we examine the extensive use of pre-training in many medical scenarios. Moreover, based on the summary of recent pre-training studies, we identify several challenges in this field to provide insights for future studies.

Keywords: Medical data, pre-training, transfer learning, self-supervised learning, medical image data, electrocardiograms (ECG) data

Acknowledgements

This work was supported by 2021 UQ School of Information Technology and Electrical Engineering (ITEE) Research Support Funding, Cyber Research Seed Funding (No. 2021-R3), the University of Adelaide (No. 1531570) and New Staff Research Start-up Funds (No. NS-2102).

Footnotes

Yixuan Qiu received the M. Sc. degree in electrical engineering from The University of Queensland, Australia in 2020. Currently, he is a Ph. D. degree candidate in data science at School of information Technology and Electrical Engineering, The University of Queensland, Australia.

His research interests include medical data analytic, self-supervised learning and federated learning.

Feng Lin received the M. Sc. degree from The University of Queensland, Australia in 2022. He is currently working at Wipro, Australia.

His research interests include weakly-supervised learning, data mining and deep learning.

Weitong Chen received the Ph. D. degree in computer science from The University of Queensland, Australia in 2020. He is currently a lecturer at The University of Adelaide, Australia.

His research interests include machine learning and its application to medical domains.

Miao Xu received the Ph. D. degree in machine learning from Nanjing University, China in 2017. She is a lecturer at The University of Queensland (UQ), Australia. Before joining UQ, she was a postdoctoral researcher at RIKEN, Japan.

Her research interests include weakly supervised learning and its application to the medical and cyber-security domains.

Contributor Information

Yixuan Qiu, Email: y.qiu@uq.edu.au.

Feng Lin, Email: feng.lin@uq.net.au.

Weitong Chen, Email: chen@adelaide.edu.au.

Miao Xu, Email: miao.xu@uq.edu.au.

References

  • [1].LeCun Y, Boser B, Denker J S, Henderson D, Howard R E, Hubbard W, Jackel L D. Backpropagation applied to handwritten zip code recognition. Neural Computation. 1989;1(4):541–551. doi: 10.1162/neco.1989.1.4.541. [DOI] [Google Scholar]
  • [2].K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. [Online], Available: https://arxiv.org/abs/1409.1556, 2014.
  • [3].Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural networks; Montreal, Canada: ACM; 2014. pp. 3104–3112. [Google Scholar]
  • [4].J. Chung, C. Gulcehre, K. Cho, Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. [Online], Available: https://arxiv.org/abs/1412.3555, 2014.
  • [5].Amin S U, Alsulaiman M, Muhammad G, Bencherif M A, Hossain M S. Multilevel weighted feature fusion using convolutional neural networks for EEG motor imagery classification. IEEE Access. 2019;7:18940–18950. doi: 10.1109/ACCESS.2019.2895688. [DOI] [Google Scholar]
  • [6].Jaderberg M, Simonyan K, Zisserman A, Kavukcuoglu K. Spatial transformer networks; Montreal, Canada: ACM; 2015. pp. 2017–2025. [Google Scholar]
  • [7].Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM. 2017;60(6):84–90. doi: 10.1145/3065386. [DOI] [Google Scholar]
  • [8].Hirschberg J, Manning C D. Advances in natural language processing. Science. 2015;349(6245):261–266. doi: 10.1126/science.aaa8685. [DOI] [PubMed] [Google Scholar]
  • [9].Wang G T, Zuluaga M A, Li W Q, Pratt R, Patel P A, Aertsen M, Doel T, David A L, Deprest J, Ourselin S, Vercauteren T. DeepIGeoS: A deep interactive geodesic framework for medical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019;41(7):1559–1572. doi: 10.1109/TPAMI.2018.2840695. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [10].Minaee S, Boykov Y Y, Porikli F, Plaza A, Kehtarnavaz N, Terzopoulos D. Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022;44(7):3523–3542. doi: 10.1109/TPAMI.2021.3059968. [DOI] [PubMed] [Google Scholar]
  • [11].Greenspan H, Van Ginneken B, Summers R M. Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Transactions on Medical Imaging. 2016;35(5):1153–1159. doi: 10.1109/TMI.2016.2553401. [DOI] [Google Scholar]
  • [12].Wang Y D, Chen W T, Pi D C, Yue L. Adversarially regularized medication recommendation model with multi-hop memory network. Knowledge and Information Systems. 2021;63(1):125–142. doi: 10.1007/s10115-020-01513-9. [DOI] [Google Scholar]
  • [13].Y. D. Wang, W. T. Chen, D. C. Pi, L. Yue, S. Wang, M. Xu. Self-supervised adversarial distribution regularization for medication recommendation. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, Montreal, Canada, pp. 3134–3140, 2021. DOI: 10.24963/ijcai.2021/431.
  • [14].Qiu Y X, Chen W T, Yue L, Xu M, Zhu B F. STCT: Spatial-temporal conv-transformer network for cardiac arrhythmias recognition; Sydney, Australia: Springer; 2022. pp. 86–100. [Google Scholar]
  • [15].Ripoll V J R, Wojdel A, Romero E, Ramos P, Brugada J. ECG assessment based on neural networks with pretraining. Applied Soft Computing. 2016;49:399–406. doi: 10.1016/j.asoc.2016.08.013. [DOI] [Google Scholar]
  • [16].Scientific Reports. 2021.
  • [17].X. S. Wang, Z. Y. Xu, L. Tam, D. Yang, D. G. Xu. Self-supervised image-text pre-training with mixed data in chest X-rays. [Online], Available: https://arxiv.org/abs/2103.16022, 2021.
  • [18].X. S. Wang, Y. F. Peng, L. Lu, Z. Y. Lu, M. Bagheri, R. M. Summers. ChestX-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp.2097-2106, 2017. DOI: 10.1109/CVPR.2017.369.
  • [19].J. H. Moon, H. Lee, W. Shin, Y. H. Kim, E. Choi. Multi-modal understanding and generation for medical images and text via vision-language pre-training. IEEE Journal of Biomedical and Health Informatics, to be published. DOI: 10.1109/JBHI.2022.3207502. [DOI] [PubMed]
  • [20].B. Yan, M. T. Pei. Clinical-BERT: Vision-language pre-training for radiograph diagnosis and reports generation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 3, pp. 2982–2990, 2022. DOI: 10.1609/aaai.v36i3.20204.
  • [21].L. Hou, D. Samaras, T. M. Kurc, Y. Gao, J. E. Davis, J. H. Saltz. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 2424–2433, 2016. DOI: 10.1109/CVPR.2016.266. [DOI] [PMC free article] [PubMed]
  • [22].Shin H C, Roth H R, Gao M C, Lu L, Xu Z Y, Nogues I, Yao J H, Mollura D, Summers R M. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging. 2016;35(5):1285–1298. doi: 10.1109/TMI.2016.2528162. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [23].Würfl T, Ghesu F C, Christlein V, Maier A. Deep learning computed tomography; Athens, Greece: Springer; 2016. pp. 432–440. [Google Scholar]
  • [24].Ramsundar B, Eastman P, Walters P, Pande V. Deep Learning for the Life Sciences: Applying Deep Learning to Genomics, Microscopy, Drug Discovery, and More. Sebastopol, USA: O’Reilly Media; 2019. [Google Scholar]
  • [25].Han X, Zhang Z Y, Ding N, Gu Y X, Liu X, Huo Y Q, Qiu J Z, Yao Y, Zhang A, Zhang L, Han W T, Huang M L, Jin Q, Lan Y Y, Liu Y, Liu Z Y, Lu Z W, Qiu X P, Song R H, Tang J, Wen J R, Yuan J H, Zhao W X, Zhu J. Pre-trained models: Past, present and future. AI Open. 2021;2:225–250. doi: 10.1016/j.aiopen.2021.08.002. [DOI] [Google Scholar]
  • [26].Torrey L, Shavlik J. Transfer learning. In: Olivas E S, Guerrero J D M, Martinez-Sober M, Magdalena-Benedito J R, López A J S, editors. Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques. Hershey, USA: IGI Global; 2010. pp. 242–264. [Google Scholar]
  • [27].Pan S J, Yang Q. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 2010;22(10):1345–1359. doi: 10.1109/TKDE.2009.191. [DOI] [Google Scholar]
  • [28].V. Jain, E. Learned-Miller. Online domain adaptation of a pre-trained cascade of classifiers. In Proceedings of IEEE Computer Vision and Pattern Recognition, Colorado Springs, USA, pp. 577–584, 2011. DOI: 10.1109/CV-PR.2011.5995317.
  • [29].Newell A, Deng J. How useful is self-supervised pretraining for visual tasks?; Seattle, USA: IEEE; 2020. pp. 7345–7354. [Google Scholar]
  • [30].Yang Y Z, Xu Z. Rethinking the value of labels for improving class-imbalanced learning; Vancouver, Canada: ACM; 2020. pp. 19290–19301. [Google Scholar]
  • [31].H. Liu, J. Z. HaoChen, A. Gaidon, T. Y. Ma. Self-supervised learning is more robust to dataset imbalance. In Proceedings of the 10th International Conference on Learning Representations, 2022.
  • [32].Schlegl T, Ofner J, Langs G. Unsupervised pre-training across image domains improves lung tissue classification; Cambridge, USA: Springer; 2014. pp. 82–93. [Google Scholar]
  • [33].Meng Y W, Speier W, Ong M K, Arnold C W. Bidirectional representation learning from transformers using multimodal electronic health record data to predict depression. IEEE Journal of Biomedical and Health Informatics. 2021;25(8):3121–3129. doi: 10.1109/JBHI.2021.3063721. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [34].Azizi S, Mustafa B, Ryan F, Beaver Z, Freyberg J, Deaton J, Loh A, Karthikesalingam A, Kornblith S, Chen T, Natarajan V, Norouzi M. Big self-supervised models advance medical image classification; Montreal, Canada: IEEE; 2021. pp. 3478–3488. [Google Scholar]
  • [35].Thinsungnoen T, Kerdprasop K, Kerdprasop N. Deep autoencoder networks optimized with genetic algorithms for efficient ECG clustering. International Journal of Machine Learning and Computing. 2018;8(2):112–116. doi: 10.18178/ijmlc.2018.8.2.672. [DOI] [Google Scholar]
  • [36].Cheplygina V, de Bruijne M, Pluim J P W. Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical Image Analysis. 2019;54:280–296. doi: 10.1016/j.media.2019.03.009. [DOI] [PubMed] [Google Scholar]
  • [37].Scientific Reports. 2020.
  • [38].P. Y. Chen. Representation learning for electronic health records: A survey. Journal of Physics: Conference Series, vol. 1487, Article number 012015, 2020. DOI: 10.1088/1742-6596/1487/1/012015.
  • [39].S. Shurrab, R. Duwairi. Self-supervised learning methods and applications in medical imaging analysis: A survey. PeerJ Computer Science, vol. 8, Article number e1045, 2022. DOI: 10.7717/peerj-cs.1045. [DOI] [PMC free article] [PubMed]
  • [40].PLoS Digital Health. 2022. [DOI] [PMC free article] [PubMed]
  • [41].Scientific Data. 2018.
  • [42].Scientific Data. 2016.
  • [43].Goldberger A L, Amaral L A, Glass L, Hausdorff J M, Ivanov P C, Mark R G, Mietus J E, Moody G B, Peng C K, Stanley H E. PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation. 2000;101(23):e215–e220. doi: 10.1161/01.CIR.101.23.e215. [DOI] [PubMed] [Google Scholar]
  • [44].A. E. Kavur, N. S. Gezer, M. Barιş, S. Aslan, P. H. Conze, V. Groza, D. D. Pham, S. Chatterjee, P. Ernst, S. Özkan, B. Baydar, D. Lachinov, S. Han, J. Pauli, F. Isensee, M. Perkonigg, R. Sathish, R. Rajan, D. Sheet, G. Dovletov, O. Speck, A. Nürnberger, K. H. Maier-Hein, G. Bozdağı Akar, G. Ünal, O. Dicle, M. A. Selver. CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation. Medical Image Analysis, vol. 69, Article number 101950, 2021. DOI: 10.1016/j.media.2020.101950. [DOI] [PubMed]
  • [45].Sinha A, Dolz J. Multi-scale self-guided attention for medical image segmentation. IEEE Journal of Biomedical and Health Informatics. 2021;25(1):121–130. doi: 10.1109/JBHI.2020.2986926. [DOI] [PubMed] [Google Scholar]
  • [46].Roth H R, Lu L, Farag A, Shin H C, Liu J M, Turkbey E B, Summers R M. DeepOrgan: Multi-level deep convolutional networks for automated pancreas segmentation; Munich, Germany: Springer; 2015. pp. 556–564. [Google Scholar]
  • [47].P. Bilic, P. F. Christ, E. Vorontsov, G. Chlebus, H. Chen, Q. Dou, C. W. Fu, X. Han, P. A. Heng, J. Hesser, S. Kadoury, T. K. Konopczynski, M. Le, C. M. Li, X. M. Li, J. Lipková, J. S. Lowengrub, H. Meine, J. H. Moltz, C. Pal, M. Piraud, X. J. Qi, J. Qi, M. Rempfler, K. Roth, A. Schenk, A. Sekuboyina, P. Zhou, C. Hülsemeyer, M. Beetz, F. Ettlinger, F. Grün, G. Kaissis, F. Lohöfer, R. Braren, J. Holch, F. Hofmann, W. H. Sommer, V. Heinemann, C. Jacobs, G. E. H. Mamani, B. van Ginneken, G. Chartrand, A. Tang, M. Drozdzal, A. Ben-Cohen, E. Klang, M. M. Amitai, E. Konen, H. Greenspan, J. Moreau, A. Hostettler, L. Soler, R. Vivanti, A. Szeskin, N. Lev-Cohain, J. Sosna, L. Joskowicz, B. H. Menze. The liver tumor segmentation benchmark (LiTS). [Online], Available: https://arxiv.org/abs/1901.04056, 2019.
  • [48].Li X M, Chen H, Qi X J, Dou Q, Fu C W, Heng P A. H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes. IEEE Transactions on Medical Imaging. 2018;37(12):2663–2674. doi: 10.1109/TMI.2018.2845918. [DOI] [PubMed] [Google Scholar]
  • [49].Kermany D S, Goldbaum M, Cai W J, Valentim C C S, Liang H Y, Baxter S L, McKeown A, Yang G, Wu X K, Yan F B, Dong J, Prasadha M K, Pei J, Ting M Y L, Zhu J, Li C, Hewett S, Dong J, Ziyar I, Shi A, Zhang R Z, Zheng L H, Hou R, Shi W, Fu X, Duan Y O, Huu V A N, Wen C, Zhang E D, Zhang C L, Li O L, Wang X B, Singer M A, Sun X D, Xu J, Tafreshi A, Lewis M A, Xia H M, Zhang K. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell. 2018;172(5):1122–1131. doi: 10.1016/j.cell.2018.02.010. [DOI] [PubMed] [Google Scholar]
  • [50].W. Al-Dhabyani, M. Gomaa, H. Khaled, A. Fahmy. Dataset of breast ultrasound images. Data in Brief, vol. 28, Article number 104863, 2020. DOI: 10.1016/j.dib.2019.104863. [DOI] [PMC free article] [PubMed]
  • [51].W. K. Moon, Y. W. Lee, H. H. Ke, S. H. Lee, C. S. Huang, R. F. Chang. Computer-aided diagnosis of breast ultrasound images using ensemble learning from convolutional neural networks. Computer Methods and Programs in Biomedicine, vol. 190, Article number 105361, 2020. DOI: 10.1016/j.cmpb.2020.105361. [DOI] [PubMed]
  • [52].P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, M. P. Lungren, A. Y. Ng. CheXNet: Radiologist-level pneumonia detection on chest X-Rays with deep learning. [Online], Available: https://arxiv.org/abs/1711.05225, 2017.
  • [53].Wang Z, Yin Y X, Shi J P, Fang W, Li H S, Wang X G. Zoom-in-Net: Deep mining lesions for diabetic retinopathy detection; Quebec City, Canada: Springer; 2017. pp. 267–275. [Google Scholar]
  • [54].N. C. F. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler, A. Halpern. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). In Proceedings of 15th IEEE International Symposium on Biomedical Imaging, Washington, USA, pp. 168–172, 2018. DOI: 10.1109/ISBI.2018.8363547.
  • [55].N. Gessert, M. Nielsen, M. Shaikh, R. Werner, A. Schlaefer. Skin lesion classification using ensembles of multi-resolution efficientNets with meta data. MethodsX, vol. 7, Article number 100864, 2020. DOI: 10.1016/j.mex.2020.100864. [DOI] [PMC free article] [PubMed]
  • [56].Kandoth C, McLellan M D, Vandin F, Ye K, Niu B F, Lu C, Xie M C, Zhang Q Y, McMichael J F, Wyczalkowski M A, Leiserson M D M, Miller C A, Welch J S, Walter M J, Wendl M C, Ley T J, Wilson R K, Raphael B J, Ding L. Mutational landscape and significance across 12 major cancer types. Nature. 2013;502(7471):333–339. doi: 10.1038/nature12634. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [57].Yao J W, Zhu X L, Zhu F Y, Huang J Z. Deep correlational learning for survival prediction from multi-modality data; Quebec City, Canada: Springer; 2017. pp. 406–414. [Google Scholar]
  • [58].Mobadersany P, Yousefi S, Amgad M, Gutman D A, Barnholtz-Sloan J S, Velázquez Vega J E, Brat D J, Cooper L A D. Predicting cancer outcomes from histology and genomics using convolutional networks. Proceedings of the National Academy of Sciences of the United States of America. 2018;115(13):E2970–E2979. doi: 10.1073/pnas.1717139115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [59].National Lung Screening Trial Research Team The national lung screening trial: Overview and study design. Radiology. 2011;258(1):243–253. doi: 10.1148/radiol.10091808. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [60].J. W. Yao, X. L. Zhu, J. Jonnagaddala, N. Hawkins, J. Z. Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, vol. 65, Article number 101789, 2020. DOI: 10.1016/j.media.2020.101789. [DOI] [PubMed]
  • [61].Veeling B S, Linmans J, Winkens J, Cohen T, Welling M. Rotation equivariant CNNs for digital pathology; Granada, Spain: Springer; 2018. pp. 210–218. [Google Scholar]
  • [62].Moody G B, Mark R G. The impact of the MIT-BIH arrhythmia database. IEEE Engineering in Medicine and Biology Magazine. 2001;20(3):45–50. doi: 10.1109/51.932724. [DOI] [PubMed] [Google Scholar]
  • [63].Scientific Data. 2020.
  • [64].Moody G B, Muldrow W K, Mark R G. A noise stress test for arrhythmia detectors. Computers in Cardiology. 1984;11(3):381–384. [Google Scholar]
  • [65].Taddei A, Distante G, Emdin M, Pisani P, Moody G B, Zeelenberg C, Marchesi C. The European ST-T database: Standard for evaluating systems for the analysis of ST-T changes in ambulatory electrocardiography. European Heart Journal. 1992;13(9):1164–1172. doi: 10.1093/oxfordjournals.eurheartj.a060332. [DOI] [PubMed] [Google Scholar]
  • [66].Clifford G D, Liu C Y, Moody B, Lehman L W H, Silva I, Li Q, Johnson A E, Mark R G. AF classification from a short single lead ECG recording: The physioNet/computing in cardiology challenge 2017; Rennes, France: IEEE; 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [67].Andreotti F, Carr O, Pimentel M A F, Mahdi A, De Vos M. Comparing feature-based classifiers and convolutional neural networks to detect arrhythmia from short segments of ECG; Rennes, France: IEEE; 2017. pp. 1–4. [Google Scholar]
  • [68].R. Bousseljot, D. Kreiseler, A. Schnabel. Nutzung der EKG-signaldatenbank cardiodat der PTB über das internet. Biomedizinische Technik, vol. 40, Article number 317, 1995. DOI: 10.1515/bmte.1995.40.s1.317.
  • [69].Sharma L, Tripathy R, Dandapat S. Multiscale energy and eigenspace approach to detection and localization of myocardial infarction. IEEE Transactions on Biomedical Engineering. 2015;62(7):1827–1837. doi: 10.1109/TBME.2015.2405134. [DOI] [PubMed] [Google Scholar]
  • [70].Moody G B, Mark R G. Development and evaluation of a 2-lead ECG analysis program. Computers in Cardiology. 1982;1982:39–44. [Google Scholar]
  • [71].Liu F F, Liu C Y, Zhao L N, Zhang X Y, Wu X L, Xu X Y, Liu Y L, Ma C Y, Wei S S, He Z Q, Li J Q, Yin Kwee E N. An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection. Journal of Medical Imaging and Health Informatics. 2018;8(7):1368–1373. doi: 10.1166/jmihi.2018.2442. [DOI] [Google Scholar]
  • [72].iScience. 2020.
  • [73].Miranda-Correa J A, Abadi M K, Sebe N, Patras I. AMIGOS: A dataset for affect, personality and mood research on individuals and groups. IEEE Transactions on Affective Computing. 2021;12(2):479–493. doi: 10.1109/TAFFC.2018.2884461. [DOI] [Google Scholar]
  • [74].Santamaria-Granados L, Munoz-Organero M, Ramirez-González G, Abdulhay E, Arunkumar N. Using deep convolutional neural network for emotion detection on a physiological signals dataset (AMIGOS) IEEE Access. 2018;7:57–67. doi: 10.1109/ACCESS.2018.2883213. [DOI] [Google Scholar]
  • [75].Subramanian R, Wache J, Abadi M K, Vieriu R L, Winkler S, Sebe N. ASCERTAIN: Emotion and personality recognition using commercial sensors. IEEE Transactions on Affective Computing. 2018;9(2):147–160. doi: 10.1109/TAFFC.2016.2625250. [DOI] [Google Scholar]
  • [76].L. Zhang, S. Walter, X. Y. Ma, P. Werner, A. Al-Hamadi, H. C. Traue, S. Gruss. “BioVid Emo DB”: A multimodal database for emotion analyses validated by subjective ratings. In Proceedings of IEEE Symposium Series on Computational Intelligence, Athens, Greece, 2016. DOI: 10.1109/SSCI.2016.7849931.
  • [77].Cheng Z, Shu L, Xie J Y, Chen C L P. A novel ECG-based real-time detection method of negative emotions in wearable applications; Shenzhen, China: IEEE; 2017. pp. 296–301. [Google Scholar]
  • [78].Koelstra S, Muhl C, Soleymani M, Lee J S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I. DEAP: A database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing. 2012;3(1):18–31. doi: 10.1109/T-AFFC.2011.15. [DOI] [Google Scholar]
  • [79].Yin Z, Zhao M Y, Wang Y X, Yang J D, Zhang J H. Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Computer Methods and Programs in Biomedicine. 2017;140:93–110. doi: 10.1016/j.cmpb.2016.12.005. [DOI] [PubMed] [Google Scholar]
  • [80].Katsigiannis S, Ramzan N. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE Journal of Biomedical and Health Informatics. 2018;22(1):98–107. doi: 10.1109/JBHI.2017.2688239. [DOI] [PubMed] [Google Scholar]
  • [81].Song T F, Zheng W M, Song P, Cui Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing. 2020;11(3):532–541. doi: 10.1109/TAFFC.2018.2817622. [DOI] [Google Scholar]
  • [82].Soleymani M, Lichtenauer J, Pun T, Pantic M. A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing. 2012;3(1):42–55. doi: 10.1109/T-AFFC.2011.25. [DOI] [Google Scholar]
  • [83].X. B. Li, J. Chen, G. Y. Zhao, M. Pietikäinen. Remote heart rate measurement from face videos under realistic situations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Columbus, USA, pp. 4264–4271, 2014. DOI: 10.1109/CVPR.2014.543.
  • [84].Song T F, Zheng W M, Lu C, Zong Y, Zhang X L, Cui Z. MPED: A multi-modal physiological emotion database for discrete emotion recognition. IEEE Access. 2019;7:12177–12191. doi: 10.1109/ACCESS.2019.2891579. [DOI] [Google Scholar]
  • [85].Zheng W L, Lu B L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development. 2015;7(3):162–175. doi: 10.1109/TAMD.2015.2431497. [DOI] [Google Scholar]
  • [86].I. Obeid, J. Picone. The temple university hospital EEG data corpus. Frontiers in Neuroscience, vol. 10, Article number 196, 2016. DOI: 10.3389/fnins.2016.00196. [DOI] [PMC free article] [PubMed]
  • [87].Schalk G, McFarland D J, Hinterberger T, Birbaumer N, Wolpaw J R. BCI2000: A general-purpose brain-computer interface (BCI) system. IEEE Transactions on Biomedical Engineering. 2004;51(6):1034–1043. doi: 10.1109/TBME.2004.827072. [DOI] [PubMed] [Google Scholar]
  • [88].Supratak A, Dong H, Wu C, Guo Y K. DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2017;25(11):1998–2008. doi: 10.1109/TNSRE.2017.2721116. [DOI] [PubMed] [Google Scholar]
  • [89].Zhang G Q, Cui L C, Mueller R, Tao S Q, Kim M, Rueschman M, Mariani S, Mobley D, Redline S. The national sleep research resource: Towards a sleep data commons. Journal of the American Medical Informatics Association. 2018;25(10):1351–1358. doi: 10.1093/jamia/ocy064. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [90].Quan S F, Howard B V, Iber C, Kiley J P, Nieto F J, O’Connor G T, Rapoport D M, Redline S, Robbins J, Samet J M, Wahl P W. The sleep heart health study: Design, rationale, and methods. Sleep. 1997;20(12):1077–1085. [PubMed] [Google Scholar]
  • [91].Sors A, Bonnet S, Mirek S, Vercueil L, Payen J F. A convolutional neural network for sleep stage scoring from raw single-channel EEG. Biomedical Signal Processing and Control. 2018;42:107–114. doi: 10.1016/j.bspc.2017.12.001. [DOI] [Google Scholar]
  • [92].Bengio Y, Lamblin P, Popovici D, Larochelle H. Greedy layer-wise training of deep networks; Vancouver, Canada: ACM; 2006. pp. 153–160. [Google Scholar]
  • [93].Ranzato M, Boureau Y L, LeCun Y. Sparse feature learning for deep belief networks; Vancouver, Canada: ACM; 2007. pp. 1185–1192. [Google Scholar]
  • [94].He K M, Girshick R, Dollár P. Rethinking ImageNet pre-training; Seoul, Republic of Korea: IEEE; 2019. pp. 4918–4927. [Google Scholar]
  • [95].Raghu M, Zhang C Y, Kleinberg J, Bengio S. Transfusion: Understanding transfer learning for medical imaging; Vancouver, Canada: ACM; 2019. pp. 3347–3357. [Google Scholar]
  • [96].Thrun S, Pratt L. Learning to learn: Introduction and overview. In: Thrun S, Pratt L, editors. Learning to Learn. Boston, USA: Springer; 1998. pp. 3–17. [Google Scholar]
  • [97].Scudder H. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory. 1965;11(3):363–371. doi: 10.1109/TIT.1965.1053799. [DOI] [Google Scholar]
  • [98].K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Vegas, USA, pp. 770–778, 2016. DOI: 10.1109/CV-PR.2016.90.
  • [99].C. Szegedy, W. Liu, Y. Q. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich. Going deeper with convolutions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 1–9, 2015. DOI: 10.1109/CVPR.2015.7298594.
  • [100].G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 2261–2269, 2017. DOI: 10.1109/CVPR.2017.243.
  • [101].J. Sarzynska-Wawer, A. Wawer, A. Pawlak, J. Szymanowska, I. Stefaniak, M. Jarkiewicz, L. Okruszek. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, vol. 304, Article number 114135, 2021. DOI: 10.1016/j.psychres.2021.114135. [DOI] [PubMed]
  • [102].Scientific Reports. 2017.
  • [103].Esteva A, Kuprel B, Novoa R A, Ko J, Swetter S M, Blau H M, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542(7639):115–118. doi: 10.1038/nature21056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [104].De Fauw J, Ledsam J R, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, Askham H, Glorot X, O’Donoghue B, Visentin D, Van Den Driessche G, Lakshminarayanan B, Meyer C, Mackinder F, Bouton S, Ayoub K, Chopra R, King D, Karthikesalingam A, Hughes C O, Raine R, Hughes J, Sim D A, Egan C, Tufail A, Montgomery H, Hassabis D, Rees G, Back T, Khaw P T, Suleyman M, Cornebise J, Keane P A, Ronneberger O. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature Medicine. 2018;24(9):1342–1350. doi: 10.1038/s41591-018-0107-6. [DOI] [PubMed] [Google Scholar]
  • [105].Treder M, Lauermann J L, Eter N. Automated detection of exudative age-related macular degeneration in spectral domain optical coherence tomography using deep learning. Graefe’s Archive for Clinical and Experimental Ophthalmology. 2018;256(2):259–265. doi: 10.1007/s00417-017-3850-3. [DOI] [PubMed] [Google Scholar]
  • [106].Apostolopoulos I D, Mpesiana T A. Covid-19: Automatic detection from X-ray images utilizing transfer learning with convolutional neural networks. Physical and Engineering Sciences in Medicine. 2020;43(2):635–640. doi: 10.1007/s13246-020-00865-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [107].Al Rahhal M M, Bazi Y, Al Zuair M, Othman E, BenJdira B. Convolutional neural networks for electrocardiogram classification. Journal of Medical and Biological Engineering. 2018;38(6):1014–1025. doi: 10.1007/s40846-018-0389-7. [DOI] [Google Scholar]
  • [108].Health Information Science and Systems. 2020. [DOI] [PMC free article] [PubMed]
  • [109].H. T. Shi, H. R. Wang, C. J. Qin, L. Q. Zhao, C. L. Liu. An incremental learning system for atrial fibrillation detection based on transfer learning and active learning. Computer Methods and Programs in Biomedicine, vol. 187, Article number 105219, 2020. DOI: 10.1016/j.cmpb.2019.105219. [DOI] [PubMed]
  • [110].Shyam A, Ravichandran V, Preejith S P, Joseph J, Sivaprakasam M. PPGnet: Deep network for device independent heart rate estimation from photoplethysmogram; Berlin, Germany: IEEE; 2019. pp. 1899–1902. [DOI] [PubMed] [Google Scholar]
  • [111].Scientific Reports. 2020.
  • [112].Matsoukas C, Haslum J F, Sorkhei M, Söderberg M, Smith K. What makes transfer learning work for medical images: Feature reuse & other factors. 2022;ProceedingsofIEEE/CVFConferenceonComputerVisionandPatternRecognition:9215–9224. [Google Scholar]
  • [113].T. Chen, S. Kornblith, M. Norouzi, G. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1597–1607, 2020.
  • [114].Grill J B, Strub F, Altché F, Tallec C, Richemond P H, Buchatskaya E, Doersch C, Avila Pires B, Guo Z D, Gheshlaghi Azar M, Piot B, Kavukcuoglu K, Munos R, Valko M. Bootstrap your own latent a new approach to self-supervised learning; Vancouver, Canada: ACM; 2020. pp. 21271–21284. [Google Scholar]
  • [115].M. Ishan, L. V. D. Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707–6717. 2020. DOI: 10.1109/cvpr42600.2020.00674.
  • [116].Caron M, Bojanowski P, Mairal J, Joulin A. Unsupervised pre-training of image features on non-curated data. 2019;IEEE:2959–2968. [Google Scholar]
  • [117].T. Chen, S. Kornblith, K. Swersky, M. Norouzi, G. E. Hinton. Big self-supervised models are strong semi-supervised learners. In Proceedings of the 34th Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 22243–22255, 2020.
  • [118].A. Van Den Oord, Y. Z. Li, O. Vinyals. Representation learning with contrastive predictive coding. [Online], Available: https://arxiv.org/abs/1807.03748, 2018.
  • [119].He K M, Fan H Q, Wu Y X, Xie S N, Girshick R. Momentum contrast for unsupervised visual representation learning; Seattle, USA: IEEE; 2020. pp. 9729–9738. [Google Scholar]
  • [120].Caron M, Misra I, Mairal J, Goyal P, Bojanowski P, Joulin A. Unsupervised learning of visual features by contrasting cluster assignments; Vancouver, Canada: ACM; 2020. pp. 9912–9924. [Google Scholar]
  • [121].He K M, Chen X L, Xie S N, Li Y H, Dollár P, Girshick R. Masked autoencoders are scalable vision learners; New Orleans, USA: IEEE; 2022. pp. 15979–15988. [Google Scholar]
  • [122].Chen X L, He K M. Exploring simple Siamese representation learning. 2021;ProceedingsofIEEE/CVFConferenceonComputerVisionandPatternRecognition:15745–15753. [Google Scholar]
  • [123].O. J. Hénaff. Data-efficient image recognition with contrastive predictive coding. In Proceedings of the 37th International Conference on Machine Learning, pp. 4182–4192, 2020.
  • [124].X. L. Chen, H. Q. Fan, R. Girshick, K. M. He. Improved baselines with momentum contrastive learning. [Online], Available: https://arxiv.org/abs/2003.04297, 2020.
  • [125].Chen X L, Xie S N, He K M. An empirical study of training self-supervised vision transformers; Montreal, Canada: IEEE; 2021. pp. 9620–9629. [Google Scholar]
  • [126].NPJ Digital Medicine. 2021. [DOI] [PMC free article] [PubMed]
  • [127].Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser Ł, Polosukhin I. Attention is all you need; Long Beach, USA: ACM; 2017. pp. 6000–6010. [Google Scholar]
  • [128].J. Devlin, M. W. Chang, K. Lee, K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, USA, pp. 4171–4186, 2019. DOI: 10.18653/v1/N19-1423.
  • [129].Journal of Medical Systems. 2018.
  • [130].R. Paul, S. H. Hawkins, L. O. Hall, D. B. Goldgof, R. J. Gillies. Combining deep neural network and traditional image features to improve survival prediction accuracy for lung cancer patients from diagnostic CT. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, Budapest, Hungary, pp. 2570–2575, 2016. DOI: 10.1109/SMC.2016.7844626.
  • [131].M. W. Ren, N. Dey, M. A. Styner, K. Botteron, G. Gerig. Local spatiotemporal representation learning for longitudinally-consistent neuroimage analysis. [Online], Available: https://arxiv.org/abs/2206.04281, 2022. [PMC free article] [PubMed]
  • [132].Bhandary A, Prabhu G A, Rajinikanth V, Thanaraj K P, Satapathy S C, Robbins D E, Shasky C, Zhang Y D, Tavares J M R, Raja N S M. Deep-learning framework to detect lung abnormality-A study with chest X-ray and lung CT scan images. Pattern Recognition Letters. 2020;129:271–278. doi: 10.1016/j.patrec.2019.11.013. [DOI] [Google Scholar]
  • [133].D. S. Reddy, R. Bharath, P. Rajalakshmi. A novel computer-aided diagnosis framework using deep learning for classification of fatty liver disease in ultrasound imaging. In Proceedings of the 20th IEEE International Conference on E-health Networking, Applications and Services (Healthcom), Ostrava, Czech Republic, pp. 1–5, 2018. DOI: 10.1109/HealthCom.2018.8531118.
  • [134].Wu C Z, Sun J, Wang J, Xu L F, Zhan S. Encoding-decoding network with pyramid self-attention module for retinal vessel segmentation. International Journal of Automation and Computing. 2021;18(6):973–980. doi: 10.1007/s11633-020-1277-0. [DOI] [Google Scholar]
  • [135].Ker J, Wang L P, Rao J, Lim T. Deep learning applications in medical image analysis. IEEE Access. 2017;6:9375–9389. doi: 10.1109/ACCESS.2017.2788044. [DOI] [Google Scholar]
  • [136].Fernandez-Quilez A. Deep Learning for an Improved Diagnostic Pathway of Prostate Cancer in a Small Multi-Parametric Magnetic Resonance Data Regime. Stavanger, Norway: University of Stavanger; 2022. [Google Scholar]
  • [137].K. B. Ahmed, L. O. Hall, D. B. Goldgof, R. H. Liu, R. A. Gatenby. Fine-tuning convolutional deep features for MRI based brain tumor classification. In Proceedings of SPIE 10134, Medical Imaging 2017: Computer-Aided Diagnosis, Orlando, USA, Article number 101342E, 2017. DOI: 10.1117/12.2253982.
  • [138].Prakash R M, Kumari R S S. Classification of MR brain images for detection of tumor with transfer learning from pre-trained CNN models; Chennai, India: IEEE; 2019. pp. 508–511. [Google Scholar]
  • [139].Khan H A, Jue W, Mushtaq M, Mushtaq M U. Brain tumor classification in MRI image using convolutional neural network. Mathematical Biosciences and Engineering. 2020;17(5):6203–6216. doi: 10.3934/mbe.2020328. [DOI] [PubMed] [Google Scholar]
  • [140].Sajjad M, Khan S, Muhammad K, Wu W Q, Ullah A, Baik S W. Multi-grade brain tumor classification using deep CNN with extensive data augmentation. Journal of Computational Science. 2019;30:174–182. doi: 10.1016/j.jocs.2018.12.003. [DOI] [Google Scholar]
  • [141].S. Deepak, P. M. Ameer. Brain tumor classification using deep CNN features via transfer learning. Computers in Biology and Medicine, vol. 111, Article number 103345, 2019. DOI: 10.1016/j.compbiomed.2019.103345. [DOI] [PubMed]
  • [142].Noreen N, Palaniappan S, Qayyum A, Ahmad I, Imran M, Shoaib M. A deep learning model based on concatenation approach for the diagnosis of brain tumor. IEEE Access. 2020;8:55135–55144. doi: 10.1109/ACCESS.2020.2978629. [DOI] [Google Scholar]
  • [143].J. Cheng. Brain tumor dataset. Figshare, [Online], Available: 10.6084/m9.figshare.1512427.v5, 2017.
  • [144].Swati Z N K, Zhao Q H, Kabir M, Ali F, Ali Z, Ahmed S, Lu J F. Brain tumor classification for MR images using transfer learning and fine-tuning. Computerized Medical Imaging and Graphics. 2019;75:34–46. doi: 10.1016/j.compmedimag.2019.05.001. [DOI] [PubMed] [Google Scholar]
  • [145].Healthcare. 2021.
  • [146].Wang S D, Dong L Y, Wang X, Wang X G. Classification of pathological types of lung cancer from CT images by deep residual neural networks with transfer learning strategy. Open Medicine. 2020;15(1):190–197. doi: 10.1515/med-2020-0028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [147].Marentakis P, Karaiskos P, Kouloulias V, Kelekis N, Argentos S, Oikonomopoulos N, Loukas C. Lung cancer histology classification from CT images based on radiomics and deep learning models. Medical & Biological Engineering & Computing. 2021;59(1):215–226. doi: 10.1007/s11517-020-02302-w. [DOI] [PubMed] [Google Scholar]
  • [148].Sensors. 2019.
  • [149].Byra M, Styczynski G, Szmigielski C, Kalinowski P, Michałowski Ł, Palusekiewicz R, Ziarkiewicz-Wróblewska B, Zieniewicz K, Sobieraj P, Nowicki A. Transfer learning with deep convolutional neural network for liver steatosis assessment in ultrasound images. International Journal of Computer Assisted Radiology and Surgery. 2018;13(12):1895–1903. doi: 10.1007/s11548-018-1843-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [150].M. Alkhaleefah, S. C. Ma, Y. L. Chang, B. Huang, P. K. Chittem, V. P. Achhannagari. Double-shot transfer learning for breast cancer classification from X-ray images. Applied Sciences, vol. 10, no. 11, Article number 3999, 2020. DOI: 10.3390/app10113999.
  • [151].Armato III S G, McLennan G, Bidaut L, McNitt-Gray M F, Meyer C R, Reeves A P, Zhao B, Aberle D R, Henschke C I, Hoffman E A, Kazerooni E A, Macmahon H, Van Beek E J R, Yankelevitz D, Biancardi A M, Bland P H, Brown M S, Engelmann R M, Laderach G E, Max D, Pais R C, Qing D P Y, Roberts R Y, Smith A R, Starkey A, Batra P, Caligiuri P, Farooqi A, Gladish G W, Jude C M, Munden R F, Petkovska I, Quint L E, Schwartz L H, Sundaram B, Dodd L E, Fenimore C, Gur D, Petrick N, Freymann J, Kirby J, Hughes B, Vande Casteele A, Gupte S, Sallam M, Heath M D, Kuhn M H, Dharaiya E, Burns R, Fryd D S, Salganicoff M, Anand V, Shreter U, Vastagh S, Croft B Y, Clarke L P. The lung image database consortium (LIDC) and image database resource initiative (IDRI): A completed reference database of long nodules on CT scans. Medical Physics. 2011;38(2):915–931. doi: 10.1118/1.3528204. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [152].S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. Meyer, A. P. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, E. A. Hoffman, E. A. Kazerooni, H. MacMahon, E. J. R. Van Beek, D. Yankelevitz, A. M. Biancardi, P. H. Bland, M. S. Brown, R. M. Engelmann, G. E. Laderach, D. Max, R. C. Pais, D. P. Y. Qing, R. Y. Roberts, A. R. Smith, A. Starkey, P. Batra, P. Caligiuri, A. Farooqi, G. W. Gladish, C. M. Jude, R. F. Munden, I. Petkovska, L. E. Quint, L. H. Schwartz, B. Sundaram, L. E. Dodd, C. Fenimore, D. Gur, N. Petrick, J. Freymann, J. Kirby, B. Hughes, A. V. Casteele, S. Gupte, M. Sallam, M. D. Heath, M. H. Kuhn, E. Dharaiya, R. Burns, D. S., Fryd, M. Salganicoff, V. Anand, U. Shreter, S. Vastagh, B. Y. Croft, Clarke, L. P. Data From LIDC-ID-RI [Data set]. The Cancer Imaging Archive. [Online], Available: 10.7937/K9/TCIA.2015.LO9QL9SX, 2015.
  • [153].P. F. Christ, F. Ettlinger, F. Grün, M. E. A. Elshaera, J. Lipkova, S. Schlecht, F. Ahmaddy, S. Tatavarty, M. Bickel, P. Bilic, M. Rempfler, F. Hofmann, M. D. Anastasi, S. A. Ahmadi, G. Kaissis, J. Holch, W. Sommer, R. Braren, V. Heinemann, B. Menze. Automatic liver and tumor segmentation of CT and MRI volumes using cascaded fully convolutional neural networks. [Online], Available: https://arxiv.org/abs/1702.05970, 2017.
  • [154].Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation; Munich, Germany: Springer; 2015. pp. 234–241. [Google Scholar]
  • [155].P. H. Conze, A. E. Kavur, E. Cornec-Le Gall, N. S. Gezer, Y. Le Meur, M. A. Selver, F. Rousseau. Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks. Artificial Intelligence in Medicine, vol. 117, Article number 102109, 2021. DOI: 10.1016/j.artmed.2021.102109. [DOI] [PubMed]
  • [156].Li M J, Cai W J, Verspoor K, Pan S R, Liang X D, Chang X J. Cross-modal clinical graph transformer for ophthalmic report generation; New Orleans, USA: IEEE; 2022. pp. 20656–20665. [Google Scholar]
  • [157].W. Gómez-Flores, W. C. de Albuquerque Pereira. A comparative study of pre-trained convolutional neural networks for semantic segmentation of breast tumors in ultrasound. Computers in Biology and Medicine, vol. 126, Article number 104036, 2020. DOI: 10.1016/j.compbiomed.2020.104036. [DOI] [PubMed]
  • [158].Piotrzkowska-Wróblewska H, Dobruch-Sobczak K, Byra M, Nowicki A. Open access database of raw ultrasonic signals acquired from malignant and benign breast lesions. Medical Physics. 2017;44(11):6105–6109. doi: 10.1002/mp.12538. [DOI] [PubMed] [Google Scholar]
  • [159].Hijab A, Rushdi M A, Gomaa M M, Eldeib A. Breast cancer classification in ultrasound images using transfer learning; Tripoli, Lebanon: IEEE; 2019. [Google Scholar]
  • [160].Cancers. 2021.
  • [161].Diagnostics. 2022.
  • [162].S. Sudharson, P. Kokil. An ensemble of deep neural networks for kidney ultrasound image classification. Computer Methods and Programs in Biomedicine, vol. 197, Article number 105709, 2020. DOI: 10.1016/j.cmpb.2020.105709. [DOI] [PubMed]
  • [163].Bai W J, Chen C, Tarroni G, Duan J M, Guitton F, Petersen S E, Guo Y K, Matthews P M, Rueckert D. Self-supervised learning for cardiac MR image segmentation by anatomical position prediction; Shenzhen, China: Springer; 2019. pp. 541–549. [Google Scholar]
  • [164].Li Y X, Chen J W, Xie X P, Ma K, Zheng Y F. Self-loop uncertainty: A novel pseudo-label for semi-supervised medical image segmentation; Lima, Peru: Springer; 2020. pp. 614–623. [Google Scholar]
  • [165].C. Doersch, A. Gupta, A. A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, pp. 1422–1430, 2015. DOI: 10.1109/ICCV.2015.167.
  • [166].D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 2536–2544, 2016. DOI: 10.1109/CVPR.2016.278.
  • [167].L. Chen, P. Bentley, K. Mori, K. Misawa, M. Fujiwara, D. Rueckert. Self-supervised learning for medical image analysis using image context restoration. Medical Image Analysis, vol. 58, Article number 101539, 2019. DOI: 10.1016/j.media.2019.101539. [DOI] [PMC free article] [PubMed]
  • [168].X. L. Zhu, J. W. Yao, J. Z. Huang. Deep convolutional neural network for survival analysis with pathological images. In Proceedings of IEEE International Conference on Bioinformatics and Biomedicine, Shenzhen, China, pp. 544–547, 2016. DOI: 10.1109/BIBM.2016.7822579.
  • [169].X. L. Zhu, J. W. Yao, F. Y. Zhu, J. Z. Huang. WSISA: Making survival prediction from whole slide histopathological images. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 7234–7242, 2017. DOI: 10.1109/CVPR.2017.725.
  • [170].Genome Medicine. 2021.
  • [171].Li Y, Wang L, Wang J, Ye J P, Reddy C K. Transfer learning for survival analysis via efficient L2, 1-norm regularized cox regression; Barcelona, Spain: IEEE; 2016. pp. 231–240. [Google Scholar]
  • [172].Agravat R R, Raval M S. Brain tumor segmentation and survival prediction; Shenzhen, China: Springer; 2019. pp. 338–348. [Google Scholar]
  • [173].Setio A A A, Traverso A, De Bel T, Berens M S N, Van Den Bogaard C, Cerello P, Chen H, Dou Q, Fantacci M E, Geurts B, Van Den Gugten R, Heng P A, Jansen B, De Kaste M M J, Kotov V, Lin J Y H, Manders J T M C, Sóñora-Mengana A, García-Naranjo J C, Papavasileiou E, Prokop M, Saletta M, Schaefer-Prokop C M, Scholten E T, Scholten L, Snoeren M M, Torres E L, Vandemeulebroucke J, Walasek N, Zuidhof G C A, Van Ginneken B, Jacobs C. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: The LUNA16 challenge. Medical Image Analysis. 2017;42:1–13. doi: 10.1016/j.media.2017.06.015. [DOI] [PubMed] [Google Scholar]
  • [174].Chen R J, Chen C K, Li Y C, Chen T Y, Trister A D, Krishnan R G, Mahmood F. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning; New Orleans, USA: IEEE; 2022. pp. 16123–16134. [Google Scholar]
  • [175].Xu Y W, Hosny A, Zeleznik R, Parmar C, Coroller T, Franco I, Mak R H, Aerts H J W L. Deep learning predicts lung cancer treatment response from serial medical imaging. Clinical Cancer Research. 2019;25(11):3266–3275. doi: 10.1158/1078-0432.CCR-18-2495. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [176].Scientific Reports. 2021.
  • [177].Konwer A, Xu X, Bae J, Chen C, Prasanna P. Temporal context matters: Enhancing single image prediction with disease progression representations; New Orleans, USA: IEEE; 2022. pp. 18802–18813. [Google Scholar]
  • [178].Gao R Q, Huo Y K, Bao S X, Tang Y C, Antic S L, Epstein E S, Balar A B, Deppen S, Paulson A B, Sandler K L, Massion P P, Landman B A. Distanced LSTM: Time-distanced gates in long short-term memory models for lung cancer detection; Shenzhen, China: Springer; 2019. pp. 310–318. [PMC free article] [PubMed] [Google Scholar]
  • [179].Ouyang J, Zhao Q Y, Adeli E, Sullivan E V, Pfefferbaum A, Zaharchuk G, Pohl K M. Self-supervised longitudinal neighbourhood embedding; Strasbourg, France: Springer; 2021. pp. 80–89. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [180].K. Antczak. Deep recurrent neural networks for ECG signal denoising. [Online], Available: https://arxiv.org/abs/1807.11551, 2018.
  • [181].K. Antczak. A generative adversarial approach to ECG synthesis and denoising. [Online], Available: https://arxiv.org/abs/2009.02700, 2020.
  • [182].Rodrigues R, Couto P. Proceedings of Computing in Cardiology. Brno, Czech Republic: IEEE; 2021. Semi-supervised learning for ECG classification. [Google Scholar]
  • [183].Jang J H, Kim T Y, Yoon D. Effectiveness of transfer learning for deep learning-based electrocardiogram analysis. Healthcare Informatics Research. 2021;27(1):19–28. doi: 10.4258/hir.2021.27.1.19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [184].Almalchy M T, ALGayar S M S, Popescu N. Atrial fibrillation automatic diagnosis based on ECG signal using pretrained deep convolution neural network and SVM multiclass model; Bucharest, Romania: IEEE; 2020. pp. 197–202. [Google Scholar]
  • [185].Qayyum A, Mériaudeau F, Chan G C Y. Classification of atrial fibrillation with pre-trained convolutional neural network models; Sarawak, Malaysia: IEEE; 2018. pp. 594–599. [Google Scholar]
  • [186].D. Kiyasseh, T. T. Zhu, D. A. Clifton. CLOCS: Contrastive learning of cardiac signals across space, time, and patients. In Proceedings of the 38th International Conference on Machine Learning, pp. 5606–5615, 2021.
  • [187].Gedon D, Ribeiro A H, Wahlström N, Schön T B. First steps towards self-supervised pretraining of the 12-lead ECG; Brno, Czech Republic: IEEE; 2021. [Google Scholar]
  • [188].T. Mehari, N. Strodthoff. Self-supervised representation learning from 12-lead ECG data. Computers in Biology and Medicine, vol. 141, Article number 105114, 2022. DOI: 10.1016/j.compbiomed.2021.105114. [DOI] [PubMed]
  • [189].H. Liu, Z. B. Zhao, Q. She. Self-supervised ECG pre-training. Biomedical Signal Processing and Control, vol. 70, Article number 103010, 2021. DOI: 10.1016/j.bspc.2021.103010.
  • [190].J. Y. Cheng, H. Goh, K. Dogrusoz, O. Tuzel, E. Azemi. Subject-aware contrastive learning for biosignals. [Online], Available: https://arxiv.org/abs/2007.04871, 2020.
  • [191].X. Zhang, Z. Y. Zhao, T. Tsiligkaridis, M. Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. [Online], Available: https://arxiv.org/abs2206.08496, 2022.
  • [192].Chen H, Wang G J, Zhang G D, Zhang P, Yang H Z. CLECG: A novel contrastive learning framework for electrocardiogram arrhythmia classification. IEEE Signal Processing Letters. 2021;28:1993–1997. doi: 10.1109/LSP.2021.3114119. [DOI] [Google Scholar]
  • [193].K. Radhika, V. R. M. Oruganti. Transfer learning for subject-independent stress detection using physiological signals. In Proceedings of the 17th IEEE India Council International Conference, New Delhi, India, 2020. DOI: 10.1109/INDICON49873.2020.9342505.
  • [194].Scientific Reports. 2021.
  • [195].Sarkar P, Etemad A. Self-supervised ECG representation learning for emotion recognition. IEEE Transactions on Affective Computing. 2022;13(3):1541–1554. doi: 10.1109/TAFFC.2020.3014842. [DOI] [Google Scholar]
  • [196].Aston P J, Lyle J V, Bonet-Luz E, Huang C L, Zhang Y M, Jeevaratnam K, Nandi M. Deep learning applied to attractor images derived from ECG signals for detection of genetic mutation; Singapore: IEEE; 2019. pp. 1–4. [Google Scholar]
  • [197].Sensors. 2020.
  • [198].S. Bagherzadeh, K. Maghooli, A. Shalbaf, A. Maghsoudi. Recognition of emotional states using frequency effective connectivity maps through transfer learning approach from electroencephalogram signals. Biomedical Signal Processing and Control, vol. 75, Article number 103544, 2022. DOI: 10.1016/j.bspc.2022.103544.
  • [199].M. N. Mohsenvand, M. R. Izadi, P. Maes. Contrastive representation learning for electroencephalogram classification. In Proceedings of the Machine Learning for Health, pp. 238–253, 2020.
  • [200].Raghu S, Sriraam N, Temel Y, Rao S V, Kubben P L. EEG based multi-class seizure type classification using convolutional neural network and transfer learning. PMLR Neural Networks. 2020;124:202–212. doi: 10.1016/j.neunet.2020.01.017. [DOI] [PubMed] [Google Scholar]
  • [201].Nogay H S, Adeli H. Detection of epileptic seizure using pretrained deep convolutional neural network and transfer learning. European Neurology. 2020;83(6):602–614. doi: 10.1159/000512985. [DOI] [PubMed] [Google Scholar]
  • [202].S. Y. Tang, J. Dunnmon, K. K. Saab, X. Zhang, Q. Y. Huang, F. Dubost, D. Rubin, C. Lee-Messer. Self-supervised graph neural networks for improved electroencephalographic seizure analysis. In Proceedings of the 10th International Conference on Learning Representations, 2022.
  • [203].J. J. Xu, Y. J. Zheng, Y. F. Mao, R. X. Wang, W. S. Zheng. Anomaly detection on electroencephalography with self-supervised learning. In Proceedings of IEEE International Conference on Bioinformatics and Biomedicine, Seoul, Republic of Korea, pp. 363–368, 2020. DOI: 10.1109/BIBM49941.2020.9313163.
  • [204].Journal of Neural Engineering. 2021.
  • [205].M. T. Sadiq, M. Z. Aziz, A. Almogren, A. Yousaf, S. Siuly, A. U. Rehman. Exploiting pretrained CNN models for the development of an EEG-based robust BCI framework. Computers in Biology and Medicine, vol. 143, Article number 105242, 2022. DOI: 10.1016/j.compbiomed.2022.105242. [DOI] [PubMed]
  • [206].Ou Y H, Sun S Q, Gan H T, Zhou R, Yang Z. An improved self-supervised learning for EEG classification. Mathematical Biosciences and Engineering. 2022;19(7):6907–6922. doi: 10.3934/mbe.2022325. [DOI] [PubMed] [Google Scholar]
  • [207].Phan H, Chén O Y, Koch P, Lu Z Q, McLoughlin I, Mertins A, De Vos M. Towards more accurate automatic sleep staging via deep transfer learning. IEEE Transactions on Biomedical Engineering. 2021;68(6):1787–1798. doi: 10.1109/TBME.2020.3020381. [DOI] [PubMed] [Google Scholar]
  • [208].Jiang X, Zhao J H, Du B, Yuan Z Y. Self-supervised contrastive learning for EEG-based sleep staging; Shenzhen, China: IEEE; 2021. [Google Scholar]
  • [209].N. Wagh, J. H. Wei, S. Rawal, B. M. Berry, L. Barnard, B. Brinkmann, G. Worrell, D. Jones, Y. Varatharajah. Domain-guided self-supervision of EEG data improves downstream classification performance and generalizability. In Proceedings of Machine Learning for Health, pp. 130–142, 2021.
  • [210].Picard R W, Vyzas E, Healey J. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2001;23(10):1175–1191. doi: 10.1109/34.954607. [DOI] [Google Scholar]
  • [211].Simkin D R, Thatcher R W, Lubar J. Quantitative EEG and neurofeedback in children and adolescents: Anxiety disorders, depressive disorders, comorbid addiction and attention-deficit/hyperactivity disorder, and brain injury. Child and Adolescent Psychiatric Clinics of North America. 2014;23(3):427–464. doi: 10.1016/j.chc.2014.03.001. [DOI] [PubMed] [Google Scholar]
  • [212].Zhao G Z, Ge Y, Shen B Y, Wei X J, Wang H. Emotion analysis for personality inference from EEG signals. IEEE Transactions on Affective Computing. 2017;9(3):362–371. doi: 10.1109/TAFFC.2017.2786207. [DOI] [Google Scholar]
  • [213].Lu N, Li T F, Ren X D, Miao H Y. A deep learning scheme for motor imagery classification based on restricted boltzmann machines. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2017;25(6):566–576. doi: 10.1109/TNSRE.2016.2601240. [DOI] [PubMed] [Google Scholar]
  • [214].M. McDermott, B. Nestor, E. Kim, W. C. Zhang, A. Goldenberg, P. Szolovits, M. Ghassemi. A comprehensive EHR timeseries pre-training benchmark. In Proceedings of the Conference on Health, Inference, and Learning, ACM, pp. 257–278, 2021. DOI: 10.1145/3450439.3451877.
  • [215].Digital Medicine. 2021.
  • [216].X. Xu, X. Xu, Y. Y. Sun, X. S. Liu, X. Li, G. T. Xie, F. Wang. Predictive modeling of clinical events with mutual enhancement between longitudinal patient records and medical knowledge graph. In Proceedings of IEEE International Conference on Data Mining, Auckland, New Zealand, pp. 777–786, 2021. DOI: 10.1109/ICDM51629.2021.00089.
  • [217].Y. Xue, N. Du, A. Mottram, M. Seneviratne, A. M. Dai. Learning to select best forecast tasks for clinical outcome prediction. In Proceedings of the 34th Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 15031–15041, 2020.
  • [218].ACM Transactions on Knowledge Discovery from Data. 2022.
  • [219].H. X. Ren, J. Y. Wang, W. X. Zhao, N. Wu. RAPT: Pre-training of time-aware transformer for learning robust healthcare representation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Singapore, pp. 3503–3511, 2021. DOI: 10.1145/3447548.3467069.
  • [220].B. van Aken, J. M. Papaioannou, M. Mayrdorfer, K. Budde, F. Gers, A. Löser. Clinical outcome prediction from admission notes using self-supervised knowledge integration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pp.881–893, 2021. DOI: 10.18653/v1/2021.eacl-main.75.
  • [221].C. Lu, C. K. Reddy, Y. Ning. Self-supervised graph learning with hyperbolic embedding for temporal health event prediction. IEEE Transactions on Cybernetics, to be published. DOI: 10.1109/TCYB.2021.3109881. [DOI] [PubMed]
  • [222].K. Hur, J. Lee, J. Oh, W. Price, Y. Kim, E. Choi. Unifying heterogeneous electronic health records systems via text-based code embedding. In Proceedings of Conference on Health, Inference, and Learning, pp.183–203, 2022.
  • [223].Biswal S, Xiao C, Glass L M, Milkovits E, Sun J M. Doctor2Vec: Dynamic doctor representation learning for clinical trial recruitment. Proceedings of AAAI Conference on Artificial Intelligence. 2020;34(1):557–564. doi: 10.1609/aaai.v34i01.5394. [DOI] [Google Scholar]
  • [224].Journal of Medical Internet Research. 2021.
  • [225].E. Lehman, S. Jain, K. Pichotta, Y. Goldberg, B. C. Wallace. Does BERT pretrained on clinical notes reveal sensitive data? In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 946–959, 2021. DOI: 10.18653/v1/2021.naacl-main.73.
  • [226].Zhang X Y, Xiao C, Glass L M, Sun J M. DeepEnroll: Patient-trial matching with deep embedding and entailment prediction; Taipei, China: ACM; 2020. pp. 1029–1037. [Google Scholar]
  • [227].H. D. Hlynsson, S. Ellertsson, J. F. Daðason, E. L. Sigurdsson, H. Loftsson. Semi-self-supervised automated ICD coding. [Online], Available: https://arxiv.org/abs/2205.10088, 2022.
  • [228].Z. Zhang, J. S. Liu, N. Razavian. BERT-XML: Large scale automated ICD coding using BERT pretraining. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pp. 24–34, 2020. DOI: 10.18653/v1/2020.clinicalnlp-1.3.
  • [229].Y. Q. Su, Y. L. Shi, W. Lee, L. Cheng, H. M. Guo. TAHDNet: Time-aware hierarchical dependency network for medication recommendation. Journal of Biomedical Informatics, vol. 129, Article number 104069, 2022. DOI: 10.1016/j.jbi.2022.104069. [DOI] [PubMed]
  • [230].J. Y. Shang, T. F. Ma, C. Xiao, J. M. Sun. Pre-training of graph augmented transformers for medication recommendation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China, pp. 5953–5959, 2019. DOI: 10.24963/ijcai.2019/825.
  • [231].Z. Sun, S. L. Peng, Y. N. Yang, X. Q. Wang, F. Li. A general fine-tuned transfer learning model for predicting clinical task acrossing diverse EHRs datasets. In Proceedings of IEEE International Conference on Bioinformatics and Biomedicine, San Diego, USA, pp. 490–495, 2019. DOI: 10.1109/BIBM47256.2019.8983098.
  • [232].Ma L T, Ma X Y, Gao J Y, Jiao X F, Yu Z H, Zhang C H, Ruan W J, Wang Y S, Tang W, Wang J T. Distilling knowledge from publicly available online EMR data to emerging epidemic for prognosis; Ljubljana, Slovenia: ACM; 2021. pp. 3558–3568. [Google Scholar]
  • [233].Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi J C, Saunders L D, Beck C A, Feasby T E, Ghali W A. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Medical Care. 2005;43(11):1130–1139. doi: 10.1097/01.mlr.0000182534.19832.83. [DOI] [PubMed] [Google Scholar]
  • [234].Y. Y. Zhang, X. Wu, Q. Fang, S. S. Qian, C. S. Xu. Knowledge-enhanced attributed multi-task learning for medicine recommendation. ACM Transactions on Inormation Systems, to be published. DOI: 10.1145/3527662.
  • [235].Y. K. Li, H. Y. Wang, Y. Luo. A comparison of pre-trained vision-and-language models or multimodal representation learning across medical images and reports. In Proceedings of IEEE International Conference on Bioinformatics and Biomedicine, Seoul, Republic of Korea, pp. 1999–2004, 2020. DOI: 10.1109/BIBM49941.2020.9313289.
  • [236].L. H. Li, M. Yatskar, D. Yin, C. J. Hsieh, K. W. Chang. VisualBERT: A simple and performant baseline or vision and language. [Online], Available: https://arxiv.org/abs/1908.03557, 2019.
  • [237].Y. C. Chen, L. J. Li, L. C. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, J. J. Liu. UNITER: UNiversal Image-TExt representation learning. [Online], Available: https://arxiv.org/abs/1909.11740, 2019.
  • [238].H. Tan, M. Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China, pp. 5100–5111, 2019. DOI: 10.18653/v1/D19-1514.
  • [239].Z. C. Huang, Z. Y. Zeng, B. Liu, D. M. Fu, J. L. Fu. Pixel-BERT: Aligning image pixels with text by deep multi-modal transformers. [Online], Available: https://arxiv.org/abs/2004.00849, 2020.
  • [240].Y. Khare, V. Bagal, M. Mathew, A. Devi, U. D. Priyakumar, C. V. Jawahar. MMBERT: Multimodal BERT pre-training for improved medical VQA. In Proceedings of the 18th IEEE International Symposium on Biomedical Imaging, Nice, France, pp. 1033–1036, 2021. DOI: 10.1109/ISBI48211.2021.9434063.
  • [241].Digital Medicine. 2020.
  • [242].M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, L. Zhang. Deep learning with differential privacy. In Proceedings of ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, pp. 308–318, 2016. DOI: 10.1145/2976749.2978318.

Articles from Machine Intelligence Research are provided here courtesy of Nature Publishing Group

RESOURCES