Table 3.

Result of the literature search.

Reference	Method/Scale which was based	Database used (Sample)	Discrimination of facial regions	Sensitivity and specificity
14	Developed in the Delphi environment, based on image recognition of pain-related facial actions. /Scale: NFCS	Own Base - UNIFESP University Hospital (30 newborns between 35 and 41 weeks of gestational age).	Bulging brow; narrowing of the lid slit; deepening of the nasolabial furrow; open lips; mouth stretching.	The software exhibited 85% sensitivity and 100% specificity in detecting neutral facial expressions in their sting state and 100% sensitivity and specificity in detecting procedural pain in neonates.
23	A computational model with face detection, data augmentation, and classification model with transfer learning to a CNN architecture pre-trained by adding fully connected layers specifically trained with neonatal face images. The application was developed for the Android operating system using the Android Studio IDE. /Scale: NFCS	UNIFESP University Hospital (30 newborns between 35 and 41 weeks of gestational age) and Infant COPE (26 Caucasian neonates).	Not applicable.	This model achieved 93.07% accuracy, 0.9431 F1 Score, and 0.9254 AUC.
30	Report /Scale: Not applicable.	Not applicable.	Not applicable.	Not applicable.
30 cited 20	Compares the use of a novel Convolutional Neural Networks Neonatal along with others (ResNet50 and VGG-16) for pain assessment application. /Scale: NIPS	Infant COPE (26 Caucasian neonates) and NPAD (31 neonates between 32 and 40 weeks of gestational age).	Not applicable.	Assessing neonatal pain using LBP features achieved 86.8% average accuracy; Assessing neonatal pain using HOG features with Support vector machines achieved 81.29% average accuracy; Proposed N-CNN, which extracts features directly from the images, achieved state-of-the-art results and outperformed ResNet, VGG-16, as well as handcrafted descriptors.
30 cited 26	Existing static methods have been divided into two categories: handcrafted-representation-based methods and deep-representation-based methods. /Scale: NIPS	Infant COPE (26 Caucasian neonates).	Not applicable.	The system achieved 95.56% accuracy using decision fusion of different pain responses that were recorded in a challenging clinical environment.
31	Uses handcrafted algorithms and deep-learned features. /Scale: NIPS and NFCS	Own Base - APN-db (213 newborns between 26 and 41 weeks of gestational age).	Brow bulge, eye squeeze, nasolabial furrow, open lips, stretch mouth (vertical), stretch mouth (horizontal), lip purse, taut tongue, chin quiver.	The system performs well with an RMSE of 1.94 compared to human error of 1.65 on the same dataset, demonstrating its potential application to newborn health care.
32	The behavioral parameters related to movement and expression are measured using computer vision techniques. /Scale: NIPS, BPSN, DAN, NFCS, PIPP and CRIES	Not reported.	Head movement, expression of pain, frowning, lips movement, eyes open/closed, cheek frowning.	Not reported.
33	Uses facial electromyography to record facial muscle activity-related infant pain. /Scale: N-PASS, PIPP-R, NFCS, FLACC and VAS	Own Base (The painful procedures will be a minimum of 60 newborns and infants averaging 6 months).	Forehead, cheek, eyebrow puffing, eye pinch, and nasolabial sulcus.	Tests will be performed in further studies.
34	It was implemented with the client-server model and designed to run on the mobile nursing personal digital assistant device. /Scale: NIPS	Own Base (232 newborns with a mean gestational age of 33.93 ± 4.77 weeks).	Frown, eye squeezing, nasolabial fold deepening, mouth stretching, and tongue tightening.	The accuracies of the NIPS pain score and pain grade given by the automated NPA system were 88.79% and 95.25%, with kappa values of 0.92 and 0.90 (p< 0.001), respectively.
35	Identifying, transforming, and extracting the regions of interest from the face, assembling an average face of the newborns, and using similarity metrics to check for artifacts. /Scale: NFCS	UNIFESP University Hospital (30 newborns between 35 and 41 weeks of gestational age).	Eyebrows, eyes, nose, the region between the eyes, mouth, nasolabial folds, cheeks, and forehead.	Not reported. However, all images could be mapped and segmented by region.
36	The system consists of several algorithmic components, ranging from face detection, determination of the region of interest, and facial feature extraction to behavior stage classification. /Scale: Unmentioned	Own Base (newborn with different conditions)	Eyes, eyebrows, and mouth.	The algorithm can operate with approximately 88% accuracy.
37	The Local Binary Pattern features are computed in the Fuzzy k-NN classifier employed to classify newborn pain. /Scale: Unmentioned	Infant COPE (26 Caucasian neonates)	Not applicable.	Using the HOMO method, the sensitivity is 96.667%. Specificity ranged from 96.5% to 97.2% and accuracy ranged from 93.3% to 97.2% depending on the illumination. The fastest time consumption was obtained by Conventional Validation under 100 illumination levels with 0.065s.
38	Facial features were extracted through different image processing methods: placement and tracking of landmarks, edge detection, and binary thresholding. /Scale: NFCS, PIPP and DAN	Own Base - Ordine Mauriziano Hospital (15 healthy full-term neonates between 48 and 72 hours of life).	Eye squeeze (between mid-eyebrow and mid-lower eyelid), cheek raise (between eye medial corner and nose corner), brow bulging (between eyebrows medial border).	The overall result is not reported, but some operators' evaluations were particularly inconsistent regarding some parameters like face furrowing. For these parameters, the scores had very low consistency (about 40%).
39	The commonly used face detection methods are introduced first, and then, the convolutional neural network in deep learning is analyzed and improved and then applied to the facial recognition of newborns. /Scale: Used in Hubei hospital	Own Base - Hubei hospital (40 newborns with the age of no more than 7 days).	Not applicable.	The accuracy of the improved algorithm is 0.889, higher by 0.036 in contrast to other models; the area under the curve (AUC) of success rate reaches 0.748, higher by 0.075 compared with other algorithms.
40	Compare five pre-trained face detection models, proposing two new NICUface models. /Scale: Unmentioned	CHEO (33 newborns), COPE (27 newborns), and NBHR (257 patients)	Not applicable.	The proposed NICUface models outperform previous state-of-the-art models for neonatal face detection and are robust to many identified complex NICU scenes.

Note: APN-db, Acute Pain in Neonates; AUC, Area Under the Cuve; BPSN, Bernese Pain Scale for Neonates; CHEO, Children's Hospital of Eastern Ontario; COPE, Classification of Pain Expression; CRIES, C–Crying; R–Requires increased oxygen administration; I–Increased vital signs; E–Expression; S–Sleeplessness; DAN, DouleurAigue Nouveau-Né; FLACC, Face, Legs, Activity, Cry, Consolability; Fuzzy k-NN, Fuzzy K-Nearest Neighbor; HOG, Histogram of Oriented Gradients; HOMO, Homomorphic Filter; IDE, Integrated Development Environment; LBP, Local BinaryPattern; NBHR, Newborn Baby Heart Rate; N-CNN, Neonatal - Convolutional Neural Networks; NFCS, Neonatal FacialCoding System; NIPS, Neonatal Infant Pain Scale; NPA, Neonatal Pain Assessment; NPAD, Neonatal Pain Assessment Dataset; N-PASS, Neonatal Pain and Sedation Scale; PIPP, Premature Infant Pain Profile; PIPP-R, Premature Infant Pain Profile-Revised; RMSE, Root Mean Square Error; UNIFESP, Universidade Federal de São Paulo; VAS, Visual Analogue Scale; VGG, Visual Geometry Group.