Table 2.

Datasets.

Acronym	Dataset/Database Name	Environment	Modality	Samples	Emotions
AffectNet [67]	Facial Affect from the InterNet	Wild	Facial	456,000 images	80,276 Neutral, 146,198 Happy, 29,487 Sad, 16,288 Surprise, 8191 Fear, 5264 Disgust, 28,130 Anger, 5135 Contempt, 35,322 None, 13,163 Uncertain, 88,895 Non-Face
AFEW 4.0 [68]	Acted Facial Expression in the Wild	Wild	Facial, audio, and posture	1426 video clips	194 Anger, 123 Disgust, 156 Fear, 165 Sadness, 387 Happiness, 257 Neutral, 144 Surprise
AFEW 5.0 [69]	Acted Facial Expression in the Wild	Wild	Facial, audio, and posture	1645 video clips	Angry, Disgust, Fear, Sadness, Happiness, Neutral, Surprise
AFEW 7.0 [70]	Acted Facial Expression in the Wild	Wild	Facial, audio, and posture	1809 video clips	295 Angry, 154 Disgust, 197 Fear, 258 Sadness, 357 Happiness, 400 Neutral, 148 Surprise
AffWild2 [57]	AffWild2 database	Wild	Facial	558 Videos with $2.8$ M frames	22,699 Anger, 16,067 Disgust, 17,488 Fear, 129,974 Happiness, 259,456 Neutral, 272,310 Other, 103,908 Sadness, 43,947 Surprise
CK+ [71]	Extended Cohn- Kanade Dataset (CK+)	Lab	Facial	327 labeled facial videos	45 Angry, 18 Contempt, 59 Disgust, 25 Fear, 69 Happy, 28 Sadness, 83 Surprise
FER-2013 [72]	Facial Expression Recognition 2013 (FER-2013)	Wild	Facial	35,887 images	4953 Anger, 547 Disgust, 5121 Fear, 8989 Happiness, 6077 Sadness, 4002 Surprise, 6198 Neutral
HEU-part1 [41]	Multimodal emotion	Wild	Face and posture	16,569 videos	1631 Anger, 691 Bored, 1662 Confused, 1097 Disappointed, 881 Disgust, 1268 Fear, 4313 Happy, 2568 Neutral, 1529 Sad, 929 Surprise
HEU-part2 [41]	Multimodal emotion	Wild	Face, speech and posture	2435 videos	389 Anger, 50 Bored, 131 Confused, 108 Disappointed, 162 Disgust, 221 Fear, 460 Happy, 350 Neutral, 364 Sad, 243 Surprise
SFEW 2.0 [73]	Static Facial Expression in Wild	Wild	Facial	1322 samples	255 anger, 75 disgust, 124 fear, 228 neutral, 256 happiness, 234 sadness and 150 surprise
RAF-DB [74]	Real-world Affective Faces Database (RAF-DB)	Wild	Facial	30,000 facial images	1619 Surprised, 355 Fearful, 877 Disgusted, 5957 Happy, 2460 Sad, 867 Angry, 560 Fearfully surprised, 148 Disgustedly surprised, 697 Happily surprised, 86 Sadly surprised, 176 Angrily surprised, 129 Sadly fearful, 8 Fearfully disgusted, 266 Happily disgusted, 738 Sadly disgusted, 841 Angrily disgusted, 150 Fearfully angry, 163 Sadly angry
MELD [52]	Multimodal EmotionLines Dataset	Wild	Audio, visual, and textual	13,158 videos	1607 anger, 361 disgust, 1002 sadness, 2308 joy, 6436 neutral, 1636 surprise, 358 fear
CMU-MOSEI [75]	CMU Multimodal Opinion Sentiment and Emotion Intensity	Wild	Language, vision, and audio	23,453 annotated sentences from more than 1000 online speakers and 250 different topics	1438 happy, 502 sad, 384 angry, 55 fear, 150 disgust, 78 surprise
IEMOCAP [76]	Interactive emotional dyadic motion capture database	Lab	Audio, video and facial	10,038 samples	1229 anger, 1182 sadness, 495 happiness, 575 neutral, 2505 excited, 24 surprise, 135 fear, 4 disgust, 3830 frustration, 59 other