Table 3:
MultiZoo provides a standardized implementation of the following multimodal methods to enable accessibility for new researchers and reproducibility of results. These approaches span advances in data processing, fusion paradigms, optimization objectives, and training procedures. We choose these approaches since they offer complementary perspectives towards tacking the fundamental challenges in multimodal fusion: (1) aligning signals across modalities at the right granularity, (2) learning complementary information across aligned signals, and (3) maintaining robustness in the presence of noisy and missing modalities.
| Category | Method | Alignment | Complementarity | Robustness |
|---|---|---|---|---|
| Data | WordAlign [26] | ✓ | ✗ | ✗ |
|
| ||||
| Model | EF, LF [10] | ✗ | ✓ | ✗ |
| TF [179], LRTF [106] | ✗ | ✓ | ✗ | |
| MI-Matrix, MI-Vector, MI-Scalar [77] | ✗ | ✓ | ✗ | |
| NL Gate [167] | ✗ | ✓ | ✗ | |
| MulT [154] | ✓ | ✓ | ✗ | |
| MFAS [122] | ✗ | ✓ | ✗ | |
|
| ||||
| Objective | CCA [7] | ✓ | ✗ | ✗ |
| RefNet [135] | ✓ | ✗ | ✗ | |
| MFM [155] | ✓ | ✓ | ✗ | |
| MVAE [168] | ✗ | ✓ | ✗ | |
| MCTN [123] | ✗ | ✗ | ✓ | |
|
| ||||
| Training | GRadBlend [167] | ✗ | ✓ | ✓ |
| RMFE [53] | ✗ | ✓ | ✓ | |