Multimodal AI System Combines Six Encoder Families Including Gemini 2.0 to Recognize Blended Emotions with 32% Accuracy

System uses late probability fusion of six encoder families: S4D-ViTMoE face encoder, Wav2Vec2 audio features, TimeSformer and VideoMAE body-language encoders, and Gemini Embedding 2.0

Frozen Wav2Vec2 prosody layers (6-12) outperform fine-tuning approaches (0.207 vs 0.161 score) by avoiding irrelevant phonetic feature extraction

Gemini Embedding 2.0 achieves competitive 0.320 ACCP accuracy using only 2 seconds of video input, marking first use of large multimodal models in emotion recognition

Post-processing salience threshold varies significantly across data folds (0.05-0.43), indicating personalized expression styles require adaptive approaches