research_article September 30, 2026

A Unified Uncertainty-Aware and Imbalance-Resilient Framework for Robust Multimodal Emotion Recognition

Qasim Mahdi Haref, Murteza Hanoon Tuama, Wahhab Muslim Mashloosh
Volume1
Issue3
Pages138-144
DOI10.36099/ijaac.v1i3.002
Abstract

The multimodal emotion recognition has become a key field of research because of its importance in human–computer interaction, affective computing and intelligent decision support systems. But these techniques are generally hampered by modality imbalance, class distribution imbalance and propagation of uncertainty across heterogeneous feature space. To solve these problems, the proposed study provides a unified and modular multimodal learning framework based on four complementary enhancement mechanisms to boost the robustness, generalization, and interpretability. The proposed framework consists of (i) an uncertainty-aware adaptive gating mechanism to control the contributions from the different modality, (ii) a multi-sample dropout strategy for better regularization and to prevent overfitting, (iii) a contrastive representation alignment module to ensure the consistency of the different modalities, and (iv) a logit-adjusted learning scheme to deal with the issue of class imbalance. All are organized as separate components, which allows for flexible activation and for systematic ablation analysis. The proposed framework consistently outperforms baseline emotion models and all individual variants of the proposed enhancement on the weighted F1-score, macro F1-score, and overall classification accuracy on extensive experiments conducted on a large-scale multimodal emotion dataset. The ablation results also validate the contribution of uncertainty modeling and imbalance-aware learning, as they yield the best performance. Interestingly, the full configuration works well on CPU-based environments and provides the best generalization performance. The work presented in this thesis makes a significant contribution to the scalable, interpretable, and practically deployable multimodal emotion recognition framework, and provides insights into future research for developing adaptive multimodal learning and powerful affective computing systems.

Keywords
Multimodal Emotion RecognitionUncertainty-Aware LearningClass Imbalance MitigationAdaptive FusionContrastive Representation LearningAffective Computing
Download Article PDF
References
  1. Tuama, M. H., Mashloosh, W. M., Albehadili, H. M., Alazzawi, M., & Al-Shareeda, M. A. (2023). Beyond polarity: The potential applications and impacts of sentiment analysis and emotion detection. AlKadhim Journal for Computer Science, 1(2), 44–51. https://doi.org/10.61710/akjs.v1i2.51
  2. Wu, Y., Mi, Q., & Gao, T. (2025). Multimodal emotion recognition: Techniques, challenges, and future directions. Biomimetics, 10(7), 418. https://doi.org/10.3390/biomimetics10070418
  3. Wu, C.-Y., Cai, Y., Liu, Y., Peng, Z., Xue, Y., & Gong, Z. (2025). Multimodal emotion recognition in conversations: A survey of methods, trends, challenges and prospects. Findings of the Association for Computational Linguistics: EMNLP 2025, 6257–6274.
  4. Hamidi, M. A. M., Taqa, A. Y., & Ibrahim, Y. I. (2025). A systematic review of multimodal sentiment analysis based on text–image fusion. Sinkron: Jurnal Dan Penelitian Teknik Informatika, 9(2), 987–999. https://doi.org/10.33395/sinkron.v9i2.14840
  5. Devarajan, K. (2025). Enhancing emotion recognition through multimodal data and graph neural networks. Intelligent Systems with Applications, 17, 200208. https://doi.org/10.1016/j.iswa.2025.200208
  6. Chen, G. (2025). Multimodal emotion recognition via the fusion of Mamba and liquid-Mamba architectures. Electronics, 14(18), 3638. https://doi.org/10.3390/electronics14183638
  7. Al-Azani, S., El-Alfy, E.-S. M., & Al-Dhelaan, A. (2025). A review and critical analysis of multimodal datasets for emotion recognition. Artificial Intelligence Review, 58, 11271–11310. https://doi.org/10.1007/s10462-025-11271-1
  8. Zhang, R., Liu, J., & Huang, X. (2025). A comprehensive review of multimodal sentiment analysis in online public opinion. Big Data and Cognitive Computing, 9(1), 10. https://doi.org/10.3390/bdcc9010010
  9. Mandal, S., & Li, X. (2025). Rethinking multimodal sentiment analysis: A high-accuracy simplified fusion architecture. arXiv preprint. https://arxiv.org/abs/2505.04642
  10. Lee, J., Park, S., & Kim, H. (2025). Dynamic multimodal sentiment analysis leveraging cross-modal attention. arXiv preprint. https://arxiv.org/abs/2501.08085
  11. Li, Z., Wang, Y., Zhang, Q., & Zhao, D. (2024). M2SE: A multistage multitask instruction tuning strategy for unified sentiment and emotion analysis. arXiv preprint. https://arxiv.org/abs/2412.08049
  12. Farhadipour, A., Ranjbar, H., Chapariniya, M., Vukovic, T., Ebling, S., & Dellwo, V. (2025). Multimodal emotion recognition and sentiment analysis in multi-party conversation contexts. arXiv preprint. https://arxiv.org/abs/2503.06805
  13. Kumar, A., & Sharma, P. (2025). Survey on multimodal emotion recognition systems. International Journal of Computer Science and Information Technology, 17(2), 45–62.
  14. Ahmed, S., & Rahman, M. (2025). Systematic review of recent advances in multimodal sentiment analysis. arXiv preprint. https://arxiv.org/abs/2504.09321
  15. Liu, L. (2025). Multimodal emotion recognition method in complex contexts based on T5 transformer. Array, 18, 100331. https://doi.org/10.1016/j.array.2025.100331
  16. Hamidi, M. A. M., Taqa, A. Y., & Ibrahim, Y. I. (2025). A systematic review of multimodal sentiment analysis based on text–image fusion. Sinkron: Jurnal Dan Penelitian Teknik Informatika, 9(2), 987–999. https://doi.org/10.33395/sinkron.v9i2.14840
  17. Wu, C.-Y., Cai, Y., Liu, Y., Peng, Z., Xue, Y., & Gong, Z. (2025). Multimodal emotion recognition in conversations: A survey of methods, trends, challenges and prospects. Findings of the Association for Computational Linguistics: EMNLP 2025, 6257–6274.
  18. Chen, G. (2025). Multimodal emotion recognition via the fusion of Mamba and liquid-Mamba architectures. Electronics, 14(18), 3638. https://doi.org/10.3390/electronics14183638
  19. Al-Azani, S., El-Alfy, E.-S. M., & Al-Dhelaan, A. (2025). A review and critical analysis of multimodal datasets for emotion recognition. Artificial Intelligence Review, 58, 11271–11310. https://doi.org/10.1007/s10462-025-11271-1
  20. Liu, L. (2025). Multimodal emotion recognition method in complex contexts based on T5 transformer. Array, 18, 100331. https://doi.org/10.1016/j.array.2025.100331
  21. Li, Z., Wang, Y., Zhang, Q., & Zhao, D. (2024). M2SE: A multistage multitask instruction tuning strategy for unified sentiment and emotion analysis. arXiv preprint. https://arxiv.org/abs/2412.08049
  22. Farhadipour, A., Ranjbar, H., Chapariniya, M., Vukovic, T., Ebling, S., & Dellwo, V. (2025). Multimodal emotion recognition and sentiment analysis in multi-party conversation contexts. arXiv preprint. https://arxiv.org/abs/2503.06805
  23. Mandal, S., & Li, X. (2025). Rethinking multimodal sentiment analysis: A high-accuracy simplified fusion architecture. arXiv preprint. https://arxiv.org/abs/2505.04642
  24. Lee, J., Park, S., & Kim, H. (2025). Dynamic multimodal sentiment analysis leveraging cross-modal attention. arXiv preprint. https://arxiv.org/abs/2501.08085
  25. Wu, Y., Mi, Q., & Gao, T. (2025). Multimodal emotion recognition: Techniques, challenges, and future directions. Biomimetics, 10(7), 418. https://doi.org/10.3390/biomimetics10070418
  26. Zhang, R., Liu, J., & Huang, X. (2025). A comprehensive review of multimodal sentiment analysis in online public opinion. Big Data and Cognitive Computing, 9(1), 10. https://doi.org/10.3390/bdcc9010010
  27. Ahmed, S., & Rahman, M. (2025). Systematic review of recent advances in multimodal sentiment analysis. arXiv preprint. https://arxiv.org/abs/2504.09321
Scroll to Top