Advances in Multimodal Models: Continual Learning and Trustworthiness

Multimodal models are recently gaining popularity and become widely used in both academia and industries. However, currently, most models face two challenges:

1. Fixed models do not have up-to-date knowledge, but direct finetuning will lead to catastrophic forgetting of models' pre-trained knowledge.
2. Models do not address trustworthy concerns such as safety and privacy.

In this tutorial, we aim to give a structured presentation of works to address the challenges above.

Continual learning (CL) aims to empower machine learning models to learn continually from new data, while building upon previously acquired knowledge without forgetting. As recent machine learning models have evolved from small to large pre-trained architectures, and from supporting unimodal to multimodal data, it has become more imperative and important to develop AI systems capable of learning continually from new datasets while maintaining computational and resource efficiency, hence the rise of multimodal continual learning (MMCL). We will present our novel works and also give descriptions of other people's excellent works, allowing the audience to gain a holistic perspective of this field.

Multimodal trustworthiness primarily focuses on enhancing the reliability and stability of multimodal systems, aiming to ensure more precise and credible outcomes during multimodal data fusion and decision-making processes. With the increased accessibility of multimodal data and the continuous advancement of multimodal large-scale model technologies, the trustworthiness and security of models have become increasingly significant, prompting extensive research interest within the academic community. We will present novel works which address various aspects of trustworthiness, including explainability, privacy, security, robustness, and fairness.