Multi-modal self-paced learning for image classification

Abstract

Self-paced learning (SPL) is a powerful framework, where samples from easy ones to more complex ones are gradually involved in the learning process. Its superiority is significant when dealing with challenging vision tasks, like natural image classification. However, SPL based image classification can not deal with information from multiple modalities. As images are usually characterized by visual feature descriptors from multiple modalities, only exploiting one of them may lose some complementary information from other modalities. To overcome the above problem, we propose a multi-modal self-paced learning (MSPL) framework for image classification which jointly trains SPL and multi-modal learning into one framework. Specifically, the multi-modal learning process with curriculum information and the curriculum learning process with multi-modal information are iteratively performed until the final mature multi-modal curriculum is learned. As this multi-modal curriculum can grasp the easy to hard knowledge from both the sample level and the modality level, a better model can be learned. Experimental results on four real-world datasets demonstrate the effectiveness of the proposed approach.