before_revision: We propose UniViLM: a Unified Video and Language pre-training Model for multimodal understanding and generation. Motivated by the recent success of BERT based pre-training technique for NLP and image-language tasks, VideoBERT and CBT are proposed to exploit BERT model for video and language pre-training using narrated instructional videos. Different from their works which only pre-train understanding task, we propose a unified video-language pre-training model for both understanding and generation tasks . Our model comprises of 4 components including two single-modal encoders, a cross encoder and a decoder with the Transformer backbone. We first pre-train our model to learn the universal representation for both video and language on a large instructional video dataset. Then we fine-tune the model on two multimodal tasks including understanding task (text-based video retrieval) and generation task (multimodal video captioning). Our extensive experiments show that our method can improve the performance of both understanding and generation tasks and achieves the state-of-the art results.
after_revision: With the recent success of pre-training technique for NLP and image-linguistic tasks, there are still few works on video-linguistic pre-training . Besides, most of the existing multimodal models are pre-trained for understanding task, which leads to a pretrain-finetune discrepency for generation tasks. In this paper, we propose UniViLM: a Unified Video and Language pre-training Model for both multimodal understanding and generation . Our model comprises of 4 components including two single-modal encoders, a cross encoder and a decoder with the Transformer backbone. We first pre-train our model to learn the universal representation for both video and language on a large instructional video dataset. Then we fine-tune the model on two multimodal tasks including understanding task (text-based video retrieval) and generation task (multimodal video captioning). Our extensive experiments show that our method can improve the performance of both understanding and generation tasks and achieves the state-of-the art results.
edit_actions: R
edit_actions: We propose UniViLM: a Unified Video and Language pre-training Model for multimodal understanding and generation. Motivated by
edit_actions: With
edit_actions: coherence
edit_actions: coherence clarity coherence
edit_actions: D
edit_actions: BERT based
edit_actions: coherence
edit_actions: style coherence coherence
edit_actions: R
edit_actions: image-language tasks, VideoBERT and CBT are proposed to exploit BERT model for video and language
edit_actions: image-linguistic tasks, there are still few works on video-linguistic
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: using narrated instructional videos. Different from their works which only pre-train
edit_actions: . Besides, most of the existing multimodal models are pre-trained for
edit_actions: coherence
edit_actions: coherence coherence clarity
edit_actions: R
edit_actions: we propose a unified video-language
edit_actions: which leads to a pretrain-finetune discrepency for generation tasks. In this paper, we propose UniViLM: a Unified Video and Language
edit_actions: meaning-changed
edit_actions: coherence meaning-changed meaning-changed
edit_actions: R
edit_actions: model for both
edit_actions: Model for both multimodal
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: D
edit_actions: tasks
edit_actions: clarity
edit_actions: coherence clarity clarity