before_revision: With the recent success of  pre-training technique for NLP and image-linguistic tasks, there are still few works on video-linguistic pre-training . Besides , most of the existing multimodal models are pre-trained for understanding task, which leads to a pretrain-finetune discrepency for generation tasks. In this paper , we propose UniViLM : a Unified Video and Language pre-training Model for both multimodal understanding and generation. Our model comprises of 4 components including two single-modal encoders, a cross encoder  and a decoder with the Transformer backbone. We first pre-train our model to learn the universal representation for both video and language on a large instructional video dataset. Then we fine-tune the model on two multimodal tasks including understanding task (text-based video retrieval) and generation task (multimodal video captioning). Our extensive experiments show that our method can improve the performance of both understanding and generation tasks and achieves the state-of-the art results .
after_revision: With the recent success of the pre-training technique for NLP and image-linguistic tasks, some video-linguistic pre-training works are gradually developed to improve video-text related downstream tasks. However , most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks. This paper proposes UniVL : a Unified Video and Language pre-training model for both multimodal understanding and generation. It comprises four components, including two single-modal encoders, a cross encoder , and a decoder with the Transformer backbone. Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The pre-train is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that the UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks .
edit_actions: A
edit_actions: the
edit_actions: coherence
edit_actions: fluency coherence coherence
edit_actions: R
edit_actions: there are still few works on
edit_actions: some
edit_actions: clarity
edit_actions: clarity clarity coherence
edit_actions: R
edit_actions: . Besides
edit_actions: works are gradually developed to improve video-text related downstream tasks. However
edit_actions: meaning-changed
edit_actions: meaning-changed coherence meaning-changed
edit_actions: R
edit_actions: task, which leads
edit_actions: tasks, leading
edit_actions: fluency
edit_actions: fluency style fluency
edit_actions: R
edit_actions: discrepency
edit_actions: discrepancy
edit_actions: fluency
edit_actions: fluency fluency fluency
edit_actions: R
edit_actions: In this paper , we propose UniViLM
edit_actions: This paper proposes UniVL
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: Model
edit_actions: model
edit_actions: fluency
edit_actions: fluency fluency fluency
edit_actions: R
edit_actions: Our model comprises of 4 components
edit_actions: It comprises four components,
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: A
edit_actions: ,
edit_actions: fluency
edit_actions: fluency fluency fluency
edit_actions: R
edit_actions: We first
edit_actions: Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: our model to learn the universal representation for both video and language on a large instructional video dataset. Then we fine-tune the model on two multimodal tasks including understanding task (text-based video retrieval) and generation task (multimodal video captioning). Our extensive experiments show that our method can improve the performance of both understanding and generation tasks and achieves
edit_actions: is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that
edit_actions: clarity
edit_actions: clarity meaning-changed clarity
edit_actions: R
edit_actions: state-of-the art results
edit_actions: UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed style