before_revision: An end-to-end speech-to-text translation (ST) takes audio in a source language and outputs the text in a target language. Inspired by neuroscience, humans have perception systems and cognitive systems to process different information , we propose LUT, Listen-Understand-Translate,  a unified framework with triple supervision  to decouple the end-to-end speech-to-text translation task. In addition to the target language sentence translation loss, LUT includes two auxiliary supervising signals to guide the acoustic encoder to extracts acoustic features from the input, and the semantic encoder to extract semantic features relevant to the source transcription text. We do experiments on English-French,  English-German and  English-Chinese speech translation benchmarks and the results demonstrate the reasonability of LUT . Our code is available at URL
after_revision: An end-to-end speech-to-text translation (ST) takes audio in a source language and outputs the text in a target language. Existing methods are limited by the amount of parallel corpus. Can we build a system to fully utilize signals in a parallel ST corpus? We are inspired by human understanding system which is composed of auditory perception and cognitive processing. In this paper , we propose Listen-Understand-Translate, (LUT), a unified framework with triple supervision signals to decouple the end-to-end speech-to-text translation task. LUT is able to guide the acoustic encoder to extract as much information from the auditory input. In addition, LUT utilizes a pre-trained BERT model to enforce the upper encoder to produce as much semantic information as possible, without extra data. We perform experiments on a diverse set of speech translation benchmarks, including Librispeech English-French, IWSLT English-German and TED English-Chinese . Our results demonstrate LUT achieves the state-of-the-art performance, outperforming previous methods. The code is available at URL
edit_actions: R
edit_actions: Inspired by neuroscience, humans have perception systems and cognitive systems to process different information
edit_actions: Existing methods are limited by the amount of parallel corpus. Can we build a system to fully utilize signals in a parallel ST corpus? We are inspired by human understanding system which is composed of auditory perception and cognitive processing. In this paper
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: D
edit_actions: LUT,
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: A
edit_actions: (LUT),
edit_actions: clarity
edit_actions: meaning-changed clarity clarity
edit_actions: A
edit_actions: signals
edit_actions: clarity
edit_actions: clarity clarity meaning-changed
edit_actions: R
edit_actions: In addition to the target language sentence translation loss, LUT includes two auxiliary supervising signals
edit_actions: LUT is able
edit_actions: coherence
edit_actions: coherence clarity coherence
edit_actions: R
edit_actions: extracts acoustic features from the input, and the semantic encoder to extract semantic features relevant to the source transcription text. We do experiments on
edit_actions: extract as much information from the auditory input. In addition, LUT utilizes a pre-trained BERT model to enforce the upper encoder to produce as much semantic information as possible, without extra data. We perform experiments on a diverse set of speech translation benchmarks, including Librispeech
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: A
edit_actions: IWSLT
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed
edit_actions: A
edit_actions: TED
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed
edit_actions: R
edit_actions: speech translation benchmarks and the results demonstrate the reasonability of LUT . Our
edit_actions: . Our results demonstrate LUT achieves the state-of-the-art performance, outperforming previous methods. The
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed