before_revision: We introduce sentenceMIM, a probabilistic auto-encoder for language modelling , trained with Mutual Information Machine (MIM) learning  . Previous attempts to learn variational auto-encoders for language data have had mixed success, with empirical performance well below state-of-the-art auto-regressive models, a key barrier being the occurrence of posterior collapsewith VAEs. The recently proposed MIM framework encourages high mutual information between observations and latent variables, and is more robust against posterior collapse. This paper formulates a MIM model for text data, along with a corresponding learning algorithm. We demonstrate excellent perplexity (PPL) results on several datasets, and show that the framework learns a rich latent space, allowing for interpolation between sentences of different lengths with a fixed-dimensional latent representation. We also demonstrate the versatility of sentenceMIM by utilizing a trained model for question-answering , a transfer learningtask , without fine-tuning . To the best of our knowledge, this is the first latent variable model (LVM) for text modelling that achieves competitive performance with non-LVM models .
after_revision: SentenceMIM is a probabilistic auto-encoder for language data , trained with Mutual Information Machine (MIM) learning to provide a fixed length representation of variable length language observations (ie, similar to VAE) . Previous attempts to learn VAEs for language data faced challenges due to posterior collapse. MIM learning encourages high mutual information between observations and latent variables, and is robust against posterior collapse. As such, it learns informative representations whose dimension can be an order of magnitude higher than existing language VAEs. Importantly, the SentenceMIM loss has no hyper-parameters, simplifying optimization. We compare sentenceMIM with VAE, and AE on multiple datasets. SentenceMIM yields excellent reconstruction, comparable to AEs, with a rich structured latent space, comparable to VAEs. The structured latent representation is demonstrated with interpolation between sentences of different lengths . We demonstrate the versatility of sentenceMIM by utilizing a trained model for question-answering and transfer learning , without fine-tuning , outperforming VAE and AE with similar architectures .
edit_actions: R
edit_actions: We introduce sentenceMIM,
edit_actions: SentenceMIM is
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: modelling
edit_actions: data
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: A
edit_actions: to provide a fixed length representation of variable length language observations (ie, similar to VAE)
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: variational auto-encoders
edit_actions: VAEs
edit_actions: clarity
edit_actions: clarity clarity style
edit_actions: R
edit_actions: have had mixed success, with empirical performance well below state-of-the-art auto-regressive models, a key barrier being the occurrence of posterior collapsewith VAEs. The recently proposed MIM framework
edit_actions: faced challenges due to posterior collapse. MIM learning
edit_actions: clarity
edit_actions: clarity clarity coherence
edit_actions: D
edit_actions: more
edit_actions: clarity
edit_actions: clarity clarity fluency
edit_actions: R
edit_actions: This paper formulates a MIM model for text data, along with a corresponding learning algorithm. We demonstrate excellent perplexity (PPL) results on several datasets, and show that the framework learns a rich
edit_actions: As such, it learns informative representations whose dimension can be an order of magnitude higher than existing language VAEs. Importantly, the SentenceMIM loss has no hyper-parameters, simplifying optimization. We compare sentenceMIM with VAE, and AE on multiple datasets. SentenceMIM yields excellent reconstruction, comparable to AEs, with a rich structured
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed clarity
edit_actions: R
edit_actions: allowing for
edit_actions: comparable to VAEs. The structured latent representation is demonstrated with
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: with a fixed-dimensional latent representation. We also
edit_actions: . We
edit_actions: clarity
edit_actions: clarity coherence clarity
edit_actions: R
edit_actions: , a transfer learningtask
edit_actions: and transfer learning
edit_actions: clarity
edit_actions: fluency clarity clarity
edit_actions: R
edit_actions: . To the best of our knowledge, this is the first latent variable model (LVM) for text modelling that achieves competitive performance with non-LVM models
edit_actions: , outperforming VAE and AE with similar architectures
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed