before_revision: We propose a novel methodology for analyzing the encoding of grammatical structure in neural language models through transfer learning. We test how a language model can leverage its internal representations to transfer knowledge across languages and symbol systems. We train LSTMs on non-linguistic , structured data and test their performance on human language to assess which kinds of data induce generalizable encodings that LSTMs can use for natural language. We find that models trained on structured data such as music and Java codehave internal representations that help in modelling human language, and that, surprisingly, adding minimal amounts of structure to the training data makes a large difference in transfer to natural language . Further experiments on transfer between human languages show that zero-shot performance on a test language is highly correlated with  syntactic similarity to the training language, even after removing any vocabulary overlap. This suggests that the internal representations induced from natural languages are typologically coherent: they encode the features and differences outlined in typological studies . Our results provide insights into how neural networks represent linguistic structure, and also about the kinds of structural biases that give learners the ability to model language.
after_revision: We propose transfer learning as a method for analyzing the encoding of grammatical structure in neural language models . We train LSTMs on non-linguistic data and evaluate their performance on natural language to assess which kinds of data induce generalizable structural features that LSTMs can use for natural language. We find that training on non-linguistic data with latent structure (MIDI music or Java code) improves test performance on natural language, despite no overlap in surface form or vocabulary. Training on artificial languages containing recursion (hierarchical structure) also improves performance on natural language, again with no vocabulary overlap. Surprisingly, training on artificial languages consisting of sets of separated pairs of words, but with no recursion, improves performance on natural language as well as recursive languages do. Experiments on transfer between natural languages show that zero-shot performance on a test language is highly correlated with typological syntactic similarity to the training language, suggesting that representations induced from natural languages correspond to the cross-linguistic syntactic properties studied in linguistic typology . Our results provide insights into the ways that neural models represent abstract syntactic structure, and also about the kind of structural inductive biases which a learner needs to model language.
edit_actions: R
edit_actions: a novel methodology
edit_actions: transfer learning as a method
edit_actions: clarity
edit_actions: clarity style clarity
edit_actions: R
edit_actions: through transfer learning. We test how a language model can leverage its internal representations to transfer knowledge across languages and symbol systems. We
edit_actions: . We
edit_actions: clarity
edit_actions: clarity style clarity
edit_actions: R
edit_actions: , structured data and test
edit_actions: data and evaluate
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: human
edit_actions: natural
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: encodings
edit_actions: structural features
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: R
edit_actions: models trained on structured data such as music and Java codehave internal representations that help in modelling human language, and that, surprisingly, adding minimal amounts of structure to the training data makes a large difference in transfer to natural language . Further experiments
edit_actions: training on non-linguistic data with latent structure (MIDI music or Java code) improves test performance on natural language, despite no overlap in surface form or vocabulary. Training on artificial languages containing recursion (hierarchical structure) also improves performance on natural language, again with no vocabulary overlap. Surprisingly, training on artificial languages consisting of sets of separated pairs of words, but with no recursion, improves performance on natural language as well as recursive languages do. Experiments
edit_actions: meaning-changed
edit_actions: meaning-changed clarity meaning-changed
edit_actions: R
edit_actions: human
edit_actions: natural
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: A
edit_actions: typological
edit_actions: meaning-changed
edit_actions: meaning-changed coherence meaning-changed
edit_actions: R
edit_actions: even after removing any vocabulary overlap. This suggests that the internal
edit_actions: suggesting that
edit_actions: clarity
edit_actions: fluency clarity clarity
edit_actions: R
edit_actions: are typologically coherent: they encode the features and differences outlined in typological studies
edit_actions: correspond to the cross-linguistic syntactic properties studied in linguistic typology
edit_actions: style
edit_actions: clarity style style
edit_actions: R
edit_actions: how neural networks represent linguistic
edit_actions: the ways that neural models represent abstract syntactic
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: kinds of structural biases that give learners the ability
edit_actions: kind of structural inductive biases which a learner needs
edit_actions: clarity
edit_actions: meaning-changed clarity clarity