before_revision: We propose transfer learning as a method for analyzing the encoding of grammatical structure in neural language models. We train LSTMs on non-linguistic data and evaluate their performance on natural language to assess which kinds of data induce generalizable structural features that LSTMs can use for natural language. We find that training on non-linguistic data with latent structure (MIDI music or Java code) improves test performance on natural language, despite no overlap in surface form or vocabulary. Training on artificial languages containing recursion (hierarchical structure) also improves performance on natural language, again with no vocabulary overlap . Surprisingly, training  on artificial languages consisting of sets of separated pairs of words, but with no recursion, improves performance on natural language as well as recursive languages do. Experiments on transfer between natural languages  show that zero-shot performance on a test language is highly correlated with typological syntactic similarity to the training language, suggesting that representations induced from natural languages correspond to the cross-linguistic syntactic properties studied in linguistic typology . Our results provide insights into the ways that neural models represent abstract syntactic structure, and also about the kind of structural inductive biases which a learner needs to model language .
after_revision: We propose transfer learning as a method for analyzing the encoding of grammatical structure in neural language models. We train LSTMs on non-linguistic data and evaluate their performance on natural language to assess which kinds of data induce generalizable structural features that LSTMs can use for natural language. We find that training on non-linguistic data with latent structure (MIDI music or Java code) improves test performance on natural language, despite no overlap in surface form or vocabulary. To pinpoint the kinds of abstract structure that models may be encoding to lead to this improvement, we run similar experiments with two artificial parentheses languages: one which has a hierarchical recursive structure, and a control which has paired tokens but no recursion . Surprisingly, training a model on either of these artificial languages leads to the same substantial gains when testing on natural language . Further experiments on transfer between natural languages controlling for vocabulary overlap show that zero-shot performance on a test language is highly correlated with typological syntactic similarity to the training language, suggesting that representations induced by pre-training correspond to the cross-linguistic syntactic properties . Our results provide insights into the ways that neural models represent abstract syntactic structure, and also about the kind of structural inductive biases which allow for natural language acquisition .
edit_actions: R
edit_actions: Training on artificial languages containing recursion (hierarchical structure) also improves performance on natural language, again with no vocabulary overlap
edit_actions: To pinpoint the kinds of abstract structure that models may be encoding to lead to this improvement, we run similar experiments with two artificial parentheses languages: one which has a hierarchical recursive structure, and a control which has paired tokens but no recursion
edit_actions: clarity
edit_actions: clarity clarity meaning-changed
edit_actions: A
edit_actions: a model on either of these artificial languages leads to the same substantial gains when testing
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: D
edit_actions: artificial languages consisting of sets of separated pairs of words, but with no recursion, improves performance on
edit_actions: coherence
edit_actions: coherence clarity coherence
edit_actions: R
edit_actions: as well as recursive languages do. Experiments
edit_actions: . Further experiments
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: A
edit_actions: controlling for vocabulary overlap
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: from natural languages
edit_actions: by pre-training
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: D
edit_actions: studied in linguistic typology
edit_actions: clarity
edit_actions: clarity clarity coherence
edit_actions: R
edit_actions: a learner needs to model language
edit_actions: allow for natural language acquisition
edit_actions: style
edit_actions: style clarity style