before_revision: We present Shapeshifter Networks (SSNs), a flexible neural network framework that improves performance and reduces memory requirements on a diverse set of scenarios over standard neural networks. Our approach is based on the observation that many neural networks are severely overparameterized, resulting in significant waste in computational resources as well as being susceptible to overfitting. SSNs address this by learning where and how to share parameters between layers in a neural network while avoiding degenerate solutions that result in underfitting. Specifically, we automatically construct parameter groups that identify where parameter sharing is most beneficial. Then, we map each group's weights to construct layerswith learned combinations of candidates from a shared parameter pool. SSNs can share parameters across layers even when they have different sizes , perform different operations , and/or operate on features from different modalities . We evaluate our approach on a diverse set of tasks , including image classification, bidirectional image-sentence retrieval, and phrase grounding, creating high performing models even when using as little as 1\% of the parameters . We also apply SSNs to knowledge distillation, where we obtain state-of-the-art results when combined with traditional distillation methods .
after_revision: Fitting a model into GPU memory during training is an increasing concern as models continue to grow. To address this issue, we present Shapeshifter Networks (SSNs), a flexible neural network framework that decouples layers from model weights, enabling us to implement any neural network with an arbitrary number of parameters. In SSNs each layer obtains weights from a parameter store that decides where and how to allocate parameters to layers. This can result in sharing parameters across layers even when they have different sizes or perform different operations . SSNs do not require any modifications to a model's loss function or architecture, making them easy to use. Our approach can create parameter efficient networks by using a relatively small number of weights, or can improve a model's performance by adding additional model capacity during training without affecting the computational resources required at test time . We evaluate SSNs using seven network architectures across diverse tasks that include image classification, bidirectional image-sentence retrieval, and phrase grounding, creating high performing models even when using as little as 1\% of the parameters .
edit_actions: R
edit_actions: We
edit_actions: Fitting a model into GPU memory during training is an increasing concern as models continue to grow. To address this issue, we
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: improves performance and reduces memory requirements on a diverse set of scenarios over standard neural networks. Our approach is based on the observation that many neural networks are severely overparameterized, resulting in significant waste in computational resources as well as being susceptible to overfitting. SSNs address this by learning
edit_actions: decouples layers from model weights, enabling us to implement any neural network with an arbitrary number of parameters. In SSNs each layer obtains weights from a parameter store that decides
edit_actions: meaning-changed
edit_actions: meaning-changed style meaning-changed
edit_actions: R
edit_actions: share parameters between layers in a neural network while avoiding degenerate solutions that result in underfitting. Specifically, we automatically construct parameter groups that identify where parameter sharing is most beneficial. Then, we map each group's weights to construct layerswith learned combinations of candidates from a shared parameter pool. SSNs can share
edit_actions: allocate parameters to layers. This can result in sharing
edit_actions: clarity
edit_actions: clarity clarity coherence
edit_actions: R
edit_actions: ,
edit_actions: or
edit_actions: fluency
edit_actions: fluency meaning-changed fluency
edit_actions: R
edit_actions: , and/or operate on features from different modalities
edit_actions: . SSNs do not require any modifications to a model's loss function or architecture, making them easy to use. Our approach can create parameter efficient networks by using a relatively small number of weights, or can improve a model's performance by adding additional model capacity during training without affecting the computational resources required at test time
edit_actions: meaning-changed
edit_actions: meaning-changed clarity meaning-changed
edit_actions: R
edit_actions: our approach on a diverse set of tasks , including
edit_actions: SSNs using seven network architectures across diverse tasks that include
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed
edit_actions: D
edit_actions: . We also apply SSNs to knowledge distillation, where we obtain state-of-the-art results when combined with traditional distillation methods
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed