before_revision: Fitting a model into GPU memory during training is an increasing concern as models continue to grow. To address this issue, we present Shapeshifter Networks (SSNs), a flexible neural network framework that decouples layers from model weights, enabling us to implement any neural network with an arbitrary number of parameters . In SSNseach layer obtains weights from a parameter store that decides where and how to allocate parameters to layers . This can result in sharing parameters across layers even when they have different sizes or perform different operations. SSNs do not require any modifications to a model's loss function or architecture  , making them easy to use. Our approach can create parameter efficient networks by using a relatively small number of weights, or can improve a model's performance by adding additional model capacity during training without affecting the computational resources required at test time. We evaluate SSNs  using seven network architectures across diverse tasks that include image classification, bidirectional image-sentence retrieval, and phrase grounding, creating high performing models even when using as little as 1\% of the parameters.
after_revision: Fitting a model into GPU memory during training is an increasing concern as models continue to grow. Parameter sharing can reduce memory requirements, but existing methods only share parameters between identical layers, limiting their impact. This paper removes these restrictions with a novel task called Neural Parameter Allocation Search (NPAS), where the goal is to generate weights for a network using a given parameter budget. NPAS requires new techniques to morph available parameters to fit any architecture. To address this new task we introduce Shapeshifter Networks (SSNs), which automatically learns where and how to share parameters between all layers in a network, even between layers of varying sizes and operations. SSNs do not require any loss function or architecture modifications , making them easy to use. We evaluate SSNs in key NPAS settings using seven network architectures across diverse tasks including image classification, bidirectional image-sentence retrieval, and phrase grounding, creating high performing models even when using as little as 1\% of the parameters.
edit_actions: R
edit_actions: To address this issue, we present Shapeshifter Networks (SSNs), a flexible neural network framework that decouples layers from model weights, enabling us to implement any neural network with an arbitrary number of parameters . In SSNseach layer obtains weights from a parameter store that decides
edit_actions: Parameter sharing can reduce memory requirements, but existing methods only share parameters between identical layers, limiting their impact. This paper removes these restrictions with a novel task called Neural Parameter Allocation Search (NPAS), where the goal is to generate weights for a network using a given parameter budget. NPAS requires new techniques to morph available parameters to fit any architecture. To address this new task we introduce Shapeshifter Networks (SSNs), which automatically learns
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: allocate parameters to layers . This can result in sharing parameters across layers even when they have different sizes or perform different
edit_actions: share parameters between all layers in a network, even between layers of varying sizes and
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: D
edit_actions: modifications to a model's
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: A
edit_actions: modifications
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed clarity
edit_actions: D
edit_actions: Our approach can create parameter efficient networks by using a relatively small number of weights, or can improve a model's performance by adding additional model capacity during training without affecting the computational resources required at test time.
edit_actions: coherence
edit_actions: coherence clarity coherence
edit_actions: A
edit_actions: in key NPAS settings
edit_actions: meaning-changed
edit_actions: meaning-changed clarity meaning-changed
edit_actions: R
edit_actions: that include
edit_actions: including
edit_actions: clarity
edit_actions: fluency clarity clarity