Impact statement: Our work aims at providing fundamental guidelines which can assist all fields that employ Transformer-based architectures to use more efficient models. This way, these fields can achieve their goals while consuming less resources. Additionally, this work made an effort to provide a theoretical interpretation by examining the (many) empirical signals already published by others, while providing only a required minimum of further experimentation. This was done under the belief that while experiments are crucial for the advancement of the field, it is important not to conduct them superfluously as they incur an environmental price [Schwartz et al., 2019].