The language modeling head is a linear layer that accepts the encoder's hidden states and transforms them into logits.