We equip BERT with this mixed attention design and build a ConvBERT model.