GeLinear

GeLinear is the implementation of LoLCATs, but with Gemma 2 model. Unlike the original LoLCATs approach that linearizes all attention layers, I focuses on linearizing only the global attention layer, while retaining Gemma 2’s built-in Sliding Window Attention (SWA) for local context (since the time complexity for this already scaled linearly with sequence length).

To run the model:

from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
model = AutoModelForCausalLM.from_pretrained("jacksonkek/GeLinear",trust_remote_code=True,torch_dtype=torch.bfloat16,device_map="sequential")
tokenizer = AutoTokenizer.from_pretrained("jacksonkek/GeLinear")

x = "tell me a joke"
input_text = [{"role":"user","content":x}]
input_ids = tokenizer.apply_chat_template(input_text, add_generation_prompt=True,return_tensors="pt").to("cuda")
text_streamer = TextStreamer(tokenizer)
_ = model.generate(input_ids, streamer = text_streamer, do_sample=False,max_new_tokens = 8192)