---
license: mit
---
# GeLinear

GeLinear is the implementation of [LoLCATs](https://arxiv.org/pdf/2410.10254), but with Gemma 2 model.
Unlike the original LoLCATs approach that linearizes all attention layers, I focuses on linearizing only the global attention layer, while retaining Gemma 2’s built-in Sliding Window Attention (SWA) for local context (since the time complexity for this already scaled linearly with sequence length). 

To run the model:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
model = AutoModelForCausalLM.from_pretrained("jacksonkek/GeLinear",trust_remote_code=True,torch_dtype=torch.bfloat16,device_map="sequential")
tokenizer = AutoTokenizer.from_pretrained("jacksonkek/GeLinear")

x = "tell me a joke"
input_text = [{"role":"user","content":x}]
input_ids = tokenizer.apply_chat_template(input_text, add_generation_prompt=True,return_tensors="pt").to("cuda")
text_streamer = TextStreamer(tokenizer)
_ = model.generate(input_ids, streamer = text_streamer, do_sample=False,max_new_tokens = 8192)

```