Granite-3.3-2B-Instruct-exl2

Original model: granite-3.3-2b-instruct by Granite Team, IBM

Quants

4bpw h6 (main)
4.5bpw h6
5bpw h6
6bpw h6
8bpw h8

Quantization notes

Made with Exllamav2 0.2.8 with default dataset.
These quants can be used with TabbyAPI or Text-Generation-WebUI and RTX GPU (Windows) or RTX/ROCm (AMD).

Original model card

Granite-3.3-2B-Instruct

Model Summary: Granite-3.3-2B-Instruct is a 2-billion parameter 128K context length language model fine-tuned for improved reasoning and instruction-following capabilities. Built on top of Granite-3.3-2B-Base, the model delivers significant gains on benchmarks for measuring generic performance including AlpacaEval-2.0 and Arena-Hard, and improvements in mathematics, coding, and instruction following. It supports structured reasoning through <think></think> and <response></response> tags, providing clear separation between internal thoughts and final outputs. The model has been trained on a carefully balanced combination of permissively licensed data and curated synthetic tasks.

• Developers: Granite Team, IBM
• GitHub Repository: ibm-granite/granite-3.3-language-models
• Website: Granite Docs
• Release Date: April 16th, 2025
• License: Apache 2.0

Supported Languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese. However, users may finetune this Granite model for languages beyond these 12 languages.

Intended Use: This model is designed to handle general instruction-following tasks and can be integrated into AI assistants across various domains, including business applications.

Capabilities

• Thinking
• Summarization
• Text classification
• Text extraction
• Question-answering
• Retrieval Augmented Generation (RAG)
• Code related tasks
• Function-calling tasks
• Multilingual dialog use cases
• Fill-in-the-middle
• Long-context tasks including long document/meeting summarization, long document QA, etc.

Generation: This is a simple example of how to use Granite-3.3-2B-Instruct model.

Install the following libraries:

pip install torch torchvision torchaudio
pip install accelerate
pip install transformers

Then, copy the snippet from the section that is relevant for your use case.

from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
import torch

model_path="ibm-granite/granite-3.3-2b-instruct"
device="cuda"
model = AutoModelForCausalLM.from_pretrained(
        model_path,
        device_map=device,
        torch_dtype=torch.bfloat16,
    )
tokenizer = AutoTokenizer.from_pretrained(
        model_path
)

conv = [{"role": "user", "content":"Take a current environmental issue and work backward to devise an innovative prevention strategy that could have averted the problem."}]

input_ids = tokenizer.apply_chat_template(conv, return_tensors="pt", thinking=True, return_dict=True, add_generation_prompt=True).to(device)

set_seed(42)
output = model.generate(
    **input_ids,
    max_new_tokens=8192,
)

prediction = tokenizer.decode(output[0, input_ids["input_ids"].shape[1]:], skip_special_tokens=True)
print(prediction)

Example Outputs Example Outputs

• thinking=True

<think>To address this task, I'll choose the environmental issue of plastic pollution in our oceans as a current problem. I'll then work backward to devise an innovative prevention strategy that could have mitigated this issue.

### Current Environmental Issue: Plastic Pollution in Oceans

**Problem Description:**
- Millions of tons of plastic waste enter oceans annually, causing severe harm to marine life, ecosystems, and human health.
- Plastic debris persists in the environment for centuries, breaking down into microplastics that contaminate food chains.

### Working Backward to Develop a Prevention Strategy

1. **Identify Root Causes:**
   - **Consumer Behavior:** Over-reliance on single-use plastics (e.g., bags, bottles, straws).
   - **Manufacturing Practices:** Production of non-biodegradable plastics.
   - **Waste Management:** Inadequate recycling infrastructure and littering.

2. **Develop a Prevention Strategy:**
   - **Innovative Product Design:**
     - **Biodegradable Alternatives:** Develop and promote the use of biodegradable plastics made from renewable resources like plant starches, seaweed, or algae. These materials would decompose naturally in marine environments without causing harm.
     - **Reusable and Recyclable Packaging:** Encourage manufacturers to design products with minimal packaging or use fully recyclable materials. Implement a robust recycling system that ensures materials are effectively recycled into new products.

   - **Policy and Regulation:**
     - **Bans and Restrictions:** Implement strict bans on single-use plastics in retail and public spaces. Encourage governments to adopt policies that incentivize the use of sustainable materials.
     - **Extended Producer Responsibility (EPR):** Introduce EPR policies where manufacturers are responsible for the entire lifecycle of their products, including disposal and recycling.

   - **Education and Awareness:**
     - **Public Campaigns:** Launch global campaigns to educate consumers about the impact of plastic pollution and promote sustainable alternatives.
     - **School and Community Programs:** Integrate environmental education into school curricula and community programs to foster a culture of sustainability from a young age.

   - **Technological Innovations:**
     - **Advanced Recycling Technologies:** Invest in research and development of advanced recycling technologies that can effectively break down plastic waste into usable materials, reducing the need for virgin plastics.
     - **Ocean Cleanup Technologies:** Support the development and deployment of technologies that can remove plastic waste from oceans, such as autonomous drones or floating barriers.

3. **Implementation and Monitoring:**
   - **Collaboration:** Partner with governments, NGOs, and private sectors to implement these strategies at scale.
   - **Monitoring and Evaluation:** Establish metrics to track the effectiveness of interventions, such as reductions in plastic waste and improvements in marine health.

### Conclusion

By focusing on innovative product design, policy changes, public education, and technological advancements, a comprehensive prevention strategy could have significantly reduced plastic pollution in our oceans. This approach not only addresses the root causes but also fosters a sustainable shift in consumer and industrial behavior, ultimately protecting marine ecosystems and human health.</think><response>### Innovative Prevention Strategy for Plastic Pollution in Oceans

**Current Environmental Issue:** Plastic pollution in oceans, causing severe harm to marine life, ecosystems, and human health.

**Working Backward to Develop a Prevention Strategy:**

1. **Identify Root Causes:**
   - **Consumer Behavior:** Over-reliance on single-use plastics.
   - **Manufacturing Practices:** Production of non-biodegradable plastics.
   - **Waste Management:** Inadequate recycling infrastructure and littering.

2. **Develop a Comprehensive Prevention Strategy:**

   - **Innovative Product Design:**
     - **Biodegradable Alternatives:**
       - Develop and promote the use of biodegradable plastics from renewable resources like plant starches, seaweed, or algae. These materials would decompose naturally in marine environments.
       - Encourage the use of compostable materials for food packaging and other disposable items.
     - **Reusable and Recyclable Packaging:**
       - Design products with minimal packaging or use fully recyclable materials.
       - Implement a robust recycling system that ensures materials are effectively recycled into new products.

   - **Policy and Regulation:**
     - **Bans and Restrictions:**
       - Implement strict bans on single-use plastics in retail and public spaces.
       - Encourage governments to adopt policies that incentivize the use of sustainable materials.
     - **Extended Producer Responsibility (EPR):**
       - Introduce EPR policies where manufacturers are responsible for the entire lifecycle of their products, including disposal and recycling.

   - **Education and Awareness:**
     - **Public Campaigns:**
       - Launch global campaigns to educate consumers about the impact of plastic pollution and promote sustainable alternatives.
     - **School and Community Programs:**
       - Integrate environmental education into school curricula and community programs to foster a culture of sustainability from a young age.

   - **Technological Innovations:**
     - **Advanced Recycling Technologies:**
       - Invest in research and development of advanced recycling technologies that can effectively break down plastic waste into usable materials, reducing the need for virgin plastics.
     - **Ocean Cleanup Technologies:**
       - Support the development and deployment of technologies that can remove plastic waste from oceans, such as autonomous drones or floating barriers.

3. **Implementation and Monitoring:**
   - **Collaboration:** Partner with governments, NGOs, and private sectors to implement these strategies at scale.
   - **Monitoring and Evaluation:** Establish metrics to track the effectiveness of interventions, such as reductions in plastic waste and improvements in marine health.

### Conclusion

By focusing on innovative product design, policy changes, public education, and technological advancements, a comprehensive prevention strategy could have significantly reduced plastic pollution in our oceans. This approach not only addresses the root causes but also fosters a sustainable shift in consumer and industrial behavior, ultimately protecting marine ecosystems and human health.</response>

• thinking=False

Current Environmental Issue: Plastic Pollution in Oceans

Prevention Strategy: Implementing a Global Extended Producer Responsibility (EPR) Program for Single-Use Plastics

Background: Plastic pollution in oceans is a pressing environmental issue, with millions of tons of plastic waste entering marine ecosystems annually. This pollution harms marine life, disrupts food chains, and poses risks to human health through microplastics in seafood and contaminated seafood products.

Innovative Prevention Strategy: Implementing a Global Extended Producer Responsibility (EPR) Program for Single-Use Plastics

1. Establish a Global EPR Framework: Create an international agreement among governments, industries, and environmental organizations to establish a comprehensive EPR program for single-use plastics. This framework would hold manufacturers, distributors, and retailers accountable for the entire lifecycle of their plastic products, including collection, recycling, and disposal.

2. Set Clear Targets and Timelines: Define specific, measurable, achievable, relevant, and time-bound (SMART) targets for reducing single-use plastic waste, such as phasing out certain items (e.g., plastic bags, straws, and cutlery) within a set timeframe (e.g., 5 years).

3. Incentivize Sustainable Alternatives: Encourage the development, production, and adoption of eco-friendly alternatives to single-use plastics by offering financial incentives, tax breaks, or subsidies to companies that invest in sustainable packaging and materials.

4. Strengthen Collection and Recycling Infrastructure: Invest in and expand waste management systems, particularly in developing countries, to ensure proper collection, sorting, and recycling of plastic waste. This includes supporting the development of advanced recycling technologies, such as chemical recycling, to handle hard-to-recycle plastics.

5. Promote Consumer Awareness and Education: Launch global campaigns to raise public awareness about the environmental impacts of single-use plastics and encourage responsible consumption. Educate consumers on proper waste disposal, recycling, and the benefits of choosing sustainable alternatives.

6. Enforce Strict Regulations and Penalties: Implement and enforce stringent regulations on the production, distribution, and sale of single-use plastics, with severe penalties for non-compliance. This includes banning or taxing harmful plastic products and promoting transparency in supply chains.

7. Foster International Collaboration: Encourage information sharing, best practices, and joint initiatives among countries to address plastic pollution effectively. This includes supporting research and development of innovative solutions, such as biodegradable plastics and ocean cleanup technologies.

8. Monitor and Evaluate Progress: Regularly assess the effectiveness of the EPR program and make necessary adjustments to ensure continuous improvement. This includes tracking key performance indicators (KPIs) such as plastic waste reduction, recycling rates, and the adoption of sustainable alternatives.

By implementing this innovative prevention strategy, we can significantly reduce the amount of single-use plastic waste entering our oceans, mitigating the environmental and health impacts associated with plastic pollution.

Evaluation Results:

Comparison with different models over various benchmarks¹. Scores of AlpacaEval-2.0 and Arena-Hard are calculated with thinking=True

Models	Arena-Hard	AlpacaEval-2.0	MMLU	PopQA	TruthfulQA	BigBenchHard2	DROP3	GSM8K	HumanEval	HumanEval+	IFEval	AttaQ
Granite-3.1-2B-Instruct	23.3	27.17	57.11	20.55	59.79	61.82	20.99	67.55	79.45	75.26	63.59	84.7
Granite-3.2-2B-Instruct	24.86	34.51	57.18	20.56	59.8	61.39	23.84	67.02	80.13	73.39	61.55	83.23
Granite-3.3-2B-Instruct	28.86	43.45	55.88	18.4	58.97	63.91	44.33	72.48	80.51	75.68	65.8	87.47
Llama-3.1-8B-Instruct	36.43	27.22	69.15	28.79	52.79	73.43	71.23	83.24	85.32	80.15	79.10	83.43
DeepSeek-R1-Distill-Llama-8B	17.17	21.85	45.80	13.25	47.43	67.39	49.73	72.18	67.54	62.91	66.50	42.87
Qwen-2.5-7B-Instruct	25.44	30.34	74.30	18.12	63.06	69.19	64.06	84.46	93.35	89.91	74.90	81.90
DeepSeek-R1-Distill-Qwen-7B	10.36	15.35	50.72	9.94	47.14	67.38	51.78	78.47	79.89	78.43	59.10	42.45
Granite-3.1-8B-Instruct	37.58	30.34	66.77	28.7	65.84	69.87	58.57	79.15	89.63	85.79	73.20	85.73
Granite-3.2-8B-Instruct	55.25	61.19	66.79	28.04	66.92	71.86	58.29	81.65	89.35	85.72	74.31	84.7
Granite-3.3-8B-Instruct	57.56	62.68	65.54	26.17	66.86	69.13	59.36	80.89	89.73	86.09	74.82	88.5

Math Benchmarks

Models	AIME24	MATH-500
Granite-3.1-2B-Instruct	0.89	35.07
Granite-3.2-2B-Instruct	0.89	35.54
Granite-3.3-2B-Instruct	3.28	58.09
Granite-3.1-8B-Instruct	1.97	48.73
Granite-3.2-8B-Instruct	2.43	52.8
Granite-3.3-8B-Instruct	8.12	69.02

Training Data: Overall, our training data is largely comprised of two key sources: (1) publicly available datasets with permissive license, (2) internal synthetically generated data targeted to enhance reasoning capabilites.

Infrastructure: We train Granite-3.3-2B-Instruct using IBM's super computing cluster, Blue Vela, which is outfitted with NVIDIA H100 GPUs. This cluster provides a scalable and efficient infrastructure for training our models over thousands of GPUs.

Ethical Considerations and Limitations: Granite-3.3-2B-Instruct builds upon Granite-3.3-2B-Base, leveraging both permissively licensed open-source and select proprietary data for enhanced performance. Since it inherits its foundation from the previous model, all ethical considerations and limitations applicable to Granite-3.3-2B-Base remain relevant.

Resources

• ⭐️ Learn about the latest updates with Granite: https://www.ibm.com/granite
• 📄 Get started with tutorials, best practices, and prompt engineering advice: https://www.ibm.com/granite/docs/
• 💡 Learn about the latest Granite learning resources: https://ibm.biz/granite-learning-resources

[1] Evaluated using OLMES (except AttaQ and Arena-Hard scores)

[2] Added regex for more efficient asnwer extraction.

[2] Modified the implementation to handle some of the issues mentioned here