README.md

---

license: mit
datasets:
- fka/awesome-chatgpt-prompts
language:
- zho
- eng
- fra
- spa
- por
- deu
- ita
- rus
- jpn
- kor
- vie
- tha
- ara
base_model:
- Qwen/Qwen2.5-1.5B-Instruct
pipeline_tag: text-generation
---

# Quantized Qwen2.5-1.5B-Instruct

This repository contains 8-bit and 4-bit quantized versions of the Qwen2.5-1.5B-Instruct model using GPTQ. Quantization significantly reduces the model's size and memory footprint, enabling faster inference on resource-constrained devices while maintaining reasonable performance.


## Model Description

The Qwen2.5-1.5B-Instruct is a powerful language model developed by Qwen for instructional tasks. These quantized versions offer a more efficient way to deploy and utilize this model.


## Quantization Details

* **Quantization Method:** GPTQ (Generative Pretrained Transformer Quantization)
* **Quantization Bits:** 8-bit and 4-bit versions available.
* **Dataset:** The model was quantized using a subset of the "fka/awesome-chatgpt-prompts" dataset.


## Usage

To use the quantized models, follow these steps:

**Install Dependencies:**
```bash

pip install transformers accelerate bitsandbytes auto-gptq optimum

```
## Performance

The quantized models offer a significant reduction in size and memory usage compared to the original model. While there might be a slight decrease in performance, the trade-off is often beneficial for deployment on devices with limited resources.


## Disclaimer

These quantized models are provided for research and experimentation purposes. We do not guarantee their performance or suitability for specific applications.


## Acknowledgements

* **Qwen:** For developing the original Qwen2.5-1.5B-Instruct model.
* **Hugging Face:** For providing the platform and tools for model sharing and quantization.
* **GPTQ Authors:** For developing the GPTQ quantization method.