Create README.md

README.md

@@ -0,0 +1,109 @@
+---
+license: mit
+library_name: vllm
+base_model:
+- deepseek-ai/DeepSeek-R1
+pipeline_tag: text-generation
+tags:
+- deepseek
+- neuralmagic
+- redhat
+- llmcompressor
+- quantized
+- INT4
+- GPTQ
+---
+
+# DeepSeek-R1-quantized.w4a16
+
+## Model Overview
+- **Model Architecture:** DeepseekV3ForCausalLM
+  - **Input:** Text
+  - **Output:** Text
+- **Model Optimizations:**
+  - **Activation quantization:** None
+  - **Weight quantization:** INT4
+- **Release Date:** 04/15/2025
+- **Version:** 1.0
+- **Model Developers:** RedHat (Neural Magic)
+
+
+### Model Optimizations
+
+This model was obtained by quantizing weights of [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1) to INT4 data type.
+This optimization reduces the number of bits used to represent weights from 8 to 4, reducing GPU memory requirements (by approximately 50%).
+Weight quantization also reduces disk size requirements by approximately 50%.
+
+
+## Deployment
+
+This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
+
+```python
+from vllm import LLM, SamplingParams
+from transformers import AutoTokenizer
+
+model_id = "RedHatAI/DeepSeek-R1-quantized.w4a16"
+number_gpus = 8
+
+sampling_params = SamplingParams(temperature=0.7, top_p=0.8, max_tokens=256)
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+prompt = "Give me a short introduction to large language model."
+
+llm = LLM(model=model_id, tensor_parallel_size=number_gpus)
+
+outputs = llm.generate(prompt, sampling_params)
+
+generated_text = outputs[0].outputs[0].text
+print(generated_text)
+```
+
+vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
+
+
+## Evaluation
+
+The model was evaluated on the OpenLLM leaderboard task (v1) via [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), and on popular reasoning tasks (AIME 2024, MATH-500, GPQA-Diamond) via [LightEval](https://github.com/huggingface/open-r1).
+For reasoning evaluations, we estimate pass@1 based on 10 runs with different seeds.
+
+<details>
+  <summary>Evaluation details</summary>
+
+  **OpenLLM v1**
+  ```
+  lm_eval \
+    --model vllm \
+    --model_args pretrained="RedHatAI/DeepSeek-R1-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+    --tasks openllm \
+    --batch_size auto
+  ```
+
+  **Reasoning Benchmarks**
+  ```
+  export MODEL_ARGS="pretrained=RedHatAI/DeepSeek-R1-quantized.w4a16,dtype=bfloat16,max_model_length=38768,gpu_memory_utilization=0.8,tensor_parallel_size=8,add_special_tokens=false,generation_parameters={\"max_new_tokens\":32768,\"temperature\":0.6,\"top_p\":0.95,\"seed\":42}"
+  export VLLM_WORKER_MULTIPROC_METHOD=spawn
+  lighteval vllm $MODEL_ARGS "custom|aime24|0|0,custom|math_500|0|0,custom|gpqa:diamond|0|0" \
+      --custom-tasks src/open_r1/evaluate.py \
+      --use-chat-template \
+      --output-dir $OUTPUT_DIR
+  ```
+
+</details>
+
+### Accuracy
+
+|                             | Recovery (%) | deepseek/DeepSeek-R1 | RedHatAI/DeepSeek-R1-quantized.w4a16<br>(this model) |
+| --------------------------- | :----------: | :------------------: | :--------------------------------------------------: |
+| ARC-Challenge<br>25-shot    | 100.00          | 72.53                | 72.53                                                |
+| GSM8k<br>5-shot             | 99.76        | 95.91                | 95.68                                                |
+| HellaSwag<br>10-shot        | 100.07       | 89.30                | 89.36                                                |
+| MMLU<br>5-shot              | 99.74        | 87.22                | 86.99                                                |
+| TruthfulQA<br>0-shot        | 100.83       | 59.28                | 59.77                                                |
+| WinoGrande<br>5-shot        | 101.65       | 82.00                | 83.35                                                |
+| **OpenLLM v1<br>Average Score** | **100.30**        | **81.04**                | **81.28**                                            |
+| AIME 2024<br>pass@1         | 98.30         | 78.33                | 77.00                                                |
+| MATH-500<br>pass@1          | 99.84        | 97.24                | 97.08                                                |
+| GPQA Diamond<br>pass@1      | 98.01        | 73.38                | 71.92                                                |
+| **Reasoning<br>Average Score**  | **98.81**        | **82.99**                | **82.00**                                                |