Initial model upload

README.md

@@ -0,0 +1,141 @@
+# Introduction
+
+DeepSeek-R1-Distill-Qwen-32B-FlagOS-Cambricon  provides an all-in-one deployment solution, enabling execution of DeepSeek-R1-Distill-Qwen-32B on Cambricon GPUs. As the first-generation release for the Cambricon-MLU590, this package delivers three key features:
+
+1. Comprehensive Integration:
+   - Integrated with FlagScale (https://github.com/FlagOpen/FlagScale).
+   - Open-source inference execution code, preconfigured with all necessary software and hardware settings.
+   - Pre-built Docker image for rapid deployment on Cambricon-MLU590.
+2. Consistency Validation:
+   - Evaluation tests verifying consistency of results between the official and ours.
+
+# Technical Summary
+
+## Serving Engine
+
+We use FlagScale as the serving engine to improve the portability of distributed inference.
+
+FlagScale is an end-to-end framework for large models across multiple chips, maximizing computational resource efficiency while ensuring model effectiveness. It ensures both ease of use and high performance for users when deploying models across different chip architectures:
+
+- One-Click Service Deployment: FlagScale provides a unified and simple command execution mechanism, allowing users to fast deploy services seamlessly across various hardware platforms using the same command. This significantly reduces the entry barrier and enhances user experience.
+- Automated Deployment Optimization: FlagScale automatically optimizes distributed parallel strategies based on the computational capabilities of different AI chips, ensuring optimal resource allocation and efficient utilization, thereby improving overall deployment performance.
+- Automatic Operator Library Switching: Leveraging FlagScale's unified Runner mechanism and deep integration with FlagGems, users can seamlessly switch to the FlagGems operator library for inference by simply adding environment variables in the configuration file.
+
+## Triton Support
+
+We validate the execution of DeepSeek-R1-Distill-Qwen-32B model with a Triton-based operator library as a PyTorch alternative.
+
+We use a variety of Triton-implemented operation kernels—approximately 70%—to run the DeepSeek-R1-Distill-Qwen-32B model. These kernels come from two main sources:
+
+- Most Triton kernels are provided by FlagGems (https://github.com/FlagOpen/FlagGems). You can enable FlagGems kernels by setting the environment variable USE_FLAGGEMS. For more details, please refer to the "How to Run Locally" section.
+
+- Also included are Triton kernels from vLLM, including fused MoE.
+
+# Bundle Download
+Requested by Cambricon, the file of docker image and model files should be applied by email.
+            
+|             | Usage                                                  | Cambricon                                                                                                                             |
+| ----------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
+| Basic Image | basic software environment that supports model running | ecosystem@cambricon.com <br> Contact by email，please indicate the unit/contact person/contact information/equipment source/specific requirements |
+| Model       | model weight and configuration files                   | ecosystem@cambricon.com <br> Contact by email，please indicate the unit/contact person/contact information/equipment source/specific requirements |
+# Evaluation Results
+
+## Benchmark Result 
+
+| Metrics            | DeepSeek-R1-Distill-Qwen-32B-H100-CUDA | DeepSeek-R1-Distill-Qwen-32B-FlagOS-Cambricon |
+|:-------------------|-----------------------|--------------------------|
+| GSM8K (EM) | 87.64 | 86.81 |
+| MMLU (Acc.) | 79.33 | 79.45 |
+| CEVAL | 83.43 | 83.43 |
+| AIME 2024 (Pass@1) | 73.33 | 70.00 |
+| GPQA-Diamond (Pass@1) | 59.60 | 64.65 |
+| MATH-500 (pass@1) | 92.80 | 93.40 |
+
+
+# How to Run Locally
+## 📌 Getting Started
+### Environment Setup
+
+```bash
+# install FlagScale
+git clone https://github.com/FlagOpen/FlagScale.git
+cd FlagScale
+pip install .
+
+# download image and ckpt
+flagscale pull --image <IMAGE> --ckpt <CKPT> --ckpt-path <CKPT_PATH>
+
+# Note: For security reasons, this image does not have passwordless configuration. In multi-machine scenarios, you need to configure passwordless access for the image yourself.
+
+# build and enter the container
+docker run -e --net=host --pid=host --ipc=host -v /tmp/.X11-unix:/tmp/.X11-unix --privileged -it -v <CKPT_PATH>:<CKPT_PATH> -v /opt/data/:/opt/data/ -v /usr/bin/cnmon:/usr/bin/cnmon --name flagrelease_cambricon <IMAGE> /bin/bash
+```
+
+
+### Download and install FlagGems
+
+```bash
+git clone https://github.com/FlagOpen/FlagGems.git
+cd FlagGems
+git checkout DeepSeek-R1-Distill-Qwen-32B_release_Cambricon
+# no additional dependencies since they are already handled in the Docker environment
+pip install ./ --no-deps
+cd ../
+```
+
+### Download FlagScale and unpatch the vendor's code to build vllm
+
+```bash
+git clone https://github.com/FlagOpen/FlagScale.git
+cd FlagScale
+# please set the name and email in git config in advance, for example: git config --global user.name "your_name"; git config --global user.email "your_email"
+python tools/patch/unpatch.py --device-type Cambricon --commit-id 57637057 --dir build
+cd build/Cambricon/FlagScale/vllm
+pip install -e . -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
+cd vllm_mlu
+pip install -e . -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
+cd ../../
+```
+### Serve
+
+```bash
+# config the DeepSeek-R1-Distill-Qwen-32B yaml
+FlagScale/
+├── examples/
+│   └── DeepSeek-R1-Distill-Qwen-32B/
+│       └── conf/
+│           └── config_DeepSeek-R1-Distill-Qwen-32B.yaml # set hostfile and ssh_port(optional), if it is passwordless access between containers, the docker field needs to be removed
+│           └── serve/
+│               └── DeepSeek-R1-Distill-Qwen-32B.yaml # set model parameters and server port
+
+# install flagscale
+pip install .
+# serve
+flagscale serve DeepSeek-R1-Distill-Qwen-32B
+```
+
+# Usage Recommendations
+When custom service parameters, users can run:
+
+```bash
+flagscale serve <MODEL_NAME> <MODEL_CONFIG_YAML>
+```
+
+# Contributing
+
+We warmly welcome global developers to join us:
+1. Submit Issues to report problems
+2. Create Pull Requests to contribute code
+3. Improve technical documentation
+4. Expand hardware adaptation support
+
+# 📞 Contact Us
+
+Scan the QR code below to add our WeChat group
+send "FlagRelease"
+
+![WeChat](image/group.png)
+
+# License
+
+This project and related model weights are licensed under the MIT License.

configuration.json

@@ -0,0 +1,4 @@
+{
+    "framework": "Pytorch",
+    "task": "any-to-any"
+}