Enhance model card for Seg-Zero-7B with detailed overview, usage, and training info (#2)

- Enhance model card for Seg-Zero-7B with detailed overview, usage, and training info (47c9e152bf76be58e7e29843084b0ab77d4ba048)


Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>

README.md

@@ -1,11 +1,10 @@
-
 ---
 datasets:
 - reasonseg
 language: en
+library_name: transformers
 license: other
 pipeline_tag: image-segmentation
-library_name: transformers
 tags:
 - vision
 - segmentation
@@ -13,13 +12,44 @@ tags:
 
 # Seg-Zero-7B
 
-This model is based on the paper [Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement](https://huggingface.co/papers/2503.06520). It uses a decoupled architecture with a reasoning model and a segmentation model. It's trained via reinforcement learning using GRPO without explicit reasoning data, leading to robust zero-shot generalization and emergent test-time reasoning.
+This model is based on the paper [Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement](https://huggingface.co/papers/2503.06520).
 
 Code: https://github.com/dvlab-research/Seg-Zero
 
-## Description
+## Model Overview
+
+Seg-Zero introduces a novel framework for reasoning segmentation that addresses the limitations of traditional supervised fine-tuning methods, which often struggle with out-of-domain generalization and lack explicit reasoning processes. The framework features a decoupled architecture consisting of a reasoning model and a segmentation model. The reasoning model interprets user intentions, generates explicit reasoning chains, and produces positional prompts, which are subsequently used by the segmentation model to generate precise pixel-level masks.
+
+Seg-Zero is trained exclusively via reinforcement learning with GRPO, without explicit reasoning data, achieving robust zero-shot generalization and emergent test-time reasoning capabilities. A sophisticated reward mechanism integrating both format and accuracy rewards guides the optimization. Experiments show that Seg-Zero-7B achieves a zero-shot performance of 57.5 on the ReasonSeg benchmark, surpassing the prior LISA-7B by 18%.
+
+<div align=center>
+<img width="98%" src="https://huggingface.co/Ricky06662/Seg-Zero-7B/resolve/main/assets/overview.png"/>
+</div>
+
+Seg-Zero demonstrates the following key features:
+1.  **Emergent Test-Time Reasoning**: It generates a reasoning chain before producing the final segmentation mask.
+2.  **Reinforcement Learning Only**: Trained exclusively using reinforcement learning, without any explicit supervised reasoning data.
+3.  **Superior Generalization**: Achieves superior performance on both in-domain and out-of-domain data compared to supervised fine-tuning methods.
+
+### Highlight Code Features
 
-This is a Seg-Zero-7B model.  It introduces a decoupled architecture consisting of a reasoning model and a segmentation model. The reasoning model interprets user intentions, generates explicit reasoning chains, and produces positional prompts, which are subsequently used by the segmentation model to generate pixel-level masks.
+*   Based on EasyR1 and veRL, which supports model split during sampling and is more GPU memory friendly.
+*   Supports both Qwen2-VL and Qwen2.5-VL series models.
+*   Already implementing commonly used rewards in Object Detection and Object Segmentation, including IoU reward and L1 reward.
+
+## Model Architecture
+
+Seg-Zero employs a decoupled architecture, including a reasoning model and segmentation model. We manually design a sophisticated reward mechanism that integrates both the format and accuracy rewards.
+
+<div align=center>
+<img width="98%" src="https://huggingface.co/Ricky06662/Seg-Zero-7B/resolve/main/assets/pipeline.png"/>
+</div>
+
+## Examples
+
+<div align=center>
+<img width="98%" src="https://huggingface.co/Ricky06662/Seg-Zero-7B/resolve/main/assets/examples.png"/>
+</div>
 
 ## Usage
 
@@ -37,26 +67,152 @@ tokenizer = Qwen2_5_VLForConditionalGeneration.from_pretrained("Ricky06662/Seg-Z
 ```bash
 git clone https://github.com/dvlab-research/Seg-Zero.git
 cd Seg-Zero
-conda create -n seg_zero python=3.11
-conda activate seg_zero
-pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1
+conda create -n visionreasoner python=3.12
+conda activate visionreasoner
+pip install torch==2.6.0 torchvision==0.21.0
 pip install -e .
-pip install sam2
-pip install matplotlib
 ```
 
 ## Inference
 
+Download pretrained models using the following scripts:
+```bash
+mkdir pretrained_models
+cd pretrained_models
+git lfs install
+git clone https://huggingface.co/Ricky06662/VisionReasoner-7B
+```
+
+> [!TIP]
+> If you encounter issues with connecting to Hugging Face, consider using `export HF_ENDPOINT=https://hf-mirror.com`.
+
+Then run inference using:
+```bash
+python inference_scripts/infer_multi_object.py
+```
+The default question is
+> "What can I have if I'm thirsty?"
+
+You will get the thinking process in command line, like:
+
+> "The question asks for items that can be consumed if one is thirsty. In the image, there are two glasses that appear to contain beverages, which are the most likely candidates for something to drink. The other items, such as the salad, fruit platter, and sandwich, are not drinks and are not suitable for quenching thirst."
+
+And the mask will be presented in **inference_scripts** folder.
+
+<div align=center>
+<img width="98%" src="https://huggingface.co/Ricky06662/Seg-Zero-7B/resolve/main/assets/test_output_multiobject.png"/>
+</div>
+
+You can also provide your own image_path and text by:
+```bash
+python inference_scripts/infer_multi_object.py --image_path "your_image_path" --text "your question text"
+```
+
+## Evaluation
+
+Evaluation Data: [🤗 ReasonSeg-Test](https://huggingface.co/datasets/Ricky06662/ReasonSeg_test) [🤗 ReasonSeg-Val](https://huggingface.co/datasets/Ricky06662/ReasonSeg_val)
+
+```bash
+bash evaluation_scripts/eval_reasonseg_visionreasoner.sh
+```
+Adjusting '--batch_size' in the bash scripts based on your GPU. And you will see the gIoU in your command line.
+<div align=center>
+<img width="98%" src="https://huggingface.co/Ricky06662/Seg-Zero-7B/resolve/main/assets/val_results.png"/>
+</div>
+
+> [!NOTE]
+> Results in VisionReasoner are evaluated within one checkpoint. We recommend you to [VisionReasoner](https://github.com/dvlab-research/VisionReasoner) for evaluation on more tasks and more benchmarks.
+
+> [!NOTE]
+> However, in Seg-Zero, the best results on different benchmark are evaluated using different checkpoint. We just evaluate all available checkpoints and write down their value. For someone who may care about the performance, we suggest you can evaluate all benchmark within one model and compare the value (of our released checkpoint) in your environment.
+
+## Training
+
+### 1. GRPO Training
+
+> [!NOTE]
+> The recommended training requirement for 7B model is a 4x80G GPUs server or a 8x46G GPUs server.
+
+Training Data: [🤗 MultiObject-1K](https://huggingface.co/datasets/Ricky06662/VisionReasoner_multi_object_1k_840) [🤗 MultiObject-7K](https://huggingface.co/datasets/Ricky06662/VisionReasoner_multi_object_7k_840)
+Download dataset using this script:
 ```bash
-python inference_scripts/infer.py
+python training_scripts/download_dataset.py
 ```
 
-The default question is:
+> [!TIP]
+> Try resize the image and re-calculate the corresponding bbox/point coordinates if you have lower GPU memory. Remember changing the corresponding resize_size in evaluation and inference.
 
-> "the unusual object in the image."
+Download pretrained models using the following scripts:
+```bash
+mkdir pretrained_models
+cd pretrained_models
+git lfs install
+git clone https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct
+```
 
-You will get the thinking process in the command line and the mask will be saved in the **inference_scripts** folder.  You can also provide your own image_path and text:
+Start training using this script:
+```bash
+bash training_scripts/run_visionreasoner_7b_4x80G.sh
+```
+(Optional) Or you can use:
+```bash
+bash training_scripts/run_visionreasoner_7b_8x46G.sh
+```
 
+You can try change the following hyper-parameters if you have a large GPU memory.
 ```bash
-python inference_scripts/infer.py --image_path "your_image_path" --text "your question text"
+worker.actor.micro_batch_size_per_device_for_update=1 or 2 or 4 or 8 or 16 \
+worker.actor.micro_batch_size_per_device_for_experience=1 or2 or 4 or 8 or 16 \
 ```
+If your GPU has less memory, you can change the following config. The number is depend on your GPU memory.
+```bash
+worker.rollout.tensor_parallel_size=[your number between 1-4]
+worker.rollout.gpu_memory_utilization=[your number between 0-1]
+worker.rollout.n=[your number between 2-32]
+```
+
+(Optional) If you have 8x140G GPUs, you can try:
+```bash
+bash training_scripts/run_visionreasoner_7b.sh
+```
+
+### 2. Merge Checkpoint in Hugging Face Format
+
+```bash
+python3 training_scripts/model_merger.py --local_dir [path_to_your_actor_checkpoint]
+```
+
+## The GRPO Algorithm
+
+<div align=center>
+<img width="48%" src="https://huggingface.co/Ricky06662/Seg-Zero-7B/resolve/main/assets/rl_sample.png"/>
+</div>
+
+Seg-Zero generates several samples, calculates the rewards and then optimizes towards samples that achieve higher rewards.
+
+> [!TIP]
+> To learn more about the GRPO algorithm, you can refer to [Hugging Face's blog](https://huggingface.co/docs/trl/v0.15.2/en/grpo_trainer).
+
+## Citation
+
+```bibtex
+@article{liu2025segzero,
+  title        = {Seg-Zero: Reasoning-Chain Guided  Segmentation via Cognitive Reinforcement},
+  author       = {Liu, Yuqi and Peng, Bohao and Zhong, Zhisheng and Yue, Zihao and Lu, Fanbin and Yu, Bei and Jia, Jiaya},
+  journal      = {arXiv preprint arXiv:2503.06520},
+  year         = {2025}
+}
+
+@article{liu2025visionreasoner,
+  title        = {VisionReasoner: Unified Visual Perception and Reasoning via Reinforcement Learning},
+  author       = {Liu, Yuqi and Qu, Tianyuan and Zhong, Zhisheng and Peng, Bohao and Liu, Shu and Yu, Bei and Jia, Jiaya},
+  journal = {arXiv preprint arXiv:2505.12081},
+  year         = {2025}
+}
+```
+
+## Acknowledgement
+We would like to thank the following repos for their great work:
+
+*   This work is built upon the [EasyR1](https://github.com/hiyouga/EasyR1) and [veRL](https://github.com/volcengine/verl).
+*   This work utilizes models from [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct), [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) and [SAM2](https://huggingface.co/facebook/sam2-hiera-large).