RedHatAI
/

Qwen2-VL-72B-Instruct-quantized.w8a8

Image-Text-to-Text

text-generation-inference

8-bit precision

compressed-tensors

Model card Files Files and versions

shubhrapandit commited on Mar 31

Commit

c06a2c8

·

verified ·

1 Parent(s): e7e43af

Update README.md

Files changed (1) hide show

README.md +1 -0

README.md CHANGED Viewed

@@ -212,6 +212,7 @@ lm_eval \
   --model vllm \
   --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
   --tasks mgsm_cot_native \
   --num_fewshot 0 \
   --batch_size auto \
   --output_path output_dir

   --model vllm \
   --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
   --tasks mgsm_cot_native \
+  --apply_chat_template \
   --num_fewshot 0 \
   --batch_size auto \
   --output_path output_dir