quantizing into int8 precision

by tanvij - opened 19 days ago

19 days ago

•

I'm looking into quantizing this model to int8 precision and I'm wondering if I should manually quantize the weights or use an automated technique like AWQ or bitsandbytes. Any recommendations on which method works best for this model? Thanks! @klldmofashi

tanvij changed discussion title from quantizing to quantizing into int8 precision 19 days ago

Louym

Efficient-Large-Model org 19 days ago

This comment has been hidden (marked as Off-Topic)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment