README.md

metadata

base_model:
  - SanghyukChun/ProLIP-ViT-B-16-DC-1B-12_8B
datasets:
  - Lin-Chen/ShareGPT4V
license: mit
tags:
  - pytorch_model_hub_mixin
  - model_hub_mixin
pipeline_tag: zero-shot-image-classification
library_name: prolip

Official implementation of ViT-B/16 LongProLIP on ShareGPT4V, HYPE medium and DFN medium

• This LongProLIP weight fine-tuned on ShareGPT4V + HYPE medium + DFN medium 128M samples
• Pre-training datasets
  • ShareGPT4V / HYPE medium + DFN medium Seen samples 128M
  • DFN medium (https://huggingface.co/datasets/apf1/datafilteringnetworks_2b/tree/main)
  • HYPE medium (https://huggingface.co/dandelin/hype-sampler/tree/main/medium_scale)

Overview

• LongProLIP Paper: https://arxiv.org/abs/2503.08048
• ProLIP Paper: https://huggingface.co/papers/2410.18857
• GitHub: https://github.com/naver-ai/prolip
• More models are available at https://huggingface.co/collections/SanghyukChun/prolip-6712595dfc87fd8597350291

Performance overview (main results)

• Zero-shot ImageNet-1k top-1 accuracy: 74.52% (before fine-tuning: 74.6%)
• Zero-shot ImageNet distribution shifts: 62.52% (before fine-tuning: 63.0%)
• Zero-shot VTAB performance: 63.01% (before fine-tuning: 63.7%)
• Zero-shot retrieval performance: 61.88% (before fine-tuning: 59.6%)
• Average zero-shot performance on 38 tasks: 63.34% (before fine-tuning: 63.3%)

Performance overview (additional results)

• Urban-1k: 77.5% (before fine-tuning: 65.4%)
• ECCV mAP@R: 34.6% (before fine-tuning: 34.1%)

import requests
from PIL import Image

import torch
from prolip.model import ProLIPHF
from transformers import CLIPProcessor
from prolip.tokenizer import HFTokenizer

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch16")
model = ProLIPHF.from_pretrained("SanghyukChun/LongProLIP-ViT-B-16-S24M")
tokenizer = HFTokenizer("timm/ViT-B-16-SigLIP", context_length=64, clean="canonicalize")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, return_tensors="pt", padding=True)
texts = ["A couple of cats laying on top of a pink blanket.", "A man walks through a flooded road during a rainstorm", "photo"]
texts = tokenizer(texts)

outputs = model(image=inputs["pixel_values"], text=texts)

l2_logit = outputs["image_features"]["mean"] @ outputs["text_features"]["mean"].T
i_unc = torch.exp(outputs["image_features"]["std"]).sum(dim=-1)
t_unc = torch.exp(outputs["text_features"]["std"]).sum(dim=-1)
csd_logit = l2_logit - 0.5 * t_unc
csd_logit2 = l2_logit.T - 0.5 * i_unc
print("Mean-only image-to-text logits (by L2 distance):", l2_logit)
print("Uncertainty-aware image-to-text logits (by CSD):", csd_logit)
print("Uncertainty-aware text-to-image logits (by CSD):", csd_logit2.T)
print("Image uncertainty: ", i_unc)
print("Text uncertainty: ", t_unc)

Citation

@inproceedings{chun2025prolip,
    title={Probabilistic Language-Image Pre-Training},
    author={Chun, Sanghyuk and Kim, Wonjae and Park, Song and Yun, Sangdoo},
    year={2025},
    booktitle={International Conference on Learning Representations (ICLR)},
}

@inproceedings{chun2025longprolip,
    title={LongProLIP: A Probabilistic Vision-Language Model with Long Context Text},
    author={Chun, Sanghyuk and Yun, Sangdoo},
    year={2025},
    booktitle={ICLR Workshop on Quantify Uncertainty and Hallucination in Foundation Models},
}