---
license: mit
datasets:
- mteb/scifact
language:
- en
pipeline_tag: text-retrieval
library_name: sentence-transformers
tags:
- mteb
- text
- transformers
- text-embeddings-inference
- CSR
model-index:
- name: CSR
  results:
    - dataset:
        name: MTEB SciFact
        type: mteb/scifact
        revision: 0228b52cf27578f30900b9e5271d331663a030d7
        config: default
        split: test
        languages:
          - eng-Latn
      metrics:
        - type: ndcg@1
          value: 0.59333
        - type: ndcg@3
          value: 0.65703
        - type: ndcg@5
          value: 0.67072
        - type: ndcg@10
          value: 0.68412
        - type: ndcg@20
          value: 0.69238
        - type: ndcg@100
          value: 0.70514
        - type: ndcg@1000
          value: 0.71517
        - type: map@1
          value: 0.5675
        - type: map@3
          value: 0.63602
        - type: map@5
          value: 0.64712
        - type: map@10
          value: 0.65301
        - type: map@20
          value: 0.65552
        - type: map@100
          value: 0.65778
        - type: map@1000
          value: 0.65815
        - type: recall@1
          value: 0.5675
        - type: recall@3
          value: 0.69772
        - type: recall@5
          value: 0.73367
        - type: recall@10
          value: 0.77333
        - type: recall@20
          value: 0.80367
        - type: recall@100
          value: 0.86667
        - type: recall@1000
          value: 0.945
        - type: precision@1
          value: 0.59333
        - type: precision@3
          value: 0.25667
        - type: precision@5
          value: 0.164
        - type: precision@10
          value: 0.08667
        - type: precision@20
          value: 0.04533
        - type: precision@100
          value: 0.0099
        - type: precision@1000
          value: 0.00107
        - type: mrr@1
          value: 0.59333
        - type: mrr@3
          value: 0.64667
        - type: mrr@5
          value: 0.65333
        - type: mrr@10
          value: 0.65883
        - type: mrr@20
          value: 0.66105
        - type: mrr@100
          value: 0.66254
        - type: mrr@1000
          value: 0.66292
        - type: main_score
          value: 0.68412
      task:
        type: Retrieval
---

For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our [Github](https://github.com/neilwen987/CSR_Adaptive_Rep).


## Usage
📌 **Tip**: For NV-Embed-V2, using Transformers versions **later** than 4.47.0 may lead to performance degradation, as ``model_type=bidir_mistral`` in ``config.json``  is no longer supported.

We recommend using ``Transformers 4.47.0.``

### Sentence Transformers Usage
You can evaluate this model loaded by Sentence Transformers with the following code snippet:
```python
import mteb
from sentence_transformers import SparseEncoder
model = SparseEncoder(
    "Y-Research-Group/CSR-NV_Embed_v2-Retrieval-SciFACT ",
    trust_remote_code=True
)
model.prompts = {
    "SciFact-query": "Instrcut: Given a scientific claim, retrieve documents that support or refute the claim\nQuery:"
}
task = mteb.get_tasks(tasks=["SciFact"])
evaluation = mteb.MTEB(tasks=task)
evaluation.run(
    model,
    eval_splits=["test"],
    output_folder="./results/SciFact",
    show_progress_bar=True
    encode_kwargs={"convert_to_sparse_tensor": False, "batch_size": 8},
)  # MTEB don't support sparse tensors yet, so we need to convert to dense tensors
```

## Citation
```bibtex
@inproceedings{wenbeyond,
  title={Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation},
  author={Wen, Tiansheng and Wang, Yifei and Zeng, Zequn and Peng, Zhong and Su, Yudi and Liu, Xinyang and Chen, Bo and Liu, Hongwei and Jegelka, Stefanie and You, Chenyu},
  booktitle={Forty-second International Conference on Machine Learning}
}
```