Hugging Face's logo

smartmind
/
KURE-v1

Feature Extraction
sentence-transformers
Safetensors
xlm-roberta
sentence-similarity
Generated from Trainer
dataset_size:1879136
loss:CachedGISTEmbedLoss
text-embeddings-inference
Model card Files Community
Jake-Network commited on
Commit
3dc0bd8
·
verified ·
1 Parent(s): 40061d7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -159
README.md CHANGED
@@ -18,29 +18,6 @@ library_name: sentence-transformers
18
 
19
  # 🔎 KURE-v1
20
 
21
- Introducing Korea University Retrieval Embedding model, KURE-v1
22
- It has shown remarkable performance in Korean text retrieval, speficially overwhelming most multilingual embedding models.
23
- To our knowledge, It is one of the best publicly opened Korean retrieval models.
24
-
25
- For details, visit the [KURE repository](https://github.com/nlpai-lab/KURE)
26
-
27
- ---
28
-
29
- ## Model Versions
30
- | Model Name | Dimension | Sequence Length | Introduction |
31
- |:----:|:---:|:---:|:---:|
32
- | [KURE-v1](https://huggingface.co/nlpai-lab/KURE-v1) | 1024 | 8192 | Fine-tuned [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) with Korean data via [CachedGISTEmbedLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedgistembedloss)
33
- | [KoE5](https://huggingface.co/nlpai-lab/KoE5) | 1024 | 512 | Fine-tuned [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) with [ko-triplet-v1.0](https://huggingface.co/datasets/nlpai-lab/ko-triplet-v1.0) via [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) |
34
-
35
- ## Model Description
36
-
37
- This is the model card of a 🤗 transformers model that has been pushed on the Hub.
38
-
39
- - **Developed by:** [NLP&AI Lab](http://nlp.korea.ac.kr/)
40
- - **Language(s) (NLP):** Korean, English
41
- - **License:** MIT
42
- - **Finetuned from model:** [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3)
43
-
44
  ## Example code
45
  ### Install Dependencies
46
  First install the Sentence Transformers library:
@@ -74,139 +51,3 @@ print(similarities)
74
  # [0.6967, 1.0000, 0.4427],
75
  # [0.5306, 0.4427, 1.0000]])
76
  ```
77
-
78
- ## Training Details
79
-
80
- ### Training Data
81
-
82
- #### KURE-v1
83
- - Korean query-document-hard_negative(5) data
84
- - 2,000,000 examples
85
-
86
- ### Training Procedure
87
- - **loss:** Used **[CachedGISTEmbedLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedgistembedloss)** by sentence-transformers
88
- - **batch size:** 4096
89
- - **learning rate:** 2e-05
90
- - **epochs:** 1
91
-
92
- ## Evaluation
93
- ### Metrics
94
- - Recall, Precision, NDCG, F1
95
- ### Benchmark Datasets
96
- - [Ko-StrategyQA](https://huggingface.co/datasets/taeminlee/Ko-StrategyQA): 한국어 ODQA multi-hop 검색 데이터셋 (StrategyQA 번역)
97
- - [AutoRAGRetrieval](https://huggingface.co/datasets/yjoonjang/markers_bm): 금융, 공공, 의료, 법률, 커머스 5개 분야에 대해, pdf를 파싱하여 구성한 한국어 문서 검색 데이터셋
98
- - [MIRACLRetrieval]([url](https://huggingface.co/datasets/miracl/miracl)): Wikipedia 기반의 한국어 문서 검색 데이터셋
99
- - [PublicHealthQA]([url](https://huggingface.co/datasets/xhluca/publichealth-qa)): 의료 및 공중보건 도메인에 대한 한국어 문서 검색 데이터셋
100
- - [BelebeleRetrieval]([url](https://huggingface.co/datasets/facebook/belebele)): FLORES-200 기반의 한국어 문서 검색 데이터셋
101
- - [MrTidyRetrieval](https://huggingface.co/datasets/mteb/mrtidy): Wikipedia 기반의 한국어 문서 검색 데이터셋
102
- - [MultiLongDocRetrieval](https://huggingface.co/datasets/Shitao/MLDR): 다양한 도메인의 한국어 장문 검색 데이터셋
103
- - [XPQARetrieval](https://huggingface.co/datasets/jinaai/xpqa): 다양한 도메인의 한국어 문서 검색 데이터셋
104
-
105
- ## Results
106
-
107
- 아래는 모든 모델의, 모든 벤치마크 데이터셋에 대한 평균 결과입니다.
108
- 자세한 결과는 [KURE Github](https://github.com/nlpai-lab/KURE/tree/main/eval/results)에서 확인하실 수 있습니다.
109
- ### Top-k 1
110
- | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
111
- |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
112
- | **nlpai-lab/KURE-v1** | **0.52640** | **0.60551** | **0.60551** | **0.55784** |
113
- | dragonkue/BGE-m3-ko | 0.52361 | 0.60394 | 0.60394 | 0.55535 |
114
- | BAAI/bge-m3 | 0.51778 | 0.59846 | 0.59846 | 0.54998 |
115
- | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.51246 | 0.59384 | 0.59384 | 0.54489 |
116
- | nlpai-lab/KoE5 | 0.50157 | 0.57790 | 0.57790 | 0.53178 |
117
- | intfloat/multilingual-e5-large | 0.50052 | 0.57727 | 0.57727 | 0.53122 |
118
- | jinaai/jina-embeddings-v3 | 0.48287 | 0.56068 | 0.56068 | 0.51361 |
119
- | BAAI/bge-multilingual-gemma2 | 0.47904 | 0.55472 | 0.55472 | 0.50916 |
120
- | intfloat/multilingual-e5-large-instruct | 0.47842 | 0.55435 | 0.55435 | 0.50826 |
121
- | intfloat/multilingual-e5-base | 0.46950 | 0.54490 | 0.54490 | 0.49947 |
122
- | intfloat/e5-mistral-7b-instruct | 0.46772 | 0.54394 | 0.54394 | 0.49781 |
123
- | Alibaba-NLP/gte-multilingual-base | 0.46469 | 0.53744 | 0.53744 | 0.49353 |
124
- | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.46633 | 0.53625 | 0.53625 | 0.49429 |
125
- | openai/text-embedding-3-large | 0.44884 | 0.51688 | 0.51688 | 0.47572 |
126
- | Salesforce/SFR-Embedding-2_R | 0.43748 | 0.50815 | 0.50815 | 0.46504 |
127
- | upskyy/bge-m3-korean | 0.43125 | 0.50245 | 0.50245 | 0.45945 |
128
- | jhgan/ko-sroberta-multitask | 0.33788 | 0.38497 | 0.38497 | 0.35678 |
129
-
130
- ### Top-k 3
131
- | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
132
- |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
133
- | **nlpai-lab/KURE-v1** | **0.68678** | **0.28711** | **0.65538** | **0.39835** |
134
- | dragonkue/BGE-m3-ko | 0.67834 | 0.28385 | 0.64950 | 0.39378 |
135
- | BAAI/bge-m3 | 0.67526 | 0.28374 | 0.64556 | 0.39291 |
136
- | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.67128 | 0.28193 | 0.64042 | 0.39072 |
137
- | intfloat/multilingual-e5-large | 0.65807 | 0.27777 | 0.62822 | 0.38423 |
138
- | nlpai-lab/KoE5 | 0.65174 | 0.27329 | 0.62369 | 0.37882 |
139
- | BAAI/bge-multilingual-gemma2 | 0.64415 | 0.27416 | 0.61105 | 0.37782 |
140
- | jinaai/jina-embeddings-v3 | 0.64116 | 0.27165 | 0.60954 | 0.37511 |
141
- | intfloat/multilingual-e5-large-instruct | 0.64353 | 0.27040 | 0.60790 | 0.37453 |
142
- | Alibaba-NLP/gte-multilingual-base | 0.63744 | 0.26404 | 0.59695 | 0.36764 |
143
- | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.63163 | 0.25937 | 0.59237 | 0.36263 |
144
- | intfloat/multilingual-e5-base | 0.62099 | 0.26144 | 0.59179 | 0.36203 |
145
- | intfloat/e5-mistral-7b-instruct | 0.62087 | 0.26144 | 0.58917 | 0.36188 |
146
- | openai/text-embedding-3-large | 0.61035 | 0.25356 | 0.57329 | 0.35270 |
147
- | Salesforce/SFR-Embedding-2_R | 0.60001 | 0.25253 | 0.56346 | 0.34952 |
148
- | upskyy/bge-m3-korean | 0.59215 | 0.25076 | 0.55722 | 0.34623 |
149
- | jhgan/ko-sroberta-multitask | 0.46930 | 0.18994 | 0.43293 | 0.26696 |
150
-
151
- ### Top-k 5
152
- | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
153
- |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
154
- | **nlpai-lab/KURE-v1** | **0.73851** | **0.19130** | **0.67479** | **0.29903** |
155
- | dragonkue/BGE-m3-ko | 0.72517 | 0.18799 | 0.66692 | 0.29401 |
156
- | BAAI/bge-m3 | 0.72954 | 0.18975 | 0.66615 | 0.29632 |
157
- | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.72962 | 0.18875 | 0.66236 | 0.29542 |
158
- | nlpai-lab/KoE5 | 0.70820 | 0.18287 | 0.64499 | 0.28628 |
159
- | intfloat/multilingual-e5-large | 0.70124 | 0.18316 | 0.64402 | 0.28588 |
160
- | BAAI/bge-multilingual-gemma2 | 0.70258 | 0.18556 | 0.63338 | 0.28851 |
161
- | jinaai/jina-embeddings-v3 | 0.69933 | 0.18256 | 0.63133 | 0.28505 |
162
- | intfloat/multilingual-e5-large-instruct | 0.69018 | 0.17838 | 0.62486 | 0.27933 |
163
- | Alibaba-NLP/gte-multilingual-base | 0.69365 | 0.17789 | 0.61896 | 0.27879 |
164
- | intfloat/multilingual-e5-base | 0.67250 | 0.17406 | 0.61119 | 0.27247 |
165
- | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.67447 | 0.17114 | 0.60952 | 0.26943 |
166
- | intfloat/e5-mistral-7b-instruct | 0.67449 | 0.17484 | 0.60935 | 0.27349 |
167
- | openai/text-embedding-3-large | 0.66365 | 0.17004 | 0.59389 | 0.26677 |
168
- | Salesforce/SFR-Embedding-2_R | 0.65622 | 0.17018 | 0.58494 | 0.26612 |
169
- | upskyy/bge-m3-korean | 0.65477 | 0.17015 | 0.58073 | 0.26589 |
170
- | jhgan/ko-sroberta-multitask | 0.53136 | 0.13264 | 0.45879 | 0.20976 |
171
-
172
- ### Top-k 10
173
- | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
174
- |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
175
- | **nlpai-lab/KURE-v1** | **0.79682** | **0.10624** | **0.69473** | **0.18524** |
176
- | dragonkue/BGE-m3-ko | 0.78450 | 0.10492 | 0.68748 | 0.18288 |
177
- | BAAI/bge-m3 | 0.79195 | 0.10592 | 0.68723 | 0.18456 |
178
- | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.78669 | 0.10462 | 0.68189 | 0.18260 |
179
- | intfloat/multilingual-e5-large | 0.75902 | 0.10147 | 0.66370 | 0.17693 |
180
- | nlpai-lab/KoE5 | 0.75296 | 0.09937 | 0.66012 | 0.17369 |
181
- | BAAI/bge-multilingual-gemma2 | 0.76153 | 0.10364 | 0.65330 | 0.18003 |
182
- | jinaai/jina-embeddings-v3 | 0.76277 | 0.10240 | 0.65290 | 0.17843 |
183
- | intfloat/multilingual-e5-large-instruct | 0.74851 | 0.09888 | 0.64451 | 0.17283 |
184
- | Alibaba-NLP/gte-multilingual-base | 0.75631 | 0.09938 | 0.64025 | 0.17363 |
185
- | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.74092 | 0.09607 | 0.63258 | 0.16847 |
186
- | intfloat/multilingual-e5-base | 0.73512 | 0.09717 | 0.63216 | 0.16977 |
187
- | intfloat/e5-mistral-7b-instruct | 0.73795 | 0.09777 | 0.63076 | 0.17078 |
188
- | openai/text-embedding-3-large | 0.72946 | 0.09571 | 0.61670 | 0.16739 |
189
- | Salesforce/SFR-Embedding-2_R | 0.71662 | 0.09546 | 0.60589 | 0.16651 |
190
- | upskyy/bge-m3-korean | 0.71895 | 0.09583 | 0.60258 | 0.16712 |
191
- | jhgan/ko-sroberta-multitask | 0.61225 | 0.07826 | 0.48687 | 0.13757 |
192
- <br/>
193
-
194
- ## Citation
195
-
196
- If you find our paper or models helpful, please consider cite as follows:
197
- ```text
198
- @misc{KURE,
199
- publisher = {Youngjoon Jang, Junyoung Son, Taemin Lee},
200
- year = {2024},
201
- url = {https://github.com/nlpai-lab/KURE}
202
- },
203
-
204
- @misc{KoE5,
205
- author = {NLP & AI Lab and Human-Inspired AI research},
206
- title = {KoE5: A New Dataset and Model for Improving Korean Embedding Performance},
207
- year = {2024},
208
- publisher = {Youngjoon Jang, Junyoung Son, Taemin Lee},
209
- journal = {GitHub repository},
210
- howpublished = {\url{https://github.com/nlpai-lab/KoE5}},
211
- }
212
- ```
 
18
 
19
  # 🔎 KURE-v1
20
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  ## Example code
22
  ### Install Dependencies
23
  First install the Sentence Transformers library:
 
51
  # [0.6967, 1.0000, 0.4427],
52
  # [0.5306, 0.4427, 1.0000]])
53
  ```