|
|
|
mLUKE |
|
Overview |
|
The mLUKE model was proposed in mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models by Ryokan Ri, Ikuya Yamada, and Yoshimasa Tsuruoka. It's a multilingual extension |
|
of the LUKE model trained on the basis of XLM-RoBERTa. |
|
It is based on XLM-RoBERTa and adds entity embeddings, which helps improve performance on various downstream tasks |
|
involving reasoning about entities such as named entity recognition, extractive question answering, relation |
|
classification, cloze-style knowledge completion. |
|
The abstract from the paper is the following: |
|
Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual |
|
alignment information from Wikipedia entities. However, existing methods only exploit entity information in pretraining |
|
and do not explicitly use entities in downstream tasks. In this study, we explore the effectiveness of leveraging |
|
entity representations for downstream cross-lingual tasks. We train a multilingual language model with 24 languages |
|
with entity representations and show the model consistently outperforms word-based pretrained models in various |
|
cross-lingual transfer tasks. We also analyze the model and the key insight is that incorporating entity |
|
representations into the input allows us to extract more language-agnostic features. We also evaluate the model with a |
|
multilingual cloze prompt task with the mLAMA dataset. We show that entity-based prompt elicits correct factual |
|
knowledge more likely than using only word representations. |
|
This model was contributed by ryo0634. The original code can be found here. |
|
Usage tips |
|
One can directly plug in the weights of mLUKE into a LUKE model, like so: |
|
thon |
|
from transformers import LukeModel |
|
model = LukeModel.from_pretrained("studio-ousia/mluke-base") |
|
|
|
Note that mLUKE has its own tokenizer, [MLukeTokenizer]. You can initialize it as follows: |
|
thon |
|
from transformers import MLukeTokenizer |
|
tokenizer = MLukeTokenizer.from_pretrained("studio-ousia/mluke-base") |
|
|
|
As mLUKE's architecture is equivalent to that of LUKE, one can refer to LUKE's documentation page for all |
|
tips, code examples and notebooks. |
|
|
|
MLukeTokenizer |
|
[[autodoc]] MLukeTokenizer |
|
- call |
|
- save_vocabulary |