knowledge_base/model_doc_marian.txt

MarianMT

Overview
A framework for translation models, using the same models as BART. Translations should be similar, but not identical to output in the test set linked to in each model card.
This model was contributed by sshleifer.
Implementation Notes

Each model is about 298 MB on disk, there are more than 1,000 models.
The list of supported language pairs can be found here.
Models were originally trained by Jörg Tiedemann using the Marian C++ library, which supports fast training and translation.
All models are transformer encoder-decoders with 6 layers in each component. Each model's performance is documented
  in a model card.
The 80 opus models that require BPE preprocessing are not supported.

The modeling code is the same as [BartForConditionalGeneration] with a few minor modifications:

static (sinusoid) positional embeddings (MarianConfig.static_position_embeddings=True)

no layernorm_embedding (MarianConfig.normalize_embedding=False)
the model starts generating with pad_token_id (which has 0 as a token_embedding) as the prefix (Bart uses
    <s/>),
Code to bulk convert models can be found in convert_marian_to_pytorch.py.

Naming

All model names use the following format: Helsinki-NLP/opus-mt-{src}-{tgt}
The language codes used to name models are inconsistent. Two digit codes can usually be found here, three digit codes require googling "language
  code {code}".
Codes formatted like es_AR are usually code_{region}. That one is Spanish from Argentina.
The models were converted in two stages. The first 1000 models use ISO-639-2 codes to identify languages, the second
  group use a combination of ISO-639-5 codes and ISO-639-2 codes.

Examples

Since Marian models are smaller than many other translation models available in the library, they can be useful for
  fine-tuning experiments and integration tests.
Fine-tune on GPU

Multilingual Models

All model names use the following format: Helsinki-NLP/opus-mt-{src}-{tgt}:
If a model can output multiple languages, and you should specify a language code by prepending the desired output
  language to the src_text.
You can see a models's supported language codes in its model card, under target constituents, like in opus-mt-en-roa.
Note that if a model is only multilingual on the source side, like Helsinki-NLP/opus-mt-roa-en, no language
  codes are required.

New multi-lingual models from the Tatoeba-Challenge repo
require 3 character language codes:
thon

from transformers import MarianMTModel, MarianTokenizer
src_text = [
     ">>fra<< this is a sentence in english that we want to translate to french",
     ">>por<< This should go to portuguese",
     ">>esp<< And this to Spanish",
 ]
model_name = "Helsinki-NLP/opus-mt-en-roa"
tokenizer = MarianTokenizer.from_pretrained(model_name)
print(tokenizer.supported_language_codes)
['>>zlm_Latn<<', '>>mfe<<', '>>hat<<', '>>pap<<', '>>ast<<', '>>cat<<', '>>ind<<', '>>glg<<', '>>wln<<', '>>spa<<', '>>fra<<', '>>ron<<', '>>por<<', '>>ita<<', '>>oci<<', '>>arg<<', '>>min<<']
model = MarianMTModel.from_pretrained(model_name)
translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))
[tokenizer.decode(t, skip_special_tokens=True) for t in translated]
["c'est une phrase en anglais que nous voulons traduire en français",
 'Isto deve ir para o português.',
 'Y esto al español']

Here is the code to see all available pretrained models on the hub:
thon
from huggingface_hub import list_models
model_list = list_models()
org = "Helsinki-NLP"
model_ids = [x.modelId for x in model_list if x.modelId.startswith(org)]
suffix = [x.split("/")[1] for x in model_ids]
old_style_multi_models = [f"{org}/{s}" for s in suffix if s != s.lower()]

Old Style Multi-Lingual Models
These are the old style multi-lingual models ported from the OPUS-MT-Train repo: and the members of each language
group:
python no-style
['Helsinki-NLP/opus-mt-NORTH_EU-NORTH_EU',
 'Helsinki-NLP/opus-mt-ROMANCE-en',
 'Helsinki-NLP/opus-mt-SCANDINAVIA-SCANDINAVIA',
 'Helsinki-NLP/opus-mt-de-ZH',
 'Helsinki-NLP/opus-mt-en-CELTIC',
 'Helsinki-NLP/opus-mt-en-ROMANCE',
 'Helsinki-NLP/opus-mt-es-NORWAY',
 'Helsinki-NLP/opus-mt-fi-NORWAY',
 'Helsinki-NLP/opus-mt-fi-ZH',
 'Helsinki-NLP/opus-mt-fi_nb_no_nn_ru_sv_en-SAMI',
 'Helsinki-NLP/opus-mt-sv-NORWAY',
 'Helsinki-NLP/opus-mt-sv-ZH']
GROUP_MEMBERS = {
 'ZH': ['cmn', 'cn', 'yue', 'ze_zh', 'zh_cn', 'zh_CN', 'zh_HK', 'zh_tw', 'zh_TW', 'zh_yue', 'zhs', 'zht', 'zh'],
 'ROMANCE': ['fr', 'fr_BE', 'fr_CA', 'fr_FR', 'wa', 'frp', 'oc', 'ca', 'rm', 'lld', 'fur', 'lij', 'lmo', 'es', 'es_AR', 'es_CL', 'es_CO', 'es_CR', 'es_DO', 'es_EC', 'es_ES', 'es_GT', 'es_HN', 'es_MX', 'es_NI', 'es_PA', 'es_PE', 'es_PR', 'es_SV', 'es_UY', 'es_VE', 'pt', 'pt_br', 'pt_BR', 'pt_PT', 'gl', 'lad', 'an', 'mwl', 'it', 'it_IT', 'co', 'nap', 'scn', 'vec', 'sc', 'ro', 'la'],
 'NORTH_EU': ['de', 'nl', 'fy', 'af', 'da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],
 'SCANDINAVIA': ['da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],
 'SAMI': ['se', 'sma', 'smj', 'smn', 'sms'],
 'NORWAY': ['nb_NO', 'nb', 'nn_NO', 'nn', 'nog', 'no_nb', 'no'],
 'CELTIC': ['ga', 'cy', 'br', 'gd', 'kw', 'gv']
}
Example of translating english to many romance languages, using old-style 2 character language codes
thon

from transformers import MarianMTModel, MarianTokenizer
src_text = [
     ">>fr<< this is a sentence in english that we want to translate to french",
     ">>pt<< This should go to portuguese",
     ">>es<< And this to Spanish",
 ]
model_name = "Helsinki-NLP/opus-mt-en-ROMANCE"
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)
translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))
tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]
["c'est une phrase en anglais que nous voulons traduire en français", 
 'Isto deve ir para o português.',
 'Y esto al español']

Resources

Translation task guide
Summarization task guide
Causal language modeling task guide

MarianConfig
[[autodoc]] MarianConfig
MarianTokenizer
[[autodoc]] MarianTokenizer
    - build_inputs_with_special_tokens

MarianModel
[[autodoc]] MarianModel
    - forward
MarianMTModel
[[autodoc]] MarianMTModel
    - forward
MarianForCausalLM
[[autodoc]] MarianForCausalLM
    - forward

TFMarianModel
[[autodoc]] TFMarianModel
    - call
TFMarianMTModel
[[autodoc]] TFMarianMTModel
    - call

FlaxMarianModel
[[autodoc]] FlaxMarianModel
    - call
FlaxMarianMTModel
[[autodoc]] FlaxMarianMTModel
    - call