Hugging Face's logo

InfoTokenizers
/
tokenizers

Model card Files Community
Zeb commited on
Commit
d30eee2
·
1 Parent(s): 29dda54

Fix multilingual frequency

Browse files
Files changed (8) hide show
  1. frequency_64000/merges.txt +0 -0
  2. frequency_64000/merges_data.csv +0 -0
  3. frequency_64000/tokenizer.json +0 -0
  4. frequency_64000/vocab.json +0 -0
  5. frequencymulti_64000/merges.txt +0 -0
  6. frequencymulti_64000/merges_data.csv +0 -0
  7. frequencymulti_64000/tokenizer.json +0 -0
  8. frequencymulti_64000/vocab.json +0 -0
frequency_64000/merges.txt CHANGED
The diff for this file is too large to render. See raw diff
 
frequency_64000/merges_data.csv CHANGED
The diff for this file is too large to render. See raw diff
 
frequency_64000/tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
frequency_64000/vocab.json CHANGED
The diff for this file is too large to render. See raw diff
 
frequencymulti_64000/merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
frequencymulti_64000/merges_data.csv ADDED
The diff for this file is too large to render. See raw diff
 
frequencymulti_64000/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
frequencymulti_64000/vocab.json ADDED
The diff for this file is too large to render. See raw diff