Hugging Face's logo

InfoTokenizers
/
tokenizers

Model card Files Community
Zeb commited on
Commit
ad21efd
·
1 Parent(s): be82611

Rename frequency tokenizer

Browse files
Files changed (6) hide show
  1. {fw57M_Entropy_frequency_16000 → frequency_16000}/merges.txt +0 -0
  2. {fw57M_Entropy_frequency_16000 → frequency_16000}/merges_data.csv +0 -0
  3. {fw57M_Entropy_frequency_16000 → frequency_16000}/special_tokens_map.json +0 -0
  4. {fw57M_Entropy_frequency_16000 → frequency_16000}/tokenizer.json +0 -0
  5. {fw57M_Entropy_frequency_16000 → frequency_16000}/tokenizer_config.json +0 -0
  6. {fw57M_Entropy_frequency_16000 → frequency_16000}/vocab.json +0 -0
{fw57M_Entropy_frequency_16000 → frequency_16000}/merges.txt RENAMED
File without changes
{fw57M_Entropy_frequency_16000 → frequency_16000}/merges_data.csv RENAMED
File without changes
{fw57M_Entropy_frequency_16000 → frequency_16000}/special_tokens_map.json RENAMED
File without changes
{fw57M_Entropy_frequency_16000 → frequency_16000}/tokenizer.json RENAMED
File without changes
{fw57M_Entropy_frequency_16000 → frequency_16000}/tokenizer_config.json RENAMED
File without changes
{fw57M_Entropy_frequency_16000 → frequency_16000}/vocab.json RENAMED
File without changes