ByteSpanTokenisers
/

tokenizers

Model card Files Files and versions Community

Ctrl+K

Ctrl+K

3 contributors

History: 21 commits

codebyzeb's picture

Upload folder using huggingface_hub

a4af786 verified 3 months ago

bytelevel
Upload folder using huggingface_hub 3 months ago
frequency_16000
Rename frequency tokenizer 3 months ago
fw57M_Entropy_frequency-mean-post-merge_16000
Upload folder using huggingface_hub 3 months ago
fw57M_Entropy_frequency_16000
Upload folder using huggingface_hub 3 months ago
fw57M_Entropy_frequency_8064
Upload folder using huggingface_hub 3 months ago
fw57M_Entropy_min-mean-post-merge_16000
Upload folder using huggingface_hub 3 months ago
fw57M_Entropy_min-mean-post-merge_8064
Upload folder using huggingface_hub 3 months ago
fw57M_Entropy_min-mean-pre-merge_16000
Upload folder using huggingface_hub 3 months ago
fw57M_Space Probability_frequency-mean-post-merge_16000
Upload folder using huggingface_hub 3 months ago
fw57M_Space Probability_min-mean-post-merge_16000
Upload folder using huggingface_hub 3 months ago
fw57M_Space Probability_min-mean-post-merge_32000
Upload folder using huggingface_hub 3 months ago
fw57M_Space Probability_min-mean-post-merge_64000
Upload folder using huggingface_hub 3 months ago
fw57M_Space Probability_min-mean-post-merge_8064
Upload folder using huggingface_hub 3 months ago
fw57M_Space Probability_min-mean-pre-merge_16000
Upload folder using huggingface_hub 3 months ago
fw57M_Surprisal_frequency-mean-post-merge_16000
Upload folder using huggingface_hub 3 months ago
fw57M_Surprisal_min-mean-post-merge_16000
Upload folder using huggingface_hub 3 months ago
fw57M_Surprisal_min-mean-pre-merge_16000
Upload folder using huggingface_hub 3 months ago
.gitattributes

1.52 kB

initial commit 3 months ago