Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
pietrolesci
/
tokenisers
like
0
Model card
Files
Files and versions
xet
Community
main
tokenisers
/
wordpiece32000minipile
/
special_tokens_map.json
pietrolesci
Upload folder using huggingface_hub
c5527f0
verified
6 months ago
raw
Copy download link
history
blame
contribute
delete
Safe
61 Bytes
{
"eos_token"
:
"<|endoftext|>"
,
"unk_token"
:
"<|unk|>"
}