ByteSpanTokenisers
/

tokenizers

Model card Files Files and versions Community

tokenizers / bytelevel /special_tokens_map.json

pietrolesci's picture

Upload folder using huggingface_hub

592b2b4 verified 4 months ago

history blame contribute delete

65 Bytes

	{
	"eos_token": "<\|endoftext\|>",
	"pad_token": "<\|padding\|>"
	}