ByteSpanTokenisers
/

tokenizers

Model card Files Files and versions Community

Commit History

Fix spaceless tokenizers

c58de58

Zeb commited on May 8

Remove ngram tokenizers

7487a89

Zeb commited on May 8

Delete fw57M_Entropy_thresholdM_500

a802c9a
verified

codebyzeb commited on May 8

Fix multilingual frequency

d30eee2

Zeb commited on May 8

Upload folder using huggingface_hub

29dda54
verified

codebyzeb commited on May 8

Delete fw57M_Entropy_thresholdM_500

b3b5b2d
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

ad3d889
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

7e118b6
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

62a0567
verified

codebyzeb commited on May 8

Fix tokenizers

f9c3417

Zeb commited on May 8

Separate frequency tokenizers

f25f2a5

Zeb commited on May 8

Upload folder using huggingface_hub

f9c85cd
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

280544f
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

3bd5966
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

77097ea
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

67b58c4
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

f0fd6ae
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

72c0e05
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

114a45f
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

06adb6a
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

1b719ca

codebyzeb commited on May 8

Upload folder using huggingface_hub

853f739
verified

codebyzeb commited on May 8

Upload folder using huggingface_hub

60e26ce
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

4763edc
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

d2ac827
verified

codebyzeb commited on May 7

Delete fw57M_Entropy_thresholdBX_600

a4fb558
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

53333c9
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

d000429
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

0322999
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

c404f26
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

58beac7
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

57b71d2
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

d7e180f
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

7c9b416
verified

codebyzeb commited on May 7

Upload folder using huggingface_hub

77f8c2e
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

ed6a20c
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

006a6ab
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

36edef8
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

8d254aa
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

420e742
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

4a56766
verified

codebyzeb commited on May 6

Remove faulty tokenizers

faf7f42

Zeb commited on May 6

Upload folder using huggingface_hub

8994231
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

c8158ef
verified

codebyzeb commited on May 6

Rename and remove old tokenizers

24ade47

Zeb commited on May 6

Upload folder using huggingface_hub

f554e97
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

0f89be7
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

ccf1210
verified

codebyzeb commited on May 6

Upload folder using huggingface_hub

8a22353
verified

codebyzeb commited on May 5

Upload folder using huggingface_hub

161dd71
verified

codebyzeb commited on May 5