InfoTokenizers
/

tokenizers

codebyzeb commited on May 16

Commit

8c571b6

verified ·

1 Parent(s): 0c66f52

Upload folder using huggingface_hub

Files changed (4) hide show

fw57M_Entropy_thresholdB_8064/stats.csv CHANGED Viewed

The diff for this file is too large to render. See raw diff

fw57M_Entropy_thresholdB_8064/tokenizer.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

fw57M_Entropy_thresholdB_8064/tokenizer_config.json CHANGED Viewed

@@ -17,7 +17,7 @@
       "single_word": false,
       "special": true
     },
-    "514": {
       "content": "<|unk|>",
       "lstrip": false,
       "normalized": false,

       "single_word": false,
       "special": true
     },
+    "699": {
       "content": "<|unk|>",
       "lstrip": false,
       "normalized": false,

fw57M_Entropy_thresholdB_8064/vocab.json CHANGED Viewed

The diff for this file is too large to render. See raw diff