dbrx-base-tokenizer / tokenizer.json
Qubitium's picture
add pad and extra_N special tokens
3a0d911
This file is stored with LFS . It is too big to display, but you can still download it.

Large File Pointer Details

( Raw pointer file )
SHA256:
a974ee932171faced419ceee9a4f8e7de782c4bbae3884a69bb5df59080e92e6
Pointer size:
132 Bytes
·
Size of remote file:
4.24 MB

Git Large File Storage (LFS) replaces large files with text pointers inside Git, while storing the file contents on a remote server. More info.