File size: 64 Bytes
5fa1a76
1
A tokenizer splits text into tokens according to a set of rules.