Ahmadzei's picture
added 3 more tables for large emb model
5fa1a76
Encoding text as a sequence of tokens requires a tokenizer, which is typically created as an independent artifact from
the model.