Ahmadzei's picture
added 3 more tables for large emb model
5fa1a76
raw
history blame contribute delete
194 Bytes
When the tokenizer is a pure python
tokenizer, this class behaves just like a standard python dictionary and holds the various model inputs computed by
these methods (input_ids, attention_mask).