| PreTrainedTokenizerBase | |
| [[autodoc]] tokenization_utils_base.PreTrainedTokenizerBase | |
| - call | |
| - all | |
| SpecialTokensMixin | |
| [[autodoc]] tokenization_utils_base.SpecialTokensMixin | |
| Enums and namedtuples | |
| [[autodoc]] tokenization_utils_base.TruncationStrategy | |
| [[autodoc]] tokenization_utils_base.CharSpan | |
| [[autodoc]] tokenization_utils_base.TokenSpan |