| Utilities for Tokenizers | |
| This page lists all the utility functions used by the tokenizers, mainly the class | |
| [~tokenization_utils_base.PreTrainedTokenizerBase] that implements the common methods between | |
| [PreTrainedTokenizer] and [PreTrainedTokenizerFast] and the mixin | |
| [~tokenization_utils_base.SpecialTokensMixin]. |