Ahmadzei's picture
added 3 more tables for large emb model
5fa1a76
tokenizer = BrandNewBertTokenizer.from_pretrained("/path/to/tokenizer/folder/")
input_ids = tokenizer(input_str).input_ids
When both input_ids yield the same values, as a final step a tokenizer test file should also be added.