Ahmadzei's picture
added 3 more tables for large emb model
5fa1a76
This ensures the text is split the same way as the pretraining corpus, and uses the same corresponding tokens-to-index (usually referred to as the vocab) during pretraining.