paragraph: Most tokenization schemes have two parts: a token learner, and a token segmenter. The token learner takes a raw training corpus (sometimes roughly preseparated into words, for example by whitespace) and induces a vocabulary, a set of tokens. The token segmenter takes a raw test sentence and segments it into the tokens in the vocabulary. Three algorithms are widely used: byte-pair encoding (Sennrich et al., 2016) , unigram language modeling (Kudo, 2018) , and WordPiece (Schuster and Nakajima, 2012) ; there is also a SentencePiece library that includes implementations of the first two of the three (Kudo and Richardson, 2018) .
answer: a token learner, and a token segmenter