File size: 251 Bytes
5fa1a76
 
 
 
1
2
3
4
You can now use a second preprocessing function to

concatenate all the sequences
split the concatenated sequences into shorter chunks defined by block_size, which should be both shorter than the maximum input length and short enough for your GPU RAM.