Tools · MarkTechPost ·
Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers
Gigatoken is an MIT-licensed Rust BPE tokenizer that reportedly processes GPT-2 text at 24.53 GB/s on a 144-core AMD EPYC 9565. Its performance gains come from a hand-written SWAR pretokenizer and pretoken caching rather than changes to the BPE merge loop.