Open Source · MarkTechPost ·
Perplexity AI Open-Sources Unigram Tokenizer That Achieves 5x Lower p50 Latency Than Hugging Face tokenizers Crate
Perplexity AI has open-sourced a rewritten Unigram tokenizer, reporting 5x lower p50 latency and 5-6x reduction in CPU utilization compared to the Hugging Face tokenizers crate.