huggingface-tokenizers
•published 2026/01/23
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
📦 Install Skill
Command line install
Install directly with the `skills` CLI.
npx skills add davila7/claude-code-templates/cli-tool/components/skills/ai-research/tokenization-huggingface-tokenizersFile Explorer
No file selected
Select a file to view its content