VocabLoader — loads a WordPiece vocabulary from tokenizer.json.
The vocab maps token strings → integer IDs. Special tokens [CLS], [SEP], [UNK], [PAD], [MASK] are extracted. Provides O(1) lookup via Map<string, number>.
Readonly
Map from token string to ID
Map from ID to token string
Vocabulary size
Special token IDs
Whether a token at a given ID is a continuation subword (starts with ##)
VocabLoader — loads a WordPiece vocabulary from tokenizer.json.
The vocab maps token strings → integer IDs. Special tokens [CLS], [SEP], [UNK], [PAD], [MASK] are extracted. Provides O(1) lookup via Map<string, number>.