Class WordPieceTokenizer

constructor

• new WordPieceTokenizer(vocab: VocabInfo, maxLength?: number): WordPieceTokenizer

  Parameters

  • vocab: VocabInfo
  • maxLength: number = 512

  Returns WordPieceTokenizer

ReadonlymaxLength

padTokenId

• get padTokenId(): number

  Returns number

clsTokenId

• get clsTokenId(): number

  Returns number

sepTokenId

• get sepTokenId(): number

  Returns number

unkTokenId

• get unkTokenId(): number

  Returns number

maskTokenId

• get maskTokenId(): number

  Returns number

vocabSize

• get vocabSize(): number

  Returns number

StaticfromJSON

• fromJSON(json: Record<string, any>, maxLength?: number): WordPieceTokenizer

  Load from a parsed tokenizer.json object

  Parameters

  • json: Record<string, any>
  • maxLength: number = 512

  Returns WordPieceTokenizer

StaticfromFile

• fromFile(path: string, maxLength?: number): WordPieceTokenizer

  Load from a tokenizer.json file path (Node.js)

  Parameters

  • path: string
  • maxLength: number = 512

  Returns WordPieceTokenizer

StaticfromBuffer

• fromBuffer(buf: ArrayBuffer, maxLength?: number): WordPieceTokenizer

  Load from a buffer (embeddable scenarios)

  Parameters

  • buf: ArrayBuffer
  • maxLength: number = 512

  Returns WordPieceTokenizer

tokenize

• tokenize(text: string): TokenizedInput

  Tokenize a single text. Returns inputIds, attentionMask, tokenTypeIds as Int32Array(length=maxLength).

  Parameters

  • text: string

  Returns TokenizedInput

tokenizeBatch

• tokenizeBatch(texts: string[], maxLength?: number): TokenizedInput

  Tokenize a batch of texts with padding to maxLength.

  Parameters

  • texts: string[]
  • OptionalmaxLength: number

  Returns TokenizedInput