agentix-embed-code-ts - v0.1.1
    Preparing search index...

    Function preTokenize

    • Pre-tokenize text into word units (BertPreTokenizer).

      Steps:

      1. Lowercase (BertNormalizer with lowercase=true)
      2. Split on whitespace
      3. Split punctuation from word boundaries
      4. Handle Chinese/CJK characters (split each as own token)

      Parameters

      • text: string

      Returns string[]

    Settings

    Member Visibility