انتقل إلى المحتوى الرئيسي

Function: cjkSentenceTokenizer()

cjkSentenceTokenizer(sentence): string[]

Tokenizes sentences. Suitable for Chinese, Japanese, and Korean. Use instead of defaultSentenceTokenizer.

Parameters​

• sentence: string

Returns​

string[]

string[]

Source​

packages/core/src/TextSplitter.ts:41