Skip to main content

Class: SentenceSplitter

SentenceSplitter is our default text splitter that supports splitting into sentences, paragraphs, or fixed length chunks with overlap.

One of the advantages of SentenceSplitter is that even in the fixed length chunks it will try to keep sentences together.

Constructors​

new SentenceSplitter()​

new SentenceSplitter(options?): SentenceSplitter

Parameters​

• options?

• options.chunkOverlap?: number

• options.chunkSize?: number

• options.chunkingTokenizerFn?

• options.paragraphSeparator?: string

• options.splitLongSentences?: boolean

• options.tokenizer?: any

• options.tokenizerDecoder?: any

Returns​

SentenceSplitter

Source​

packages/core/src/TextSplitter.ts:76

Properties​

chunkOverlap​

chunkOverlap: number

Source​

packages/core/src/TextSplitter.ts:69

chunkSize​

chunkSize: number

Source​

packages/core/src/TextSplitter.ts:68

chunkingTokenizerFn()​

private chunkingTokenizerFn: (text) => string[]

Parameters​

• text: string

Returns​

string[]

Source​

packages/core/src/TextSplitter.ts:73

paragraphSeparator​

private paragraphSeparator: string

Source​

packages/core/src/TextSplitter.ts:72

splitLongSentences​

private splitLongSentences: boolean

Source​

packages/core/src/TextSplitter.ts:74

tokenizer​

private tokenizer: Tokenizer

Source​

packages/core/src/TextSplitter.ts:71

Methods​

combineTextSplits()​

combineTextSplits(newSentenceSplits, effectiveChunkSize): TextSplit[]

Parameters​

• newSentenceSplits: SplitRep[]

• effectiveChunkSize: number

Returns​

TextSplit[]

Source​

packages/core/src/TextSplitter.ts:214

getEffectiveChunkSize()​

private getEffectiveChunkSize(extraInfoStr?): number

Parameters​

• extraInfoStr?: string

Returns​

number

Source​

packages/core/src/TextSplitter.ts:112

getParagraphSplits()​

getParagraphSplits(text, effectiveChunkSize?): string[]

Parameters​

• text: string

• effectiveChunkSize?: number

Returns​

string[]

Source​

packages/core/src/TextSplitter.ts:130

getSentenceSplits()​

getSentenceSplits(text, effectiveChunkSize?): string[]

Parameters​

• text: string

• effectiveChunkSize?: number

Returns​

string[]

Source​

packages/core/src/TextSplitter.ts:156

processSentenceSplits()​

private processSentenceSplits(sentenceSplits, effectiveChunkSize): SplitRep[]

Splits sentences into chunks if necessary.

This isn't great behavior because it can split down the middle of a word or in non-English split down the middle of a Unicode codepoint so the splitting is turned off by default. If you need it, please set the splitLongSentences option to true.

Parameters​

• sentenceSplits: string[]

• effectiveChunkSize: number

Returns​

SplitRep[]

Source​

packages/core/src/TextSplitter.ts:185

splitText()​

splitText(text, extraInfoStr?): string[]

Parameters​

• text: string

• extraInfoStr?: string

Returns​

string[]

Source​

packages/core/src/TextSplitter.ts:308

splitTextWithOverlaps()​

splitTextWithOverlaps(text, extraInfoStr?): TextSplit[]

Parameters​

• text: string

• extraInfoStr?: string

Returns​

TextSplit[]

Source​

packages/core/src/TextSplitter.ts:280