Ga naar hoofdinhoud

Class: SentenceSplitter

SentenceSplitter is our default text splitter that supports splitting into sentences, paragraphs, or fixed length chunks with overlap.

One of the advantages of SentenceSplitter is that even in the fixed length chunks it will try to keep sentences together.

Constructors

new SentenceSplitter()

new SentenceSplitter(options?): SentenceSplitter

Parameters

options?

options.chunkOverlap?: number

options.chunkSize?: number

options.chunkingTokenizerFn?

options.paragraphSeparator?: string

options.splitLongSentences?: boolean

options.tokenizer?: any

options.tokenizerDecoder?: any

Returns

SentenceSplitter

Source

packages/core/src/TextSplitter.ts:76

Properties

chunkOverlap

chunkOverlap: number

Source

packages/core/src/TextSplitter.ts:69


chunkSize

chunkSize: number

Source

packages/core/src/TextSplitter.ts:68


chunkingTokenizerFn()

private chunkingTokenizerFn: (text) => string[]

Parameters

text: string

Returns

string[]

Source

packages/core/src/TextSplitter.ts:73


paragraphSeparator

private paragraphSeparator: string

Source

packages/core/src/TextSplitter.ts:72


splitLongSentences

private splitLongSentences: boolean

Source

packages/core/src/TextSplitter.ts:74


tokenizer

private tokenizer: Tokenizer

Source

packages/core/src/TextSplitter.ts:71

Methods

combineTextSplits()

combineTextSplits(newSentenceSplits, effectiveChunkSize): TextSplit[]

Parameters

newSentenceSplits: SplitRep[]

effectiveChunkSize: number

Returns

TextSplit[]

Source

packages/core/src/TextSplitter.ts:214


getEffectiveChunkSize()

private getEffectiveChunkSize(extraInfoStr?): number

Parameters

extraInfoStr?: string

Returns

number

Source

packages/core/src/TextSplitter.ts:112


getParagraphSplits()

getParagraphSplits(text, effectiveChunkSize?): string[]

Parameters

text: string

effectiveChunkSize?: number

Returns

string[]

Source

packages/core/src/TextSplitter.ts:130


getSentenceSplits()

getSentenceSplits(text, effectiveChunkSize?): string[]

Parameters

text: string

effectiveChunkSize?: number

Returns

string[]

Source

packages/core/src/TextSplitter.ts:156


processSentenceSplits()

private processSentenceSplits(sentenceSplits, effectiveChunkSize): SplitRep[]

Splits sentences into chunks if necessary.

This isn't great behavior because it can split down the middle of a word or in non-English split down the middle of a Unicode codepoint so the splitting is turned off by default. If you need it, please set the splitLongSentences option to true.

Parameters

sentenceSplits: string[]

effectiveChunkSize: number

Returns

SplitRep[]

Source

packages/core/src/TextSplitter.ts:185


splitText()

splitText(text, extraInfoStr?): string[]

Parameters

text: string

extraInfoStr?: string

Returns

string[]

Source

packages/core/src/TextSplitter.ts:308


splitTextWithOverlaps()

splitTextWithOverlaps(text, extraInfoStr?): TextSplit[]

Parameters

text: string

extraInfoStr?: string

Returns

TextSplit[]

Source

packages/core/src/TextSplitter.ts:280