TextChunker emits a trailing chunk that is fully contained in the previous chunk #7063

Expected Behavior

Once a chunk already covers the last word of the text, TextChunker should stop. Every emitted chunk should add at least one word that the previous chunk did not have.

Current Behavior

The sliding window in TextChunker._chunk_by_words keeps advancing after a chunk has reached the end of the text. If the leftover window still has at least min_chunk_size words, it is emitted as another chunk, even though all of its words are already in the previous chunk.

With the default ChunkingConfig (chunk_size=100, chunk_overlap=20, min_chunk_size=20):

• a 100-word document gives two chunks: words 0-99 and words 80-99
• a 180-word document gives three chunks: words 0-99, 80-179 and 160-179

TextChunker is the default chunker of DocEmbedder, so the extra chunk is embedded and written to the online store as its own row, and retrieval can return it as a duplicate of the previous chunk.

Steps to reproduce

from feast import ChunkingConfig, TextChunker

text = " ".join(f"w{i}" for i in range(100))
chunks = TextChunker(ChunkingConfig(max_chunk_chars=None)).load_parse_and_chunk(
    source=text, source_id="doc", source_column="text"
)
for c in chunks:
    words = c["text"].split()
    print(c["chunk_id"], words[0], words[-1], len(words))

Output on master (e304a84):

doc_0 w0 w99 100
doc_1 w80 w99 20

doc_1 is fully contained in doc_0.

Specifications

• Version: master (e304a84)
• Platform: macOS, Python 3.12
• Subsystem: sdk/python/feast/chunker.py (TextChunker._chunk_by_words)

Possible Solution

In the loop in TextChunker._chunk_by_words, break once i + chunk_size >= len(words), right after the current chunk is appended. Outputs for texts that do not hit this boundary stay the same. I have this fix with a unit test locally and can open a PR.

Disclosure: found, reproduced and the draft fix tested locally with the help of an AI coding agent (Claude Code); reviewed before filing.