fix(nlp): match single-letter bullets in the enumerated bullet pattern - #4636

Summary

ENUMERATED_BULLETS_RE cannot match a single-letter bullet, so lettered lists come out as one ListItem instead of one per item.

unstructured/nlp/patterns.py:111 compiled

ENUMERATED_BULLETS_RE = re.compile(r"(?:(?:\d{1,3}|[a-z][A-Z])\.?){1,3}")

[a-z][A-Z] is two single-character classes placed next to each other, so it matches exactly two characters (aB) and never a single a or A. The two ordered-bullet helpers in this repo both accept a single letter as a marker:

• unstructured/cleaners/core.py:53 clean_ordered_bullets — docstring says "accounting numeric and alphanumeric types", with the example a.b This is a very important point -> This is a very important point.
• unstructured/cleaners/extract.py:104 extract_ordered_bullets — docstring shows a.1 This is a very important point -> ("a", "1", None).

The splitter regex used by the list path does not, so the marker stays in the text:

from unstructured.partition.common.common import layout_list_to_list_items

[e.text for e in layout_list_to_list_items("a. foo\nb. bar", None, None, None, None)]
# ['a. foo\nb. bar']      expected ['foo', 'bar']

[e.text for e in layout_list_to_list_items("1. foo\n2. bar", None, None, None, None)]
# ['foo', 'bar']          unchanged

It is user visible through partition_image and partition_pdf(strategy="ocr_only"): ocr_data_to_elements -> normalize_layout_element -> layout_list_to_list_items (unstructured/partition/common/common.py:137).

Change

Two files change; the numeric branch and every existing behaviour stay byte for byte.

unstructured/nlp/patterns.py gains a lettered branch next to the untouched numeric one:

ENUMERATED_BULLETS_RE = re.compile(
    r"(?:(?:\d{1,3})\.?){1,3}"
    rf"|(?:(?:(?<=\n)|\A){NON_NEWLINE_WHITESPACE}[a-zA-Z]\."
    rf"(?=[^\S\n\r]|$)(?![^\S\n\r]+[a-zA-Z]\.))",
)

Each restriction is load-bearing:

• The line anchor: an unanchored [a-zA-Z] makes every letter a separator and "1. foo\n2. bar" splits into [].
• The literal dot: \d{1,3}\.? keeps the dot optional for numbers, but "a" without a dot is a word, not a marker.
• The lookahead (?=[^\S\n\r]|$) requires same-line whitespace or the end of the text after the dot.
• The guard (?![^\S\n\r]+[a-zA-Z]\.) rejects a second letter and dot on the same line. Without it "e. g. spaced example" becomes ['g. spaced example'] and "i. e. the letter" becomes ['e. the letter']. It uses [^\S\n\r]+ rather than a single space because extracted text writes abbreviations with repeated spaces or tabs as often as with one space, and it must not reach across a line break.

unstructured/partition/common/common.py also has to split mixed blocks. A block can hold enumerated markers and bullet glyphs at the same time:

layout_list_to_list_items("a. heading\n\u2022 first\n\u2022 second", None, None, None, None)
# ['heading', 'first', 'second']

Before, the enumerated split produced two segments and the bullet glyphs stayed glued to the second one, because the glyph split only ran when the enumerated split found nothing. The list path now splits every remaining segment on the glyph pattern as well, so both orders of the two marker families produce one item per item, and blocks that use a single family are unaffected.

Behaviour on the existing inputs is unchanged: "1. foo\n2. bar", "1.1 ...", "1.1.1 Deep point", "Page 1 of 2", "1 foo 2 bar", "a cat sat on a mat.", "See section A. Appendix", "Fig. 3 shows the trend", "Note. This is a note.", "b) alt paren", "e. g. spaced example", "i. e. the letter" all behave exactly as before.

Test

test_unstructured/nlp/test_patterns.py is new and covers the regex, the converter, mixed marker families, and the abbreviation cases that must not split.

python -m pytest test_unstructured/nlp/test_patterns.py -q

pristine (main 910d9f41): 12 failed, 16 passed
patched:                  28 passed

ruff check reports All checks passed! and ruff format --check reports 3 files already formatted for the two source files and the new test.

Fixes #4635