← 输入框|github.com|刷新
Skip to content

Tags: Unstructured-IO/unstructured

Tags

0.27.36

Toggle 0.27.36's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
chore: release 0.27.36 (#4631)

0.27.35

Toggle 0.27.35's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
chore: release 0.27.35 (#4606)

Release PR: changes `unstructured/__version__.py` from `0.27.35-dev` to
`0.27.35`, per CONTRIBUTING.md release-day step 1. No changelog edit
(CHANGELOG.md is frozen).

This is the first release since the post-release dev-bump automation
landed (#4590). After the `0.27.35` GitHub Release is published, the
workflow should open a `chore: start 0.27.36-dev after 0.27.35 release`
PR automatically.

0.27.34

Toggle 0.27.34's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
chore: release 0.27.34 (#4582)

Drops the `-dev6` suffix so 0.27.34 can be tagged and published. There
are no code changes.

Since 0.27.33, this release contains:

- Free decode buffers sooner when restoring original elements (#4524)
- Free image buffers sooner during figure extraction (#4523)
- Honor `encoding=` in `partition_json` and `partition_ndjson` (#4483)
- Read a PowerPoint line break as a newline (#4571)
- Drop the empty entry of an email address group (#4576)
- Read CSV and TSV cells as the file holds them (#4574)
- Classify text as an email address only when all of it is one (#4577)

After merge, create the `0.27.34` GitHub release from the merge commit.
That publishes to PyPI.

<!-- This is an auto-generated description by cubic. -->
<a
href="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4582?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4582?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-dark.svg%2522%253E%253Csource" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-light.svg%2522%253E%253Cimg" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

0.27.33

Toggle 0.27.33's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(deps): bump certifi to 2026.10.19 (#4564)

## Why

Eight eMudhra intermediate CA private keys were confirmed compromised
([article](https://thehackernews.com/2026/10/eight-emudhra-ca-keys-compromised.html)).
certifi 2026.10.19 drops the **emSign ECC Root CA - G3** root.
`requests` and `httpx` trust certifi's bundle and do not check
revocation, so the older bundle accepts certificates chaining to that
root.

## What

- `uv.lock`: `certifi` upgraded to 2026.10.19 (`uv lock
--upgrade-package certifi`; no other packages changed).
- Version bumped to `0.27.33` with a CHANGELOG entry.

Tests: lock consistency checked locally; full suites left to CI.

Also requires `certifi>=2026.10.19` in `[project].dependencies`, so pip
installs get the new bundle, not only the lockfile.

**Release note:** this cuts `0.27.33` from main's `0.27.33-dev0`, so it
also releases the chunking fix already on main (folded into the
`0.27.33` CHANGELOG section).

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated description by cubic. -->
<a
href="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4564?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4564?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-dark.svg%2522%253E%253Csource" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-light.svg%2522%253E%253Cimg" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

0.27.32

Toggle 0.27.32's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
perf(xlsx): reduce dense-worksheet partitioning memory (#4563)

## What

`partition_xlsx()` with subtable detection (the default) needs
substantially less memory on dense worksheets, and the saving grows with
sheet size. Output is unchanged.

On synthetic dense sheets (a ten-column sheet split into three dense
subtables, half integers, half short Unicode strings), peak RSS in a
fresh process drops by roughly 40–60% depending on size, with comparable
runtime. Working memory per populated cell drops to roughly a third.
Serialized element JSON is byte-identical before and after.

## Why

Two whole-sheet intermediates dominated memory on dense sheets:

1. **Subtable detection built a NetworkX graph** with a node per
populated cell and an edge per neighbor pair. It now uses an iterative
four-neighbor flood fill over populated cells (the approach from #4357),
with the same component admission order, cell membership, row-overlap
merging, table order and bounding boxes.
2. **Table HTML was rendered for the whole table at once** (pandas
`to_html` string, its parsed DOM and a full-table token list). Rows are
now formatted by pandas' formatter for the whole column, so
float/date/string formatting is unchanged, and each row is rendered and
parsed on its own before the next.

`networkx` is no longer needed at runtime by the XLSX partitioner, so it
moves from the `xlsx` extra to the test dependencies (it's still used by
other extras that depend on it).

## Behavior

- Same elements, text, `text_as_html` and metadata; exact-output tests
cover mixed types, NaN/empty cells, wide floats that switch to
scientific notation in another row, dates and HTML escaping.
- The renderer uses pandas' private formatter classes, as the existing
reader already does for `OpenpyxlReader`. Verified on pandas 2.1.4
(renderer-parity comparisons), 2.2.3 and 2.3.3.
- The worksheet cell limit, empty-row compaction, empty-sheet handling
and XLS padding handling are unchanged.

## Validation

- XLSX and dense-memory suites pass. New tests compare component
detection and rendered HTML against the previous implementation on
generated sheets.
- Ruff check/format pass.

## Release

Sets the release to **0.27.32** (0.27.31 was taken by #4538, which
merged first). It merges after #4419, #4516, #4517 and #4562, and its
CHANGELOG collects their entries and this one into a single 0.27.32
section. #4524 and #4523 are not included and will go into a later
release.

Supersedes #4357.


<!-- This is an auto-generated description by cubic. -->
<a
href="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4563?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4563?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-dark.svg%2522%253E%253Csource" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-light.svg%2522%253E%253Cimg" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

0.27.30

Toggle 0.27.30's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(email): keep non-ASCII sender and recipient names readable (#4543)

## What

`partition_email()` now keeps non-ASCII sender and recipient names, and
non-ASCII addresses, as written in the `sent_from`, `sent_to`,
`cc_recipient` and `bcc_recipient` metadata. Closes #4542.

```
From: José García <jose@ejemplo.es>      (RFC 2047-encoded on the wire, as mail clients write it)
To:   张伟 <zhang@example.cn>

Before: sent_from=['=?utf-8?b?Sm9zw6kgR2FyY8OtYQ==?= <jose@ejemplo.es>']  sent_to=['=?utf-8?b?5byg5Lyf?= <zhang@example.cn>']
After:  sent_from=['José García <jose@ejemplo.es>']                     sent_to=['张伟 <zhang@example.cn>']

From: José <josé@ejemplo.es>               (RFC 6532 address)
Before: UnicodeEncodeError, no elements
After:  sent_from=['José <josé@ejemplo.es>']
```

## Why

The headers are already decoded (`email.policy.default`), but the four
address properties formatted them with `email.utils.formataddr()`. It
writes a non-ASCII display name as an RFC 2047 encoded-word, and it
raises `UnicodeEncodeError` on a non-ASCII address. They now use
`_format_address()`, which quotes and escapes the display name with the
same rules as `formataddr()` but leaves non-ASCII text as is. For ASCII
names and addresses the result is unchanged: on 50,000 random ASCII
name/address pairs it matched `formataddr()` exactly.

## Validation

- `test_partition_email_keeps_non_ascii_display_names_readable` covers
From, To (Chinese and German names), a quoted Cc name with a comma, and
Bcc. `test_partition_email_accepts_non_ascii_addresses` covers an RFC
6532 From and To. Both fail on `main` (the encoded-word, and the
`UnicodeEncodeError`) and pass here.
- `test_unstructured/partition/test_email.py`: 75 passed, 1 skipped.
`test_msg.py` gives the same result as on `main`: one test fails locally
on both because the image and PDF extras aren't installed.
- `ruff check` and `ruff format --check` pass on the changed files.
- CHANGELOG entry under 0.27.27, with the version bumped.


<!-- This is an auto-generated description by cubic. -->
<a
href="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4543?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4543?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-dark.svg%2522%253E%253Csource" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-light.svg%2522%253E%253Cimg" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

0.27.29

Toggle 0.27.29's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(xlsx): read .xls files whose Workbook stream ends in padding (#4545)

## What

`partition_xlsx()` now reads `.xls` files whose `Workbook` stream ends
in a few bytes of padding, such as many files written by `xlwt`. Before,
these raised `struct.error: unpack requires a buffer of 4 bytes` from
the msoffcrypto encryption check. Closes #4544.

```
xlwt workbook, 3 rows (example-docs/xlwt-padded-workbook.xls)

Before: struct.error: unpack requires a buffer of 4 bytes
After:  [Table("Fruit Count Apple 3 Pear 7")]
```

## Why

msoffcrypto's `is_encrypted()` scans an XLS `Workbook` stream as 4-byte
record headers, looking for a `FilePass` record. `xlwt` pads the stream
with zeros to a 4096-byte boundary, and when fewer than 4 bytes of
padding are left, the scan raises (nolze/msoffcrypto-tool#83). A
`struct.error` from that check is now read as "not encrypted". This does
not let a password-protected `.xls` through: its `FilePass` record
directly follows the `BOF` record, so the scan finds it before any
padding. msoffcrypto's `rc4cryptoapi_password.xls` and
`xor_password_123456789012345.xls` samples are still rejected with "XLSX
file is password protected.", and its `plain.xls` sample still
partitions.

## Validation

- `test_partition_xlsx_reads_xls_whose_workbook_stream_ends_in_padding`
partitions a 5.6 KB `xlwt` fixture. It fails on `main` with the
`struct.error` and passes here.
- `test_unstructured/partition/test_xlsx.py`: 108 passed.
- `ruff check` and `ruff format --check` pass on the changed files.
- CHANGELOG entry under 0.27.27, with the version bumped. #4543 also
bumps to 0.27.27; I'll rebase whichever merges second.


<!-- This is an auto-generated description by cubic. -->
<a
href="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4545?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4545?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-dark.svg%2522%253E%253Csource" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-light.svg%2522%253E%253Cimg" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

0.27.28

Toggle 0.27.28's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(ocr): include underlying error when loading OCR agent fails (#4491)

### Summary

Fixes #3276.

When `OCRAgent.get_instance()` fails to import or load a requested OCR
agent module (e.g. missing dependencies or an `AttributeError`), it
previously raised a generic `RuntimeError` without the original
exception details or exception chaining (`from e`):

```python
raise RuntimeError(
    "Could not get the OCRAgent instance. Please check the OCR package and the "
    "OCR_AGENT environment variable."
)
```

As reported in #3276 and discussed with @scanny, this masks the
underlying error (such as `ImportError: No module named '...'`), making
troubleshooting difficult.

### Changes

1. In `unstructured/partition/utils/ocr_models/ocr_interface.py`,
include `{e}` in the raised `RuntimeError` message and chain the
original exception with `raise ... from e`.
2. In
`test_unstructured/partition/utils/ocr_models/test_ocr_interface.py`,
update `and_it_raises_when_the_requested_agent_cannot_be_loaded` to
verify that the underlying exception message is present in the raised
`RuntimeError` and that `__cause__` references the original exception.
3. Update `CHANGELOG.md` under `## 0.27.8-dev0` / `### Fixes`.

### Testing

- Ran targeted unit tests:
  ```bash
pytest
test_unstructured/partition/utils/ocr_models/test_ocr_interface.py -v
  ```
  (All 10 tests passed)
- Ran regression test suites:
  ```bash
pytest test_unstructured/cleaners/test_core.py
test_unstructured/partition/test_text.py
  ```
  (All 197 tests passed)
- Linting and formatting checks:
  ```bash
ruff check unstructured/partition/utils/ocr_models/ocr_interface.py
test_unstructured/partition/utils/ocr_models/test_ocr_interface.py
ruff format --check
unstructured/partition/utils/ocr_models/ocr_interface.py
test_unstructured/partition/utils/ocr_models/test_ocr_interface.py
  ```


<!-- This is an auto-generated description by cubic. -->
<a
href="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4491?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4491?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-dark.svg%2522%253E%253Csource" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-light.svg%2522%253E%253Cimg" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

0.27.27

Toggle 0.27.27's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix: preserve MathML content in HTML partitioning (#4478)

## Summary

This PR adds support for preserving inline MathML content in the v1 HTML
parser used by `partition_html()`.

Previously, `<math>` was not assigned a custom element class. As a
result, it fell back to `DefaultElement`, whose contents are
intentionally skipped. This could silently remove mathematical
expressions from otherwise valid extracted text.

For example, HTML such as:

```html
<p>
    The error rate is
    <math>
        <semantics>
            <annotation encoding="application/x-tex">
                10^{-4}
            </annotation>
        </semantics>
    </math>
    per token.
</p>
```

could be extracted as:

```text
The error rate is per token.
```

This PR introduces a `Math` phrasing element that preserves a textual
representation of the expression when one is available.

## Changes

* Adds a `Math` class derived from `Phrasing`.
* Registers `<math>` elements with the v1 HTML parser.
* Searches for recognized TeX/LaTeX annotations within the MathML
subtree.
* Supports the following annotation encodings:

  * `application/x-tex`
  * `tex`
  * `latex`
  * `application/x-latex`
* Falls back to the `<math>` element's `alttext` attribute when no
supported TeX annotation is available.
* Preserves text following the closing `</math>` element by emitting the
element tail using the existing phrasing behavior.

The parser prefers an explicit TeX representation because flattening
presentation MathML with `itertext()` can lose mathematical structure.
For example, a superscript expression such as `10^-4` could otherwise be
flattened into ambiguous plain text such as `10-4`.

## Why

The HTML parser's documented principle is that the browser rendering is
the document.

However, before this change, standards-based MathML content could be
rendered correctly by a browser while being completely removed during
HTML partitioning because `<math>` was treated as an unrecognized
element.

This is especially problematic because the failure is silent.
Surrounding paragraph text remains intact, so consumers of
`partition_html()` may receive syntactically valid but incomplete text.

Treating `<math>` as a specialized phrasing element allows mathematical
content to participate naturally in surrounding paragraph text without
requiring the parser to understand every individual Presentation MathML
element such as `<mrow>`, `<msup>`, `<mfrac>`, `<mi>`, or `<mn>`.

## Scope

This PR intentionally does not attempt to implement a complete
MathML-to-text converter.

It currently preserves mathematical content when a reliable textual
representation is directly available through:

1. a recognized TeX/LaTeX `<annotation>`, or
2. the `<math>` element's `alttext` attribute.

Other representations, such as Content MathML, arbitrary
`annotation-xml` formats, OpenMath, SVG, or unsupported custom
annotation encodings, are outside the scope of this change.

## Example

Input:

```html
<p>
    Complexity decreases from
    <math alttext="O(n^2)"></math>
    to
    <math>
        <semantics>
            <annotation encoding="application/x-tex">
                O(n^2/r)
            </annotation>
        </semantics>
    </math>.
</p>
```

Before:

```text
Complexity decreases from to .
```

After:

```text
Complexity decreases from O(n^2) to O(n^2/r).
```

## Testing

Tests have been added for MathML behavior including:

* extracting `application/x-tex` annotations,
* supported TeX/LaTeX encoding variants,
* falling back to `alttext`,
* preferring a supported TeX annotation when both annotation and
`alttext` are present,
* preserving surrounding and tail text,
* multiple inline `<math>` elements in one paragraph,
* ignoring unsupported annotation encodings,
* realistic nested Presentation MathML,
* interaction with existing phrasing/emphasis behavior,
* and regression coverage for ordinary phrasing elements.

Reviewer verification can also be performed with:

```python
from unstructured.partition.html import partition_html

html = r"""
<p>
    The loss difference is on the order of
    <math>
        <semantics>
            <annotation encoding="application/x-tex">
                10^{-4}
            </annotation>
        </semantics>
    </math>.
</p>
"""

elements = partition_html(text=html)

for element in elements:
    print(element.text)
```

Expected output:

```text
The loss difference is on the order of 10^{-4}.
```

<!-- This is an auto-generated description by cubic. -->
<a
href="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4478?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4478?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-dark.svg%2522%253E%253Csource" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-light.svg%2522%253E%253Cimg" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

0.27.25

Toggle 0.27.25's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
feat(image): partition WEBP images like PNG and JPEG (#4537)

## What

`.webp` images now partition like PNG and JPEG.

```
partition("photo.webp")
Before: UnsupportedFileFormatError: Partitioning is not supported for the FileType.UNK file type.
After:  [Title(...), NarrativeText(...), ...]   # each with metadata.filetype == "image/webp"
```

## Why

`FileType` had no WEBP member, so detection fell through to `UNK` and
`partition()` rejected every `.webp`. Pillow already decodes WEBP and
`partition_image()` already accepts it, so the change is one
`FileType.WEBP` member, which detection by content type, by libmagic or
`filetype`, and by extension all read. `webp` also joins the image types
that skip table inference by default, with PNG, JPEG and HEIC, and the
types runtime telemetry counts as images.

Two behaviours change beyond files named `.webp`:

- `partition_email()` and `partition_msg()` used to skip a WEBP
attachment with a warning. With the `image` extra installed they now
partition it; without the extra they still skip it with a warning. A
WEBP attachment that fails image processing, for example because
Tesseract is missing, now raises and fails the whole call, as a PNG or
JPEG attachment already does.
- A WEBP saved under another image extension is detected as `image/webp`
from its content, where its extension decided before.

`IMAGE_URL_PATTERN` stays as it is. Its only user is
`extract_image_urls_from_html()`, a cleaner that partitioning never
calls, so it doesn't decide whether a `.webp` partitions. Adding `webp`
to it is a separate change.

## Validation

- `detect_filetype()`: WEBP joins each table HEIC is in, reading the new
`example-docs/img/DA-1p.webp` (a WEBP copy of `DA-1p.jpg`). A WEBP
copied to `renamed.png` is detected as WEBP, where `main` says PNG.
- `test_auto_partition_webp_from_filename` partitions a lossless WEBP
copy of `layout-parser-paper-fast.jpg` through `partition()` under
`auto` and `ocr_only`, as the BMP test does with its own copy.
- `test_auto_partition_webp_skips_table_inference_by_default_like_png`
checks the `infer_table_structure` value `partition()` passes to the
image partitioner: true for BMP, false for HEIC, JPEG, PNG and WEBP.
Dropping `"webp"` from the default fails it.
- Two `partition_email()` tests attach `DA-1p.webp` to a message built
with `EmailMessage`. When the image dependency check fails, the
attachment is skipped with an `ImportError` warning from
`partition_image()`. With the image partitioner stubbed, the
attachment's elements carry `filetype` `image/webp`, `filename`
`DA-1p.webp` and `attached_to_filename`. `partition_msg()` follows the
same attachment rule but has no WEBP test of its own.
- The telemetry strategy test gains a `webp` row.

Against `main`'s source every new WEBP case fails. The table test's BMP,
HEIC, JPEG and PNG rows pass there, since they pin the existing
defaults.

CI ran all 19 new items and passed them. On Python 3.11, 3.12 and 3.13,
`test_unit` reports 4209 passed and 15 skipped, 19 items more than
`main`'s push run of the same base (4195 passed, 10 skipped).
Pull-request runs skip five tests that push runs execute.

The docs' open-source supported-file-types and partitioning-strategies
pages list `.heic` but not `.webp`; that update belongs in the docs
repository.


<!-- This is an auto-generated description by cubic. -->
<a
href="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4537?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4537?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-dark.svg%2522%253E%253Csource" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="/?u=https%3A%2F%2Fgithub.com%2F%253Ca href="/?u=https%3A%2F%2Fwww.cubic.dev%2Fbuttons%2Freview-in-cubic-light.svg%2522%253E%253Cimg" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="/?u=https%3A%2F%2Fgithub.com%2F%253Ca%2520href%3D"https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->