← 输入框|github.com|刷新
Skip to content

build: replace rockylinux with chainguard/wolfi as a base image - #423

Merged
christinestraub merged 14 commits into
mainfrom
OSS-12/wolfi-base-image
Jun 17, 2024
Merged

christinestraub merged 14 commits into
mainfrom
OSS-12/wolfi-base-image

Conversation

@christinestraub

@christinestraub christinestraub commented Jun 11, 2024 •

Copy link
Copy Markdown
Contributor

Summary

Updates the Dockerfile to use the Chainguard wolfi-base image to reduce CVEs. Also adds a step in the docker publish job that scans the images and checks for CVEs before publishing.

Testing

Run make docker-build and make docker-start-api, then try:

from unstructured.partition.api import partition_via_api

elements = partition_via_api(
    filename=filename,
    api_url="http://localhost:8000/general/v0/general",
    api_key="<API-KEY>",
    strategy="hi_res",
)

print("\n\n".join([str(el) for el in elements]))

@christinestraub
christinestraub marked this pull request as ready for review June 14, 2024 22:12

@MthwRobinson MthwRobinson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Main comment is I don't think we need the download packages step now since all of that gets installed in the base images. We can remove that from unstructured now as well.

Comment thread .github/workflows/ci.yml Outdated
- name: Test Dockerfile
run: |
source .venv/bin/activate
make docker-dl-packages

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we pull from base-images I don't think we'll need this anymore.

Comment thread .github/workflows/docker-publish.yml Outdated
# Clear some space (https://github.com/actions/runner-images/issues/2840)
sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/share/boost

make docker-dl-packages

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here on download packages.

Comment thread .gitignore Outdated
tmp*

# APK packages for the docker build
docker-packages/*

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here on download packages.

Comment thread CHANGELOG.md Outdated
@@ -1,3 +1,7 @@
## 0.0.71-dev0

* replace rockylinux with chainguard/wolfie as a base image for `amd64`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit: wolfi instead of wolfie.

Comment thread Makefile Outdated
docker-build:
PIP_VERSION=${PIP_VERSION} PIPELINE_FAMILY=${PIPELINE_FAMILY} PIPELINE_PACKAGE=${PIPELINE_PACKAGE} ./scripts/docker-build.sh

.PHONY: docker-dl-packages

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here on download packages

Comment thread scripts/docker-dl-packages.sh Outdated
@@ -0,0 +1,22 @@
#!/bin/bash

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here on download packages.

@christinestraub christinestraub changed the title build: replace rockylinux with chainguard/wolfie as a base image build: replace rockylinux with chainguard/wolfi as a base image Jun 17, 2024

@MthwRobinson MthwRobinson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@christinestraub
christinestraub merged commit 2bdd52a into main Jun 17, 2024
@christinestraub
christinestraub deleted the OSS-12/wolfi-base-image branch June 17, 2024 20:50
tjtanaa added a commit to EmbeddedLLM/unstructured-api-executable that referenced this pull request Jul 11, 2024
…indows (#2)

## PR Summary
1. Merged changes from upstream.
2. Update `unstructuredio_api.spec`.
3. Update `unstructuredio_api.py`.
4. Add additional setup dependencies to the `docs/Windows.md`. 

* build(deps): version bumps for maintenance (Unstructured-IO#424)

### Summary
Version bumps for regular maintenance and to address moderate CVEs from
security scans.
- bump `unstructured` to `0.14.6`
- bump `unstructured-inference` to `0.7.35`

* build: replace rockylinux with chainguard/wolfi as a base image (Unstructured-IO#423)

### Summary
Updates the Dockerfile to use the Chainguard wolfi-base image to reduce
CVEs. Also adds a step in the docker publish job that scans the images
and checks for CVEs before publishing.

### Testing
Run `make docker-build` and  `make docker-start-api`, then try:
```
from unstructured.partition.api import partition_via_api

elements = partition_via_api(
    filename=filename,
    api_url="http://localhost:8000/general/v0/general",
    api_key="<API-KEY>",
    strategy="hi_res",
)

print("\n\n".join([str(el) for el in elements]))
```

* fix: build and push workflow failing due to missing `-f` option `buildx build` command (Unstructured-IO#425)

I noticed that images on main branch are failing to build (and push) due
to missing `-f` parameter in `docker buildx build`. By default it
expects `Dockerfile` to exist, but we only have `Dockerfile-amd64` and
`Dockerfile-arm64`


![image](https://github.com/Unstructured-IO/unstructured-api/assets/64484917/4527165a-909e-498d-b0ee-8bba4b1a13e4)

---------

Co-authored-by: christinestraub <christinemstraub@gmail.com>

* fix: update SHA for the base images (both architectures) after `base-images` repo update (Unstructured-IO#427)

build and publish CI steps are failing, because the base images have
changed in quay (their SHAs)

![image](https://github.com/Unstructured-IO/unstructured-api/assets/64484917/fc4e9aac-0820-4c90-9ad9-68cc6d9aad03)


![image](https://github.com/Unstructured-IO/unstructured-api/assets/64484917/fafe2ca4-dab2-4610-a26b-a7a4d56723a5)

* fix: revert to rockylinux SHA that works (arm64) (Unstructured-IO#428)

unnecessary SHA update introduced in
Unstructured-IO#427 that needs
to be reverted

* fix: re-add `DOCKER_IMAGE` env var in `Test image` step (Unstructured-IO#429)

shell syntax error occurs in docker-publish.yml workflow

* fix: invalid env var setting in `docker-publish` workflow (Unstructured-IO#430)

bug introduced in previous PR causing build failure on main

* fix: `docker-publish` workflow failing on main due to inexisting `ARCH` env var (Unstructured-IO#431)

* build(deps): bump dependency versions (Unstructured-IO#434)

### Summary

Bumps dependency versions for the API. Closes Unstructured-IO#432.

* fix/Fix MS Office filetype errors and harden docker smoketest (Unstructured-IO#436)

# Changes
**Fix for docx and other office files returning `{"detail":"File type
None is not supported."}`**
After moving to the wolfi base image, the `mimetypes` lib no longer
knows about these file extensions. To avoid issues like this, let's add
an explicit mapping for all the file extensions we care about. I added a
`filetypes.py` and moved `get_validated_mimetype` over. When this file
is imported, we'll call `mimetypes.add_type` for all file extensions we
support.

**Update smoke test coverage**
This bug snuck past because we were already providing the mimetype in
the docker smoke test. I updated `test_happy_path` to test against the
container with and without passing `content_type`. I added some missing
filetypes, and sorted the test params by extension so we can see when
new types are missing.

# Testing
The new smoke test will verify that all filetypes are working. You can
also `make docker-build && make docker-start-api`, and test out the docx
in the sample docs dir. On `main`, this file will give you the error
above.
```
curl 'http://localhost:8000/general/v0/general' \
--form 'files=@"fake.docx"'
```

* merge main; validated format: xml, txv, csv, xml, json, html, docs, docx, ppt, pptx, xlsx, xls, pdf

* compilable setting

* update Windows markdown

* Disable debug mode in `unstructuredio_api.spec`

* Enable pdf test case in `test_app.py`

---------

Co-authored-by: Christine Straub <christinemstraub@gmail.com>
Co-authored-by: Michał Martyniak <64484917+micmarty-deepsense@users.noreply.github.com>
Co-authored-by: Matt Robinson <mrobinson@unstructured.io>
Co-authored-by: Austin Walker <austin@unstructured.io>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
agarctfi added a commit that referenced this pull request Aug 27, 2026
scripts/app-start.sh invokes GNU `timeout` with --preserve-status and
--foreground when MAX_LIFETIME_SECONDS is set. The Wolfi base only ships
BusyBox `timeout`, which rejects those flags, so the server never starts
and the container restart-loops. GNU coreutils rode in for free on the old
RockyLinux base and was lost in the RockyLinux -> Wolfi switch (PR #423).

Add coreutils to the image to restore GNU `timeout`. Verified against the
real app-start.sh: the server now starts and runs for the full lifetime.
grype shows adding coreutils introduces 0 new CVEs (6 Chainguard-maintained
packages, ~10 MB).

Reported in Pylon 2657. Fixes ENG-1428.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
agarctfi added a commit that referenced this pull request Sep 9, 2026
#586)

## Summary

`MAX_LIFETIME_SECONDS` is broken on the published `unstructured-api`
image and has been since the base image moved from RockyLinux to
Chainguard Wolfi (#423).

`scripts/app-start.sh` runs GNU `timeout` with `--preserve-status` and
`--foreground` when `MAX_LIFETIME_SECONDS` is set. The Wolfi base only
provides BusyBox `timeout`, which does not support those flags, so
BusyBox rejects the first one and exits non-zero. uvicorn never
launches, the script falls through to its shutdown echoes, and a
container with a restart policy loops.

GNU coreutils shipped for free as part of the old RockyLinux userland
and was lost in the distro swap; it has never been an explicit package
in the Dockerfile. This PR adds it back.

Reported in Pylon 2657. Fixes ENG-1428.

## Reproduction

Runs the real `scripts/app-start.sh` with a stub standing in for
uvicorn.

Current base (BusyBox `timeout`), server never starts:

```
$ docker run --rm -e MAX_LIFETIME_SECONDS=3 -v "$PWD:/work" \
    --entrypoint sh cgr.dev/chainguard/wolfi-base:latest \
    -c 'apk add --no-cache bash >/dev/null 2>&1; export PATH=/work/bin:$PATH; cd /work; bash app-start.sh'
Server's lifetime set to 3 seconds.
timeout: unrecognized option '--preserve-status'
BusyBox v1.38.0 multi-call binary.
Usage: timeout [-s SIG] [-k KILL_SECS] SECS PROG ARGS
Server was shutdown
Reached timeout of 3 seconds
```

The stub "server started" line never prints.

With this fix (base + coreutils), server launches and runs for the full
lifetime:

```
$ docker run --rm -e MAX_LIFETIME_SECONDS=3 -v "$PWD:/work" \
    --entrypoint sh cgr.dev/chainguard/wolfi-base:latest \
    -c 'apk add --no-cache bash coreutils >/dev/null 2>&1; export PATH=/work/bin:$PATH; cd /work; bash app-start.sh'
Server's lifetime set to 3 seconds.
[stub-uvicorn] server started, args: prepline_general.api.app:app --log-config logger_config.yaml --host 0.0.0.0 --port 8000 --workers 1
Server was shutdown
Reached timeout of 3 seconds
```

## CVE impact

`apk add coreutils` was scanned with grype (DB 27 August 2026) on
`cgr.dev/chainguard/wolfi-base:latest`:

| Image | apk packages | CVEs |
| --- | --- | --- |
| base as-is | 15 | 0 |
| base + coreutils | 21 | 0 |

It pulls in 6 Chainguard-maintained packages (coreutils, libacl1,
libattr1, libpcre2-8-0, libselinux, libsepol), all with no known
vulnerabilities. Image size grows about 10 MB. Adding coreutils does not
reintroduce the RockyLinux CVE surface that #423 was shedding.

## Notes

- The guard in `app-start.sh` (`command -v timeout`) is also broken
independent of the base image: BusyBox `timeout` satisfies the check, so
the intended `gtimeout` fallback never runs. This PR fixes the reported
bug by making GNU `timeout` present; hardening the script to detect
BusyBox vs GNU would make it robust regardless of base and can be a
follow-up.
- The `core-product` mirror of this Dockerfile (base
`cgr.dev/unstructured.io/python-fips:3.12-dev`) is also Wolfi-based and
lacks coreutils, so it needs the same change.

## Test plan

- [x] Reproduced the failure with the real `app-start.sh` on the current
base
- [x] Verified the server starts and runs for the full lifetime with
coreutils
- [x] Confirmed 0 new CVEs via grype

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants