Repository navigation
build: replace rockylinux with chainguard/wolfi as a base image - #423
Merged
Merged
Conversation
christinestraub
marked this pull request as ready for review
June 14, 2024 22:12
MthwRobinson
suggested changes
Jun 17, 2024
MthwRobinson
left a comment
Contributor
There was a problem hiding this comment.
Looks good! Main comment is I don't think we need the download packages step now since all of that gets installed in the base images. We can remove that from unstructured now as well.
| - name: Test Dockerfile | ||
| run: | | ||
| source .venv/bin/activate | ||
| make docker-dl-packages |
Contributor
There was a problem hiding this comment.
If we pull from base-images I don't think we'll need this anymore.
| # Clear some space (https://github.com/actions/runner-images/issues/2840) | ||
| sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/share/boost | ||
|
|
||
| make docker-dl-packages |
Contributor
There was a problem hiding this comment.
Same here on download packages.
| tmp* | ||
|
|
||
| # APK packages for the docker build | ||
| docker-packages/* |
Contributor
There was a problem hiding this comment.
Same here on download packages.
| @@ -1,3 +1,7 @@ | |||
| ## 0.0.71-dev0 | |||
|
|
|||
| * replace rockylinux with chainguard/wolfie as a base image for `amd64` | |||
Contributor
There was a problem hiding this comment.
Small nit: wolfi instead of wolfie.
| docker-build: | ||
| PIP_VERSION=${PIP_VERSION} PIPELINE_FAMILY=${PIPELINE_FAMILY} PIPELINE_PACKAGE=${PIPELINE_PACKAGE} ./scripts/docker-build.sh | ||
|
|
||
| .PHONY: docker-dl-packages |
Contributor
There was a problem hiding this comment.
Same here on download packages
| @@ -0,0 +1,22 @@ | |||
| #!/bin/bash | |||
Contributor
There was a problem hiding this comment.
Same here on download packages.
tjtanaa
added a commit
to EmbeddedLLM/unstructured-api-executable
that referenced
this pull request
Jul 11, 2024
…indows (#2) ## PR Summary 1. Merged changes from upstream. 2. Update `unstructuredio_api.spec`. 3. Update `unstructuredio_api.py`. 4. Add additional setup dependencies to the `docs/Windows.md`. * build(deps): version bumps for maintenance (Unstructured-IO#424) ### Summary Version bumps for regular maintenance and to address moderate CVEs from security scans. - bump `unstructured` to `0.14.6` - bump `unstructured-inference` to `0.7.35` * build: replace rockylinux with chainguard/wolfi as a base image (Unstructured-IO#423) ### Summary Updates the Dockerfile to use the Chainguard wolfi-base image to reduce CVEs. Also adds a step in the docker publish job that scans the images and checks for CVEs before publishing. ### Testing Run `make docker-build` and `make docker-start-api`, then try: ``` from unstructured.partition.api import partition_via_api elements = partition_via_api( filename=filename, api_url="http://localhost:8000/general/v0/general", api_key="<API-KEY>", strategy="hi_res", ) print("\n\n".join([str(el) for el in elements])) ``` * fix: build and push workflow failing due to missing `-f` option `buildx build` command (Unstructured-IO#425) I noticed that images on main branch are failing to build (and push) due to missing `-f` parameter in `docker buildx build`. By default it expects `Dockerfile` to exist, but we only have `Dockerfile-amd64` and `Dockerfile-arm64`  --------- Co-authored-by: christinestraub <christinemstraub@gmail.com> * fix: update SHA for the base images (both architectures) after `base-images` repo update (Unstructured-IO#427) build and publish CI steps are failing, because the base images have changed in quay (their SHAs)   * fix: revert to rockylinux SHA that works (arm64) (Unstructured-IO#428) unnecessary SHA update introduced in Unstructured-IO#427 that needs to be reverted * fix: re-add `DOCKER_IMAGE` env var in `Test image` step (Unstructured-IO#429) shell syntax error occurs in docker-publish.yml workflow * fix: invalid env var setting in `docker-publish` workflow (Unstructured-IO#430) bug introduced in previous PR causing build failure on main * fix: `docker-publish` workflow failing on main due to inexisting `ARCH` env var (Unstructured-IO#431) * build(deps): bump dependency versions (Unstructured-IO#434) ### Summary Bumps dependency versions for the API. Closes Unstructured-IO#432. * fix/Fix MS Office filetype errors and harden docker smoketest (Unstructured-IO#436) # Changes **Fix for docx and other office files returning `{"detail":"File type None is not supported."}`** After moving to the wolfi base image, the `mimetypes` lib no longer knows about these file extensions. To avoid issues like this, let's add an explicit mapping for all the file extensions we care about. I added a `filetypes.py` and moved `get_validated_mimetype` over. When this file is imported, we'll call `mimetypes.add_type` for all file extensions we support. **Update smoke test coverage** This bug snuck past because we were already providing the mimetype in the docker smoke test. I updated `test_happy_path` to test against the container with and without passing `content_type`. I added some missing filetypes, and sorted the test params by extension so we can see when new types are missing. # Testing The new smoke test will verify that all filetypes are working. You can also `make docker-build && make docker-start-api`, and test out the docx in the sample docs dir. On `main`, this file will give you the error above. ``` curl 'http://localhost:8000/general/v0/general' \ --form 'files=@"fake.docx"' ``` * merge main; validated format: xml, txv, csv, xml, json, html, docs, docx, ppt, pptx, xlsx, xls, pdf * compilable setting * update Windows markdown * Disable debug mode in `unstructuredio_api.spec` * Enable pdf test case in `test_app.py` --------- Co-authored-by: Christine Straub <christinemstraub@gmail.com> Co-authored-by: Michał Martyniak <64484917+micmarty-deepsense@users.noreply.github.com> Co-authored-by: Matt Robinson <mrobinson@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io> Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
3 tasks done
agarctfi
added a commit
that referenced
this pull request
Aug 27, 2026
scripts/app-start.sh invokes GNU `timeout` with --preserve-status and --foreground when MAX_LIFETIME_SECONDS is set. The Wolfi base only ships BusyBox `timeout`, which rejects those flags, so the server never starts and the container restart-loops. GNU coreutils rode in for free on the old RockyLinux base and was lost in the RockyLinux -> Wolfi switch (PR #423). Add coreutils to the image to restore GNU `timeout`. Verified against the real app-start.sh: the server now starts and runs for the full lifetime. grype shows adding coreutils introduces 0 new CVEs (6 Chainguard-maintained packages, ~10 MB). Reported in Pylon 2657. Fixes ENG-1428. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
agarctfi
added a commit
that referenced
this pull request
Sep 9, 2026
#586) ## Summary `MAX_LIFETIME_SECONDS` is broken on the published `unstructured-api` image and has been since the base image moved from RockyLinux to Chainguard Wolfi (#423). `scripts/app-start.sh` runs GNU `timeout` with `--preserve-status` and `--foreground` when `MAX_LIFETIME_SECONDS` is set. The Wolfi base only provides BusyBox `timeout`, which does not support those flags, so BusyBox rejects the first one and exits non-zero. uvicorn never launches, the script falls through to its shutdown echoes, and a container with a restart policy loops. GNU coreutils shipped for free as part of the old RockyLinux userland and was lost in the distro swap; it has never been an explicit package in the Dockerfile. This PR adds it back. Reported in Pylon 2657. Fixes ENG-1428. ## Reproduction Runs the real `scripts/app-start.sh` with a stub standing in for uvicorn. Current base (BusyBox `timeout`), server never starts: ``` $ docker run --rm -e MAX_LIFETIME_SECONDS=3 -v "$PWD:/work" \ --entrypoint sh cgr.dev/chainguard/wolfi-base:latest \ -c 'apk add --no-cache bash >/dev/null 2>&1; export PATH=/work/bin:$PATH; cd /work; bash app-start.sh' Server's lifetime set to 3 seconds. timeout: unrecognized option '--preserve-status' BusyBox v1.38.0 multi-call binary. Usage: timeout [-s SIG] [-k KILL_SECS] SECS PROG ARGS Server was shutdown Reached timeout of 3 seconds ``` The stub "server started" line never prints. With this fix (base + coreutils), server launches and runs for the full lifetime: ``` $ docker run --rm -e MAX_LIFETIME_SECONDS=3 -v "$PWD:/work" \ --entrypoint sh cgr.dev/chainguard/wolfi-base:latest \ -c 'apk add --no-cache bash coreutils >/dev/null 2>&1; export PATH=/work/bin:$PATH; cd /work; bash app-start.sh' Server's lifetime set to 3 seconds. [stub-uvicorn] server started, args: prepline_general.api.app:app --log-config logger_config.yaml --host 0.0.0.0 --port 8000 --workers 1 Server was shutdown Reached timeout of 3 seconds ``` ## CVE impact `apk add coreutils` was scanned with grype (DB 27 August 2026) on `cgr.dev/chainguard/wolfi-base:latest`: | Image | apk packages | CVEs | | --- | --- | --- | | base as-is | 15 | 0 | | base + coreutils | 21 | 0 | It pulls in 6 Chainguard-maintained packages (coreutils, libacl1, libattr1, libpcre2-8-0, libselinux, libsepol), all with no known vulnerabilities. Image size grows about 10 MB. Adding coreutils does not reintroduce the RockyLinux CVE surface that #423 was shedding. ## Notes - The guard in `app-start.sh` (`command -v timeout`) is also broken independent of the base image: BusyBox `timeout` satisfies the check, so the intended `gtimeout` fallback never runs. This PR fixes the reported bug by making GNU `timeout` present; hardening the script to detect BusyBox vs GNU would make it robust regardless of base and can be a follow-up. - The `core-product` mirror of this Dockerfile (base `cgr.dev/unstructured.io/python-fips:3.12-dev`) is also Wolfi-based and lacks coreutils, so it needs the same change. ## Test plan - [x] Reproduced the failure with the real `app-start.sh` on the current base - [x] Verified the server starts and runs for the full lifetime with coreutils - [x] Confirmed 0 new CVEs via grype Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the Dockerfile to use the Chainguard wolfi-base image to reduce CVEs. Also adds a step in the docker publish job that scans the images and checks for CVEs before publishing.
Testing
Run
make docker-buildandmake docker-start-api, then try: