← 输入框|github.com|刷新
Skip to content

[pipeline-manager] Serve prebuilt pipeline binaries from the compiler - #7885

Merged
bestouff merged 4 commits into
mainfrom
gz/prebuilt-pipeline-binaries
Nov 7, 2026
Merged

bestouff merged 4 commits into
mainfrom
gz/prebuilt-pipeline-binaries

Conversation

@gz

@gz gz commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

The compiler server can deliver a pipeline binary built ahead of time instead of running cargo. Environments that cannot compile when a pipeline starts, such as the Antithesis deterministic simulation (one slow CPU, no crate registry), still run a fixed set of SQL programs.

Flag Effect
--prebuilt-binaries-directory <dir> Before a Rust compilation, the compiler looks for a binary of the same program in <dir> and delivers it if present.
--prebuild-sql-program <file> (repeatable, with --precompile) --precompile also compiles each file as the nil pipeline and stores its binary in the directory under its lookup key.
Topic Behaviour
Lookup key The source checksum, with the generated code hashed by file paths and contents and under the crate names of the nil pipeline. The plain source checksum cannot find a prebuilt binary: it hashes encoded generated code that carries modification times and pipeline-specific crate names.
Key normalization Rust identifiers and Cargo crate references are normalized structurally; SQL literals are never rewritten, so two programs that differ only in a literal keep distinct keys. Duplicate normalized archive paths are refused.
Cargo target The compiler passes CARGO_BUILD_TARGET and reads artifacts from target/<triple>/, so RUSTFLAGS apply to target artifacts only and coverage instrumentation no longer breaks build scripts and proc macros.
Default Off. Without the flag the compiler behaves as before.

The four commits come from the branch antithesis-prebuilt-binaries, rebased on main with the lock file regenerated; authorship is kept.

Verification

Check Result
git range-diff against the original branch Identical apart from toml crate versions in Cargo.lock
cargo test -p pipeline-manager --no-default-features --features feldera-enterprise --lib prebuilt_binary 4 passed
cargo check -p pipeline-manager (default features) ok
pre-commit run --files <changed files> all hooks passed (clippy, fmt, machete, OpenAPI, docs)

gz and others added 4 commits November 6, 2026 17:23
Some environments cannot run cargo when a pipeline starts. A
deterministic simulation test runs on a single slow CPU with no crate
registry, yet still needs to start pipelines for a known set of SQL
programs.

The compiler server gains --prebuilt-binaries-directory. Before running
cargo, it looks there for a binary of the same program and, if one
exists, delivers it instead. A binary is valid for any pipeline with the
same SQL, but the source checksum cannot find it: it hashes the encoded
generated code, which carries file modification times, and the
generated crate names embed the pipeline id. The lookup key is
therefore the source checksum with the generated code hashed by file
paths and contents, under the crate names of the nil pipeline.

--precompile fills the directory: each --prebuild-sql-program file is
compiled as the nil pipeline, like the dummy program, and its binary is
copied there under its lookup key.

The compiler now also passes CARGO_BUILD_TARGET to cargo and reads
artifacts from target/<triple>/. With an explicit target, RUSTFLAGS
apply only to target artifacts, so coverage instrumentation flags no
longer break the linking of build scripts and proc macros.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
… key

find_prebuilt_binary receives the prebuilt binary key, which differs
from the source checksum its parameter and test were named after.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Update the test-only PostgreSQL connector to the current TLS API and
use the Rust 2024-compatible random helper. This lets prebuilt cache
unit tests compile without installing an embedded database.
Blind namespace replacement can change SQL literals and conflate distinct
programs. Normalize generated Rust identifiers and Cargo crate references
structurally, including workspace members, while preserving literal data.
Reject duplicate normalized archive paths instead of overwriting them.

Cover namespace independence, literal collisions, Cargo references and
malformed generated sources. The four prebuilt cache tests pass, and real
SQL compiler output normalizes identically across pipeline IDs.
@gz
gz requested a review from bestouff November 7, 2026 01:31

let prebuilt_binary_key = match config.prebuilt_binaries_directory {
None => None,
Some(_) => Some(prebuilt_binary_key(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): the cache lookup is optional, but the ? makes any failure here fatal. If proc_macro2 can't tokenize a generated file, if an archive entry isn't UTF-8 (read_to_string), or if two archive paths normalize to the same path, the whole compilation fails with a SystemError about the "prebuilt cache". Without the cache, cargo would just have run. Suggest logging a warning and treating it as a miss (None), plus a test with an untokenizable main_rust that checks the compilation still goes through cargo.

#[arg(long)]
pub precompile: bool,

/// Directory of pipeline binaries built ahead of time, each named by the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (low-medium, user-visible --help): this doc string is out of date after the second commit. Files are named by the prebuilt binary key, which is the source checksum with the generated code normalized. The plain source checksum never matches (the PR description says so too). Anyone who populates the directory by hand from this text will never get a hit. Please describe the key, or say that the directory should only be filled with --precompile --prebuild-sql-program. The same goes for "A Rust compilation whose source checksum names a file…" below.

@@ -2922,6 +3057,151 @@ mod test {
}

/// Tests the binary delivery mode configuration.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the new tests were inserted under this doc line, so /// Tests the binary delivery mode configuration. now documents prebuilt_binary_key_depends_only_on_program. Please move it back down to binary_delivery_mode_from_config.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran: cargo test --locked -p pipeline-manager --lib prebuilt (debug, default features). All 4 tests pass. I also checked that toml 1.2 is MIT/Apache-2.0 and pulls in no crypto. It is pinned in the crate, not in the workspace deps (fine).

Gaps in the tests (missing, medium):

Case Covered?
Key is stable across pipeline IDs and mtimes; literals are preserved ✅
find_prebuilt_binary hit, miss, and directory entry ✅
perform_rust_compilation hit path: delivers the prebuilt binary and program info, skips cargo, returns rustc_result ❌
prebuild_program: copies from cargo_profile_dir of the nil pipeline, skips when the file already exists ❌
Key computation fails (bad tokens, non-UTF-8, duplicate path) → should fall back to cargo (see inline) ❌
CARGO_BUILD_TARGET set → cargo_profile_dir and cleanup use target/<triple>/<profile> ❌

Pitfall (low-medium): forwarding CARGO_BUILD_TARGET is not behind the new flag. It changes where every compiler reads artifacts from whenever that variable is set in its environment, so "Default: off" applies only to the binary cache. Please mention it as a behaviour change, or gate it.

No unsafe code, no CI workflow changes, nothing breaking for users. The new flags are internal and test-oriented, so I'm not asking for docs beyond fixing the --help text flagged inline.

@bestouff
bestouff added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit a9104c1 Nov 7, 2026
3 checks passed
@bestouff
bestouff deleted the gz/prebuilt-pipeline-binaries branch November 7, 2026 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants