Repository navigation
[pipeline-manager] Serve prebuilt pipeline binaries from the compiler - #7885
Conversation
Some environments cannot run cargo when a pipeline starts. A deterministic simulation test runs on a single slow CPU with no crate registry, yet still needs to start pipelines for a known set of SQL programs. The compiler server gains --prebuilt-binaries-directory. Before running cargo, it looks there for a binary of the same program and, if one exists, delivers it instead. A binary is valid for any pipeline with the same SQL, but the source checksum cannot find it: it hashes the encoded generated code, which carries file modification times, and the generated crate names embed the pipeline id. The lookup key is therefore the source checksum with the generated code hashed by file paths and contents, under the crate names of the nil pipeline. --precompile fills the directory: each --prebuild-sql-program file is compiled as the nil pipeline, like the dummy program, and its binary is copied there under its lookup key. The compiler now also passes CARGO_BUILD_TARGET to cargo and reads artifacts from target/<triple>/. With an explicit target, RUSTFLAGS apply only to target artifacts, so coverage instrumentation flags no longer break the linking of build scripts and proc macros. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
… key find_prebuilt_binary receives the prebuilt binary key, which differs from the source checksum its parameter and test were named after. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Update the test-only PostgreSQL connector to the current TLS API and use the Rust 2024-compatible random helper. This lets prebuilt cache unit tests compile without installing an embedded database.
Blind namespace replacement can change SQL literals and conflate distinct programs. Normalize generated Rust identifiers and Cargo crate references structurally, including workspace members, while preserving literal data. Reject duplicate normalized archive paths instead of overwriting them. Cover namespace independence, literal collisions, Cargo references and malformed generated sources. The four prebuilt cache tests pass, and real SQL compiler output normalizes identically across pipeline IDs.
|
|
||
| let prebuilt_binary_key = match config.prebuilt_binaries_directory { | ||
| None => None, | ||
| Some(_) => Some(prebuilt_binary_key( |
There was a problem hiding this comment.
pitfall (medium): the cache lookup is optional, but the ? makes any failure here fatal. If proc_macro2 can't tokenize a generated file, if an archive entry isn't UTF-8 (read_to_string), or if two archive paths normalize to the same path, the whole compilation fails with a SystemError about the "prebuilt cache". Without the cache, cargo would just have run. Suggest logging a warning and treating it as a miss (None), plus a test with an untokenizable main_rust that checks the compilation still goes through cargo.
| #[arg(long)] | ||
| pub precompile: bool, | ||
|
|
||
| /// Directory of pipeline binaries built ahead of time, each named by the |
There was a problem hiding this comment.
bug (low-medium, user-visible --help): this doc string is out of date after the second commit. Files are named by the prebuilt binary key, which is the source checksum with the generated code normalized. The plain source checksum never matches (the PR description says so too). Anyone who populates the directory by hand from this text will never get a hit. Please describe the key, or say that the directory should only be filled with --precompile --prebuild-sql-program. The same goes for "A Rust compilation whose source checksum names a file…" below.
| @@ -2922,6 +3057,151 @@ mod test { | |||
| } | |||
|
|
|||
| /// Tests the binary delivery mode configuration. | |||
There was a problem hiding this comment.
nit: the new tests were inserted under this doc line, so /// Tests the binary delivery mode configuration. now documents prebuilt_binary_key_depends_only_on_program. Please move it back down to binary_delivery_mode_from_config.
|
What I ran: Gaps in the tests (missing, medium):
Pitfall (low-medium): forwarding No unsafe code, no CI workflow changes, nothing breaking for users. The new flags are internal and test-oriented, so I'm not asking for docs beyond fixing the |
The compiler server can deliver a pipeline binary built ahead of time instead of running
cargo. Environments that cannot compile when a pipeline starts, such as the Antithesis deterministic simulation (one slow CPU, no crate registry), still run a fixed set of SQL programs.--prebuilt-binaries-directory <dir><dir>and delivers it if present.--prebuild-sql-program <file>(repeatable, with--precompile)--precompilealso compiles each file as the nil pipeline and stores its binary in the directory under its lookup key.CARGO_BUILD_TARGETand reads artifacts fromtarget/<triple>/, soRUSTFLAGSapply to target artifacts only and coverage instrumentation no longer breaks build scripts and proc macros.The four commits come from the branch
antithesis-prebuilt-binaries, rebased on main with the lock file regenerated; authorship is kept.Verification
git range-diffagainst the original branchtomlcrate versions inCargo.lockcargo test -p pipeline-manager --no-default-features --features feldera-enterprise --lib prebuilt_binarycargo check -p pipeline-manager(default features)pre-commit run --files <changed files>