Repository navigation
[manager] Support for executing Gen-2 pipelines. - #6978
Conversation
| /// whichever test runs first then enables it for the whole binary, whatever the order. | ||
| #[cfg(test)] | ||
| pub(crate) fn enable_test_unstable_features() { | ||
| if unstable_features().is_none() { |
There was a problem hiding this comment.
Check-then-act on a OnceLock whose setter is UNSTABLE_FEATURES.set(...).expect("UNSTABLE_FEATURES already initialized"). Before this PR config.rs was the only caller, so no race; now pipeline_automata::test::crucible_multihost_refused_before_provisioning is a second caller in the same lib test binary, and cargo runs tests on parallel threads. Two tests can both observe None and the loser panics.
I ran the two tests together 25× without hitting it (the window is a few instructions), but it's a real flaky-panic source.
| if unstable_features().is_none() { | |
| let _ = UNSTABLE_FEATURES.set(HashSet::from_iter(["runtime_version", "testing"])); |
(or make platform_enable_unstable tolerant of a repeat set, or use get_or_init).
| let stdout_sink = if runtime_selector.is_crucible() { | ||
| create_new_file(&output_jit_file_path).await? | ||
| } else { | ||
| output_stdout_file | ||
| }; |
There was a problem hiding this comment.
Capturing the IR by redirecting the JVM's stdout is fragile and costs the diagnostics file. Only --crates is incompatible with --jit; CompilerMain.getOutputStream() honours -o otherwise (CompilerMain.java:135-147, :240-256), so .arg("-o").arg(&output_jit_file_path) writes the IR to the file directly and leaves stdout.log for real stdout.
As written, any stray byte on stdout (a JVM warning, an -XX message, an Xlint note) lands inside jit.json and the failure surfaces as json_deserialization_error("jit.json from SQL compiler") — which names the symptom, not the cause — while stdout.log is left empty.
| let (main_rust, stubs) = if runtime_selector.is_crucible() { | ||
| // Crucible skips Rust codegen, so there is no generated Rust to package. | ||
| (String::new(), String::new()) | ||
| } else { |
There was a problem hiding this comment.
Crucible silently drops UDFs. A pipeline with udf_rust / udf_toml set plus runtime_version: "crucible" reaches Success with main_rust = "" and udf_stubs = "", and the rust/crates tree is never even created (l. 749). The user gets a running pipeline whose UDFs do nothing, with no message anywhere.
Better to refuse at the top of perform_sql_compilation: "crucible does not support Rust UDFs; remove udf_rust/udf_toml or select a different runtime version." Same class of gap as the multihost refusal added in commit 2.
| let program_ir = self.dataflow.as_ref().map(|d| ProgramIr { | ||
| mir: d.mir.clone(), | ||
| program_schema: self.schema.clone(), | ||
| circuit_ir: self.circuit_ir.clone(), | ||
| }); |
There was a problem hiding this comment.
circuit_ir only reaches the pipeline when dataflow is Some — it rides inside the ProgramIr that this map builds, and local_runner copies it straight into deployment_config.program_ir. So the crucible engine's whole reason for existing is smuggled in as a passenger of an unrelated, optional field.
Today it works because the crucible command line still passes --dataflow, but nothing enforces that pairing: drop --dataflow for crucible as an optimisation (dataflow is several MiB and crucible does not need it) and crucible pipelines start with an empty circuit, with no error to point at the cause. Worth either lifting circuit_ir out of ProgramIr, or asserting dataflow.is_some() when circuit_ir.is_some().
| // Under --jit the index value drops the index-key columns, so key ++ value holds each | ||
| // column once (matching the deduped source); otherwise the value is the whole row. | ||
| boolean dedupKeys = this.compiler.options.ioOptions.interpreterJson; | ||
| List<Integer> valueColumnIndexes = new ArrayList<>(); | ||
| List<ViewColumnMetadata> valueColumns = new ArrayList<>(); | ||
| if (dedupKeys) { | ||
| for (int c = 0; c < struct.fields.size(); c++) { | ||
| if (!keyColumnIndexes.contains(c)) { | ||
| valueColumnIndexes.add(c); | ||
| valueColumns.add(view.metadata.columns.get(c)); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| DBSPExpression valueTuple = dedupKeys | ||
| ? new DBSPTupleExpression(Linq.map( | ||
| valueColumnIndexes, c -> var.deref().field(c).applyCloneIfNeeded()), false) |
There was a problem hiding this comment.
DedupKeysTests covers only DBSPSourceMapOperator; nothing pins this CREATE INDEX branch, which is the one that changes a user-visible output shape. I checked it by hand against the built jar:
CREATE TABLE t(a INT, id INT NOT NULL, b VARCHAR);
CREATE VIEW v AS SELECT * FROM t;
CREATE INDEX vi ON v(b, id);--jit → MapIndex key = 2 fields, value = 1 (a); without --jit → value = all 3. So the same CREATE INDEX yields a different index payload on crucible than on Rust. That is presumably the intent, but it deserves a test in DedupKeysTests alongside the source-map cases, and the divergence is worth stating somewhere a user will find it.
Two cases the test should cover: an index key given out of row order (as above — valueColumnIndexes is ascending while keyColumnIndexes follows the index order), and an index over every column (empty value tuple).
| throw new UnimplementedException( | ||
| "LATENESS on the primary-key table " + viewOrTable.singleQuote() + | ||
| " is not yet supported with --jit", | ||
| operator.getRelNode()); |
There was a problem hiding this comment.
Verified this fires as intended (exit=1, error_type: "Not yet implemented"), but the message names a flag the user never typed. Someone who set runtime_version: "crucible" in a program config sees:
LATENESS on the primary-key table 't' is not yet supported with --jit
and has no way to connect --jit to anything they wrote. Name the runtime instead — e.g. "… is not yet supported by the crucible runtime; remove the LATENESS annotation or select a different runtime version" — so the message says what to change.
| if configured.is_absolute() || configured.components().count() > 1 { | ||
| if configured.is_file() { | ||
| return Ok(configured.to_path_buf()); | ||
| } | ||
| } else if let Some(paths) = path_env { | ||
| for dir in std::env::split_paths(&paths) { | ||
| let candidate = dir.join(configured); | ||
| if candidate.is_file() { | ||
| return Ok(candidate); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
is_file() is weaker than "the OS would spawn this": a path that exists but lacks the execute bit passes the check and then fails with exactly the opaque process-spawn error this commit set out to replace (EACCES from Command::new). Since CRUCIBLE_BINARY is operator-supplied, a chmod-forgotten binary is the likely way this goes wrong.
Checking PermissionsExt::mode() & 0o111 != 0 on the candidate (both branches) closes the gap.
| /// Delivers the crucible program info artifact so the runner can fetch its circuit | ||
| /// IR. Crucible builds no binary, so the artifact is named by its own integrity | ||
| /// checksum; the runner rebuilds the same name. Returns that checksum. The SQL | ||
| /// compiler completes a crucible program without Rust compilation. | ||
| async fn deliver_crucible_program_info( | ||
| common_config: &CommonConfig, | ||
| config: &CompilerConfig, | ||
| pipeline_id: PipelineId, | ||
| program_version: Version, | ||
| program_info_value: &serde_json::Value, | ||
| ) -> Result<String, String> { | ||
| let program_info = validate_program_info(program_info_value) | ||
| .map_err(|error| format!("crucible program info is not valid: {error}"))? | ||
| .to_pipeline_config_program_info(); | ||
| let program_info_str = serde_json::to_string(&program_info) | ||
| .map_err(|e| format!("failed to serialize crucible program info: {e}"))?; | ||
| let program_info_integrity_checksum = checksum_buffer(program_info_str.as_bytes()) | ||
| .await | ||
| .map_err(|e| format!("failed to checksum crucible program info: {e:?}"))?; | ||
|
|
||
| // The compiler's HTTP server serves program info artifacts from this directory, | ||
| // so deliver here for the runner's program_info_url to resolve. | ||
| let pipeline_binaries_dir = config | ||
| .working_dir() | ||
| .join("rust-compilation") | ||
| .join("pipeline-binaries"); | ||
| create_dir_if_not_exists(&pipeline_binaries_dir) | ||
| .await | ||
| .map_err(|e| format!("failed to create pipeline-binaries directory: {e:?}"))?; | ||
|
|
||
| let program_info_metadata = FileUploadMetadata { | ||
| pipeline_id, | ||
| program_version, | ||
| source_checksum: program_info_integrity_checksum.clone(), | ||
| integrity_checksum: program_info_integrity_checksum.clone(), | ||
| }; | ||
| deliver_program_info( | ||
| common_config, | ||
| &FileDeliveryMode::from_config(config), | ||
| &program_info_str, | ||
| &program_info_metadata, | ||
| &pipeline_binaries_dir, | ||
| ) | ||
| .await | ||
| .map_err(|e| format!("failed to deliver crucible program info: {e:?}"))?; | ||
| Ok(program_info_integrity_checksum) | ||
| } | ||
|
|
There was a problem hiding this comment.
The crucible artifact survives GC only by accident of a fallback. cleanup_pipeline_binaries builds valid_program_info_filenames from (program_binary_source_checksum, program_info_integrity_checksum); crucible leaves the binary source checksum NULL, so it drops into the format!("program_info_{pipeline_id}_v{program_version}_") branch whose comment says it is there for the in-flight CompilingRust upload window (rust_compiler.rs:2056-2075).
It works today, but the retention of a running crucible pipeline's only artifact now depends on a prefix rule written for a transient state. Tighten that fallback — e.g. to actually scope it to CompilingRust — and crucible pipelines lose their program info on the next cleanup pass and cannot be re-provisioned. Either populate program_binary_source_checksum with the program-info checksum for crucible (it is already used as the source checksum in the artifact name and in the automaton's URLs), or add a decide_pipeline_binary_cleanup unit test pinning the (None, None, Some(checksum)) row so the coupling is not silently broken.
|
Reviewed against What I ran
I went past the arity assertions and checked the interleave actually lands columns in the right slots — Cases the tests don't cover
Docs: |
mihaibudiu
left a comment
There was a problem hiding this comment.
Approving the compiler part, but there are comments.
I would prefer not entangling the --jit flag with the circuit shape.
We should add a new @hidden compiler flag --crucible to drive this.
--jit still controls emitting the file, but not the structure of the circuit.
| valueColumns.add(view.metadata.columns.get(c)); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
I would prefer an else here filling valueColumns and homogeneous treatment afterwards.
| final DBSPTypeIndexedZSet indexedOutputType = (multisetInput != null) ? | ||
| ExpandIndexedInputs.getIndexedType(multisetInput) : null; | ||
| if (indexedOutputType != null && this.compiler.options.ioOptions.interpreterJson) { | ||
| // --jit dedups the primary-key columns out of the source value, but the waterline |
There was a problem hiding this comment.
not clear to me whether this will work when there is no lateness on a key column, maybe that should be legal?
| public void postorder(DBSPSourceMultisetOperator node) { | ||
| DBSPTypeIndexedZSet ix = getIndexedType(node); | ||
| // Under --jit the value drops the primary-key columns (the indexed z-set holds | ||
| // each column once); otherwise the value is the whole row, as before. |
| /** Rebuild the whole row from a deduped {@code (key, value)} pair: field {@code i} | ||
| * comes from the key when column {@code i} is a primary-key column, else from the | ||
| * value, each side addressed by its own compacted slot. */ | ||
| private DBSPClosureExpression interleaveKeyValue( |
There was a problem hiding this comment.
keyColumnFields could be a Set.
Can delete ", each side ... slot." from the comment above
| int keyField = this.keyColumnFields.indexOf(field.fieldNo); | ||
| if (keyField < 0) { | ||
| this.map(field, w.field(1).deref().field(field.fieldNo)); | ||
| // A non-key column. When the value dropped the key columns (--jit), |
There was a problem hiding this comment.
most of this comment should be inside the if below
397b2db to
17db101
Compare
Run crucible pipelines through the same pipeline manager as Rust-compiled ones, selected by `runtime_version: "crucible"` in the program config. A crucible program needs no Rust compilation: the SQL compiler captures the circuit IR and the runner launches the crucible engine to execute it. Doesn't bundle the actual crucible runtime yet. As a result, the integration tests in `python/tests/crucible` are disabled by default: they skip unless `FELDERA_TEST_CRUCIBLE` is set, and no CI job collects the directory. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Crucible does not support multihost pipelines yet. The provisioning gate now refuses a pipeline that has runtime_config.hosts > 1 and the crucible runtime, keeping it Stopped with a clear error. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
…is missing. The crucible engine is not always installed. The local runner now resolves the crucible executable (CRUCIBLE_BINARY, or `crucible` on PATH) before launching and refuses to provision with a clear CrucibleBinaryNotFound error when it is absent, rather than letting the failure surface as an opaque process-spawn error. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Under `--jit` a primary-key table's indexed source drops the key columns from the value, so key ++ value holds each column once, as expected by crucible, instead of duplicating the key. The non-jit Rust backend path stays unchanged, gated on `interpreterJson`. LATENESS on a primary-key table is not deduped yet, so under `--jit` it fails with a clear 'not yet implemented' error rather than emitting an indexed z-set whose value duplicates the key. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The engine is named Gen-2, so the manager calls it that. `runtime_version` takes `"gen2"` where it took `"crucible"`, and the code and comments say "Gen-2 engine" throughout. Renamed on the wire and in the operator's environment: | was | is | |----------------------------------------|-------------------------------------| | `"runtime_version": "crucible"` | `"runtime_version": "gen2"` | | `CRUCIBLE_BINARY` | `GEN2_BINARY` | | error code `CrucibleBinaryNotFound` | `Gen2BinaryNotFound` | | error code `AutomatonCrucibleMultihostUnsupported` | `AutomatonGen2MultihostUnsupported` | Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Splits the two things --jit meant. It emitted the circuit IR *and* dropped the primary-key columns from an indexed source's value; the dedup is what the Gen-2 engine needs, and asking for the IR should not change how the IR is built. | flag | emits circuit IR | dedups keys | |---|---|---| | `--jit` | yes | no (was: yes) | | `--gen2` | yes, implied | yes | `--gen2` implies `--jit` through a derived `emitCircuitIr()` rather than by mutating `interpreterJson` during argument parsing: tests set the option fields directly and never run the parser, so an implication applied only at parse time would silently not hold for them. `DedupKeysTests` pins both halves of the split, including that --jit alone leaves the whole row in the value. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
…--gen2 The compiler half of the flag split is the preceding commit: the dedup the Gen-2 engine needs now lives behind `--gen2`, so that is what the manager asks for. `--jit` alone would emit an IR whose values still carry the key columns. Review comments, in order: - The circuit IR no longer travels to the compiler through stdout. Only `--crates` conflicts with emitting the IR, so `-o` names the file directly; `stdout.log` goes back to holding stdout. A JVM warning on stdout used to land inside jit.json and surface as a deserialization error naming the symptom rather than the cause, while the diagnostics file was left empty. - A Gen-2 program with `udf_rust`/`udf_toml` set reached Success with the UDFs silently absent: the Rust half that builds them never runs. Refused up front now, as a program error naming both fields. `perform_sql_compilation` takes the two fields to make the check; the other three call sites have no UDFs. - `circuit_ir` was only delivered when `dataflow` was `Some`, riding inside the `ProgramIr` that field builds. Dropping `--dataflow` for Gen-2, which the engine ignores and which is large, would have started pipelines with no circuit and nothing to point at the cause. Either payload now builds the `ProgramIr`; the engine reads only `circuit_ir` and `program_schema`, so an empty `mir` costs nothing. - `enable_test_unstable_features` did check-then-act on a `OnceLock` whose setter panics on a second set. Cargo runs tests on parallel threads, so two callers could both observe an empty lock and the loser would panic. Installs the set with a discarded `set` instead. - Resolving the engine executable accepted a file without the execute bit, which then failed at spawn with exactly the opaque `EACCES` that resolving up front exists to avoid. Since the path is operator-supplied, a forgotten `chmod +x` is the likely way this goes wrong. - A running Gen-2 pipeline's only artifact was retained by accident. Having no binary source checksum, it missed the exact-filename branch and fell into the checksum-less prefix that exists for the transient `CompilingRust` upload window; scoping that fallback more tightly would have deleted it. Cleanup now asks whether the program is Gen-2 and rebuilds the exact name, so the two are no longer coupled. `list_pipeline_programs_across_all_tenants` reports the flag, derived from the parsed program config rather than a SQL comparison against the literal `gen2`, so what selects the engine stays defined in one place; its five-tuple becomes `PipelineProgramArtifacts`, which six call sites read. The filename lists move into functions so a test can pin the retention rule. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
…ness The database-backed tests never ran on a developer machine, failing in two ways that looked unrelated and had one cause. `setup_pg` connects through `rustls`, which panics when no process-wide crypto provider is installed. Most test modules call `ensure_default_crypto_provider` themselves, but the database harness did not, so whether a database test passed depended on some other test in the same binary having installed the provider first. The panic then fires *after* the embedded server has started, and a panicking test never stops it. Leaked servers accumulate across runs -- 155 were live on the machine this was diagnosed on -- and once they exhaust the host's shared memory every later `pg_ctl` fails with "could not start server". That reads as a port or resource problem, which is why the cause stayed hidden. Installing the provider in `setup_pg` fixes both faces: the tests connect, and they leave no servers behind. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The model's `get_pipeline_by_id_for_runner` decided readiness by comparing the requested platform version to the pipeline's for exact string equality, while the implementation asks `is_supported_runtime`: a pipeline compiled for a different platform version may still be runnable, and that function is the one place deciding it. Any request naming a different platform version therefore made the model report `Monitoring` where the implementation reported `Complete`, so `db_impl_behaves_like_model` failed whenever its generator produced such a request. The model now asks the same question. `runner_descriptor_uses_runtime_compatibility_not_string_equality` pins it deterministically, where the property test only reaches the case by chance. It drives the model and the implementation through the same steps and compares them: asserting on the implementation alone would pass either way, since the divergence was in the model. The divergence was invisible until the previous commit let the database tests run at all. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The implementation carries `deployment_id` through the transition to `Stopping` (`set_deployment_resources_status_stopping` passes `current.deployment_id`) and clears it at `Stopped`: while the deployment is being torn down, the runner still needs to know which deployment it is tearing down. The model cleared it on entering `Stopping`, so every pipeline observed after a stop request diverged from the implementation on that field, and `db_impl_behaves_like_model` failed whenever its generator stopped a provisioned pipeline and then read it back. The model's two `Stopped` transitions already clear it and are unchanged. Pre-existing, and independent of the artifact retention fix; it surfaced from the same seed once the database tests could run. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Artifact cleanup keeps the files its list of live programs names and removes every other one, with no retention grace. The list came from programs in `Success` or `CompilingRust`, which covers the window a legacy program has: it delivers its binary and program info while the row still says `CompilingRust` and carries no checksums, so the checksum-less prefix keeps them. A Gen-2 program has the same window one status earlier and it was not covered. `deliver_gen2_program_info` writes the file while the row still says `CompilingSql`; only afterwards does `transit_program_status_to_success_no_binary` record the checksum. A janitor pass inside that window found no row for the program at all, so its program info -- the only artifact it has, and the only copy of the circuit IR the engine needs -- was removed. Nothing heals that: the row then reads `Success` with a checksum, so no recompilation is triggered, and provisioning fails fetching an artifact that no longer exists until someone edits the program to bump its version. Including `CompilingSql` closes it symmetrically: such a row has no program-info checksum yet, so it falls to the same relaxed prefix that covers `CompilingRust`, and the prefix is relaxed about the checksum, not the version. Two tests, one per half of the mechanism: that the row reaches the janitor at all (`artifact_retention_covers_both_compiling_statuses`, which also pins that a `Pending` program holds nothing), and that the delivered filename survives a pass while the checksum is missing (`a_gen2_program_info_delivered_before_success_is_retained`). Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Recompiling a Gen-2 pipeline could delete the artifact it had just produced. The program ends up `Success` with its program info needed to run the program missing. Editing a program bumps its version, returns it to `Pending`, and clears `program_info` and both binary checksums, but left `program_info_integrity_checksum` holding the previous version's value. Artifact cleanup builds the filename to keep out of the row's checksums, so while the program recompiled it protected `program_info_<id>_vN+1_sc_<vN>_ic_<vN>.json`, a name that is never written, and collected the file actually delivered for N+1. Only a Gen-2 program is exposed: its artifact is named by the program-info checksum on both halves, so a stale value alone is enough to compute a wrong name. A legacy program is named partly by its binary source checksum, which the edit does clear, so it falls to the prefix and keeps everything for the version. Fixed at both ends, because either alone leaves the rule fragile: - The edit clears the info checksum, as it already clears the binary ones, so the column never describes an artifact of a different version. - Cleanup builds a filename from the checksums only for a `Success` program, and otherwise keeps every artifact of that version by prefix. What survives no longer depends on each writer of the row remembering to clear a column: an unsettled row keeps too much, the safe direction, whatever it happens to hold. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
f8008c9 to
48982a0
Compare
Signed-off-by: feldera-bot <feldera-bot@feldera.com>
Execute crucible pipelines by setting
"runtime_version": "gen2".Doesn't ship the actual crucible runtime, which will follow later.
@mihaibudiu , can you review compiler changes (the last commit)?
Describe Manual Test Plan
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes