← 输入框|github.com|刷新
Skip to content

[manager] Support for executing Gen-2 pipelines. - #6978

Merged
ryzhyk merged 13 commits into
mainfrom
crucible-integration
Aug 28, 2026
Merged

ryzhyk merged 13 commits into
mainfrom
crucible-integration

Conversation

@ryzhyk

@ryzhyk ryzhyk commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Execute crucible pipelines by setting "runtime_version": "gen2".

Doesn't ship the actual crucible runtime, which will follow later.

@mihaibudiu , can you review compiler changes (the last commit)?

Describe Manual Test Plan

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

@ryzhyk
ryzhyk requested review from gz and mihaibudiu August 27, 2026 00:53
@ryzhyk ryzhyk added the Pipeline manager Pipeline manager (API, API server, runner, compiler server) label Aug 27, 2026
Comment thread crates/pipeline-manager/src/lib.rs Outdated
/// whichever test runs first then enables it for the whole binary, whatever the order.
#[cfg(test)]
pub(crate) fn enable_test_unstable_features() {
if unstable_features().is_none() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check-then-act on a OnceLock whose setter is UNSTABLE_FEATURES.set(...).expect("UNSTABLE_FEATURES already initialized"). Before this PR config.rs was the only caller, so no race; now pipeline_automata::test::crucible_multihost_refused_before_provisioning is a second caller in the same lib test binary, and cargo runs tests on parallel threads. Two tests can both observe None and the loser panics.

I ran the two tests together 25× without hitting it (the window is a few instructions), but it's a real flaky-panic source.

Suggested change
if unstable_features().is_none() {
let _ = UNSTABLE_FEATURES.set(HashSet::from_iter(["runtime_version", "testing"]));

(or make platform_enable_unstable tolerant of a repeat set, or use get_or_init).

Comment on lines +814 to +818
let stdout_sink = if runtime_selector.is_crucible() {
create_new_file(&output_jit_file_path).await?
} else {
output_stdout_file
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capturing the IR by redirecting the JVM's stdout is fragile and costs the diagnostics file. Only --crates is incompatible with --jit; CompilerMain.getOutputStream() honours -o otherwise (CompilerMain.java:135-147, :240-256), so .arg("-o").arg(&output_jit_file_path) writes the IR to the file directly and leaves stdout.log for real stdout.

As written, any stray byte on stdout (a JVM warning, an -XX message, an Xlint note) lands inside jit.json and the failure surfaces as json_deserialization_error("jit.json from SQL compiler") — which names the symptom, not the cause — while stdout.log is left empty.

Comment on lines +976 to +979
let (main_rust, stubs) = if runtime_selector.is_crucible() {
// Crucible skips Rust codegen, so there is no generated Rust to package.
(String::new(), String::new())
} else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Crucible silently drops UDFs. A pipeline with udf_rust / udf_toml set plus runtime_version: "crucible" reaches Success with main_rust = "" and udf_stubs = "", and the rust/crates tree is never even created (l. 749). The user gets a running pipeline whose UDFs do nothing, with no message anywhere.

Better to refuse at the top of perform_sql_compilation: "crucible does not support Rust UDFs; remove udf_rust/udf_toml or select a different runtime version." Same class of gap as the multihost refusal added in commit 2.

Comment on lines 719 to 723
let program_ir = self.dataflow.as_ref().map(|d| ProgramIr {
mir: d.mir.clone(),
program_schema: self.schema.clone(),
circuit_ir: self.circuit_ir.clone(),
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

circuit_ir only reaches the pipeline when dataflow is Some — it rides inside the ProgramIr that this map builds, and local_runner copies it straight into deployment_config.program_ir. So the crucible engine's whole reason for existing is smuggled in as a passenger of an unrelated, optional field.

Today it works because the crucible command line still passes --dataflow, but nothing enforces that pairing: drop --dataflow for crucible as an optimisation (dataflow is several MiB and crucible does not need it) and crucible pipelines start with an empty circuit, with no error to point at the cause. Worth either lifting circuit_ir out of ProgramIr, or asserting dataflow.is_some() when circuit_ir.is_some().

Comment on lines +3470 to +3486
// Under --jit the index value drops the index-key columns, so key ++ value holds each
// column once (matching the deduped source); otherwise the value is the whole row.
boolean dedupKeys = this.compiler.options.ioOptions.interpreterJson;
List<Integer> valueColumnIndexes = new ArrayList<>();
List<ViewColumnMetadata> valueColumns = new ArrayList<>();
if (dedupKeys) {
for (int c = 0; c < struct.fields.size(); c++) {
if (!keyColumnIndexes.contains(c)) {
valueColumnIndexes.add(c);
valueColumns.add(view.metadata.columns.get(c));
}
}
}

DBSPExpression valueTuple = dedupKeys
? new DBSPTupleExpression(Linq.map(
valueColumnIndexes, c -> var.deref().field(c).applyCloneIfNeeded()), false)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DedupKeysTests covers only DBSPSourceMapOperator; nothing pins this CREATE INDEX branch, which is the one that changes a user-visible output shape. I checked it by hand against the built jar:

CREATE TABLE t(a INT, id INT NOT NULL, b VARCHAR);
CREATE VIEW v AS SELECT * FROM t;
CREATE INDEX vi ON v(b, id);

--jit → MapIndex key = 2 fields, value = 1 (a); without --jit → value = all 3. So the same CREATE INDEX yields a different index payload on crucible than on Rust. That is presumably the intent, but it deserves a test in DedupKeysTests alongside the source-map cases, and the divergence is worth stating somewhere a user will find it.

Two cases the test should cover: an index key given out of row order (as above — valueColumnIndexes is ascending while keyColumnIndexes follows the index order), and an index over every column (empty value tuple).

Comment on lines +1632 to +1635
throw new UnimplementedException(
"LATENESS on the primary-key table " + viewOrTable.singleQuote() +
" is not yet supported with --jit",
operator.getRelNode());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified this fires as intended (exit=1, error_type: "Not yet implemented"), but the message names a flag the user never typed. Someone who set runtime_version: "crucible" in a program config sees:

LATENESS on the primary-key table 't' is not yet supported with --jit

and has no way to connect --jit to anything they wrote. Name the runtime instead — e.g. "… is not yet supported by the crucible runtime; remove the LATENESS annotation or select a different runtime version" — so the message says what to change.

Comment on lines +121 to +132
if configured.is_absolute() || configured.components().count() > 1 {
if configured.is_file() {
return Ok(configured.to_path_buf());
}
} else if let Some(paths) = path_env {
for dir in std::env::split_paths(&paths) {
let candidate = dir.join(configured);
if candidate.is_file() {
return Ok(candidate);
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is_file() is weaker than "the OS would spawn this": a path that exists but lacks the execute bit passes the check and then fails with exactly the opaque process-spawn error this commit set out to replace (EACCES from Command::new). Since CRUCIBLE_BINARY is operator-supplied, a chmod-forgotten binary is the likely way this goes wrong.

Checking PermissionsExt::mode() & 0o111 != 0 on the candidate (both branches) closes the gap.

Comment on lines +601 to +648
/// Delivers the crucible program info artifact so the runner can fetch its circuit
/// IR. Crucible builds no binary, so the artifact is named by its own integrity
/// checksum; the runner rebuilds the same name. Returns that checksum. The SQL
/// compiler completes a crucible program without Rust compilation.
async fn deliver_crucible_program_info(
common_config: &CommonConfig,
config: &CompilerConfig,
pipeline_id: PipelineId,
program_version: Version,
program_info_value: &serde_json::Value,
) -> Result<String, String> {
let program_info = validate_program_info(program_info_value)
.map_err(|error| format!("crucible program info is not valid: {error}"))?
.to_pipeline_config_program_info();
let program_info_str = serde_json::to_string(&program_info)
.map_err(|e| format!("failed to serialize crucible program info: {e}"))?;
let program_info_integrity_checksum = checksum_buffer(program_info_str.as_bytes())
.await
.map_err(|e| format!("failed to checksum crucible program info: {e:?}"))?;

// The compiler's HTTP server serves program info artifacts from this directory,
// so deliver here for the runner's program_info_url to resolve.
let pipeline_binaries_dir = config
.working_dir()
.join("rust-compilation")
.join("pipeline-binaries");
create_dir_if_not_exists(&pipeline_binaries_dir)
.await
.map_err(|e| format!("failed to create pipeline-binaries directory: {e:?}"))?;

let program_info_metadata = FileUploadMetadata {
pipeline_id,
program_version,
source_checksum: program_info_integrity_checksum.clone(),
integrity_checksum: program_info_integrity_checksum.clone(),
};
deliver_program_info(
common_config,
&FileDeliveryMode::from_config(config),
&program_info_str,
&program_info_metadata,
&pipeline_binaries_dir,
)
.await
.map_err(|e| format!("failed to deliver crucible program info: {e:?}"))?;
Ok(program_info_integrity_checksum)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The crucible artifact survives GC only by accident of a fallback. cleanup_pipeline_binaries builds valid_program_info_filenames from (program_binary_source_checksum, program_info_integrity_checksum); crucible leaves the binary source checksum NULL, so it drops into the format!("program_info_{pipeline_id}_v{program_version}_") branch whose comment says it is there for the in-flight CompilingRust upload window (rust_compiler.rs:2056-2075).

It works today, but the retention of a running crucible pipeline's only artifact now depends on a prefix rule written for a transient state. Tighten that fallback — e.g. to actually scope it to CompilingRust — and crucible pipelines lose their program info on the next cleanup pass and cannot be re-provisioned. Either populate program_binary_source_checksum with the program-info checksum for crucible (it is already used as the source checksum in the artifact name and in the automaton's URLs), or add a decide_pipeline_binary_cleanup unit test pinning the (None, None, Some(checksum)) row so the coupling is not silently broken.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Reviewed against feldera-pr-checks. Design is sound — routing crucible around the Rust queue via CompilingSql → Success with "none" as the binary checksum sentinel is a clean fit, and the two follow-up commits (multihost refusal, missing-executable error) close the two gaps I'd have asked for. Details are inline; the only gate I'd call failed is flaky tests (Hard Rule 4): enable_test_unstable_features is a check-then-act on a OnceLock whose setter .expect()s, and this PR adds the second caller in the same test binary.

What I ran

Check Result
cargo check --locked -p pipeline-manager --all-targets pass (1 warning, pre-existing in compiler/test.rs)
cargo test -p pipeline-manager --lib -- crucible resolve_ large_program_info artifacts_present 8 pass; crucible_multihost_refused_before_provisioning fails on a rustls CryptoProvider panic — reproduced identically on the pre-existing automata::test::starting, so environmental, not yours
./build.sh + mvn test -pl SQL-compiler -Dtest=DedupKeysTests 6/6 pass
pre-commit run --files <changed> all pass
Manual --jit runs against the built jar dedup verified semantically, not just by arity

I went past the arity assertions and checked the interleave actually lands columns in the right slots — t(a,b,id PK) with SELECT id,b,a gives [key0, val1, val0], and composite PRIMARY KEY(k1,k2) over t(k1,x,k2,y) with SELECT y,k2,x,k1 gives [val1, key1, val0, key0]. Both correct. Also confirmed LATENESS-on-PK is rejected under --jit and still compiles without it.

Cases the tests don't cover

  • compileCreateIndex — the CREATE INDEX half of the dedup has no test at all, and it's the half that changes a user-visible output shape (I measured key=2/value=1 under --jit vs value=3 without).
  • Crucible program-info delivery failure → CompilingSql → SystemError. New error path, no test.
  • Crucible artifacts vs cleanup_pipeline_binaries (inline: they survive only via a fallback prefix meant for a different state).
  • UDFs + crucible — currently silently dropped rather than refused.
  • python/tests/crucible is skip-gated and uncollected by any job (tests/{platform,runtime,workloads,unit,platform_rbac} only), so integration coverage is zero until the runtime ships. Fine for now; please wire it up in the same PR that lands the engine. When you do, the round-trip test's CREATE TABLE t(id BIGINT, s VARCHAR) is the only evidence the IR survives serialization — worth widening to a full SQL-type table.

Docs: runtime_version: "crucible" and CRUCIBLE_BINARY are both user-visible and the checkbox is unchecked. Soft call given the runtime_version unstable-feature gate keeps this off by default, but the env var in particular should be documented before the engine ships. Commit messages are good — subject/body/why all present.

@mihaibudiu mihaibudiu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the compiler part, but there are comments.
I would prefer not entangling the --jit flag with the circuit shape.
We should add a new @hidden compiler flag --crucible to drive this.
--jit still controls emitting the file, but not the structure of the circuit.

valueColumns.add(view.metadata.columns.get(c));
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer an else here filling valueColumns and homogeneous treatment afterwards.

final DBSPTypeIndexedZSet indexedOutputType = (multisetInput != null) ?
ExpandIndexedInputs.getIndexedType(multisetInput) : null;
if (indexedOutputType != null && this.compiler.options.ioOptions.interpreterJson) {
// --jit dedups the primary-key columns out of the source value, but the waterline

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not clear to me whether this will work when there is no lateness on a key column, maybe that should be legal?

public void postorder(DBSPSourceMultisetOperator node) {
DBSPTypeIndexedZSet ix = getIndexedType(node);
// Under --jit the value drops the primary-key columns (the indexed z-set holds
// each column once); otherwise the value is the whole row, as before.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no ", as before"

/** Rebuild the whole row from a deduped {@code (key, value)} pair: field {@code i}
* comes from the key when column {@code i} is a primary-key column, else from the
* value, each side addressed by its own compacted slot. */
private DBSPClosureExpression interleaveKeyValue(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keyColumnFields could be a Set.
Can delete ", each side ... slot." from the comment above

int keyField = this.keyColumnFields.indexOf(field.fieldNo);
if (keyField < 0) {
this.map(field, w.field(1).deref().field(field.fieldNo));
// A non-key column. When the value dropped the key columns (--jit),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most of this comment should be inside the if below

@ryzhyk ryzhyk changed the title [manager] Support for executing crucible pipelines. [manager] Support for executing Gen-2 pipelines. Aug 27, 2026
@ryzhyk
ryzhyk force-pushed the crucible-integration branch from 397b2db to 17db101 Compare August 27, 2026 06:58
ryzhyk added 12 commits August 27, 2026 20:04
Run crucible pipelines through the same pipeline manager as Rust-compiled ones,
selected by `runtime_version: "crucible"` in the program config. A crucible
program needs no Rust compilation: the SQL compiler captures the circuit IR and
the runner launches the crucible engine to execute it.

Doesn't bundle the actual crucible runtime yet.  As a result, the integration
tests in `python/tests/crucible` are disabled by default: they skip unless
`FELDERA_TEST_CRUCIBLE` is set, and no CI job collects the directory.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Crucible does not support multihost pipelines yet. The provisioning gate now
refuses a pipeline that has runtime_config.hosts > 1 and the crucible runtime,
keeping it Stopped with a clear error.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
…is missing.

The crucible engine is not always installed. The local runner now resolves the
crucible executable (CRUCIBLE_BINARY, or `crucible` on PATH) before launching and
refuses to provision with a clear CrucibleBinaryNotFound error when it is absent,
rather than letting the failure surface as an opaque process-spawn error.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Under `--jit` a primary-key table's indexed source drops the key columns from
the value, so key ++ value holds each column once, as expected by crucible,
instead of duplicating the key.  The non-jit Rust backend path stays unchanged,
gated on `interpreterJson`.

LATENESS on a primary-key table is not deduped yet, so under `--jit` it fails
with a clear 'not yet implemented' error rather than emitting an indexed z-set
whose value duplicates the key.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The engine is named Gen-2, so the manager calls it that. `runtime_version`
takes `"gen2"` where it took `"crucible"`, and the code and comments say
"Gen-2 engine" throughout.

Renamed on the wire and in the operator's environment:

| was                                    | is                                  |
|----------------------------------------|-------------------------------------|
| `"runtime_version": "crucible"`        | `"runtime_version": "gen2"`         |
| `CRUCIBLE_BINARY`                      | `GEN2_BINARY`                       |
| error code `CrucibleBinaryNotFound`    | `Gen2BinaryNotFound`                |
| error code `AutomatonCrucibleMultihostUnsupported` | `AutomatonGen2MultihostUnsupported` |

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Splits the two things --jit meant. It emitted the circuit IR *and* dropped the
primary-key columns from an indexed source's value; the dedup is what the Gen-2
engine needs, and asking for the IR should not change how the IR is built.

| flag | emits circuit IR | dedups keys |
|---|---|---|
| `--jit` | yes | no (was: yes) |
| `--gen2` | yes, implied | yes |

`--gen2` implies `--jit` through a derived `emitCircuitIr()` rather than by
mutating `interpreterJson` during argument parsing: tests set the option fields
directly and never run the parser, so an implication applied only at parse time
would silently not hold for them. `DedupKeysTests` pins both halves of the
split, including that --jit alone leaves the whole row in the value.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
…--gen2

The compiler half of the flag split is the preceding commit: the dedup the
Gen-2 engine needs now lives behind `--gen2`, so that is what the manager asks
for. `--jit` alone would emit an IR whose values still carry the key columns.

Review comments, in order:

- The circuit IR no longer travels to the compiler through stdout. Only
  `--crates` conflicts with emitting the IR, so `-o` names the file directly;
  `stdout.log` goes back to holding stdout. A JVM warning on stdout used to land
  inside jit.json and surface as a deserialization error naming the symptom
  rather than the cause, while the diagnostics file was left empty.

- A Gen-2 program with `udf_rust`/`udf_toml` set reached Success with the UDFs
  silently absent: the Rust half that builds them never runs. Refused up front
  now, as a program error naming both fields. `perform_sql_compilation` takes the
  two fields to make the check; the other three call sites have no UDFs.

- `circuit_ir` was only delivered when `dataflow` was `Some`, riding inside the
  `ProgramIr` that field builds. Dropping `--dataflow` for Gen-2, which the
  engine ignores and which is large, would have started pipelines with no
  circuit and nothing to point at the cause. Either payload now builds the
  `ProgramIr`; the engine reads only `circuit_ir` and `program_schema`, so an
  empty `mir` costs nothing.

- `enable_test_unstable_features` did check-then-act on a `OnceLock` whose setter
  panics on a second set. Cargo runs tests on parallel threads, so two callers
  could both observe an empty lock and the loser would panic. Installs the set
  with a discarded `set` instead.

- Resolving the engine executable accepted a file without the execute bit, which
  then failed at spawn with exactly the opaque `EACCES` that resolving up front
  exists to avoid. Since the path is operator-supplied, a forgotten `chmod +x` is
  the likely way this goes wrong.

- A running Gen-2 pipeline's only artifact was retained by accident. Having no
  binary source checksum, it missed the exact-filename branch and fell into the
  checksum-less prefix that exists for the transient `CompilingRust` upload
  window; scoping that fallback more tightly would have deleted it. Cleanup now
  asks whether the program is Gen-2 and rebuilds the exact name, so the two are
  no longer coupled. `list_pipeline_programs_across_all_tenants` reports the flag,
  derived from the parsed program config rather than a SQL comparison against the
  literal `gen2`, so what selects the engine stays defined in one place; its
  five-tuple becomes `PipelineProgramArtifacts`, which six call sites read. The
  filename lists move into functions so a test can pin the retention rule.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
…ness

The database-backed tests never ran on a developer machine, failing in two ways
that looked unrelated and had one cause.

`setup_pg` connects through `rustls`, which panics when no process-wide crypto
provider is installed. Most test modules call `ensure_default_crypto_provider`
themselves, but the database harness did not, so whether a database test passed
depended on some other test in the same binary having installed the provider
first.

The panic then fires *after* the embedded server has started, and a panicking
test never stops it. Leaked servers accumulate across runs -- 155 were live on
the machine this was diagnosed on -- and once they exhaust the host's shared
memory every later `pg_ctl` fails with "could not start server". That reads as a
port or resource problem, which is why the cause stayed hidden.

Installing the provider in `setup_pg` fixes both faces: the tests connect, and
they leave no servers behind.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The model's `get_pipeline_by_id_for_runner` decided readiness by comparing the
requested platform version to the pipeline's for exact string equality, while
the implementation asks `is_supported_runtime`: a pipeline compiled for a
different platform version may still be runnable, and that function is the one
place deciding it.

Any request naming a different platform version therefore made the model report
`Monitoring` where the implementation reported `Complete`, so
`db_impl_behaves_like_model` failed whenever its generator produced such a
request. The model now asks the same question.

`runner_descriptor_uses_runtime_compatibility_not_string_equality` pins it
deterministically, where the property test only reaches the case by chance. It
drives the model and the implementation through the same steps and compares
them: asserting on the implementation alone would pass either way, since the
divergence was in the model.

The divergence was invisible until the previous commit let the database tests
run at all.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The implementation carries `deployment_id` through the transition to `Stopping`
(`set_deployment_resources_status_stopping` passes `current.deployment_id`) and
clears it at `Stopped`: while the deployment is being torn down, the runner still
needs to know which deployment it is tearing down.

The model cleared it on entering `Stopping`, so every pipeline observed after a
stop request diverged from the implementation on that field, and
`db_impl_behaves_like_model` failed whenever its generator stopped a provisioned
pipeline and then read it back. The model's two `Stopped` transitions already
clear it and are unchanged.

Pre-existing, and independent of the artifact retention fix; it surfaced from the
same seed once the database tests could run.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Artifact cleanup keeps the files its list of live programs names and removes
every other one, with no retention grace. The list came from programs in
`Success` or `CompilingRust`, which covers the window a legacy program has: it
delivers its binary and program info while the row still says `CompilingRust`
and carries no checksums, so the checksum-less prefix keeps them.

A Gen-2 program has the same window one status earlier and it was not covered.
`deliver_gen2_program_info` writes the file while the row still says
`CompilingSql`; only afterwards does `transit_program_status_to_success_no_binary`
record the checksum. A janitor pass inside that window found no row for the
program at all, so its program info -- the only artifact it has, and the only
copy of the circuit IR the engine needs -- was removed. Nothing heals that: the
row then reads `Success` with a checksum, so no recompilation is triggered, and
provisioning fails fetching an artifact that no longer exists until someone
edits the program to bump its version.

Including `CompilingSql` closes it symmetrically: such a row has no program-info
checksum yet, so it falls to the same relaxed prefix that covers
`CompilingRust`, and the prefix is relaxed about the checksum, not the version.

Two tests, one per half of the mechanism: that the row reaches the janitor at
all (`artifact_retention_covers_both_compiling_statuses`, which also pins that a
`Pending` program holds nothing), and that the delivered filename survives a
pass while the checksum is missing
(`a_gen2_program_info_delivered_before_success_is_retained`).

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Recompiling a Gen-2 pipeline could delete the artifact it had just produced. The
program ends up `Success` with its program info needed to run the program missing.

Editing a program bumps its version, returns it to `Pending`, and clears
`program_info` and both binary checksums, but left
`program_info_integrity_checksum` holding the previous version's value. Artifact
cleanup builds the filename to keep out of the row's checksums, so while the
program recompiled it protected `program_info_<id>_vN+1_sc_<vN>_ic_<vN>.json`, a
name that is never written, and collected the file actually delivered for N+1.

Only a Gen-2 program is exposed: its artifact is named by the program-info
checksum on both halves, so a stale value alone is enough to compute a wrong
name. A legacy program is named partly by its binary source checksum, which the
edit does clear, so it falls to the prefix and keeps everything for the version.

Fixed at both ends, because either alone leaves the rule fragile:

- The edit clears the info checksum, as it already clears the binary ones, so
  the column never describes an artifact of a different version.
- Cleanup builds a filename from the checksums only for a `Success` program, and
  otherwise keeps every artifact of that version by prefix. What survives no
  longer depends on each writer of the row remembering to clear a column: an
  unsettled row keeps too much, the safe direction, whatever it happens to hold.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk force-pushed the crucible-integration branch from f8008c9 to 48982a0 Compare August 28, 2026 03:04
Signed-off-by: feldera-bot <feldera-bot@feldera.com>
@ryzhyk
ryzhyk added this pull request to the merge queue Aug 28, 2026
Merged via the queue into main with commit e828414 Aug 28, 2026
1 check passed
@ryzhyk
ryzhyk deleted the crucible-integration branch August 28, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Pipeline manager Pipeline manager (API, API server, runner, compiler server)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants