← 输入框|github.com|刷新
Skip to content

[support] Option to remove source code from support bundles - #7907

Merged
mihaibudiu merged 3 commits into
mainfrom
issue5377
Nov 10, 2026
Merged

mihaibudiu merged 3 commits into
mainfrom
issue5377

Conversation

@mihaibudiu

Copy link
Copy Markdown
Contributor

Linked Issue

Fixes #5377

The code to assemble support bundles is somewhat messy. There are 3 commits, 2 fixing obvious bugs:

  • remove one of the two identical copies of the dataflow graph data from a bundle
  • do not decompress and recompress the pipeline_config when possible

The third commit adds an option to skip source code; see the commit message for details.

@mihaibudiu

Copy link
Copy Markdown
Contributor Author

We should probably redesign support bundle collection to be more streamlined. It's not trivial, since the data in the bundle is collected ahead of time, and the options required by the user may require postprocessing.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran: cargo test --locked -p pipeline-manager --lib support: 27/27 passed, including the new raw-copy and strip tests. pre-commit run on the changed files: clippy, fmt, machete, OpenAPI and the client regeneration all passed.

gap (high): source_code=false does not cover the dataflow graph. #5377 says source code can be in either the pipeline config or the dataflow graph. With this PR, dataflow_graph.json is still collected by default when the "Source code" box is unchecked. Its Calcite conditions, expressions and literals, and its MIR positions, encode the program logic. A user who unticks "Source code" will reasonably expect no SQL logic in the bundle. Either drop or strip dataflow_graph.json when source_code=false, or say clearly in the docs, the checkbox tooltip and the param doc that the dataflow graph is separate. Also note the issue thread: @ryzhyk and @Karakatiza666 wanted the code decoupled from the config in the backend, not stripped at the last minute. Please confirm the team agrees with this approach.

missing tests (medium):

Case Covered?
v4 config with source_code=false yes
v0 collection with source_code=false (output is renamed .json.gz → .json) no. I tested it locally and it passes
pre-PR collection with program_info.dataflow and source_code=false no. It fails, see inline comment
fda --no-source-code / Python source_code=False wiring no
deflated_entry_archive when the uncompressed size is ≥ 4 GiB (no ZIP64; size wraps) no. This is unrealistic for a config, so low

Repro for the inline bug (add it to the support_data_collector tests):

#[tokio::test]
async fn legacy_collection_without_source_code_drops_dataflow() {
    let params = SupportBundleParameters { source_code: false, ..Default::default() };
    let mut entries = Vec::new();
    let mut add = |name: &str, content: ZipContent<'_>| {
        entries.push((name.to_string(), entry_data(content).0));
        Ok(())
    };
    let mut data = SupportBundleData::test_data();
    data.pipeline_config = gz_compress(serde_json::to_vec(&json!({
        "program_info": {"dataflow": {"calcite_plan": {"v": {"rels": [{"condition": "LIKE 'SECRET%'"}]}}}}
    })).unwrap());
    data.push_to_zip(&mut add, &params).await.unwrap();
    let (_, bytes) = entries.iter().find(|(n, _)| n == "pipeline_config.json").unwrap();
    assert!(!String::from_utf8_lossy(bytes).contains("SECRET"));
}

pitfall (medium, breaking): pipeline_config.json no longer has program_info.dataflow. Any tool or script that reads bundles from that path will break silently, and it also gets no graph when dataflow_graph=false. This is worth a line in the changelog.

program_config.program_info holds a copy of the dataflow graph.
dataflow_graph.json has another copy.

I removed the copy from program_config.

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
…ompressing it.

A collection stores the pipeline config as gzip-compressed JSON, and the
bundle carries it as pipeline_config.json, deflated. The bundle used to
decompress the config and let the ZIP writer compress it again.

The bundle now copies the stored
deflate stream into the ZIP file unchanged, through a one-entry archive in
memory, as the zip crate copies compressed data only from another archive.

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Fixes #5377.

When the `source_code` parameter of the support bundle endpoint is false,
the bundle removes from pipeline_config.json
the SQL program, the UDF code, the generated Rust code, and the code
excerpts in the compiler messages.

fda gets --no-source-code, the Python SDK gets `source_code`, and the web
console gets a "Source code" checkbox.

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 10, 2026
@Karakatiza666

Copy link
Copy Markdown
Contributor

not stripped at the last minute

If the code does not make it to the API it's fine by me. We can implement "not collecting the code in the first place" later

Merged via the queue into main with commit 541c5e6 Nov 10, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the issue5377 branch November 10, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[webui][profile] Support bundle checklist should include "source code" explicitly

3 participants