Repository navigation
[support] Option to remove source code from support bundles - #7907
Conversation
|
We should probably redesign support bundle collection to be more streamlined. It's not trivial, since the data in the bundle is collected ahead of time, and the options required by the user may require postprocessing. |
|
What I ran: gap (high): missing tests (medium):
Repro for the inline bug (add it to the #[tokio::test]
async fn legacy_collection_without_source_code_drops_dataflow() {
let params = SupportBundleParameters { source_code: false, ..Default::default() };
let mut entries = Vec::new();
let mut add = |name: &str, content: ZipContent<'_>| {
entries.push((name.to_string(), entry_data(content).0));
Ok(())
};
let mut data = SupportBundleData::test_data();
data.pipeline_config = gz_compress(serde_json::to_vec(&json!({
"program_info": {"dataflow": {"calcite_plan": {"v": {"rels": [{"condition": "LIKE 'SECRET%'"}]}}}}
})).unwrap());
data.push_to_zip(&mut add, ¶ms).await.unwrap();
let (_, bytes) = entries.iter().find(|(n, _)| n == "pipeline_config.json").unwrap();
assert!(!String::from_utf8_lossy(bytes).contains("SECRET"));
}pitfall (medium, breaking): |
program_config.program_info holds a copy of the dataflow graph. dataflow_graph.json has another copy. I removed the copy from program_config. Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
…ompressing it. A collection stores the pipeline config as gzip-compressed JSON, and the bundle carries it as pipeline_config.json, deflated. The bundle used to decompress the config and let the ZIP writer compress it again. The bundle now copies the stored deflate stream into the ZIP file unchanged, through a one-entry archive in memory, as the zip crate copies compressed data only from another archive. Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Fixes #5377. When the `source_code` parameter of the support bundle endpoint is false, the bundle removes from pipeline_config.json the SQL program, the UDF code, the generated Rust code, and the code excerpts in the compiler messages. fda gets --no-source-code, the Python SDK gets `source_code`, and the web console gets a "Source code" checkbox. Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
ea2e6c9 to
e9b784c
Compare
If the code does not make it to the API it's fine by me. We can implement "not collecting the code in the first place" later |
Linked Issue
Fixes #5377
The code to assemble support bundles is somewhat messy. There are 3 commits, 2 fixing obvious bugs:
The third commit adds an option to skip source code; see the commit message for details.