Repository navigation
Support bundle: stream while collecting, limit each step, no blind retry - #7818
Conversation
| /// | ||
| /// The runner's limit stops when the response headers arrive, so it does not | ||
| /// stop a pipeline that sends its headers and then a slow body. For example, | ||
| /// a multihost profile streams one host at a time. |
There was a problem hiding this comment.
Not as worded: since feldera/cloud#2164 the coordinator asks all hosts at once, so "one host at a time" was misleading. It still sends the headers with the first host's profile and the rest after that, so the limit is still needed. 9252f08 rewords the example to say that.
| # A bundle that collects new data is slow and loads the pipeline, so do | ||
| # not repeat it after a failure that can come late, such as a gateway | ||
| # timeout or a dropped connection. | ||
| collect = str((params or {}).get("collect", "true")).lower() != "false" |
There was a problem hiding this comment.
this doesn't look very robust to me.
what is "true"?
There was a problem hiding this comment.
Agreed, the "true" default and the string matching were guessing at the server's parsing. 74ddc87 replaces them with a retry: bool = False argument to get_pipeline_support_bundle(). Pipeline.support_bundle() passes retry=not collect from its own bool.
003c045 to
e5f94df
Compare
| )) | ||
| .insert_header(actix_http::ContentEncoding::Identity) | ||
| .body(bundle.buffer)) | ||
| .streaming(body)) |
There was a problem hiding this comment.
pitfall (medium, UX): Before, .body(Vec) set Content-Length. A streamed response has none. In the web console, streamToDownload (pipelineManager.ts) now passes totalBytes = 0, so useDownloadProgress.percent is always null. The support bundle download progress bar is then indeterminate for the whole download. Is that intended? If so, say so in the PR description. If not, the UI could show the bytes received, or the collection-log phase.
There was a problem hiding this comment.
Intended. Without a Content-Length the bar is indeterminate, but DownloadProgressDisplay.svelte still shows the bytes received. Before, the bar showed nothing until the whole bundle was built (about 8 minutes in the reported case), and only then a percentage over a fast download. The PR description now says this. A bar for the collection phase could be a separate web console change.
|
What I ran:
Missing (medium, docs gate): The Missing (medium, tests): The tests don't cover Test diffdiff --git a/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs b/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs
index 18e4af5..cf5c927 100644
--- a/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs
+++ b/crates/pipeline-manager/src/api/endpoints/pipeline_interaction/support_bundle.rs
@@ -649,6 +649,71 @@ mod tests {
assert!(dirs.contains(format!("{base}-1").as_str()));
}
+ /// `limit` counts the new collection, and a failed collection ends the
+ /// stream with an error after the first chunk.
+ #[tokio::test(start_paused = true)]
+ async fn bundle_stream_limit_and_failed_collection() {
+ use futures_util::StreamExt;
+
+ let params = SupportBundleParameters {
+ limit: Some(1),
+ ..Default::default()
+ };
+ let collection = async { Ok(SupportBundleData::test_data()) };
+ let chunks: Vec<_> = support_bundle_stream(
+ fake_pipeline("p"),
+ vec![SupportBundleData::test_data(), SupportBundleData::test_data()],
+ Some(collection),
+ params,
+ Duration::from_secs(10),
+ )
+ .collect()
+ .await;
+ let bundle: Vec<u8> = chunks
+ .into_iter()
+ .flat_map(|c| c.unwrap().to_vec())
+ .collect();
+ let mut archive = read_zip(&bundle);
+ let mut metadata = String::new();
+ archive
+ .by_name("metadata.json")
+ .unwrap()
+ .read_to_string(&mut metadata)
+ .unwrap();
+ let metadata: serde_json::Value = serde_json::from_str(&metadata).unwrap();
+ assert_eq!(metadata["collections"].as_array().unwrap().len(), 1);
+
+ // No collection: no log entry.
+ let chunks: Vec<_> = support_bundle_stream(
+ fake_pipeline("p"),
+ vec![SupportBundleData::test_data()],
+ None::<std::future::Ready<Result<SupportBundleData, ManagerError>>>,
+ SupportBundleParameters::default(),
+ Duration::from_secs(10),
+ )
+ .collect()
+ .await;
+ let bundle: Vec<u8> = chunks
+ .into_iter()
+ .flat_map(|c| c.unwrap().to_vec())
+ .collect();
+ assert!(read_zip(&bundle).by_name(COLLECTION_LOG).is_err());
+
+ // Failed collection.
+ let failing = async { Err(bundle_error("boom", "x")) };
+ let results: Vec<_> = support_bundle_stream(
+ fake_pipeline("p"),
+ vec![],
+ Some(failing),
+ SupportBundleParameters::default(),
+ Duration::from_secs(10),
+ )
+ .collect()
+ .await;
+ assert!(results[0].is_ok());
+ assert!(results.last().unwrap().is_err());
+ }
+
/// The bundle sends bytes while a slow collection runs, not only at its
/// end, and the result is still one valid ZIP file.
#[tokio::test(start_paused = true)]There is no handler-level test that checks the streamed response itself: no |
e5f94df to
6c62f81
Compare
0613e4f to
4df17c5
Compare
4df17c5 to
1aa5aba
Compare
1a87426 to
fe698dc
Compare
|
Verdict: PR bug (high confidence). The merge queue run's test failure is caused by this PR, specifically a gap left by its own last commit. What failed
Why Fix Take it with: Or compare: support-bundle-2163...fred/observe-7818-42285488013 |
9b48c6f to
b07139d
Compare
Each support data step had a 120 s limit, but awc's limit ends when the response headers arrive. A multihost pipeline sends the headers of its JSON profile with the first host's profile and the other hosts' profiles after that, so the rest of the body had no limit. In a 50-host pipeline in a long transaction commit, the profile step took 434 s, and the bundle response was silent for all that time. Now the 120 s limit also covers the body. A step that is not complete after 120 s fails and is reported in the bundle, and the other steps are still delivered. Part of feldera/cloud#2163. Signed-off-by: Ben Pfaff <blp@feldera.com>
A GET retries on 502, 503, 504, a read timeout, or a dropped connection. For a support bundle that collects new data, each retry starts a new slow collection that loads the pipeline. Such a failure can come after the collection ran for minutes, so a retry seldom helps. A bundle with collect=false only reads stored data, so it still retries. get_pipeline_support_bundle() takes `retry`, which is False by default, and Pipeline.support_bundle() passes `retry=not collect`. Part of feldera/cloud#2163. Signed-off-by: Ben Pfaff <blp@feldera.com>
The support bundle handler sent nothing until it had collected all data and built the whole ZIP file. A slow collection, such as the profile of a large multihost pipeline in a long commit, kept the response silent for minutes, so a proxy or the client closed it first. Now the response starts when the request is valid. When the bundle collects new data, its first entry is collection_log.txt, which gets a line every 10 s until the collection is complete. Then the collections and the metadata follow, each sent when it is written. The ZIP file uses data descriptors, which the profiler UI, unzip, Python, and the zip crate read. A failure after the response starts now cuts the response short instead of setting an error status. The collection itself reports failures inside the bundle, as before. The API docs no longer list 503 examples for a failed pipeline request, because the endpoint reports such failures inside the bundle. They describe collection_log.txt and the missing Content-Length. The operations guide and the changelog describe the change too. Part of feldera/cloud#2163. Signed-off-by: Ben Pfaff <blp@feldera.com>
b07139d to
f719398
Compare
A UDF that sleeps 45 s keeps one circuit step busy. A support bundle requested during that step must send its first bytes within 15 s, and must still hold the circuit profile when the step completes. The test also checks that the bundle took more than half of the step time, so that it fails if the bundle started before the step did. Before #7818, the bundle sent nothing until it had all of its data, so a proxy could close the idle connection (feldera/cloud#2163). On v0.375.0 this test fails: the first byte arrived after 43.1 s. The CI multihost suite runs this test with 2 hosts too. Signed-off-by: Ben Pfaff <blp@feldera.com>
A UDF that sleeps 45 s keeps one circuit step busy. A support bundle requested during that step must send its first bytes within 15 s, and must still hold the circuit profile when the step completes. Before feldera#7818, the bundle sent nothing until it had all of its data, so a proxy could close the idle connection (feldera/cloud#2163). On v0.375.0 this test fails: the first byte arrived after 43.1 s. The CI multihost suite runs this test with 2 hosts too. Signed-off-by: Ben Pfaff <blp@feldera.com>
Part of feldera/cloud#2163. The coordinator part is feldera/cloud#2164.
Problem
A support bundle for a 50-host pipeline in a long transaction commit failed in the web console and in the API. The response was silent for about 8 minutes, so a proxy or the client closed it first.
Changes
within,support_data_collector.rs). The proxy endpoints keep their old limits.support_bundle()retried on 502/503/504, a read timeout, or a dropped connection. Each retry started a new slow collection.collect=falsestill retries.collection_log.txtentry gets a line every 10 s while the collection runs. Then each collection is sent when it is written.collection_log.txtexample:Behavior changes
ZipWriter::new_stream). A streamed bundle was read correctly bybut-unzip0.1.7 (the profiler UI),unzip -t, Pythonzipfile, and the zip crate (fda).SupportBundleData::collectalready reports each failure inside the bundle, so this happens only if writing the ZIP fails.Content-Length, so the web console's progress bar is indeterminate for the whole download. It still shows the bytes received (DownloadProgressDisplay.svelte). Before, the bar showed nothing until the whole bundle was built (about 8 minutes in the reported case), and then a percentage over a fast download. A bar for the collection phase could be a separate web console change.LockTookTooLong,DeadlockDetected). The changelog and the operations guide describe the changes.Tests
Each test was checked against the old behavior:
within_stops_a_body_that_never_endsbundle_streams_while_a_collection_runsbundle_streams_while_a_collection_runstest_support_bundle_retries_only_without_collectionPipeline.support_bundle()does not passretry, orretryisTrueby defaultlimit_counts_the_new_collectionbundle_without_collection_has_no_collection_logfailed_collection_ends_the_stream_with_an_errorOkcargo test -p pipeline-manager --lib support: 25 passed. 3sql_compilertests fail locally because the SQL compiler JAR is not built in this worktree. They do not touch this code.