Repository navigation
[pipeline-manager] Send PipelineInteractionConnectFailed when a pipeline connect fails - #7860
Conversation
…ine connect fails. The api-server sent 503 PipelineInteractionUnreachable both when it could not open a connection to a pipeline and when a request failed part way. Clients cannot safely retry the second case for a request that is not idempotent, so since 7e1d8fd the Python SDK retried neither. A connect timeout right after a pipeline restart then failed HTTP ingress on the first attempt. A connect failure now gets its own code, PipelineInteractionConnectFailed. The request was not sent, so the Python SDK and the Rust client retry it for all requests. Both also retry the old code with the "Failed to connect to host" message, for managers that predate this change. Fixes feldera/cloud#2175.
| error: format_disconnected_error_message(e), | ||
| } | ||
| } | ||
| SendRequestError::Connect(_) => RunnerError::PipelineInteractionConnectFailed { |
There was a problem hiding this comment.
pitfall (medium): "Retry is safe" depends on how awc works inside. I checked awc 3.7.0. h1proto.rs returns ConnectError::Disconnected after it writes the request (via ok_or(ConnectError::Disconnected)??), which is why that variant has to stay excluded. Every other ConnectError comes from the connector or the pool before any bytes are sent. If a future awc bump returns some other ConnectError after the request is sent, this would silently turn into a double-apply. The unit test only uses hand-built errors, so it can't catch that. Please add a socket-level test that pins it down (diff in the top-level comment).
There was a problem hiding this comment.
Added both tests in 03cc63f, as test_streaming_proxy_connection_refused_is_connect_failed and test_streaming_proxy_close_after_accept_is_unreachable. I checked that each one fails when the classification is broken. If the Disconnected arm is removed, the close-after-accept test fails. If all Connect errors map to Unreachable, the connection-refused test fails.
-- Claude Code, replying at @blp's request.
|
What I ran
No gate fails: the PR has tests, a changelog entry, no new dependencies and no Missing (medium): no test runs the classification against a real socket. + /// A refused connection reaches the client as the retryable code.
+ #[actix_web::test]
+ async fn test_streaming_proxy_connection_refused_is_connect_failed() {
+ setup();
+ // Bind and drop a listener to get a port with nothing listening on it.
+ let port = std::net::TcpListener::bind("127.0.0.1:0")
+ .unwrap().local_addr().unwrap().port();
+ let url = format!("http://127.0.0.1:{port}/ingress/t0");
+ let (req, payload) = test_get_request(&[]);
+ let err = streaming_proxy(&awc::Client::default(), &url, "test-pipeline",
+ &req, payload, Duration::from_secs(10)).await.unwrap_err();
+ assert_eq!(err.error_code(), "PipelineInteractionConnectFailed");
+ }
+
+ /// A pipeline that accepts the request and then closes the connection may
+ /// have applied it, so the client must not get the retryable code.
+ #[actix_web::test]
+ async fn test_streaming_proxy_close_after_accept_is_unreachable() {
+ setup();
+ let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
+ let port = listener.local_addr().unwrap().port();
+ let server = std::thread::spawn(move || {
+ use std::io::Read;
+ let (mut stream, _) = listener.accept().unwrap();
+ let _ = stream.read(&mut [0u8; 1024]);
+ // Dropping the stream closes the connection without a response.
+ });
+ let url = format!("http://127.0.0.1:{port}/ingress/t0");
+ let (req, payload) = test_get_request(&[]);
+ let err = streaming_proxy(&awc::Client::default(), &url, "test-pipeline",
+ &req, payload, Duration::from_secs(10)).await.unwrap_err();
+ server.join().unwrap();
+ assert_eq!(err.error_code(), "PipelineInteractionUnreachable");
+ }Otherwise LGTM. |
…ockets. A refused connection must give PipelineInteractionConnectFailed. A peer that reads the request and then closes the connection must give PipelineInteractionUnreachable, because it may have applied the request. These tests catch an awc change that reports a post-send failure as some other ConnectError. Signed-off-by: Ben Pfaff <blp@feldera.com>
The api-server sent 503
PipelineInteractionUnreachableboth when it could not open a connection to a pipeline and when a request failed part way. Clients cannot safely retry the second case for a request that is not idempotent, so since 7e1d8fd the Python SDK retried neither. A connect timeout right after a pipeline restart then failed HTTP ingress on the first attempt (feldera/cloud#2175, 4 CI failures on amd64).PipelineInteractionConnectFailedforSendRequestError::Connect(_), exceptDisconnected. The overall requestTimeoutkeeps the old code. The two copies of the error mapping are now one function,pipeline_send_error.openapi.jsonregenerated.wait_for_pipeline_reachableaccepts both codes.Tests: new unit tests in the manager, the Python SDK and the Rust client. Each one fails when the fix is reverted.
Fixes feldera/cloud#2175.