Repository navigation
[connectors] Kafka tests: do not fail on transient transport errors. - #7849
Conversation
|
|
||
| /// Returns true if `error` is a non-fatal transport error, such as `UnknownPartition` while | ||
| /// a new topic's metadata spreads across the brokers. The endpoint retries these itself. | ||
| fn is_transient(error: &ControllerError) -> bool { |
There was a problem hiding this comment.
pitfall (medium): check_fault_tolerance (~L2178) still uses Box::new(move |error, _| panic!("error: {error}")). It creates fresh topics with KafkaResources::create_topics, and its with_adding_partition_afterward step adds partitions while the pipeline runs. That is the same stale-metadata window, so it can hit the same non-fatal UnknownPartition flake. Should it also use is_transient? (The with_failure_expected rounds likely rely on fatal errors only, so the filter should still work for them.)
There was a problem hiding this comment.
Reply from Claude (AI assistant), on behalf of Ben.
Agreed. check_fault_tolerance now uses is_transient too (squashed into the single commit, 4c0baea). The rounds that expect failure still fail: without resume_earliest_if_data_expires, a missing checkpoint offset ends the poller with a fatal error, and the check is only that the records never arrive. All 9 ft_* tests pass against a local Redpanda.
|
Review summary. This change only touches tests. The diagnosis is right:
Apart from that, LGTM. |
check_end_to_end, check_fault_tolerance, and the buffer test panicked on every error that the controller reported, also on non-fatal ones. librdkafka can report UnknownPartition once for a new topic or a new partition, while its metadata spreads to the brokers, and the endpoint retries it. The tests now log non-fatal transport errors and fail only on fatal ones. The output checks still catch lost data. Fixes #7641. Signed-off-by: Ben Pfaff <blp@feldera.com>
9144d44 to
4c0baea
Compare
Linked Issue
Closes #7641.
check_end_to_end,check_fault_tolerance, and the buffer test incrates/connectors/src/transport/kafka/ft/test.rspanicked on every controller error, also on non-fatal ones. librdkafka can reportUnknownPartitiononce for a new topic or partition while its metadata spreads, andrefine_kafka_errorclassifies that as non-fatal, so the endpoint retries. The tests now log non-fatal transport errors (new helperis_transient) and fail only on fatal ones. The output checks still catch lost data. See the analysis in the issue.Describe Manual Test Plan
Ran
cargo test -p static-connectors --test connector_suites -- transport::kafka::ft::test::random_end_to_end transport::kafka::ft::test::bufferagainst a local Redpanda: 5 passed. Ran... -- transport::kafka::ft::test::ft_: 9 passed, includingft_with_topic_expirationandft_with_tolerance_topic_expiration, whose rounds expect failure. The original flake depends on timing and does not reproduce on demand, so this checks only that the tests still pass.Checklist
Breaking Changes?
None. Test-only change.