← 输入框|github.com|刷新
Skip to content

[connectors] Journal the input of an ad hoc INSERT removed before its step collects it. - #7672

Merged
blp merged 1 commit into
mainfrom
adhoc-orphan-journal
Oct 29, 2026
Merged

blp merged 1 commit into
mainfrom
adhoc-orphan-journal

Conversation

@blp

@blp blp commented Oct 28, 2026

Copy link
Copy Markdown
Member

An ad hoc INSERT hands its rows to the circuit when a step asks its endpoint for input (reader.queue(), which the ad hoc transport handles synchronously on the circuit thread), and then removes its endpoint as soon as its queue drains. If the removal came before the step collected the endpoint's reply, the step still processed the rows, but ControllerStatus::remove_input kept only their size (for total_processed_records), so the step's journal entry did not record them. A restart from the journal then lost rows whose INSERT had succeeded.

remove_input now keeps the removed endpoint's full StepResults, keyed by endpoint id, and stores them while it still holds the inputs lock, so the circuit thread cannot see the endpoint gone without its results. The step that waits for the endpoint claims them and journals them like any other reply; write_step already handles journaled input from an endpoint removed after its step took its input. Results that no step claims are still credited by size only.

Found by an Antithesis run of the controller harness.

A follow-up PR, stacked on this one, makes the circuit thread the only thing that removes input endpoints, which removes this window and several related special cases.

Describe Manual Test Plan

New test adhoc_insert_removed_before_collection_replays holds the step, through a new test hook that runs after the step asks its endpoints for input and before it collects their replies, until the INSERT has removed its endpoint, then restarts from the journal and checks that the row survives. Without the fix it fails with "the restart lost the INSERT's row". The controller tests pass (cargo test -p static-connectors --test connector_suites controller::, 3 runs, and cargo test -p dbsp_connectors --lib controller::), and the ad hoc tests passed 200 of 200 runs.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

@blp
blp requested a review from ryzhyk October 28, 2026 00:25
@blp blp self-assigned this Oct 28, 2026
@blp blp added bug Something isn't working ft Fault tolerant, distributed, and scale-out implementation connectors Issues related to the adapters/connectors crate adhoc Issue related to ad hoc query processing demo Feldera demos rust Pull requests that update Rust code profiler Issues related to the profiler and its APIs enterprise Issue related to Feldera Enterprise features. labels Oct 28, 2026
@blp

blp commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

This is about the nth PR to fix related races. I'm going to follow up after this one with a refactoring that makes everything associated less bad.

Here's Claude's take on the refactoring idea:

The flaw: three threads share one map, ControllerStatus::inputs, with no ordering between them:

┌─────────────────────────┬──────────────────────────────────────────────────┬───────────────┐
│          Actor          │             What it does to the map              │     When      │
├─────────────────────────┼──────────────────────────────────────────────────┼───────────────┤
│ HTTP and ad hoc INSERT  │ add and remove entries; removal also discards    │ any time,     │
│ tasks, stop, restore    │ the entry's uncollected reply                    │ from any      │
│                         │                                                  │ thread        │
├─────────────────────────┼──────────────────────────────────────────────────┼───────────────┤
│ Circuit thread,         │ snapshots the map to decide whom to poll, then   │ during the    │
│ input_step              │ collects replies from the map                    │ step          │
├─────────────────────────┼──────────────────────────────────────────────────┼───────────────┤
│ Circuit thread,         │ works out the journal's add/remove lists by      │ after the     │
│ write_step              │ comparing the map, as it is now, with the        │ step          │
│                         │ previous entry                                   │               │
└─────────────────────────┴──────────────────────────────────────────────────┴───────────────┘

The journal is supposed to say "these endpoints existed, and they supplied this input, in this step". In fact the endpoint lists come from whatever the map holds at write time, and the input comes from whatever survived in the map until collection. Any add or remove that lands between those moments makes the journal disagree with what the circuit actually did. Each bug was one such window:

┌─────────────────────────────────────┬───────────────────────────────────────────────────────┐
│                 Bug                 │                        Window                         │
├─────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ 5c7931d09 counts the input of a     │ removed after it gave rows to the circuit; the        │
│ removed endpoint (orphaned_input)   │ processed count fell short                            │
├─────────────────────────────────────┼───────────────────────────────────────────────────────┤
│                                     │ removed after input was collected but before the      │
│ W1, in #7482 (52511a30b)            │ journal write; the entry recorded the removal, so     │
│                                     │ replay could not feed the endpoint                    │
├─────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ H2 (this branch)                    │ removed after queue() but before collection; the rows │
│                                     │  were applied but not journaled                       │
├─────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ Bug 3 (3368b762b)                   │ the poll snapshot included endpoints that could not   │
│                                     │ reply                                                 │
├─────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ be5458880 duplicate-insertion race  │ two adds of the same name raced on the map            │
├─────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ bf5b9f36e, 62603b06e (#7558)        │ replay met add/remove lists that did not match the    │
│                                     │ checkpoint                                            │
├─────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ the Terminated skip in write_step   │ stop removes endpoints at the same time, so the       │
│                                     │ comparison would record live endpoints as removed     │
└─────────────────────────────────────┴───────────────────────────────────────────────────────┘

// so this step processes it and must journal it. See
// `ControllerStatus::orphaned_results`.
if let Some(results) =
self.controller.status.take_orphaned_results(&endpoint_id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): This fixes the case where the reply comes before the removal. The reverse order still loses journal data. If an async transport is removed before its reply arrives, this branch finds no results and drops the endpoint from waiting. The late reply then goes through ControllerStatus::extended → orphan_input (stats.rs ~1398), which credits size only. Any rows that endpoint already pushed into the circuit get processed but never journaled. I see the stacked follow-up is meant to close this. Could you add a test for it there (reply after removal, then replay) so the follow-up has to keep that path covered?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is correct. If an endpoint is removed before its reply arrives, ControllerStatus::extended cannot find the endpoint. It then sends the input to orphan_input, which credits only its size, so the rows get to the circuit without a journal entry. This PR closes only the window where the reply comes first.

The stacked follow-up makes the circuit thread the only thing that removes input endpoints. That closes both orders. In the follow-up, I will add these tests:

  • a replay test where the reply arrives after the endpoint is removed;
  • results that no step claims, which take_orphaned_input credits by size only;
  • an endpoint removed during stop(), where the input is now journaled but write_step skips the endpoint diff.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. The fix is correct. Lock order stays inputs → progress → orphaned_results on every path, so it can't deadlock. Results are stashed before the inputs write lock is released, so the circuit thread can't see the endpoint gone without its results. write_step already keeps logged-but-removed endpoints in the journal.

What I ran Result
cargo test -p static-connectors --test connector_suites adhoc_insert_removed 2/2 pass
New test, run 30 times 30/30 pass (~0.4 s each), no flakes
New test with take_orphaned_results stubbed to None fails at test.rs:1771 ("restart lost the INSERT's row"), so the test catches the bug
connector_suites controller:: 90/90 pass
pre-commit run --files <changed> all pass

Gates: tests ✅. Flakiness ✅: the waits are bounded and fail with clear messages. No docs needed (internal behaviour). No unsafe code, dependency changes or breaking changes.

Missing (medium): nothing tests results that no step claims, which take_orphaned_input credits by size only. Also untested: a removal during stop(). With this change that input is now journaled while the endpoint diff is skipped. The new_inputs path covers it, but no test exercises it.

Nit: the commit subject is about 100 chars and GitHub truncates it.

An ad hoc INSERT hands its rows to the circuit when a step asks its endpoint
for input, then removes the endpoint as soon as its queue drains.  If the
removal came before the step collected the endpoint's reply, the step still
processed the rows, but `remove_input` kept only their size, so the step's
journal entry did not record them.  A restart from the journal then lost
rows whose INSERT had succeeded.

`remove_input` now keeps the removed endpoint's full results, and the step
that waits for the endpoint journals them like any other reply.  Results
that no step claims are still credited by size only.

Found by an Antithesis run of the controller harness.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit b4f2367 Oct 29, 2026
1 check passed
@blp
blp deleted the adhoc-orphan-journal branch October 29, 2026 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Issue related to ad hoc query processing bug Something isn't working connectors Issues related to the adapters/connectors crate demo Feldera demos enterprise Issue related to Feldera Enterprise features. ft Fault tolerant, distributed, and scale-out implementation profiler Issues related to the profiler and its APIs rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants