← 输入框|github.com|刷新
Skip to content

[pipeline-manager] Give one reply to every rejected federated token - #7862

Merged
swanandx merged 1 commit into
mainfrom
federated-auth-uniform-rejection
Nov 12, 2026
Merged

swanandx merged 1 commit into
mainfrom
federated-auth-uniform-rejection

Conversation

@swanandx

@swanandx swanandx commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Linked Issue

None.

Describe Manual Test Plan

Every rejected federated token now gets the same message and takes 2 s, so the reply does not depend on which trust relationships exist. A key fetch for the issuer waits at most until the 2 s mark and then continues in the background. If the server has earlier keys for the issuer, a login uses them when the fetch is slow, as it does when the fetch fails. A new unit test sends seven kinds of rejected tokens at once, including one whose key fetch never completes, plus two during a database outage, and checks that the replies are identical. Undoing any one fix makes the test fail.

any rejected federated token ──> same 401, same message, 2 s

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

None. A rejected federated login no longer says which check failed. The server log still has the reason.

Comment thread crates/pipeline-manager/src/auth.rs Resolved
Comment thread crates/pipeline-manager/src/auth.rs Resolved
Comment thread crates/pipeline-manager/src/auth.rs Resolved
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test --locked -p pipeline-manager --lib auth:: (debug) ✅ 26/26 pass, including federated_rejections_do_not_reveal_trusted_issuers
QA test: an operator-trusted issuer with a slow discovery endpoint (4 s) vs. an unknown issuer ❌ 4.0 s vs. 2.0 s, so the timing still reveals that the issuer is trusted (see inline comment)

Gates: tests ✅, flakiness ✅ (the new test only asserts a lower bound on elapsed time, but it adds about 14 s of real sleeping; tokio::time::pause could avoid that if the embedded DB allows it). There is no docs or breaking-change impact apart from the changed 401 messages.

missing (medium): the PR's tests do not cover these cases:

  1. A trusted issuer whose JWKS or discovery fetch is slow or fails. The new test pre-warms the cache and starts a cooldown, so the fetch path never runs.
  2. A DB failure in match_oidc_trust, after the signature is verified.

The test I used for case 1 (it fails on this PR):

+    #[tokio::test]
+    async fn qa_slow_trusted_issuer_takes_as_long_as_unknown() {
+        use wiremock::matchers::{method, path};
+        use wiremock::{Mock, MockServer, ResponseTemplate};
+        let server = MockServer::start().await;
+        Mock::given(method("GET"))
+            .and(path("/.well-known/openid-configuration"))
+            .respond_with(ResponseTemplate::new(200).set_delay(Duration::from_secs(4)))
+            .mount(&server)
+            .await;
+        let trusted = server.uri();
+        let mut config = default_manager_test_config();
+        config.owner_trusts = crate::config::OwnerTrusts(vec![crate::config::OwnerTrust {
+            issuer: trusted.clone(),
+            subject: "repo:x".to_string(),
+            audience: None,
+        }]);
+        let (app, _state, _db) = login_app(None, None, validation("some-client", "some-iss"), "some-url", config).await;
+        let key = EncodingKey::from_rsa_pem(RsaPrivateKey::new(&mut rand::thread_rng(), 2048).unwrap()
+            .to_pkcs1_pem(LineEnding::LF).unwrap().as_bytes()).unwrap();
+        let mut times = vec![];
+        for iss in ["https://other.example.com".to_string(), trusted] {
+            let mut claim = default_claim();
+            claim.iss = iss;
+            let mut header = Header::new(Algorithm::RS256);
+            header.kid = Some("k".to_string());
+            let token = encode(&header, &claim, &key).unwrap();
+            let req = test::TestRequest::get().uri("/")
+                .insert_header((http::header::AUTHORIZATION, format!("Bearer {token}"))).to_request();
+            let sent = std::time::Instant::now();
+            let res = test::call_service(&app, req).await;
+            assert_eq!(401, res.status());
+            times.push(sent.elapsed());
+        }
+        // observed: unknown=2.00s trusted-slow=4.01s
+        assert!(times[1] < times[0] + Duration::from_millis(500));
+    }

The main idea of the fix is sound, and so is querying the DB for every issuer. Please address the timing leak and the misleading message before merging.

Comment thread crates/pipeline-manager/src/auth.rs Resolved
Comment thread crates/pipeline-manager/src/auth.rs Resolved
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Command Result
cargo test --locked -p pipeline-manager --lib auth::test (debug) 26/26 passed, including federated_rejections_do_not_reveal_trusted_issuers

The new test does not look flaky: all the timing checks are lower bounds, and the overall timeout is 10× the floor.

Gates

  • Tests, unsafe, dependencies, CI: pass.
  • docs/changelog (medium): The 401 text for federated tokens changes, and slow-issuer logins now fail (see the inline comment). Users can see both, but "Changelog updated" is unchecked. Please add a changelog line.

Missing coverage (medium)

  1. No test shows that a valid federated token still succeeds, and fast, with no 2 s floor. A regression that delays successful logins too would pass.
  2. No test covers retrying after a fetch timed out: the first login is rejected, the background refresh fills the cache, and the second login succeeds. a_dropped_request_still_finishes_its_refresh covers this only at the resolve_issuer_jwk level.
  3. Rejections before started (bad header, wrong alg, missing kid/iss) keep their own messages and come back at once. That is fine because none depends on trust, but nothing asserts it, so a future refactor could move a trust-dependent check above started without notice.

Comment thread crates/pipeline-manager/src/auth.rs Resolved
Comment thread crates/pipeline-manager/src/auth.rs Resolved
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test --locked -p pipeline-manager --lib auth:: (27 tests, including the 2 new ones) ✅ pass
rustfmt --check on auth.rs ✅

Review

  • Correctness: I traced the code and confirmed three things. The DB mutex guard is dropped before the floor sleep. The JWKS fetch really does continue in its own task (tokio::spawn in resolve_issuer_jwk), so dropping it at the timeout is safe. The rejections that come before the trust check (malformed header, wrong alg, missing kid/iss) do not depend on trust, so it is fine that they stay distinct.
  • Commit history (medium): The third commit is titled "Test a retry after a slow federated key fetch", but it also changes the rejection message and the log level. Please squash the three commits into one before merge, or retitle that one.
  • Docs (medium): The changelog entry is there, but docs/get-started/enterprise/authentication/roles.md still says nothing about the generic 401 or about retrying after a slow first key fetch. The "Documentation updated" box is unchecked even though this is a user-visible behavior change.
  • Missing test (low-medium): No test covers a valid, matching token from a tenant-registered issuer with a warm cache. That case should succeed without hitting the 2 s floor. Only the operator-issuer retry path is tested.
  • Flakiness: The elapsed < floor assert in the retry test is timing-based. In practice it has about 2 s of headroom in debug, so it is acceptable.

@blp

blp commented Nov 5, 2026

Copy link
Copy Markdown
Member

What's the reason for this change? It can make troubleshooting harder. I'm surprised there's no stated rationale.

@swanandx
swanandx force-pushed the federated-auth-uniform-rejection branch from b941690 to 1101b3c Compare November 5, 2026 21:26
@swanandx

swanandx commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

What's the reason for this change? It can make troubleshooting harder. I'm surprised there's no stated rationale.

Trust relationships are configuration that the caller cannot read, so the reply to a rejected federated token should not depend on them. For troubleshooting, the message lists what to check and says to retry, a failed verification logs at info with its reason, and roles.md describes the retry.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM.

What I ran (merge commit, debug build):

Check Result
cargo test --locked -p pipeline-manager --lib federated, 4 runs ✅ 2/2 every run, ~24 s each
cargo test --locked -p pipeline-manager --lib auth:: ✅ 27/27
pre-commit run --files <changed files> ✅ all hooks pass

Gates: all pass. The PR adds unit tests that cover each rejection path, and it updates the docs and the changelog. It adds no dependencies and no unsafe code. The new 401 text is a visible change, and the changelog records it.

What I checked: resolve_issuer_jwk already runs the fetch in a tokio::spawn that holds the refresh gate. So when timeout_at drops the lookup, the fetch keeps going and fills the cache, as the retry test shows. I couldn't find any timing assertion that is likely to flake. The rejection test measures from before started, and tokio rounds timers up. The only upper bound is the 200 path, which has a 2 s margin.

@swanandx
swanandx force-pushed the federated-auth-uniform-rejection branch from 1101b3c to d49e33e Compare November 6, 2026 08:42
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM

What I ran (debug build, embedded Postgres):

Check Result
cargo test -p pipeline-manager --lib auth::test::federated (run 3×) 2/2 pass every run, ~20–30 s
cargo test -p pipeline-manager --lib auth:: 30/30 pass
pre-commit run --files <changed> pass (clippy, fmt, machete, OpenAPI, docs)

Gates: tests added, docs and changelog updated, commit message is good, no unsafe code, no new dependencies, no breaking API change. The background fetch really does outlive the timed-out request: resolve_issuer_jwk runs it in tokio::spawn, and the task holds the refresh gate until the cache is filled.

Not covered by tests (medium at most):

  • pitfall, medium: the floor does not cover the match_oidc_trust DB error path (unavailable returns at once). Only a token with a valid signature from a trusted issuer reaches that path, so during a partial DB outage a fast 5xx could still reveal that the issuer is trusted. To close it, wait out the floor before returning that error too.
  • pitfall, medium (flakiness): assert!(elapsed < FEDERATED_REJECTION_FLOOR) on the success paths is a wall-clock bound. 2 s leaves plenty of room, but it is the one assertion that could fail on a heavily loaded CI runner.

@swanandx
swanandx force-pushed the federated-auth-uniform-rejection branch from d49e33e to 0f6b2eb Compare November 6, 2026 09:28

@jyotshnayaparla-00 jyotshnayaparla-00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@gz can review the auth logic.

Comment thread docs.feldera.com/docs/changelog.md Resolved
error continues, make sure that the token is current and that a trust relationship \
matches its issuer, subject and audience";

/// The time of every federated rejection. A request also waits for a JWKS fetch

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how much of a concern timings are here? lot of noise anyways. having sleeps in the code makes me nervous a bit. (queueing up of tasks?)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Claude, on behalf of @swanandx) Timing is the main point. Only a trusted issuer starts a JWKS fetch, and a network round trip (tens to hundreds of ms) is much larger than the noise of a local rejection, so a few requests would show whether a tenant trusts an issuer. The sleep is an async timer, not a blocked thread or a spawned task. A rejected request holds only its own connection for 2 s, and no lock is held while it waits, so nothing queues behind it. The background key fetch is limited to one per issuer.

Every rejected federated token now gets the same message and takes 2 s,
so the reply does not depend on which trust relationships exist. A key
fetch waits at most 2 s and then continues in the background. The issuer
check queries the database for every issuer.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
@swanandx
swanandx force-pushed the federated-auth-uniform-rejection branch from 0f6b2eb to b142d15 Compare November 12, 2026 07:34
@swanandx
swanandx added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 9d0eadd Nov 12, 2026
1 check passed
@swanandx
swanandx deleted the federated-auth-uniform-rejection branch November 12, 2026 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants