← 输入框|github.com|刷新
Skip to content

[pipeline-manager] Limit login JWKS refreshes - #7799

Merged
swanandx merged 1 commit into
mainfrom
login-jwks-cooldown
Nov 4, 2026
Merged

swanandx merged 1 commit into
mainfrom
login-jwks-cooldown

Conversation

@swanandx

@swanandx swanandx commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Linked Issue

A token with an unknown key id made a JWKS fetch on every request and slowed down every login.

before:  unknown kid -> lock -> fetch keys -> unlock    (every request, all logins wait)
after:   unknown kid -> fetch keys in a task, no lock  (at most once per second)

Describe Manual Test Plan

New tests cover a dropped request, concurrent misses, the cooldown after success and failure, and a valid login during a slow fetch. Each test fails when its part of the fix is reverted.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

None.

Comment thread crates/pipeline-manager/src/auth.rs Outdated
cache.insert(issuer, keys);
let keys = fetch.await;
let mut cache = jwk_cache.lock().await;
if keys.is_err() && retry_failed_fetch {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): With retry_failed_fetch = true, the "at most one refresh per 30 s" limit goes away exactly when the login IdP is failing. A caller can't cause the failure, but during an outage every forged-kid token triggers a new fetch. I checked this with a mock JWKS that always returns 503: 5 sequential forged tokens produced 5 fetches, about 2.4 s each. These fetches all queue on the issuer gate, so once the 120 s key TTL runs out, a valid login waits behind the whole queue. Could a failed fetch get a short cooldown (a few seconds) instead of none? Please also update the doc comment ("which a caller cannot make fail") to describe this trade-off.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review summary. The fix is sound: the login path now goes through the lock-free (issuer, kid) resolver with a cooldown. Login and federated tokens are routed by exact iss, so they never share cache entries or cooldown. Only one point needs action (see the inline comment). All feldera-pr-checks gates pass: tests added, commit message OK, no unsafe code, no new dependencies, and no user-visible change, so no docs are needed.

What I ran Result
cargo test --locked -p pipeline-manager --lib auth:: 22/22 pass, including both new tests
QA test I wrote: login JWKS always returns 503, 5 forged kids Passes with expect(5), so each forged token makes its own fetch while the IdP is down

Missing (medium): uncovered test cases

  1. Login JWKS endpoint keeps failing (no test pins the fetch count during an outage). A sketch of the test I used:
Mock::given(method("GET")).and(path("/jwks"))
    .respond_with(ResponseTemplate::new(503)).expect(5).mount(&jwks).await;
let (app, _db) = login_app(Some(decoding_key), None, validation("some-client", "some-iss"), &jwk_uri).await;
for i in 0..5 { /* forged token with kid u{i} */ assert_eq!(401, call(forged).await.status()); }
  1. The login provider rotates its key while the cooldown is active. A valid token with the new kid gets 401 until the cooldown ends, and nothing tests that this lasts at most 30 s.

Observation: unknown_login_kids_cost_one_fetch_and_block_nobody takes 5 s because of set_delay(5s). The margin over the 200 ms sleep is wide, so I don't see a flakiness risk, but a shorter delay (around 1 s) would keep the suite faster.

@gz gz left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some findings from LLM review -- high seems worthwhile to check if true:

  High: a client can lock out every login with one unauthenticated request every 30 seconds. The cooldown is armed before the fetch at crates/pipeline-manager/src/auth.rs:1267-1269 on the PR branch and only cleared when the fetch returns an error at lines 1273-1275. A cancelled fetch does neither. I verified that actix drops the in-flight handler when the client resets the TCP connection,
  so the attack is:

  1. Send a JWT whose iss is the login issuer with an unknown kid. No signature is needed.
  2. Reset the connection a few milliseconds later, while the manager is still fetching the JWKS.
  3. The handler is dropped mid-await. The cooldown stays armed and no keys are inserted.
  4. Repeat before the cooldown lapses.

  ┌─────────────────────┬───────┬────────────────────────────────────────────────────────────────────────────┐
  │      Constant       │ Value │                            Effect under attack                             │
  ├─────────────────────┼───────┼────────────────────────────────────────────────────────────────────────────┤
  │ Cooldown, line 1632 │ 30 s  │ Every uncached kid is rejected while armed                                 │
  ├─────────────────────┼───────┼────────────────────────────────────────────────────────────────────────────┤
  │ Key TTL, line 1626  │ 120 s │ Hits do not extend it, so cached real keys expire and are never re-fetched │
  └─────────────────────┴───────┴────────────────────────────────────────────────────────────────────────────┘

  After the TTL every login token fails, including the web console. On main the same pattern already exists for federated issuers, so a fix in resolve_issuer_jwk covers both. The experiment that proved cancellation printed fetch_finished=false cooldown_armed=true key_cached=false. 

  The fix is simpler than the current code. Arm the cooldown after the fetch completes and drop both the pre-fetch mark and clear_refreshed. The per-issuer gate already serializes concurrent misses, so nothing is lost:

  let keys = fetch.await;
  let mut cache = jwk_cache.lock().await;
  if keys.is_ok() || !retry_failed_fetch {
      cache.mark_refreshed(issuer);
  }
  cache.insert(issuer, keys?);

  A regression test can poll the resolve_issuer_jwk future once with a pending fetch, drop it, and assert the issuer is not marked as refreshed.

  Low: the login path has no rate limit while the identity provider is failing. With retry_failed_fetch set at line 1551, every forged kid costs one outbound fetch, serialized by the gate. That matches main, not a regression. A one or two second cooldown on failure would keep the bound without the 30 s lockout the PR avoids.

}
}

impl JwkCache {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there was some reason why they were separate caches, I dont remember the details though and I can't argue against why you wouldn't combine them

A login token with an unknown kid made a JWKS fetch on every request,
and held a lock that all JWT logins wait on. The fetch now runs in its
own task without the lock, and each issuer refreshes at most once per
cooldown (1 s login, 30 s federated), even if the client disconnects.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
@swanandx
swanandx force-pushed the login-jwks-cooldown branch from f5c38a0 to a955051 Compare November 3, 2026 21:46
@swanandx
swanandx added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit c5db31e Nov 4, 2026
1 check passed
@swanandx
swanandx deleted the login-jwks-cooldown branch November 4, 2026 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants