Repository navigation
[pipeline-manager] Limit login JWKS refreshes - #7799
Conversation
| cache.insert(issuer, keys); | ||
| let keys = fetch.await; | ||
| let mut cache = jwk_cache.lock().await; | ||
| if keys.is_err() && retry_failed_fetch { |
There was a problem hiding this comment.
pitfall (medium): With retry_failed_fetch = true, the "at most one refresh per 30 s" limit goes away exactly when the login IdP is failing. A caller can't cause the failure, but during an outage every forged-kid token triggers a new fetch. I checked this with a mock JWKS that always returns 503: 5 sequential forged tokens produced 5 fetches, about 2.4 s each. These fetches all queue on the issuer gate, so once the 120 s key TTL runs out, a valid login waits behind the whole queue. Could a failed fetch get a short cooldown (a few seconds) instead of none? Please also update the doc comment ("which a caller cannot make fail") to describe this trade-off.
|
Review summary. The fix is sound: the login path now goes through the lock-free
Missing (medium): uncovered test cases
Mock::given(method("GET")).and(path("/jwks"))
.respond_with(ResponseTemplate::new(503)).expect(5).mount(&jwks).await;
let (app, _db) = login_app(Some(decoding_key), None, validation("some-client", "some-iss"), &jwk_uri).await;
for i in 0..5 { /* forged token with kid u{i} */ assert_eq!(401, call(forged).await.status()); }
Observation: |
There was a problem hiding this comment.
Some findings from LLM review -- high seems worthwhile to check if true:
High: a client can lock out every login with one unauthenticated request every 30 seconds. The cooldown is armed before the fetch at crates/pipeline-manager/src/auth.rs:1267-1269 on the PR branch and only cleared when the fetch returns an error at lines 1273-1275. A cancelled fetch does neither. I verified that actix drops the in-flight handler when the client resets the TCP connection,
so the attack is:
1. Send a JWT whose iss is the login issuer with an unknown kid. No signature is needed.
2. Reset the connection a few milliseconds later, while the manager is still fetching the JWKS.
3. The handler is dropped mid-await. The cooldown stays armed and no keys are inserted.
4. Repeat before the cooldown lapses.
┌─────────────────────┬───────┬────────────────────────────────────────────────────────────────────────────┐
│ Constant │ Value │ Effect under attack │
├─────────────────────┼───────┼────────────────────────────────────────────────────────────────────────────┤
│ Cooldown, line 1632 │ 30 s │ Every uncached kid is rejected while armed │
├─────────────────────┼───────┼────────────────────────────────────────────────────────────────────────────┤
│ Key TTL, line 1626 │ 120 s │ Hits do not extend it, so cached real keys expire and are never re-fetched │
└─────────────────────┴───────┴────────────────────────────────────────────────────────────────────────────┘
After the TTL every login token fails, including the web console. On main the same pattern already exists for federated issuers, so a fix in resolve_issuer_jwk covers both. The experiment that proved cancellation printed fetch_finished=false cooldown_armed=true key_cached=false.
The fix is simpler than the current code. Arm the cooldown after the fetch completes and drop both the pre-fetch mark and clear_refreshed. The per-issuer gate already serializes concurrent misses, so nothing is lost:
let keys = fetch.await;
let mut cache = jwk_cache.lock().await;
if keys.is_ok() || !retry_failed_fetch {
cache.mark_refreshed(issuer);
}
cache.insert(issuer, keys?);
A regression test can poll the resolve_issuer_jwk future once with a pending fetch, drop it, and assert the issuer is not marked as refreshed.
Low: the login path has no rate limit while the identity provider is failing. With retry_failed_fetch set at line 1551, every forged kid costs one outbound fetch, serialized by the gate. That matches main, not a regression. A one or two second cooldown on failure would keep the bound without the 30 s lockout the PR avoids.
| } | ||
| } | ||
|
|
||
| impl JwkCache { |
There was a problem hiding this comment.
there was some reason why they were separate caches, I dont remember the details though and I can't argue against why you wouldn't combine them
A login token with an unknown kid made a JWKS fetch on every request, and held a lock that all JWT logins wait on. The fetch now runs in its own task without the lock, and each issuer refreshes at most once per cooldown (1 s login, 30 s federated), even if the client disconnects. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
f5c38a0 to
a955051
Compare
Linked Issue
A token with an unknown key id made a JWKS fetch on every request and slowed down every login.
Describe Manual Test Plan
New tests cover a dropped request, concurrent misses, the cooldown after success and failure, and a valid login during a slow fetch. Each test fails when its part of the fix is reverted.
Checklist
Breaking Changes?
None.