Repository navigation
[pipeline-manager] Keep cached JWKS keys when a refresh fails - #7865
Conversation
|
Review: LGTM. What I ran:
The gates are met: there are unit tests for both halves of the fix, and they use paused tokio time, so they cannot flake. The changelog is updated, no dependencies change (
Missing (low-medium), test coverage: no test covers requests that queue on the gate while a refresh fails. Those requests reach the + /// Concurrent stale hits during an outage share one failed fetch, and the
+ /// request queued on the gate still gets the cached key.
+ #[tokio::test(start_paused = true)]
+ async fn concurrent_stale_hits_survive_a_failed_refresh() {
+ let cache = Arc::new(Mutex::new(super::IssuerJwkCache::new()));
+ let fetches = Arc::new(AtomicUsize::new(0));
+ assert!(resolve_counting(&cache, &fetches, "real", true).await.is_ok());
+ tokio::time::advance(super::JWK_FRESH_FOR).await;
+ let (first, second) = tokio::join!(
+ resolve_counting(&cache, &fetches, "real", false),
+ resolve_counting(&cache, &fetches, "real", false),
+ );
+ assert!(first.is_ok() && second.is_ok());
+ assert_eq!(2, fetches.load(Ordering::SeqCst));
+ // Just inside the stale limit, after the cooldown: still usable.
+ tokio::time::advance(super::JWK_STALE_LIMIT - super::JWK_FRESH_FOR - Duration::from_millis(1)).await;
+ assert!(resolve_counting(&cache, &fetches, "real", false).await.is_ok());
+ tokio::time::advance(Duration::from_millis(1) + COOLDOWN).await;
+ assert!(resolve_counting(&cache, &fetches, "real", false).await.is_err());
+ } |
|
happy to rebase #7862 on top of this / when this merges |
|
Added -- Claude Code, replying at @blp's request. |
Cached signing keys expired 120 s after a fetch. The next request then refreshed them, and every refresh, success or failure, starts a cooldown that rejects cache misses without a fetch. So one failed JWKS fetch refused every token of the issuer with a 401 for the whole cooldown, 30 s for federated tokens. CI authenticates to its instances with federated GitHub tokens, and its retry of a 401 comes too soon to outlast that. A failed fetch tells us that the issuer is unreachable, not that the keys we have are wrong. Issuers publish a new key before they sign with it and keep the old one published for a time after, so keys from a recent successful fetch are almost always still correct. Refusing every token because of a network error is the worse choice. So keys older than 120 s are still refreshed on use, but if the refresh fails, or the issuer is cooling down, the keys from the last successful refresh stay in use. The cooldown stays as it was, because it is what keeps an outage to one fetch per cooldown. Fixing only the CI client would leave other federated clients with the same 30 s of 401s. The risk is a key that the issuer removed, for example after a leak. We see the removal only on a successful refresh. A successful refresh replaces all keys of the issuer, so a removed key is refused at once. While the issuer stays unreachable, the cached keys stay in use for at most 1 hour, long enough for a short outage and short enough to bound the time a removed key is still accepted. Related to #7048. Signed-off-by: Ben Pfaff <blp@feldera.com>
04de839 to
3003873
Compare
|
fixed the merge conflict in changelog, merging this |
Thank you! |
Linked Issue
Related to #7048. This is the likely cause, but it is not confirmed from the instance logs.
In run 41968241434, the client sent a token that had not expired, and its retry sent the same token. Both got 401. So the instance refused a valid token.
Cached JWKS keys expired 120 s after a fetch. A failed refresh starts the 30 s federated cooldown, and during it every cache miss is refused. So one failed fetch of the GitHub JWKS refused every CI token for 30 s:
kid, keys older than 120 skidwas removedRationale
This touches
auth.rstogether with #7862. The two changes are independent, but one of them will need a rebase.Describe Manual Test Plan
New unit tests. Each fails when its part of the fix is reverted:
a_failed_refresh_keeps_the_cached_keysusable()limited to the fresh agea_refresh_drops_keys_the_issuer_removedconcurrent_stale_hits_survive_a_failed_refresh(suggested in review)The last test covers the CI case: requests that wait on the gate while a refresh fails. It also checks both edges of the 1 h limit. All 28
auth::tests pass, and clippy is clean.Checklist
Breaking Changes?
Describe Incompatible Changes
None.