Repository navigation
[pipeline-manager] Give one reply to every rejected federated token - #7862
Conversation
|
What I ran
Gates: tests ✅, flakiness ✅ (the new test only asserts a lower bound on elapsed time, but it adds about 14 s of real sleeping; missing (medium): the PR's tests do not cover these cases:
The test I used for case 1 (it fails on this PR): + #[tokio::test]
+ async fn qa_slow_trusted_issuer_takes_as_long_as_unknown() {
+ use wiremock::matchers::{method, path};
+ use wiremock::{Mock, MockServer, ResponseTemplate};
+ let server = MockServer::start().await;
+ Mock::given(method("GET"))
+ .and(path("/.well-known/openid-configuration"))
+ .respond_with(ResponseTemplate::new(200).set_delay(Duration::from_secs(4)))
+ .mount(&server)
+ .await;
+ let trusted = server.uri();
+ let mut config = default_manager_test_config();
+ config.owner_trusts = crate::config::OwnerTrusts(vec![crate::config::OwnerTrust {
+ issuer: trusted.clone(),
+ subject: "repo:x".to_string(),
+ audience: None,
+ }]);
+ let (app, _state, _db) = login_app(None, None, validation("some-client", "some-iss"), "some-url", config).await;
+ let key = EncodingKey::from_rsa_pem(RsaPrivateKey::new(&mut rand::thread_rng(), 2048).unwrap()
+ .to_pkcs1_pem(LineEnding::LF).unwrap().as_bytes()).unwrap();
+ let mut times = vec![];
+ for iss in ["https://other.example.com".to_string(), trusted] {
+ let mut claim = default_claim();
+ claim.iss = iss;
+ let mut header = Header::new(Algorithm::RS256);
+ header.kid = Some("k".to_string());
+ let token = encode(&header, &claim, &key).unwrap();
+ let req = test::TestRequest::get().uri("/")
+ .insert_header((http::header::AUTHORIZATION, format!("Bearer {token}"))).to_request();
+ let sent = std::time::Instant::now();
+ let res = test::call_service(&app, req).await;
+ assert_eq!(401, res.status());
+ times.push(sent.elapsed());
+ }
+ // observed: unknown=2.00s trusted-slow=4.01s
+ assert!(times[1] < times[0] + Duration::from_millis(500));
+ }The main idea of the fix is sound, and so is querying the DB for every issuer. Please address the timing leak and the misleading message before merging. |
|
What I ran
The new test does not look flaky: all the timing checks are lower bounds, and the overall timeout is 10× the floor. Gates
Missing coverage (medium)
|
|
What I ran
Review
|
|
What's the reason for this change? It can make troubleshooting harder. I'm surprised there's no stated rationale. |
b941690 to
1101b3c
Compare
Trust relationships are configuration that the caller cannot read, so the reply to a rejected federated token should not depend on them. For troubleshooting, the message lists what to check and says to retry, a failed verification logs at info with its reason, and roles.md describes the retry. |
|
LGTM. What I ran (merge commit, debug build):
Gates: all pass. The PR adds unit tests that cover each rejection path, and it updates the docs and the changelog. It adds no dependencies and no What I checked: |
1101b3c to
d49e33e
Compare
|
LGTM What I ran (debug build, embedded Postgres):
Gates: tests added, docs and changelog updated, commit message is good, no unsafe code, no new dependencies, no breaking API change. The background fetch really does outlive the timed-out request: Not covered by tests (medium at most):
|
d49e33e to
0f6b2eb
Compare
jyotshnayaparla-00
left a comment
There was a problem hiding this comment.
Looks good, thanks!
@gz can review the auth logic.
| error continues, make sure that the token is current and that a trust relationship \ | ||
| matches its issuer, subject and audience"; | ||
|
|
||
| /// The time of every federated rejection. A request also waits for a JWKS fetch |
There was a problem hiding this comment.
how much of a concern timings are here? lot of noise anyways. having sleeps in the code makes me nervous a bit. (queueing up of tasks?)
There was a problem hiding this comment.
(Claude, on behalf of @swanandx) Timing is the main point. Only a trusted issuer starts a JWKS fetch, and a network round trip (tens to hundreds of ms) is much larger than the noise of a local rejection, so a few requests would show whether a tenant trusts an issuer. The sleep is an async timer, not a blocked thread or a spawned task. A rejected request holds only its own connection for 2 s, and no lock is held while it waits, so nothing queues behind it. The background key fetch is limited to one per issuer.
Every rejected federated token now gets the same message and takes 2 s, so the reply does not depend on which trust relationships exist. A key fetch waits at most 2 s and then continues in the background. The issuer check queries the database for every issuer. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
0f6b2eb to
b142d15
Compare
Linked Issue
None.
Describe Manual Test Plan
Every rejected federated token now gets the same message and takes 2 s, so the reply does not depend on which trust relationships exist. A key fetch for the issuer waits at most until the 2 s mark and then continues in the background. If the server has earlier keys for the issuer, a login uses them when the fetch is slow, as it does when the fetch fails. A new unit test sends seven kinds of rejected tokens at once, including one whose key fetch never completes, plus two during a database outage, and checks that the replies are identical. Undoing any one fix makes the test fail.
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes
None. A rejected federated login no longer says which check failed. The server log still has the reason.