Repository navigation
Conversation
| default_value_t = false, | ||
| env = "FELDERA_REQUIRE_CLIENT_CERTIFICATES" | ||
| )] | ||
| pub require_client_certificates: bool, |
There was a problem hiding this comment.
pitfall (medium): This uses the same env name, FELDERA_REQUIRE_CLIENT_CERTIFICATES, that the pipeline binary reads. The local runner does not call env_clear() (local_runner.rs ~L1331), so a non-K8s manager started with this env var passes it to every pipeline it spawns. Each pipeline then panics at startup, either on "requires --enable-https" or on "the certificate names no pipeline", because the local runner's cert has no urn:feldera:pipeline:<id> SAN.
Also, canonicalize does not check that require_client_certificates comes with enable_https, private_ca_cert_path and https_client_cert_path. Without these, the compiler and runner panic on an assert! when they start the server. If only the client cert is missing, they start fine but answer every control-plane call with 401. Please validate this in canonicalize with an actionable error. Also either strip the variable in the local runner or reject the flag when the runner is not K8s.
| let config = match (&self.https_client_cert_path, &self.https_client_key_path) { | ||
| (Some(cert_path), Some(key_path)) => builder.with_client_cert_resolver(Arc::new( | ||
| ClientCertificateFiles::load(cert_path, key_path) | ||
| .unwrap_or_else(|error| panic!("HTTPS client certificate: {error:#}")), |
There was a problem hiding this comment.
pitfall (medium): reqwest_client() is built per request in several places, for example call_compiler_validate and the binary upload in rust_compiler.rs. Each call reloads the cert files and panics here if they are unreadable. cert-manager usually swaps the secret atomically, but a half-written or temporarily missing file would panic a request handler or the compiler loop. That contradicts the "keep the previous certificate" behaviour that ClientCertificateFiles::resolve implements. Consider building the resolver once, say lazily on CommonConfig, and sharing it, or log and fall back instead of panicking.
|
Review summary: mTLS between components and the Rust UDF switch What I ran (debug builds)
Missing tests (medium)
New crate checklist (pitfall, medium)
Gates
|
|
What I ran (debug,
QA, extra tests I wrote (pass, not pushed): 401 + actionable body for an anonymous caller through let resp = test::try_call_service(&app, TestRequest::get().uri("/x").to_request()).await;
assert_eq!(resp.err().unwrap().as_response_error().status_code(), StatusCode::UNAUTHORIZED);Uncovered cases (missing, medium):
Gates: tests ✅ (aside from the gaps above), docs ✅, changelog has the breaking env-name change ✅, no |
Enterprise pipelines may select only Kubernetes service accounts that the operator lists in pipeline.allowedServiceAccountNames, and pipeline pods no longer get a Kubernetes API token or the service environment variables. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
runtime_config.env no longer accepts names that change how the shell, the dynamic loader or the C library start the pipeline process, such as BASH_ENV, PATH, and names that start with BASH_FUNC_ or LD_. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
…or Helm values. Add pipeline.allowedServiceAccountNames, pipeline.allowEnv and pipeline.deniedConnectors to the Helm chart reference. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
|
Verdict: PR bug (high confidence). Evidence
Fix pushed to Take it with: |
With --allow-rust-udfs=false (FELDERA_ALLOW_RUST_UDFS=false), the API server refuses pipelines that set udf_rust or udf_toml, the compiler does not compile them, and the runner does not start them. The API server also leaves out the demos that use Rust UDFs. The default, true, keeps the current behavior. /v0/config returns allow_rust_udfs. When Rust UDFs are turned off, the web console keeps the udf.rs and udf.toml tabs, but an empty one is read-only and shows that the operator turned Rust UDFs off. A pipeline that still has UDF code can edit it, so that its author can remove it. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
With --require-client-certificates, the compiler server accepts client certificates that the private CA signed and admits each caller only to its own routes. The control plane (URI SAN urn:feldera:control-plane) may use all routes. A pipeline (URI SAN urn:feldera:pipeline:<id>) may only download its own binary and program info. /healthz stays open for probes. Routes that the table does not list are for the control plane only. The HTTPS clients present --https-client-cert-path to servers that request a client certificate, and read the file again at most once per minute so that a renewed certificate takes effect without a restart. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Describe the mtls.enabled and caSecretRef Helm values, the route table of the compiler server, and the restart of running pipelines after mtls.enabled is set. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
--require-client-certificates (FELDERA_REQUIRE_CLIENT_CERTIFICATES) moves to the common configuration and applies to the compiler and runner servers. The runner admits only the control plane (URI SAN urn:feldera:control-plane); /healthz stays open for probes. The client certificate code moves to the client_auth module, with one route table per server and its own error type. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
With mtls.enabled the runner server also requires client certificates, and pipelines may not call it. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
With FELDERA_REQUIRE_CLIENT_CERTIFICATES, a pipeline admits only the control plane (URI SAN urn:feldera:control-plane) and its own pipeline identity, which its multihost coordinator presents. It verifies clients with the CA chain that follows the leaf in its own certificate file, so it needs no separate CA file. An environment variable, not a flag, so that pipeline binaries of older runtime versions ignore it. The client certificate identities, the verifier and the 401/403 refusal move into the new feldera-mtls crate, which the compiler, runner and pipeline servers share. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
With mtls.enabled, pipelines and their coordinator also require client certificates; clients that reach a pipeline directly, instead of through the API server, are refused. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
… crypto provider. With several rustls crypto features compiled in, rustls has no default provider unless the process installed one, and building the verifier panicked. Use the installed default, else aws-lc-rs. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
…r servers. The route tables of the compiler and runner servers move from client_auth to compiler::main and runner::main, and the TLS configuration that verifies client certificates to CommonConfig::internal_https_server_config. New tests cover percent-encoded paths. feldera_mtls::crypto_provider replaces two copies of the crypto provider fallback, and the doc comments of the client certificate options are shorter. feldera_mtls::client_certificate_verifier builds the verifier of every server and trusts only the first certificate of the CA chain, the private CA, so the root may also sign certificates for other uses. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
--extra-client-certificate-uris (FELDERA_EXTRA_CLIENT_CERTIFICATE_URIS) lists the URI SANs of clients outside Feldera, such as sidecars, that a pipeline and its coordinator admit like the control plane. A certificate with exactly one URI SAN outside urn:feldera: identifies such a client. Listed URIs must match exactly and lie outside urn:feldera:. The compiler and runner servers do not admit them. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Add mtls.extraClientUris to the Helm chart reference, and an example with a cert-manager Certificate to the HTTPS guide. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
…lines. The local runner passes its environment to the pipelines it starts. A manager started with FELDERA_REQUIRE_CLIENT_CERTIFICATES would make every local pipeline require client certificates, which fails because local pipelines share the manager's certificate and it names no pipeline. The local runner now removes FELDERA_REQUIRE_CLIENT_CERTIFICATES and FELDERA_EXTRA_CLIENT_CERTIFICATE_URIS, whose names feldera-mtls now exports. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
…hare client certificates. canonicalize refuses --require-client-certificates without --enable-https or --private-ca-cert-path, and loads the --https-client-cert-path files, so a wrong setting stops the process with an error instead of a panic when a server starts. HTTPS clients built from the same files share one client certificate resolver, so a client built per request keeps the last certificate that loaded instead of reading the files again. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Adds mutual TLS (mTLS) between Feldera components, and settings that let operators limit what pipelines may use.
Client certificates (mTLS)
With
--require-client-certificates(FELDERA_REQUIRE_CLIENT_CERTIFICATES), the compiler, runner and pipeline servers request client certificates that the private CA signed. Each server identifies the caller by the URI SAN of its certificate and admits it only to its own routes:<p>urn:feldera:control-plane<id>urn:feldera:pipeline:<id>/healthz/healthz<id>is<p>(the multihost coordinator), else none--extra-client-certificate-uris/healthz/healthz/healthz/healthzA caller without a certificate gets 401, any other refused caller 403. Routes that a server's table does not list are for the control plane only.
feldera-mtlsholds the certificate identities, the client certificate verifier and the 401/403 response. All servers share it.--https-client-cert-pathand--https-client-key-pathset the certificate that the control plane's HTTP clients present. The clients read a renewed file within a minute, without a restart.--features fips).Settings
--require-client-certificates,FELDERA_REQUIRE_CLIENT_CERTIFICATES--https-client-cert-path,--https-client-key-path--extra-client-certificate-uris,FELDERA_EXTRA_CLIENT_CERTIFICATE_URIS(pipeline)urn:feldera:--allow-rust-udfs,FELDERA_ALLOW_RUST_UDFStruefalse, the API server refuses pipelines that setudf_rustorudf_toml, the compiler does not compile them and the runner does not start them./v0/configreturnsallow_rust_udfs, and the web console shows empty UDF tabs read-onlyruntime_config.envnamesBASH_ENV,PATH,IFS,GLIBC_TUNABLES,OPENSSL_CONFand others, plus names that start withBASH_FUNC_orLD_Docs: the HTTPS guide (client certificates, external clients), the Helm chart reference (
mtls.enabled,mtls.extraClientUris,caSecretRef, and the pipeline service account, environment and connector values), and the changelog.Compatibility
Testing
--features fips, which also check that the client certificate verifier uses a FIPS provider.