← 输入框|github.com|刷新
Skip to content

Client certificates (mTLS) for the compiler, runner and pipeline servers; option to turn off Rust UDFs - #7684

Merged
gz merged 17 commits into
mainfrom
gz/mtls
Oct 29, 2026
Merged

gz merged 17 commits into
mainfrom
gz/mtls

Conversation

@gz

@gz gz commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Adds mutual TLS (mTLS) between Feldera components, and settings that let operators limit what pipelines may use.

Client certificates (mTLS)

With --require-client-certificates (FELDERA_REQUIRE_CLIENT_CERTIFICATES), the compiler, runner and pipeline servers request client certificates that the private CA signed. Each server identifies the caller by the URI SAN of its certificate and admits it only to its own routes:

Caller URI SAN Compiler Runner Pipeline <p>
Control plane (API server, runner, compiler replicas) urn:feldera:control-plane all routes all routes all routes
Pipeline <id> urn:feldera:pipeline:<id> its own binary and program info, /healthz /healthz all routes if <id> is <p> (the multihost coordinator), else none
Extra client, such as a sidecar a URI in --extra-client-certificate-uris /healthz /healthz all routes
No certificate none /healthz /healthz none

A caller without a certificate gets 401, any other refused caller 403. Routes that a server's table does not list are for the control plane only.

  • New crate feldera-mtls holds the certificate identities, the client certificate verifier and the 401/403 response. All servers share it.
  • --https-client-cert-path and --https-client-key-path set the certificate that the control plane's HTTP clients present. The clients read a renewed file within a minute, without a restart.
  • Pipelines read the switch from the environment, so pipeline binaries of older runtime versions ignore it. A pipeline verifies clients with the CA chain after the leaf in its own certificate file, so it needs no extra mount.
  • Works in FIPS builds (--features fips).

Settings

Setting Default Effect
--require-client-certificates, FELDERA_REQUIRE_CLIENT_CERTIFICATES off the table above
--https-client-cert-path, --https-client-key-path unset client certificate of the control plane
--extra-client-certificate-uris, FELDERA_EXTRA_CLIENT_CERTIFICATE_URIS (pipeline) empty comma-separated URI SANs of other clients that pipelines admit. Exact match, and nothing under urn:feldera:
--allow-rust-udfs, FELDERA_ALLOW_RUST_UDFS true with false, the API server refuses pipelines that set udf_rust or udf_toml, the compiler does not compile them and the runner does not start them. /v0/config returns allow_rust_udfs, and the web console shows empty UDF tabs read-only
reserved runtime_config.env names adds BASH_ENV, PATH, IFS, GLIBC_TUNABLES, OPENSSL_CONF and others, plus names that start with BASH_FUNC_ or LD_

Docs: the HTTPS guide (client certificates, external clients), the Helm chart reference (mtls.enabled, mtls.extraClientUris, caSecretRef, and the pipeline service account, environment and connector values), and the changelog.

Compatibility

  • Client certificates and the Rust UDF switch are off by default, so existing deployments behave as before.
  • The reserved environment names are an incompatible change: a pipeline that sets one fails to start until the variable is removed (changelog).

Testing

  • Unit tests: identities from certificates, the compiler and runner route tables (including percent-encoded paths), pipeline admission, extra client URIs, client certificate reload.
  • A TLS test against a real compiler server with no certificate, the control plane, the own pipeline, another pipeline, and a certificate from another CA.
  • The same tests with --features fips, which also check that the client certificate verifier uses a FIPS provider.
  • I disabled each new check once to confirm that its test fails.
  • End to end on Kubernetes with the Enterprise Helm chart, with client certificates on and off: compile, start, ingress, query, logs and a 2-host pipeline. The last three commits (route table move, extra client URIs, docs) are covered by unit tests only.

default_value_t = false,
env = "FELDERA_REQUIRE_CLIENT_CERTIFICATES"
)]
pub require_client_certificates: bool,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): This uses the same env name, FELDERA_REQUIRE_CLIENT_CERTIFICATES, that the pipeline binary reads. The local runner does not call env_clear() (local_runner.rs ~L1331), so a non-K8s manager started with this env var passes it to every pipeline it spawns. Each pipeline then panics at startup, either on "requires --enable-https" or on "the certificate names no pipeline", because the local runner's cert has no urn:feldera:pipeline:<id> SAN.

Also, canonicalize does not check that require_client_certificates comes with enable_https, private_ca_cert_path and https_client_cert_path. Without these, the compiler and runner panic on an assert! when they start the server. If only the client cert is missing, they start fine but answer every control-plane call with 401. Please validate this in canonicalize with an actionable error. Also either strip the variable in the local runner or reject the flag when the runner is not K8s.

let config = match (&self.https_client_cert_path, &self.https_client_key_path) {
(Some(cert_path), Some(key_path)) => builder.with_client_cert_resolver(Arc::new(
ClientCertificateFiles::load(cert_path, key_path)
.unwrap_or_else(|error| panic!("HTTPS client certificate: {error:#}")),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): reqwest_client() is built per request in several places, for example call_compiler_validate and the binary upload in rust_compiler.rs. Each call reloads the cert files and panics here if they are unreadable. cert-manager usually swaps the secret atomically, but a half-written or temporarily missing file would panic a request handler or the compiler loop. That contradicts the "keep the previous certificate" behaviour that ClientCertificateFiles::resolve implements. Consider building the resolver once, say lazily on CommonConfig, and sharing it, or log and fall back instead of panicking.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review summary: mTLS between components and the Rust UDF switch

What I ran (debug builds)

Command Result
cargo test --locked -p feldera-mtls 2/2 pass
cargo test --locked -p pipeline-manager -- client_certificate runner_route_table rust_udfs pipeline_env demos_without 15 pass, 1 fail
  • The compiler TLS test, the route tables, the cert reload, the UDF refusal in the API, compiler and runner, and the env blocklist all pass.
  • The one failure is gen2_sql_stage_leaves_rust_udfs_to_the_rust_compiler. It fetches the SQL compiler JAR over the network, which this sandbox lacks, and the PR does not touch that code. Unrelated.
  • Not run: cargo check -p feldera-adapters, web-console unit tests, pre-commit.

Missing tests (medium)

  1. The pipeline server has no TLS-level test. Nothing checks that Caller::record, Condition and authorize_pipeline_caller in crates/connectors/src/server.rs actually refuse an anonymous caller (401) or another pipeline (403) over a real handshake. Only PipelineClientAuth::is_allowed has a unit test, and the K8s run is manual. The compiler test harness could be reused for this.
  2. The runner server is covered only by its route table, not over TLS.
  3. There are no tests for config validation of --require-client-certificates without HTTPS, without a CA or without a client cert (see the inline comment).
  4. There is no web-console unit test for the locked UDF tabs when allow_rust_udfs is false.

New crate checklist (pitfall, medium)

  • feldera-mtls has publish = true, a description and a versioned root dependency, which is good.
  • Before the next release, create the crates.io entry and set up Trusted Publishing for both workflows. Otherwise the release publish fails, because feldera-adapters now depends on this crate.

Gates

  • Docs, changelog (including the incompatible env and service-account changes), and dependency licences (actix-tls and rustls-webpki are MIT/Apache) all pass.
  • No unsafe code and no workflow changes.

@gz
gz marked this pull request as ready for review October 28, 2026 19:04
Comment thread crates/pipeline-manager/src/config.rs Resolved
Comment thread crates/connectors/src/server.rs Resolved
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran (debug, --locked):

Command Result
cargo test -p feldera-mtls ✅ 2/2
cargo test -p pipeline-manager --lib (route tables, TLS handshake test, reload/shared cert, canonicalize, UDF gating, env blocklist, demos) ✅ 11/11
cargo check -p dbsp_connectors ✅
DB-backed tests (rust_udfs_refused_when_not_allowed, rust_udfs_refused_before_provisioning), FIPS build, web console ⏭️ not run

QA, extra tests I wrote (pass, not pushed): 401 + actionable body for an anonymous caller through feldera_mtls::authorize on plain HTTP, and /healthz still open; 403 status for CallerNotAllowed:

let resp = test::try_call_service(&app, TestRequest::get().uri("/x").to_request()).await;
assert_eq!(resp.err().unwrap().as_response_error().status_code(), StatusCode::UNAUTHORIZED);

Uncovered cases (missing, medium):

  • Pipeline server over TLS (see inline): own, other, anonymous and extra-URI callers, plus a malformed FELDERA_EXTRA_CLIENT_CERTIFICATE_URIS.
  • The 403 path for a pipeline cert on the runner over TLS. Only the route table is unit-tested.
  • A client cert from a sibling intermediate under the same root (see the inline comment on anchors).

Gates: tests ✅ (aside from the gaps above), docs ✅, changelog has the breaking env-name change ✅, no unsafe ✅, new deps actix-tls/rustls-webpki are MIT/Apache with features trimmed ✅. New crate feldera-mtls has publish = true: make sure the crates.io entry and Trusted Publishing are set up before the release, or ci-post-release fails.

gz added 3 commits October 28, 2026 15:45
Enterprise pipelines may select only Kubernetes service accounts that the operator lists in pipeline.allowedServiceAccountNames, and pipeline pods no longer get a Kubernetes API token or the service environment variables.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
runtime_config.env no longer accepts names that change how the shell, the dynamic loader or the C library start the pipeline process, such as BASH_ENV, PATH, and names that start with BASH_FUNC_ or LD_.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
…or Helm values.

Add pipeline.allowedServiceAccountNames, pipeline.allowEnv and pipeline.deniedConnectors to the Helm chart reference.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
@gz
gz added this pull request to the merge queue Oct 28, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 28, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: PR bug (high confidence). 12f8a9ab5 ("Add an option to turn off Rust UDFs") added page.data.feldera?.config.allow_rust_udfs in PipelineCodePanel.svelte, but the ?. only guards feldera, not config. The integration test's $app/state mock predates that field and doesn't set it, so config is undefined there.

Evidence

  • Failed job: Web Console Integration Tests
  • TypeError: Cannot read properties of undefined (reading 'allow_rust_udfs') at PipelineCodePanel.svelte:97, thrown while rendering PipelineEditLayout → PipelineCodePanel.
  • Both failures are in DeletedPipelineState.svelte.test.ts (log lines ~587, ~604), whose mock at lines 33-42 builds feldera without config.
  • TabAdHocQuery.svelte.test.ts has the same gap but passed in this run — it renders TabAdHocQuery directly, never PipelineCodePanel, so it never hit the missing field.
  • Confirmed config is always present on real layout data: buildFelderaData in +layout.ts sets it unconditionally (Configuration.allow_rust_udfs is a required field), so the mock was simply stale.

Fix pushed to fred/observe-7684-40784265377 (9206eec): add config: { allow_rust_udfs: true } to the test's mocked feldera object.

Take it with:

git fetch origin fred/observe-7684-40784265377 && git cherry-pick 9206eec1f

Compare: gz/mtls...fred/observe-7684-40784265377

gz added 11 commits October 28, 2026 17:19
With --allow-rust-udfs=false (FELDERA_ALLOW_RUST_UDFS=false), the API server refuses pipelines that set udf_rust or udf_toml, the compiler does not compile them, and the runner does not start them.  The API server also leaves out the demos that use Rust UDFs.  The default, true, keeps the current behavior.

/v0/config returns allow_rust_udfs.  When Rust UDFs are turned off, the web console keeps the udf.rs and udf.toml tabs, but an empty one is read-only and shows that the operator turned Rust UDFs off.  A pipeline that still has UDF code can edit it, so that its author can remove it.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
With --require-client-certificates, the compiler server accepts client certificates that the private CA signed and admits each caller only to its own routes. The control plane (URI SAN urn:feldera:control-plane) may use all routes. A pipeline (URI SAN urn:feldera:pipeline:<id>) may only download its own binary and program info. /healthz stays open for probes. Routes that the table does not list are for the control plane only.

The HTTPS clients present --https-client-cert-path to servers that request a client certificate, and read the file again at most once per minute so that a renewed certificate takes effect without a restart.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Describe the mtls.enabled and caSecretRef Helm values, the route table of the compiler server, and the restart of running pipelines after mtls.enabled is set.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
--require-client-certificates (FELDERA_REQUIRE_CLIENT_CERTIFICATES) moves to the common configuration and applies to the compiler and runner servers. The runner admits only the control plane (URI SAN urn:feldera:control-plane); /healthz stays open for probes. The client certificate code moves to the client_auth module, with one route table per server and its own error type.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
With mtls.enabled the runner server also requires client certificates, and pipelines may not call it.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
With FELDERA_REQUIRE_CLIENT_CERTIFICATES, a pipeline admits only the control plane (URI SAN urn:feldera:control-plane) and its own pipeline identity, which its multihost coordinator presents. It verifies clients with the CA chain that follows the leaf in its own certificate file, so it needs no separate CA file. An environment variable, not a flag, so that pipeline binaries of older runtime versions ignore it.

The client certificate identities, the verifier and the 401/403 refusal move into the new feldera-mtls crate, which the compiler, runner and pipeline servers share.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
With mtls.enabled, pipelines and their coordinator also require client certificates; clients that reach a pipeline directly, instead of through the API server, are refused.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
… crypto provider.

With several rustls crypto features compiled in, rustls has no default provider unless the process installed one, and building the verifier panicked. Use the installed default, else aws-lc-rs.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
…r servers.

The route tables of the compiler and runner servers move from client_auth to compiler::main and runner::main, and the TLS configuration that verifies client certificates to CommonConfig::internal_https_server_config. New tests cover percent-encoded paths. feldera_mtls::crypto_provider replaces two copies of the crypto provider fallback, and the doc comments of the client certificate options are shorter.

feldera_mtls::client_certificate_verifier builds the verifier of every server and trusts only the first certificate of the CA chain, the private CA, so the root may also sign certificates for other uses.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
--extra-client-certificate-uris (FELDERA_EXTRA_CLIENT_CERTIFICATE_URIS) lists the URI SANs of clients outside Feldera, such as sidecars, that a pipeline and its coordinator admit like the control plane. A certificate with exactly one URI SAN outside urn:feldera: identifies such a client. Listed URIs must match exactly and lie outside urn:feldera:. The compiler and runner servers do not admit them.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Add mtls.extraClientUris to the Helm chart reference, and an example with a cert-manager Certificate to the HTTPS guide.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
gz added 3 commits October 28, 2026 17:19
…lines.

The local runner passes its environment to the pipelines it starts. A manager started with FELDERA_REQUIRE_CLIENT_CERTIFICATES would make every local pipeline require client certificates, which fails because local pipelines share the manager's certificate and it names no pipeline. The local runner now removes FELDERA_REQUIRE_CLIENT_CERTIFICATES and FELDERA_EXTRA_CLIENT_CERTIFICATE_URIS, whose names feldera-mtls now exports.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
…hare client certificates.

canonicalize refuses --require-client-certificates without --enable-https or --private-ca-cert-path, and loads the --https-client-cert-path files, so a wrong setting stops the process with an error instead of a panic when a server starts. HTTPS clients built from the same files share one client certificate resolver, so a client built per request keeps the last certificate that loaded instead of reading the files again.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
@gz
gz enabled auto-merge October 29, 2026 00:20
@gz
gz disabled auto-merge October 29, 2026 00:22
@gz
gz added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit 36b8029 Oct 29, 2026
1 check passed
@gz
gz deleted the gz/mtls branch October 29, 2026 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants