← 输入框|github.com|刷新
Skip to content

[pipeline-manager] Add GET /pipelines/{name}/deployment - #7490

Merged
xiasongh merged 5 commits into
mainfrom
song/pipeline-deployment-endpoint-main
Nov 12, 2026
Merged

xiasongh merged 5 commits into
mainfrom
song/pipeline-deployment-endpoint-main

Conversation

@xiasongh

@xiasongh xiasongh commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Read-only endpoint that shows what the runner's platform has for a pipeline now: desired and current CPU, memory, and storage per host, with a state, reason, and message for each.

  • States and aggregation follow Kubernetes kstatus.
  • Proxied to the runner like /logs. The default executor returns the spec only. The Kubernetes runner (feldera/cloud#2088) fills in the status.

Endpoint

GET /v0/pipelines/{pipeline_name}/deployment

Status When Body
200 Always, for an existing pipeline PipelineDeployment
404 Unknown pipeline ErrorResponse
503 Runner timeout, or a runner older than the API server (no /deployment route) ErrorResponse
500 Other errors ErrorResponse

Schema

PipelineDeployment
├── spec: DeploymentSpec
│   └── resources: DeploymentSpecResources
└── status?: DeploymentStatus                (absent if the runner does not observe)
    ├── state: DeploymentState               (summary of all hosts)
    ├── reason?: string
    ├── message?: string
    └── hosts: HostStatus[]
        ├── ordinal: integer
        ├── compute: HostCompute
        │   ├── state, reason?, message?, last_transition_time?   (Transition, flattened)
        │   ├── manager: ResourceManager
        │   ├── desired?: ComputeResources
        │   └── current?: ComputeResources
        └── storage?: HostStorage                                 (absent without storage)
            ├── state, reason?, message?, last_transition_time?   (Transition, flattened)
            ├── manager: ResourceManager
            ├── desired?: StorageResources
            └── current?: StorageResources

? marks a field that can be absent or null.

DeploymentSpecResources

Copied from the pipeline's deployment_config.resources. Each *_min is a Kubernetes request and each *_max a limit.

Field Type Example
cpu_cores_min number? 1
cpu_cores_max number? 2
memory_min size string? "1G"
memory_max size string? "2G"
storage_min size string? "10G"
storage_max size string? "100G"

Transition fields (flattened into compute and storage)

Field Type Meaning
state DeploymentState kstatus state of the object
reason string? Machine-readable cause, passed through from the platform, e.g. PodResizePending, VolumeNotExpanded
message string? Human-readable detail
last_transition_time RFC 3339? When the platform last changed the condition behind state (Kubernetes lastTransitionTime)

ComputeResources and StorageResources

Normalized units, so desired and current compare directly.

Type Field Type
ComputeResources cpu_cores_min, cpu_cores_max number?
memory_mb_min, memory_mb_max integer?
StorageResources storage_mb integer?

Enums

DeploymentState Meaning
Current Matches its desired state
InProgress On its way to the desired state
Failed Will not reach the desired state without a change
Terminating Being deleted
NotFound Does not exist
Unknown Cannot be determined
ResourceManager Meaning
User The pipeline's configuration sets desired
Autoscaler The runner's autoscaler sets desired, within the configured bounds

Top-level status.state

Like kstatus AggregateStatus. Each object is compared with the state the pipeline lifecycle expects for it:

Lifecycle Compute expected Storage expected
Resources Provisioned Current
Resources Stopped NotFound
Storage InUse Current
Storage Clearing or Cleared NotFound
First rule that matches status.state reason, message from
Any object Failed Failed That object
Any object Unknown Unknown That object
Any object not in its expected state InProgress That object
Otherwise NotFound if nothing is expected to exist, else Current (none)

Example

A running single-host pipeline during an in-place CPU scale-down (Kubernetes runner):

{
  "spec": {
    "resources": {
      "cpu_cores_min": 1, "cpu_cores_max": 2,
      "memory_min": "4G", "memory_max": "8G",
      "storage_min": "10G", "storage_max": null
    }
  },
  "status": {
    "state": "InProgress",
    "reason": "InProgress",
    "hosts": [
      {
        "ordinal": 0,
        "compute": {
          "state": "InProgress",
          "reason": "InProgress",
          "last_transition_time": "2026-11-03T19:40:00Z",
          "manager": "User",
          "desired": {"cpu_cores_min": 1, "cpu_cores_max": 2, "memory_mb_min": 4000, "memory_mb_max": 8000},
          "current": {"cpu_cores_min": 4, "cpu_cores_max": 8, "memory_mb_min": 4000, "memory_mb_max": 8000}
        },
        "storage": {
          "state": "Current",
          "message": "PVC is Bound",
          "manager": "User",
          "desired": {"storage_mb": 10000},
          "current": {"storage_mb": 10000}
        }
      }
    ]
  }
}

With the default executor the response is {"spec": {...}} only.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test --locked -p feldera-types deployment ✅ 8/8 (kstatus aggregation)
cargo test --locked -p pipeline-manager deployment ✅ 6/6, including deployment_spec
cargo test --locked -p pipeline-manager rbac ✅ 18/18 (the new route has an RBAC entry)
pre-commit run --files … ⏱ timed out locally, not verified

Gates

  • missing (medium), Hard Rule 3, integration tests: the change crosses the manager→runner boundary, but nothing exercises it end to end. No test covers the proxy, the 400 when the deployment unstable feature is off, a 404 for an unknown pipeline, or the spec-only body from the default executor. A small python/tests/platform test, run with the feature enabled, would cover the happy path and the 404.
  • missing (medium), commit quality: commit 3 ("Name the GET /deployment response schema…") fixes commit 1. Please squash it into commit 1 before merge to keep history linear and clean.

Uncovered cases (see the inline comments for the details)

  1. Mapping from lifecycle to expected state (ResourcesDesiredStatus/StorageStatus → DeploymentState): untested. I posted a test diff inline.
  2. Version skew (new manager, old runner): the runner's bare 404 is passed through to the client as if the pipeline did not exist.
  3. deployment_spec with malformed resources: it silently falls back to all-None. Nothing tests that, and nothing documents it.

Docs: the endpoint is behind an unstable flag and documented through OpenAPI. That is fine for now, but it needs a docs page before the flag is removed.

@xiasongh
xiasongh force-pushed the song/pipeline-deployment-endpoint-main branch 3 times, most recently from c2f42ed to 06fef83 Compare October 16, 2026 04:20
@xiasongh
xiasongh force-pushed the song/pipeline-deployment-endpoint-main branch from 06fef83 to ff54511 Compare October 26, 2026 12:29
@xiasongh
xiasongh force-pushed the song/pipeline-deployment-endpoint-main branch from ff54511 to 5043342 Compare November 3, 2026 19:29
@xiasongh
xiasongh marked this pull request as ready for review November 3, 2026 19:29
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review: GET /pipelines/{name}/deployment

What I ran:

Check Result
cargo test -p feldera-types deployment (8 aggregate/serde tests) ✅ pass
cargo test -p pipeline-manager --lib with filters expected_states, deployment_spec, missing_route ✅ pass
rustfmt --check on the changed Rust files ✅ clean
python/tests/platform/test_pipeline_deployment.py not run (needs the compiler and manager running)

What looks good: the runner route sits behind the mTLS authorize_runner_caller middleware, RBAC is Read, the old-runner 404 is told apart from an unknown-pipeline 404, and the error message tells the user what to do.

Test cases the PR does not cover:

  • missing (medium): a pipeline with explicit resources. See the inline comment. Today the platform test passes trivially on the default executor.
  • missing (medium): a runner-side actix test for the get_deployment handler. It should cover an invalid UUID (400), a tenant/pipeline mismatch (404 ErrorResponse forwarded unchanged through the proxy), and a stopped pipeline with cleared storage.
  • missing (low→medium): a proxy test where the runner returns a non-2xx ErrorResponse, checking that the status and Content-Type pass through.

Gates: no unsafe, no new dependencies, no workflow changes, and the API addition is backwards compatible. The endpoint is documented only through the OpenAPI rustdoc. That is acceptable for a read-only debugging endpoint, but a line in the operations docs about how to read state/reason would help users.

@xiasongh
xiasongh force-pushed the song/pipeline-deployment-endpoint-main branch from 7ad60c0 to 8ced31e Compare November 3, 2026 21:01
@xiasongh

xiasongh commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Replies to the review summaries:

Item Status
Platform test: happy path and 404 Done: test_pipeline_deployment, test_pipeline_deployment_unknown_pipeline
Platform test with explicit resources Done
Squash commits Done: 1 commit
expected_states table test Done: expected_states_follow_the_lifecycle
Version skew (old runner) Done: 503 with an upgrade message
Malformed resources in deployment_spec Done: documented and tested
Proxy test for a non-2xx ErrorResponse Done: deployment_proxy_passes_the_runner_response_through
Runner-side actix test for get_deployment Not done. It needs embedded Postgres. The proxy test covers pass-through, and expected_states_follow_the_lifecycle covers the lifecycle mapping.
Docs Schema is in the PR description. A docs page comes before the endpoint is stable.

Comment thread crates/feldera-types/src/deployment.rs Resolved
Comment thread crates/feldera-types/src/deployment.rs Resolved
Comment thread crates/feldera-types/src/deployment.rs Resolved
Comment thread crates/feldera-types/src/deployment.rs Resolved
Comment thread crates/feldera-types/src/deployment.rs Resolved
Comment thread crates/feldera-types/src/deployment.rs Resolved
Desired and current CPU, memory, and storage per host, with kstatus states.
The default executor returns the spec only.
@xiasongh
xiasongh force-pushed the song/pipeline-deployment-endpoint-main branch from 8ced31e to e91d147 Compare November 12, 2026 20:52
Signed-off-by: feldera-bot <feldera-bot@feldera.com>
@xiasongh
xiasongh added this pull request to the merge queue Nov 12, 2026
@xiasongh
xiasongh removed this pull request from the merge queue due to a manual request Nov 12, 2026
@xiasongh
xiasongh enabled auto-merge November 12, 2026 21:13
@xiasongh
xiasongh disabled auto-merge November 12, 2026 21:38
@xiasongh
xiasongh added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@xiasongh
xiasongh added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
The docs site compiles OpenAPI descriptions as MDX, where <https://...> is a tag.
@xiasongh
xiasongh enabled auto-merge November 12, 2026 21:55
@xiasongh
xiasongh disabled auto-merge November 12, 2026 22:00
@xiasongh
xiasongh added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@xiasongh
xiasongh added this pull request to the merge queue Nov 12, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: infrastructure flake, not this PR. Confidence: high.

Opened #7961 to track this occurrence (new data point: longest recorded compile yet, 1133.8s, ending in a 503 rather than a timeout).

Next step: re-queueing should pass — nothing to fix in this PR.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@xiasongh
xiasongh added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 8e901d0 Nov 12, 2026
1 check passed
@xiasongh
xiasongh deleted the song/pipeline-deployment-endpoint-main branch November 12, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants