← 输入框|github.com|刷新
Skip to content

[connectors,dbsp] Use neutral names in tests that copied a reported workload - #7950

Merged
bestouff merged 2 commits into
mainfrom
neutral-delta-test-columns
Nov 12, 2026
Merged

bestouff merged 2 commits into
mainfrom
neutral-delta-test-columns

Conversation

@bestouff

@bestouff bestouff commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

The nested after struct of the Delta column-mapping tests used column names copied from a table in a problem report. The tests need only the structure of that table: a nested struct whose children are column-mapped. This PR gives the children neutral names and removes the origin from the comments.

Old name New name
transaction__id item_id
transaction__merchant_id vendor_id
transaction__merchant_name vendor_name
transaction__time event_time
transaction__status status
transaction__amount amount

The fixture versions are bumped, because a cached fixture is found by its path alone.

The second commit changes one comment in crates/dbsp/src/operator/dynamic/balance/balancer.rs. The comment named the pipeline and the join that the test costs came from. It now describes a cluster of 25 streams and a skewed join. No code changes.

Action before merge

The Unity Catalog fixture is a table that someone makes by hand. test_delta_input_uniform_unity.py fails in an environment that still points at the old table. To update it, from the python directory:

source ~/.feldera-dbx-test.env
.venv/bin/python -m tests.platform.fixtures.unity_uniform

Then set DELTA_TABLE_TEST_UNITY_UNIFORM_TABLE to the uniform_iceberg_v2 name that the script prints.

Testing

  • cargo test -p dbsp_connectors --lib project_to_logical_matches_by_field_id passes.
  • fixtures/uniform_iceberg.py builds the table locally, and the Parquet schema has the new names.
  • The platform tests were not run locally. They need a Feldera instance, Spark, and for the Unity test a Databricks workspace.

The nested `after` struct of the column-mapping tests used column names
copied from a table in a problem report. The tests need only the
structure of that table: a nested struct whose children are column-mapped.
The children now have neutral names (`item_id`, `vendor_id`,
`vendor_name`, `event_time`, `status`, `amount`), and the comments
describe the shape without its origin.

The fixture versions are bumped, because a cached fixture is found by
its path alone. The Unity Catalog fixture is a table that someone makes
by hand: run `tests.platform.fixtures.unity_uniform` again and point
DELTA_TABLE_TEST_UNITY_UNIFORM_TABLE at `uniform_iceberg_v2`.

Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. This PR only renames test fixtures. The fixture versions are bumped and no transaction__* names are left.

Ran Result
cargo test -p dbsp_connectors --lib project_to_logical_matches_by_field_id pass
pre-commit run --files <changed> pass
fixtures/uniform_iceberg.py <tmp> + pq.read_schema Parquet has item_id…amount with field ids 104–109
platform tests (need Feldera/Spark/Databricks) not run
  • pitfall (medium): test_delta_input_uniform_unity.py doesn't check that DELTA_TABLE_TEST_UNITY_UNIFORM_TABLE points at fixture.TABLE. Until someone rebuilds the table, an environment still set to uniform_iceberg_v1 fails with a SQL column-not-found error and no hint to rebuild. A one-line check in require_id_mapped_table would make that failure say what to do: assert table.endswith(fixture.TABLE).
  • missing (low, same goal as this PR): the problem-report wording is still in fixtures/unity_deletion_vectors.py:28, fixtures/unity_change_feed.py:328 and delta_table/input.rs:316.

The comment of gate_measures_only_the_collections_it_changes named the
pipeline and the join that the costs came from. The test needs only the
costs, so the comment now describes a cluster of 25 streams and a skewed
join.

Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
@bestouff bestouff changed the title [connectors] Use neutral column names in the Delta column-mapping tests [connectors,dbsp] Use neutral names in tests that copied a reported workload Nov 12, 2026
@bestouff
bestouff added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 7044da0 Nov 12, 2026
1 check passed
@bestouff
bestouff deleted the neutral-delta-test-columns branch November 12, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants