Repository navigation
Support for JSON output with index properties - #7752
Conversation
Strix Security Review✅ No open security findings. PR summary: Adds a JSON output encoder path for connectors attached to a view index: Debezium-format change events keyed by the index columns, `u` events with `before`/`after`, optional tombstones after deletes (`tombstones_on_delete`), the Debezium value schema now declaring `before`, CSV rejecting `index`, and a refactor of the profiler support-bundle SQL panel. Review summaryReviewed the new indexed JSON/Debezium encoder path ( Ruled out (6)
Review 1 on this PR · Reviewed by Strix |
|
Review: JSON Debezium output for What I ran:
The Kafka tests in the same filter failed only because no broker was running in my environment, so that failure is not from this PR. Gates:
Cases the PR's tests don't cover:
|
anandbraman
left a comment
There was a problem hiding this comment.
Tested with sample data locally and everything worked well. Thank you!
…ormat With an index, the JSON encoder writes one Debezium event for each key that a step changes: `c` with `after` for an insert, `u` with `before` and `after` for an update, and `d` with `before` for a delete. The message key holds the index columns. A tombstone follows each `d`, unless `tombstones_on_delete` is false. The output without an index does not change. Fixes #7723 Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Before, the CSV encoder ignored the index and wrote each update as a delete and an insert. The controller tests of aliased indexes now attach JSON Debezium output to the index, on an in-memory endpoint that keeps keyed messages. Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
The size check of the JSON encoder did not count the message key, the `\n` after the last record, or the `]\n` after an array. When a record did not fit, the encoder wrote `\n` again, so the message ended in an empty line. A message could be larger than the transport's limit by the size of its key plus two bytes. Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
… JSON output A Debezium delete event holds the deleted row in `before`, but the schema that the `kafka_connect_json_converter` flavor writes declared only `after` and `op`. The Kafka Connect JsonConverter builds its record from the schema, so a delete reached Kafka Connect without the row. The value schema now declares `before`, as the Debezium envelope does for every event. Fixes #7750 Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
A Kafka output connector with exactly-once fault tolerance does not write tombstones. JSON `debezium` output with an index writes a tombstone after each `d` event, unless `tombstones_on_delete` is false, so such a pipeline must set it to false. Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Since exactly-once Kafka output accepts keyed messages, Avro output with an index works in that mode. Only the `confluent_jdbc` update format cannot write its deletions, which are tombstones. Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Linked Issue
Fixes #7723
Fixes #7750
There are 4 separate commits, which should work independently.
Checklist