Redshift push ignores schema #3651

Expected Behavior

When writing to a Redshift push source, the table is written to under the correct schema.

Current Behavior

Even if schema is specified in the data source, data is pushed to a table in the default schema.

You can trace this through the code:

• RedshiftOfflineStore.offline_write_batch(...)
• aws_utils.upload_arrow_table_to_redshift(...)

Steps to reproduce

• define a redshiftsource where the schema is set to something other than the default schema
• define a pushsource
• push some data to it
• observe a new table getting created under the default schema

abc_source = RedshiftSource(
    name="abc",
    table="abc",
    timestamp_field="event_timestamp",
    created_timestamp_column="proctime",
)

abc_push_source = PushSource(
    name="abc_push_source", batch_source=abc_source
)

# ... push some data in

Specifications

• Version: v0.31.1
• Platform: Linux x64
• Subsystem:

Possible Solution

In offline_write_batch, construct a fully-qualified table name using the schema in the options, if present.