Ray integration tests are broken on master: list-typed features lose object dtype and Ray materialization writes nothing to the online store #6912

Problem

The pr-ray-integration-tests workflow (integration-test-ray job) is failing on master for every PR, regardless of what the PR changes. 12 failed, 32 passed, 4 skipped.

Example run: https://github.com/feast-dev/feast/actions/runs/36757976337/job/110032974407 (on PR #6909, which only touches sdk/python/feast/feature_store.py, sdk/python/feast/utils.py and a new unit test — nothing Ray-related).

The same job is failing on unrelated branches on the same day, which confirms it is a master-level breakage rather than a per-PR regression:

• fix/offline-server-bounded-batches
• fix/sdk-dual-stack-binds
• perf/decouple-metrics-server

All failures are on the single LOCAL:Ray:sqlite:python_fs:False configuration.

Failure 1 — list-typed features lose object dtype (10 cases)

sdk/python/tests/integration/offline_store/test_universal_types.py::test_feature_get_historical_features_types_match

sdk/python/tests/integration/offline_store/test_universal_types.py:202: in assert_feature_list_types
    assert pd.api.types.is_object_dtype(historical_features_df.dtypes["value"])
E   AssertionError: assert False

Failing parametrizations — every feature_is_list=True case:

feature_is_list=False cases pass. So get_historical_features() on the Ray offline store now returns list columns as a native Arrow/pandas list dtype instead of object.

Failure 2 — Ray materialization writes nothing to the online store (2 cases)

sdk/python/tests/integration/offline_store/test_universal_materialization.py::test_universal_materialization_consistency (both True and False parametrizations)

sdk/python/tests/integration/offline_store/test_universal_materialization.py:39: in test_universal_materialization_consistency
    validate_offline_online_store_consistency(fs, driver_stats_fv, split_dt)
sdk/python/tests/utils/e2e_test_validation.py:52: in validate_offline_online_store_consistency
    _check_offline_and_online_features(
sdk/python/tests/utils/e2e_test_validation.py:162: in _check_offline_and_online_features
    assert not df.to_dict(orient="list")[f"{fv.name}__value"] or math.isnan(
E   TypeError: must be real number, not NoneType

The online read returns None, i.e. the Ray compute engine materialization did not land rows in the sqlite online store. The TypeError is a secondary symptom — e2e_test_validation.py:162 also doesn't guard against None before calling math.isnan, so the real assertion never gets a chance to report.

Environment (from the CI run, pixi -e ray-tests)

• ray 2.58.0
• pandas 2.3.3
• pyarrow 25.0.1

Given the dtype nature of failure 1, a recent Ray / pyarrow / pandas bump in the ray-tests pixi environment is the most likely trigger.

Expected

integration-test-ray is green on master so it is a meaningful signal on PRs.

Suggested work

1. Find the commit / lockfile bump that first turned this job red and confirm the trigger.
2. Fix the Ray offline store list-column conversion so list features come back as object dtype (or update the universal type contract deliberately, if the new dtype is intended).
3. Fix Ray compute engine materialization so rows actually reach the online store.
4. Harden _check_offline_and_online_features in sdk/python/tests/utils/e2e_test_validation.py:162 to handle None so a missing online value fails with a clear assertion instead of a TypeError.