'feast apply' does not create online store tables in remote mode

Expected Behavior

With online_store.type: remote + registry.registry_type: remote, a client-side feast apply should create the corresponding feature view tables in the server-side online store, so that feast materialize can load data successfully — i.e. the schema setup done by SqliteOnlineStore.update() during a local feast apply should also take effect on the feature-server side when feature views are applied through the remote registry.

Current Behavior

feast apply (remote) succeeds and registers the feature view, but no table is created server-side:

After feast apply, online_store.db does not exist (only registry_sqlite.db is present).
feast materialize reads offline data on the client, POSTs batches to the feature server's /write-to-online-store, and gets HTTP 500. The materialize command fails:

requests.exceptions.RetryError: HTTPConnectionPool(host='localhost', port=6566): Max retries exceeded with url: /write-to-online-store (Caused by ResponseError('too many 500 error responses'))

Feature server log shows the root cause:

File ".../feast/feature_store.py", line 3126, in write_to_online_store
    provider.ingest_df(feature_view, df)
File ".../feast/infra/passthrough_provider.py", line 197, in online_write_batch
    self.online_store.online_write_batch(config, table, data, progress)
File ".../feast/infra/online_stores/sqlite.py", line 220, in online_write_batch
    conn.execute(
sqlite3.OperationalError: no such table: test_project_driver_hourly_stats

After the failed materialize, online_store.db exists but is 0 bytes with zero tables.

Root cause (in source):

RemoteOnlineStore.update() is a no-op (pass): feast/infra/online_stores/remote.py L706-715. feast apply relies on OnlineStore.update() to create tables, so remote apply never creates them on the server-side online store.
feast serve does not call SqliteOnlineStore.update() for newly-applied feature views, so the table is never created server-side either.
SqliteOnlineStore.online_write_batch() only INSERTs (no CREATE TABLE): feast/infra/online_stores/sqlite.py L152-239. Table creation happens only in update() (L291-315), which is never invoked in this flow.

Net effect: the remote online store path is unusable for materialization unless the user manually runs feast apply locally on the feature-server repo, defeating the purpose of the remote architecture.

Steps to reproduce

Setup: a server repo (test_project/feature_repo) hosting registry + online store, and a client repo (test_demo) using remote registry + remote online store.

Server repo test_project/feature_repo/feature_store.yaml:

project: test_project
registry:
  registry_type: sql
  path: sqlite:///data/registry_sqlite.db
provider: local
online_store:
    type: sqlite
    path: data/online_store.db
entity_key_serialization_version: 3
auth:
    type: no_auth

Client repo test_demo/feature_store.yaml (remote registry + remote online store):

project: test_project
registry:
  registry_type: remote
  path: localhost:6570          # feast serve_registry (gRPC)
provider: local
online_store:
    type: remote
    path: http://localhost:6566  # feast serve (HTTP feature server)
entity_key_serialization_version: 3
auth:
    type: no_auth

Client repo test_demo/feature_definitions.py (a single simple FeatureView):

from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource, Project
from feast.types import Float32, Int64

project = Project(name="test_project", description="A project for driver statistics")
driver = Entity(name="driver", join_keys=["driver_id"])

driver_stats_source = FileSource(
    name="driver_hourly_stats_source",
    path="data/driver_stats.parquet",
    timestamp_field="event_timestamp",
    created_timestamp_column="created",
)

driver_stats_fv = FeatureView(
    name="driver_hourly_stats",
    entities=[driver],
    ttl=timedelta(days=1),
    schema=[
        Field(name="conv_rate", dtype=Float32),
        Field(name="avg_daily_trips", dtype=Int64),
    ],
    online=True,
    source=driver_stats_source,
    version="latest",
)

Start the servers from the server repo:

cd test_project/feature_repo
feast serve_registry --port 6570     # gRPC registry server
feast serve --port 6566              # HTTP feature server (separate shell)

From the client repo, apply (remote):

cd test_demo
feast apply
# Applying changes for project test_project
# Deploying infrastructure for driver_hourly_stats

Inspect the server-side online store — online_store.db does not exist (only registry_sqlite.db):

ls test_project/feature_repo/data
# registry_sqlite.db   (online_store.db is absent)

From the client repo, materialize (remote):

cd test_demo
feast materialize 2021-04-12T00:00:00 2026-08-02T00:00:00

Observe the failure: client fails with too many 500 error responses against /write-to-online-store; feature server logs sqlite3.OperationalError: no such table: test_project_driver_hourly_stats; online_store.db is left as a 0-byte file with no tables.

Specifications

Version: Feast SDK 0.64.0 (Python 3.12.10)
Platform: Windows 11 (10.0.26200), AMD64 — relevant code is platform-independent, expected to reproduce on Linux/macOS too
Subsystem: infra/online_stores/remote (RemoteOnlineStore), infra/online_stores/sqlite (SqliteOnlineStore), feature server (feast serve / write_to_online_store)

Thanks for the detailed root cause, yuan1j. I've reproduced the trace and agree with the analysis: RemoteOnlineStore.update() is a no-op, feast serve never calls the server-side SqliteOnlineStore.update(), so online_write_batch hits a table that was never created.

The fix is design-entangled because there's no existing mechanism in the feature server (feast serve) for a client to say "create this FV's online table on the server" — update() is called during feast apply locally, but the remote path bypasses it entirely. Three approaches, depending on how invasive you want to be:

Option A — Minimal: add update to the /write-to-online-store handler (or a new /ensure-online-table endpoint)

The feature server already receives WriteToOnlineStoreRequest per FV per batch. Extend it to carry FeatureViewProto on the first write (or add a lightweight EnsureOnlineTableRequest with just the FV name + project). Server-side, call SqliteOnlineStore.update() for that FV before the write. This keeps the schema migration on the existing write path, no new gRPC surface. Downside: the first write for each FV pays the update() cost, and there's a race if two clients hit it simultaneously (idempotent CREATE TABLE IF NOT EXISTS in SQLite handles it, but other online stores may need locks).

Option B — Protocol extension: feast apply (remote) calls a new gRPC method on the registry server to create server-side online tables

The registry server (feast serve_registry) already holds the FV proto. Add UpdateOnlineStore(FeatureViewProto) to the RegistryServer gRPC service. RemoteOnlineStore.update() (client) calls it; server dispatches to the configured OnlineStore.update(). This mirrors the local path exactly and keeps schema ops on the registry plane. Downside: new gRPC method + proto change; registry server currently has no online store dependency, so it becomes coupled.

Option C — Push-based: feast serve (feature server) subscribes to registry changes and auto-creates tables

Feature server already watches the registry for FV changes (for the online feature cache). Extend that to call SqliteOnlineStore.update() when a new FV with online=True appears. This is clean architecturally — the server owns its online store schema — but it's a bigger refactor of the feature server's registry-watcher loop, and it makes the feature server stateful in a new way.

My lean: Option A is the smallest surface-area change that unblocks the immediate bug. Option B is cleaner long-term if you're already planning gRPC evolution. Option C is right if you're investing in the feature server as a first-class online store coordinator.

Happy to prototype whichever you prefer — just need a nod on direction before I ship a PR.

@adarshsm

Hi @adarshsm ,

Thanks for the detailed root cause analysis. I agree with your findings.

I would like to strongly advocate for Option A (Minimal) to fix this issue immediately.

Reasons:

Unblocks the Core Workflow : This bug completely breaks feast materialize in remote mode. Option A is the quickest way to resolve this critical failure path.
Low Risk : It has a minimal surface area, avoiding complex refactoring, new gRPC methods, or proto changes. This makes it easy to review and merge.
Addresses Future Concerns : While the locking/idempotency issue is valid, it is manageable. Our production environment will use Postgres or Redis (not just SQLite), where we can handle this via CREATE TABLE IF NOT EXISTS or similar mechanisms.
Clean Variant : I agree with using a dedicated endpoint (e.g., /ensure-online-table ) to avoid mixing DDL with the high-frequency write path.

Could you help proceed with implementing Option A?

Would anyone be able to help with this? Thanks in advance!

I'd like to work on this. I traced the path and it looks like there are two gaps rather than one:

Both .update() and .teardown() in RemoteOnlineStore are just pass calls, hence client-side feast apply in remote mode won't provision anything.
Feature server updates the registry using async_refresh() but never invokes update_infra(), hence the DDL in SqliteOnlineStore.update() doesn't get executed server-side either.

As DDL generation is done exclusively in update(), this should affect any online store wrapped with type: remote configuration and not just SQLite.

Just before writing some code - which way do you want me to take this?

A. New authenticated endpoint (POST /update-infra) from which RemoteOnlineStore.update() will call to mirror the behavior of local mode for any online store.
B. update_infra() should be called by the feature server in case of registry refresh when it picks up new feature views.
C. Lazily provision using online_write_batch() (smallest but limited to SQLite only and putting DDL in the write path).

Will be happy to follow whatever suits the project better. Will post a small repro soon.

Posting the repro I promised, plus a prototype of option A so the tradeoff is concrete rather than hypothetical.

Repro — same store, same feature view, same write; the only variable is whether the online store's update() runs:

CONTROL (local apply, update() runs) : write succeeded
REMOTE MODE (update() is a no-op) : sqlite3.OperationalError:
no such table: _<feature_view>

One thing worth flagging: test_remote_online_store_read_write already applies a feature view through a remote client and writes to it, and it passes. It passes because it applies driver_hourly_stats — a name the server had already provisioned during its own feast apply at setup, so the table exists before the client touches it. Applying a feature view the server has never seen fails.

One constraint that shapes the design: FeatureStore.apply() calls update_infra() (feature_store.py:1871) before registry.commit() (line 1880). So an endpoint that just tells the server to re-read the registry and provision won't work — at that point the objects aren't committed, and with a file registry they're not on disk at all. Either they travel in the request body, or commit() moves ahead of update_infra().

This is also why option B is more attractive than I first thought: a registry refresh naturally happens after the commit.

I went with A for the prototype: /update-infra and /teardown-infra on the feature server, with the objects sent as protos in the body. Authorization reuses the existing CREATE/DELETE actions, so no new permission surface. teardown() was the same no-op bug, so it's fixed too.

Still happy to redo this as B (or move the commit) if that fits the project's direction better — the integration test asserts the user-visible behaviour rather than the mechanism, so it holds either way. I'll open a PR once you've had a chance to weigh in.

@RishabhCodezZz

Hi @RishabhCodezZz

Thanks for the repro + prototype — I'd like to stay with A , on these points:

B re-triggers the exact bug. The root cause is an apply → materialize race. B makes it worse: apply returns success at registry commit, but the table only appears after the next async refresh — so an immediate materialize hits the same "no such table" window. A is synchronous: when apply returns, the table exists, materialize is safe.
Observability. Under A, DDL failure surfaces in apply . Under B it's invisible until materialize — barely better than today.
Server stays stateless. B makes the feature server mutate schema via its registry-watcher loop (= adarshsm's Option C, which he flagged as making the feature server stateful). A keeps the server stateless; client drives infra, mirroring local mode.
Commit-order isn't a problem for us. With a file registry objects aren't on disk before commit() , so they travel in the body — fine, because our project is client-driven by design; the server already trusts client-supplied objects.

Please go ahead with the option A. Thanks for your help.

test: Add failing regression test for remote online store provisioning

fix: Provision online store infra for remote feast apply

Navigation Menu

Repository navigation

'feast apply' does not create online store tables in remote mode #6693

Expected Behavior

Current Behavior

Steps to reproduce

Specifications

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

'feast apply' does not create online store tables in remote mode #6693

Description

Expected Behavior

Current Behavior

Steps to reproduce

Specifications

Activity

adarshsm commented on Aug 2, 2026

yuan1j commented on Aug 5, 2026

yuan1j commented on Aug 12, 2026

RishabhCodezZz commented on Aug 19, 2026

RishabhCodezZz commented on Aug 19, 2026

yuan1j commented on Aug 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions