← 输入框|github.com|刷新
Skip to content

fix: Stop DynamoDB update_online_store overwriting the item when its read fails - #7046

Open
ericdelorefice wants to merge 1 commit into
feast-dev:masterfrom
ericdelorefice:dynamodb-update-read-errors
Open

ericdelorefice wants to merge 1 commit into
feast-dev:masterfrom
ericdelorefice:dynamodb-update-read-errors

Conversation

@ericdelorefice

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Problem

DynamoDBOnlineStore.update_online_store and update_online_store_async (the async one is what FeatureStore.update_online_store calls) read the stored item with get_item before applying list_append. A ClientError from that read is caught with except ClientError: pass (dynamodb.py 1010-1011, async 1151-1152). The item is then treated as missing and written with put_item (1032, async 1178), which replaces the whole item with only the values in this call. The call returns normally.

For a list_append feature, the stored history is replaced by just the new elements. When the store method is called directly with only some features, the features left out are dropped as well. (FeatureStore.update_online_store requires every feature column, so from there it is the list history that goes.)

A missing item comes back as a response without "Item", not as an exception, so the handler never covered the not-found case. The update_item branch of the same methods already logs and re-raises (1060-1062, async 1211-1213).

The Devin review on #5916 pointed at this handler in January. This PR adds a reproduction, tests and the fix.

Reproduction

moto's DynamoDB server on localhost with the real clients (aiobotocore for the async path, boto3 for the sync one). Only the read failure is injected: a ProvisionedThroughputExceededException on GetItem. I have not seen this in a production deployment. A realistic cause is a provisioned table whose read capacity is throttled while its write capacity is not, since the two are set separately.

Through FeatureStore.update_online_store on master, with update_expressions={"transactions": "list_append(transactions, :new_val)"}:

stored:              transactions=['tx1', 'tx2', 'tx3']  user_score=110
update (tx4, 120) with one throttled GetItem -> returns normally
stored:              transactions=['tx4']  user_score=120
get_online_features: transactions=[['tx4']]

Calling the sync update_online_store directly with only transactions, and the throttle returned as an HTTP 400 on every GetItem attempt (botocore retried, 3 attempts, then raised):

before: {'transactions': ['tx1', 'tx2', 'tx3'], 'user_score': 100, 'segment': 'gold'}
after:  {'transactions': ['tx4']}

With this change both calls raise ProvisionedThroughputExceededException and the stored item is unchanged.

Change

Both handlers log the error and raise, the same as the update_item branch. The intended behaviour change: a failed read now raises out of update_online_store and FeatureStore.update_online_store instead of writing. Entities earlier in the same call stay written, as they already do when update_item fails. A missing item and a successful read behave as before.

Verification

Five new tests in tests/unit/infra/online_store/test_dynamodb_online_store.py. With the source reverted to master and the tests kept, 2 fail: a failed read, sync (moto) and async (mocked client), must raise and leave the item alone. The 3 that pass are the controls: an update keeps features it does not name (sync), a missing item is created (async), and an existing item is updated in place with the list merged (async). Each direction was run 5 times with the same result.

The DynamoDB online store unit tests go from 47 passed to 52 passed. ruff check and ruff format --check are clean on both files, and mypy is clean on dynamodb.py.

Which issue(s) this PR fixes:

None filed. The reproduction is above.

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Misc

The read-modify-write is still not atomic when two updates for the same entity run at once. That is a separate question, so I left it alone here.

This PR was written with AI assistance (Claude Code); the tests above were run before opening it.

@ericdelorefice
ericdelorefice requested a review from a team as a code owner October 19, 2026 11:11
@codecov-commenter

codecov-commenter commented Oct 19, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 51.06%. Comparing base (e304a84) to head (ae36aa2).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7046      +/-   ##
==========================================
+ Coverage   50.99%   51.06%   +0.07%     
==========================================
  Files         455      455              
  Lines       56994    56998       +4     
  Branches     8321     8321              
==========================================
+ Hits        29063    29106      +43     
+ Misses      25946    25903      -43     
- Partials     1985     1989       +4     
Flag Coverage Δ *Carryforward flag
go-feature-server 30.58% <ø> (ø) Carriedforward from e304a84
python-unit 52.53% <100.00%> (+0.07%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines Coverage Δ
sdk/python/feast/infra/online_stores/dynamodb.py 72.89% <100.00%> (+7.54%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e304a84...ae36aa2. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…read fails

update_online_store and update_online_store_async read the stored item
with get_item before applying list_append / prepend. A ClientError from
that read (for example ProvisionedThroughputExceededException once
botocore's retries are used up) was swallowed, so the item was treated as
missing and put_item replaced it with only the values in the call: the
list history and any feature not in the call were lost, and the call
returned normally.

A missing item is reported by the absence of "Item" in the response, not
by an exception, so the handler never covered the not-found case. Log
and re-raise, as the update_item branch of the same methods already does.

Signed-off-by: Eric Del Orefice <114540347+ericdelorefice@users.noreply.github.com>
@ericdelorefice
ericdelorefice force-pushed the dynamodb-update-read-errors branch from e244d3c to ae36aa2 Compare October 21, 2026 18:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants