Repository navigation
fix: Stop DynamoDB update_online_store overwriting the item when its read fails - #7046
Open
ericdelorefice wants to merge 1 commit into
Open
ericdelorefice wants to merge 1 commit into
ericdelorefice wants to merge 1 commit into
Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7046 +/- ##
==========================================
+ Coverage 50.99% 51.06% +0.07%
==========================================
Files 455 455
Lines 56994 56998 +4
Branches 8321 8321
==========================================
+ Hits 29063 29106 +43
+ Misses 25946 25903 -43
- Partials 1985 1989 +4
*This pull request uses carry forward flags. Click here to find out more.
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
…read fails update_online_store and update_online_store_async read the stored item with get_item before applying list_append / prepend. A ClientError from that read (for example ProvisionedThroughputExceededException once botocore's retries are used up) was swallowed, so the item was treated as missing and put_item replaced it with only the values in the call: the list history and any feature not in the call were lost, and the call returned normally. A missing item is reported by the absence of "Item" in the response, not by an exception, so the handler never covered the not-found case. Log and re-raise, as the update_item branch of the same methods already does. Signed-off-by: Eric Del Orefice <114540347+ericdelorefice@users.noreply.github.com>
ericdelorefice
force-pushed
the
dynamodb-update-read-errors
branch
from
October 21, 2026 18:28
e244d3c to
ae36aa2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Problem
DynamoDBOnlineStore.update_online_storeandupdate_online_store_async(the async one is whatFeatureStore.update_online_storecalls) read the stored item withget_itembefore applyinglist_append. AClientErrorfrom that read is caught withexcept ClientError: pass(dynamodb.py1010-1011, async 1151-1152). The item is then treated as missing and written withput_item(1032, async 1178), which replaces the whole item with only the values in this call. The call returns normally.For a
list_appendfeature, the stored history is replaced by just the new elements. When the store method is called directly with only some features, the features left out are dropped as well. (FeatureStore.update_online_storerequires every feature column, so from there it is the list history that goes.)A missing item comes back as a response without
"Item", not as an exception, so the handler never covered the not-found case. Theupdate_itembranch of the same methods already logs and re-raises (1060-1062, async 1211-1213).The Devin review on #5916 pointed at this handler in January. This PR adds a reproduction, tests and the fix.
Reproduction
moto's DynamoDB server on localhost with the real clients (aiobotocore for the async path, boto3 for the sync one). Only the read failure is injected: a
ProvisionedThroughputExceededExceptiononGetItem. I have not seen this in a production deployment. A realistic cause is a provisioned table whose read capacity is throttled while its write capacity is not, since the two are set separately.Through
FeatureStore.update_online_storeonmaster, withupdate_expressions={"transactions": "list_append(transactions, :new_val)"}:Calling the sync
update_online_storedirectly with onlytransactions, and the throttle returned as an HTTP 400 on everyGetItemattempt (botocore retried, 3 attempts, then raised):With this change both calls raise
ProvisionedThroughputExceededExceptionand the stored item is unchanged.Change
Both handlers log the error and
raise, the same as theupdate_itembranch. The intended behaviour change: a failed read now raises out ofupdate_online_storeandFeatureStore.update_online_storeinstead of writing. Entities earlier in the same call stay written, as they already do whenupdate_itemfails. A missing item and a successful read behave as before.Verification
Five new tests in
tests/unit/infra/online_store/test_dynamodb_online_store.py. With the source reverted tomasterand the tests kept, 2 fail: a failed read, sync (moto) and async (mocked client), must raise and leave the item alone. The 3 that pass are the controls: an update keeps features it does not name (sync), a missing item is created (async), and an existing item is updated in place with the list merged (async). Each direction was run 5 times with the same result.The DynamoDB online store unit tests go from 47 passed to 52 passed.
ruff checkandruff format --checkare clean on both files, and mypy is clean ondynamodb.py.Which issue(s) this PR fixes:
None filed. The reproduction is above.
Checks
git commit -s)Testing Strategy
Misc
The read-modify-write is still not atomic when two updates for the same entity run at once. That is a separate question, so I left it alone here.
This PR was written with AI assistance (Claude Code); the tests above were run before opening it.