Repository navigation
list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration #84507
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-05-22.23:15:10.063>
created_at = <Date 2020-04-19.08:24:06.095>
labels = ['extension-modules', 'type-bug', '3.7']
title = 'list(sys.modules.items()) can throw RuntimeError: dictionary changed size during iteration'
updated_at = <Date 2021-08-26.00:30:48.133>
user = 'https://github.com/mmohrhard'bugs.python.org fields:
activity = <Date 2021-08-26.00:30:48.133>
actor = 'Andrew Nelson'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-22.23:15:10.063>
closer = 'rhettinger'
components = ['Extension Modules']
creation = <Date 2020-04-19.08:24:06.095>
creator = 'Markus Mohrhard'
dependencies = []
files = []
hgrepos = []
issue_num = 40327
keywords = ['patch']
message_count = 8.0
messages = ['366762', '366798', '366804', '366854', '366952', '369660', '373399', '400301']
nosy_count = 6.0
nosy_names = ['rhettinger', 'nirs', 'serhiy.storchaka', 'miss-islington', 'Markus Mohrhard', 'Andrew Nelson']
pr_nums = ['19628', '19640']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40327'
versions = ['Python 3.7']
Activity
mmohrhard commentedon Apr 19, 2020
We have hit an issue in the pickle module where the code throws an exception in a threaded environment:
The interesting piece of the backtrace is:
File "/xxx/1004060/lib/python3.7/site-packages/numpy/core/init.py", line 130, in _ufunc_reduce
return _ufunc_reconstruct, (whichmodule(func, name), name)
File "/xxx/lib/python3.7/pickle.py", line 309, in whichmodule
for module_name, module in list(sys.modules.items()):
RuntimeError: dictionary changed size during iteration
I tried to find a code path that would explain how the dict could be changed while the list is created but have not been able to find a code path that releases the GIL.
The executable is using many threads with imports happening in random threads and a custom class loader but we already make sure that the class loader is always holding the GIL.
The issue happens quite rarely (maybe once every one thousand's execution) so I don't have a reproducer right now.
rhettinger commentedon Apr 19, 2020
Suggest making this change:
+ for module_name, module in sys.modules.copy().items():
rhettinger commentedon Apr 19, 2020
We should consider making this change just about everywhere in the standard library. Besides avoiding a race condition, running a for loop over
d.copy().items()is more memory efficient and possibly faster than a for loop overlist(d.items()).serhiy-storchaka commentedon Apr 20, 2020
I afraid that this is a part of the larger issue (see also bpo-31165). Seems the cause is that the GIL can be released when allocate or reallocate the memory. Does your project use some memory managing hooks?
Using sys.modules.copy().items() can help a lot, because you allocate memory for new dict only once. In case of list(sys.modules.items()) you allocate memory for every key-value pair, and may allocate memory several times to resize a list.
This reduces the probability of occurring the problem in many times, but not to zero. There is still a chance that the size of the original dict increase when you allocate a memory for a new dict, so copying the data to a new dict can fail or even crash or spoil the memory.
rhettinger commentedon Apr 21, 2020
New changeset 75bedbe by Raymond Hettinger in branch 'master':
bpo-40327: Improve atomicity, speed, and memory efficiency of the items() loop (GH-19628)
75bedbe
miss-islington commentedon May 22, 2020
New changeset 16d0781 by Miss Islington (bot) in branch '3.8':
bpo-40327: Improve atomicity, speed, and memory efficiency of the items() loop (GH-19628)
16d0781
nirs commentedon Jul 9, 2020
Does this really affect only python 3.7?
We see this in RHEL 8.2, using python 3.6.8:
https://bugzilla.redhat.com/show_bug.cgi?id=1837199#c69
Likely caused by:
Without locking. self._lvs is a dict that may contain 1000's of items.
I'm not sure if this is relvant now for upstream, but backport to 3.6 would be useful.
AndrewNelson commentedon Aug 26, 2021
Just to mention that we have seen similar issues whilst importing scipy in a multiprocessing context.
See scipy/scipy#11479 and https://stackoverflow.com/questions/68911221/python-runtimeerror-dictionary-changed-size-during-iteration-popping-out-rand.
Fix "dictionary changed size during iteration"
Update git submodules
Fix "dictionary changed size during iteration"
Fix "dictionary changed size during iteration"