Repository navigation
macOS 10.14 Mojave crashes in multiprocessing #79400
Description
| BPO | 35219 |
|---|---|
| Nosy | @warsaw, @ronaldoussoren, @ned-deily, @applio, @pablogsal |
| Superseder |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-11-14.20:56:16.470>
created_at = <Date 2018-11-12.17:57:32.609>
labels = ['3.7', '3.8']
title = 'macOS 10.14 Mojave crashes in multiprocessing'
updated_at = <Date 2018-11-14.20:56:16.468>
user = 'https://github.com/warsaw'bugs.python.org fields:
activity = <Date 2018-11-14.20:56:16.468>
actor = 'barry'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-14.20:56:16.470>
closer = 'barry'
components = []
creation = <Date 2018-11-12.17:57:32.609>
creator = 'barry'
dependencies = []
files = []
hgrepos = []
issue_num = 35219
keywords = []
message_count = 14.0
messages = ['329747', '329752', '329753', '329754', '329757', '329758', '329760', '329780', '329784', '329795', '329796', '329869', '329928', '329935']
nosy_count = 5.0
nosy_names = ['barry', 'ronaldoussoren', 'ned.deily', 'davin', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '33725'
type = None
url = 'https://bugs.python.org/issue35219'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']
Activity
warsaw commentedon Nov 12, 2018
As we're beginning to roll out macOS 10.14 High Sierra, we're seeing an increase in core files inside /cores. This can consume a ton of disk space if you have coredumps enabled. I don't have a short reproducer yet, but we believe this is related to changes in macOS behavior around fork(). Here are some clues:
ansible/ansible#32499 (comment)
https://stackoverflow.com/questions/50168647/multiprocessing-causes-python-to-crash-and-gives-an-error-may-have-been-in-progr/52230415#52230415
http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
[-]macOS 10.14 High Sierra crashes in multiprocessing[/-][+]macOS 10.14 Mojave crashes in multiprocessing[/+]warsaw commentedon Nov 12, 2018
That should of course be 10.14 Mojave.
pablogsal commentedon Nov 12, 2018
Apparently setting the env variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES should fix some fork-related changes. Have you tried already doing this? Does it change the behaviour of the errors in any way?
warsaw commentedon Nov 12, 2018
On Nov 12, 2018, at 13:34, Pablo Galindo Salgado <report@bugs.python.org> wrote:
Yes, sorry I got distracted before I could add a comment about this. I have indeed tested this and it does prevent the core files.
pablogsal commentedon Nov 12, 2018
Apparently, GitLab have a more permanent solution around this by enabling the Objective-C runtime before any forking happens:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16649/diffs
pablogsal commentedon Nov 12, 2018
This can a duplicate of https://bugs.python.org/issue33725
warsaw commentedon Nov 12, 2018
I'm not sure whether it's a duplicate or not, since in my case it doesn't hang, but instead core dumps. It's seems like the reasoning given in the Ruby bug is relevant to Python too, so maybe we should adopt the same workaround. For our internal projects, I'm probably just going to set the environment variable for now.
I'd love to know what Ned and Ronald thing, thus cc'ing them.
warsaw commentedon Nov 13, 2018
I've done a fair bit of testing and it seems rather inconsistent as to whether either of these work when added right before an explicit call to
os.fork():os.environ['OBJC_DISABLE_INITIALIZE_FORK_SAFETY'] = 'YES'
Neither of these seems to reliably prevent the __NSPlaceholderDate warning, nor prevent the core dumps. The best I've been able to do is to prevent them by setting the environment variable *before* the parent process starts (i.e. outside of the Python code of the process).
warsaw commentedon Nov 13, 2018
Based on my testing, the environment variable has to be set before the parent process starts. Neither os.environ nor os.putenv seem to do the trick.
warsaw commentedon Nov 13, 2018
I'm still testing this solution, but it looks like if you set the environment variable, and then double fork, the granchild won't crash. Roughly:
warsaw commentedon Nov 13, 2018
Nope, actually double fork doesn't work. It's misleading because in my testing, the first invocation of the process causes the core dump, but subsequent runs do not.
warsaw commentedon Nov 13, 2018
I am going to dupe this to 33725 after all.
applio commentedon Nov 14, 2018
Resolution is marked dupe but status is still open. Are we closing this one or is there a more specific remedy for this situation (as opposed to what bpo-33725 discusses) that would be helpful to document?
warsaw commentedon Nov 14, 2018
Closing in favor of bpo-33725