Repository navigation
Support builds on macOS without installed system header files #80412
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/ned-deily'
closed_at = <Date 2019-06-20.06:46:28.503>
created_at = <Date 2019-03-08.10:03:47.265>
labels = ['OS-mac', '3.8', 'build', '3.7']
title = 'Support builds on macOS without installed system header files'
updated_at = <Date 2019-06-20.08:22:24.734>
user = 'https://github.com/dimpase'bugs.python.org fields:
activity = <Date 2019-06-20.08:22:24.734>
actor = 'dimpase'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2019-06-20.06:46:28.503>
closer = 'ned.deily'
components = ['Build', 'macOS']
creation = <Date 2019-03-08.10:03:47.265>
creator = 'dimpase'
dependencies = []
files = ['48192']
hgrepos = []
issue_num = 36231
keywords = ['patch']
message_count = 13.0
messages = ['337461', '337540', '337680', '337906', '337912', '340202', '344407', '346008', '346049', '346062', '346101', '346104', '346108']
nosy_count = 5.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'erik.bray', 'jdemeyer', 'dimpase']
pr_nums = ['12825', '13773', '14208', '14256']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue36231'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8']
Activity
dimpase commentedon Mar 8, 2019
Neither Xcode nor its command-line tools on macOS 10.14 Mojave come with header files installed in /usr/ and other "normal" directories.
This is not documented in https://devguide.python.org/setup/#macos-and-os-x
While an extra step to handle this, i.e. to install the headers, is available (see a discussion on https://bugs.python.org/issue34956), Apple stated that this workaround will disappear.
It is thus highly desirable to provide a way to deal with headers located not at /usr/include, but at
xcrun --show-sdk-path/usr/include.A small change in setup.py along the lines of the following:
with the necessary changes to enable various modules (see attachment for a complete POC diff against the current master), the result builds and passes all the (enabled) tests on an OSX 10.13 system with empty /usr/include and /usr/local/include containing only LZMA headers.
Needless to say, a proper patch would not modify Modules/Setup, it'd adjust configure.ac etc.
dimpase commentedon Mar 8, 2019
Needless to say, subprocess is most certainly an overkill, something less involved would do the job, without the need for all the module dependencies of subprocess.
embray commentedon Mar 11, 2019
Perhaps it would be better if the
xcrun --show-sdk-paththing were run at configure-time and its result shoved into a variable we can read with sysconfig.get_config_var()dimpase commentedon Mar 14, 2019
I won't mind to provide a PR for this. but it is not clear what the goal should be. Is it to build a working OSX Python with as few external to Xcode deps (it seems that only lzma is needed) as possible?
ned-deily commentedon Mar 14, 2019
Thanks for the report and for the PR offer but let's hold off on that for the moment: I'm planning to merge a somewhat different approach.
dimpase commentedon Apr 14, 2019
In case,I have opened PR #12825
to provide our solution to this issue.
ned-deily commentedon Jun 3, 2019
New changeset 0288dd6 by Ned Deily in branch 'master':
bpo-36231: Support building on macOS without /usr/include (GH-13773)
0288dd6
ned-deily commentedon Jun 18, 2019
New changeset c730211 by Ned Deily in branch '3.7':
bpo-36231: Support building on macOS without /usr/include (GH-13773) (GH-14208)
c730211
dimpase commentedon Jun 19, 2019
I find it puzzling that a relatively clean and short solution using autotools and the Apple-approved way to get the location of the headers is rejected, and a hacky, longer way that scrapes output of the compiler is being pushed instead.
ned-deily commentedon Jun 19, 2019
Thank you again for the suggested PR. Using "xcrun --show-sdk-path" at configure time *is* appealing. Unfortunately, it does not cover all of the necessary use cases.
One, the --show-sdk-path option is not available on old versions of xcrun, versions we still build with to support older releases of macOS. Second, some users build Python for macOS with compiler tool chains other than the Apple-supplied ones (for example, current gcc) that do not necessarily support the transparent selection of header and library files location via xcrun and friends. Third, capturing the selected SDK path at configure time is no guarantee that the same SDK path will be used for extension module builds when setup.py runs. With the Apple-supplied tools, the actual SDK path used is determined each time the compiler front-end is invoked and depends on the then-current selected values (e.g. the most recent value set by 'xcode-select --switch') and the current value of environment variables (e.g. like DEVELOPER_DIR and SDKROOT). Or a different or non-Apple compiler could now be in use by overriding CC. In other words, lots of edge cases largely due to the inherent flexibility of Apple's compiler frontend.
The other issue here is the behavior of setup.py in trying to make reasonable default choices for finding header and library files for the extension modules it is building; to do so, it tries to guess what the compiler frontend is going to do and that's the real hack. Life would be much simpler if Python relied on a modern autotools build setup - although some of the same issues of dynamic SDK locations would still apply but at least they would apply consistently - and not the legacy mixture of build tools we have today. Unfortunately, it would be a big deal to replace the current build system and, while it would be desirable, that's a very big project.
One other point: while scraping the output of the compiler is hacky, it is a well-known and widely-used technique and is already in use elsewhere in setup.py; this code was adapted from that in add_multiarch_paths().
ned-deily commentedon Jun 20, 2019
New changeset c421c66 by Ned Deily in branch '2.7':
bpo-36231: Support building on macOS without /usr/include (GH-13773) (GH-14256)
c421c66
ned-deily commentedon Jun 20, 2019
Thanks again everyone. The general solution has now been merged to all active branches for release in 3.8.0, 3.7.4, and 2.7.17.
Note that bpo-19960 identifies an additional problem only on 2.7 where a few extension modules, notably zlib, are still not being built when header files are not installed; a PR for that issue is currently awaiting review.
[-]no "proper" header files on macOS 10.14 Mojave[/-][+]Support builds on macOS without installed system header files[/+]dimpase commentedon Jun 20, 2019
Thanks for the review of our PR. I think not all points made there are relevant:
on older systems, where
xcrun --show-sdk-pathis not available, one can assume that /usr/include is there, and simply set the corresponding variable to this value in ./configurebuilding Python with a non-system compiler is only possible with system headers, and thus either SDK or at least command line tools, available, and all is well.
mid-flight switching to a different SDK/compiler is dodgy, and in particular switching to a different set of headers is asking for trouble. And the gcc toolchain is not much different in this way - e.g. on Gentoo one can do gcc-config to switch from one profile to another, with provably "interesting" results if it is done mid-flight.
the Python 2.7 zlib issue is not a problem with our PR.
---------
Most probably, the desired switch to autotools can only happen gradually, by moving more of setup.py to ./configure, and our PR is a step in such direction.