Repository navigation
os.environ updates only one copy of env vars under Windows (GetEnvironmentVariable vs. getenv) #60837
Description
| BPO | 16633 |
|---|---|
| Nosy | @tjguk, @asvetlov |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2012-12-07.12:01:43.915>
labels = ['type-bug', 'docs']
title = 'os.environ updates only one copy of env vars under Windows (GetEnvironmentVariable vs. getenv)'
updated_at = <Date 2013-01-17.11:09:42.053>
user = 'https://bugs.python.org/eudoxos'bugs.python.org fields:
activity = <Date 2013-01-17.11:09:42.053>
actor = 'goungy'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2012-12-07.12:01:43.915>
creator = 'eudoxos'
dependencies = []
files = ['28243']
hgrepos = []
issue_num = 16633
keywords = []
message_count = 5.0
messages = ['177081', '177087', '177088', '177099', '180122']
nosy_count = 5.0
nosy_names = ['tim.golden', 'asvetlov', 'docs@python', 'eudoxos', 'goungy']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16633'
versions = ['Python 2.7', 'Python 3.3']
Activity
eudoxos commentedon Dec 7, 2012
On windows, environment variables exist in two copies: one is manipulated using win32 API (GetEnvironmentVariable, SetEnvironmentVariable), and another one is maintained by the C runtime (getenv, _putenv). This is explained in more depth in [1].
os.environ manipulates win32 environment variables, but *not* those seen by the CRT. This means that if I set an environment variable using os.environ and later read it, in the same process, using getenv in an extension module, it will not give the expected result. Child processes *do* see those vars in CRT, since it is copied over from the win32 version at process startup.
Setting env vars has legitimate uses, since it is one of the few ways to influence initialization of extension modules: for instance, setting OMP_NUM_THREADS sets number of threads for OpenMP runtime (which cannot be changed once the module using it is loaded).
It would be ideal to keep both CRT and win32 env vars in sync transparently. If that is not realistically achievable, this gotcha should be documented as a warning in the os.environ documentation.
A workaround to this problem to set variables in both win32 and CRT using something like
[1] http://msmvps.com/blogs/senthil/archive/2009/10/13/when-what-you-set-is-not-what-you-get-setenvironmentvariable-and-getenv.aspx
asvetlov commentedon Dec 7, 2012
I think the problem is: windows process can contain single shared win32 API and several CRT copies — one for Visual Studio 2008, other for VS 2010 etc. We cannot know which CRT version is used by particular extension, so I doubt we can initialize it properly.
Your workaround has the same problem.
tjguk commentedon Dec 7, 2012
Does the same problem obtain if you use os.putenv (which calls the crt
putenv under the covers)?
eudoxos commentedon Dec 7, 2012
I checked on Windows 7 64bit with python 2.7 (sorry, not python 3.3 installed here) with the script attached. Each line sets a variable using the method in the very left column, then it attempts to read it back using all methods.
os.environ OK OK OK -- --
win32 -- OK -- -- --
os.?etenv -- OK -- -- --
msvcrt -- OK -- OK --
msvcr? -- OK -- -- --
Methods which can read back what they also set are os.environ, win32api, msvcrt. OTOH, msvcr? (which is ctypes.util.find_msvcr(), in my case msvcr90.dll) does not read its own values, just like os.getenv/os.putenv.
It *seems* that reading with win32 API is very reliable (that is not a good news for writing cross-platform extension modules), though, but perhaps it just asks the CRT if it does not find the variable defined (my testing did not go that far).
@andrew: you're probably right, though it does not explain, why msvcr90.dll does not read back the values I set in there - that is the CRT python27.dll itself links to --
goungy commentedon Jan 17, 2013
We encountered the same issue when upgrading our application's JRE from 1.5 to 1.6.
We use a kind of grid, which has engines written in C, which uses an embedded JAVA JVM, which loads C dll. One of these uses python.
Here is what happens:
With JRE 1.5, Python is able to get correct PYTHONHOME, and can do "import os" in our script.
With JRE 1.6, this is not the case, as JRE 1.6 loads MSVCR71.dll @ step 1. JRE 1.5 did not.
As stated in the previous comments, and from my understanding, in Windows there is the "Process Environment Variables Space" and possibly several "C Runtime Environment Variables Space".
The first time a C runtime dll is loaded, it copies the Process Env Var into its own buffer.
Our JRE 1.6 loads msvcr71.dll (C runtime), and so copies env var @ step 1.
It happens before we set PYTHONHOME with JAVA @ step 2.
To correct this, we had to use the Py_SetPythonHome function before calling Py_PyInit to set PYTHONHOME ourselves
This way, Python executes our code fine when we use JRE 1.6.
But this is because we do not call any getenv functionality within python at the moment, but it could happen in the future...
As stated by eudoxos, the safest solution for windows is to use GetEnvironmentVariable (win32 kernel API).
Is there any schedule for a fix for this problem?
Thanks for you time and answer.
Regards.