Repository navigation
AssemblyManager.FindAssemblyCandidates runs without GIL and crashes with "Attempted to read or write protected memory" unpredictably #2481
Description
Environment
- Pythonnet version: 3.0.4
- Python version: 3.11.9
- Operating System: Windows 11 Pro
- .NET Runtime: .NET Core 8
Details
When assemblies are dynamically loaded in .NET, PythonNet assembly loading hook can crash with "Attempted to read or write protected memory." I'm still trying to make a repro, but maybe you'll immediately see what's wrong.
pythonnet/src/runtime/AssemblyManager.cs
Line 206 in de7a1d2
I think that the issue may be that during assembly loading, the AssemblyManager hook runs without a GIL, so it may crash with the error above. I've seen multiple stack traces happening at different times, often in code completely unrelated to PythonNet (there is multithreading, I use PythonEngine.BeginAllowThreads())
This did not happen in version 3.0.3.
-
Describe what you were trying to get done.
Run an application normally that has more than one assembly being loaded (e.g. resources)
-
What commands did you run to trigger this issue? If you can provide a
Minimal, Complete, and Verifiable example
this will help us understand the issue.
I will try to provide one, although it's difficult to reproduce reliably
- If there was a crash, please include the traceback here.
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at Python.Runtime.Runtime.PyObject_GetAttrString(Python.Runtime.BorrowedReference, System.String)
at Python.Runtime.AssemblyManager+<FindAssemblyCandidates>d__15.MoveNext()
at System.Linq.Enumerable.TryGetFirst[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable`1<System.__Canon>, Boolean ByRef)
at System.Linq.Enumerable.FirstOrDefault[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
at Python.Runtime.AssemblyManager.FindAssembly(System.String)
at Python.Runtime.AssemblyManager.LoadAssemblyPath(System.String)
at Python.Runtime.AssemblyManager.ResolveHandler(System.Object, System.ResolveEventArgs)
at System.Runtime.Loader.AssemblyLoadContext.InvokeResolveEvent(System.ResolveEventHandler, System.Reflection.RuntimeAssembly, System.String)
at System.Reflection.RuntimeAssembly.InternalLoad(System.Reflection.AssemblyName, System.Threading.StackCrawlMark ByRef, System.Runtime.Loader.AssemblyLoadContext, System.Reflection.RuntimeAssembly, Boolean)
at ...
Activity
filmor commentedon Oct 11, 2024
You are absolutely right! To help you with your repro (in case you hadn't seen this, yet), this code was introduced in #2383 to warn about badly formatted strings in paths. You can set up your path as the test does. It should actually only happen in this case.
If the call chain is indeed not GIL-protected, then it should be enough to add a line
using var _ = Py.GIL()directly above the line you mentioned.@lostmsu
acidbubbles commentedon Oct 11, 2024
The issue is that I don't control assembly loading events, some of them happen without any user code. I did figure out the cause of the error (a
\somethinginstead of\\somethingin a string) so the change is a good thing, it's just a shame the message didn't make it through :)Here's a full example of a stack trace where it's impossible to wrap in a GIL:
I could probably try and make a repro, but it would probably be much easier to reproduce in a unit test with access to internal classes.
So for now here's what I think is the recipe:
PythonEngine.PythonPathpaths, like\nfor exampleThread.Sleepwould probably work.var _ DateTimeOffset.GetUtcNow().Humanize()If you feel like making a repro is worth it v.s. using the unit tests, let me know. I'd like to contribute but I'm short on time, so if you prefer leaving it to me I can work on it later (undetermined), but I don't know how I could fix this with my current knowledge. I guess we cannot grab the GIL here because it may have been grabbed, and checking whether it has been grabbed wouldn't be thread safe and it quickly becomes tricky :)
filmor commentedon Oct 11, 2024
I'll check again, I think when looking through this code before, I found another place in the same call were we take the GIL implicitly, so it wouldn't make things worse.
acidbubbles commentedon Oct 13, 2024
My worry would be deadlocks, since C# assembly loading can happen both when calling CLR from Python (within a GIL) or outside (no GIL).
filmor commentedon Oct 13, 2024
I know that, that would be my worry as well, hence my note about another potentially GIL-protected call. It looks like the function may also get called in some resolve handler.
I suggest that we just gate the warnings with a
PyGILState_Checkand only emit warnings while the GIL is taken (e.g. during an explicitclr.AddReference).