Deleted dotnet object does not get garbage collected

Environment

Pythonnet version: 3.0.0rc3
Python version: 3.10.5
Operating System: Windows 10 20H2
.NET Runtime: 6.0.6

Details

Describe what you were trying to get done.

Create dotnet-object and delete it. If I do that a lot I get memory issues. It seems the never get cleared by the Python- or dotnet-gc. This is not a problem in pythonnet 2.5.2

What commands did you run to trigger this issue?

import gc
import clr
import psutil
import matplotlib.pyplot as plt
import System

# Create dict store count of objects left
types = [
    str,
    System.String,
]
types = {obj: 0 for obj in types}

N = 1_000
str_size = 1_000_000

memory_py = [0 for _ in range(N + 2)]
memory_py[0] = psutil.virtual_memory().used
for ii in range(N):
    s_py = str('a' * str_size)
    del s_py
    memory_py[ii + 1] = psutil.virtual_memory().used - memory_py[0]

gc.collect()
# See what types of objects are still present
for obj in gc.get_objects():
    if type(obj) in list(types.keys()):
        types[type(obj)] += 1
print(types)
memory_py[ii + 2] = psutil.virtual_memory().used - memory_py[0]
memory_py[0] = 0

{<class 'str'>: 0, <class 'System.String'>: 0}

memory_net = [0 for _ in range(N + 2)]
memory_net[0] = psutil.virtual_memory().used
s_py = str('a' * str_size)
for ii in range(N):
    s_net = System.String(s_py)
    del s_net
    memory_net[ii + 1] = psutil.virtual_memory().used - memory_net[0]

gc.collect()
# See what types of objects are still present
for obj in gc.get_objects():
    if type(obj) in list(types.keys()):
        types[type(obj)] += 1
print(types)
memory_net[ii + 2] = psutil.virtual_memory().used - memory_net[0]
memory_net[0] = 0

{<class 'str'>: 0, <class 'System.String'>: 1000}

You can also visualize the memory consumption:

fig, ax = plt.subplots()
ax.plot(memory_py, label='Python object')
ax.plot(memory_net, label='DotNet object')
ax.legend()
plt.show(fig)

You can execute the same code with pythonnet 2.5.2 (Python 3.9.13) and get the following figure

You are not calling .NET GC

@lostmsu

@lostmsu I get the same results if I add System.GC.Collect()

Full code for reference:

import gc
import clr
import psutil
import matplotlib.pyplot as plt
import System

# Create dict store count of objects left
types = [
    str,
    System.String,
]
types = {obj: 0 for obj in types}

N = 1_000
str_size = 1_000_000

memory_py = [0 for _ in range(N + 2)]
memory_py[0] = psutil.virtual_memory().used
for ii in range(N):
    s_py = str('a' * str_size)
    del s_py
    memory_py[ii + 1] = psutil.virtual_memory().used - memory_py[0]

gc.collect()
System.GC.Collect()
# See what types of objects are still present
for obj in gc.get_objects():
    if type(obj) in list(types.keys()):
        types[type(obj)] += 1
print(types)
memory_py[ii + 2] = psutil.virtual_memory().used - memory_py[0]
memory_py[0] = 0

{<class 'str'>: 0, <class 'System.String'>: 0}

memory_net = [0 for _ in range(N + 2)]
memory_net[0] = psutil.virtual_memory().used
s_py = str('a' * str_size)
for ii in range(N):
    s_net = System.String(s_py)
    del s_net
    memory_net[ii + 1] = psutil.virtual_memory().used - memory_net[0]

gc.collect()
System.GC.Collect()
# See what types of objects are still present
for obj in gc.get_objects():
    if type(obj) in list(types.keys()):
        types[type(obj)] += 1
print(types)
memory_net[ii + 2] = psutil.virtual_memory().used - memory_net[0]
memory_net[0] = 0

{<class 'str'>: 0, <class 'System.String'>: 1000}

Hmm, this tells us that Python is not releasing the wrapper objects for some reason. Can you check the refcount and referrers for these?

I used objgraph to display the backrefs. The output of gc.get_referrers(obj) is really huge or have you another advice to display the referrers?

import gc
import sys
import clr
import objgraph
import System

s_net = System.String('a')
del s_net

gc.collect()
System.GC.Collect()
for obj in gc.get_objects():
    if type(obj) == System.String:
        print(f'Refcount: {sys.getrefcount(obj)}')
        print(f'Num Referrers: {len(gc.get_referrers(obj))}')
        objgraph.show_backrefs(obj, filename='backref.png')

Refcount: 4
Num Referrers: 2

@filmor

@filmor an update from my debugging the issue:

From this code I am getting a refcount rc of 3:

import ctypes
from System import Uri

u = Uri('http://google.com')
addr = id(u)

rc = ctypes.c_long.from_address(addr).value

del u
# tp_dealloc not called

Replacing Uri with Python's str gives rc 1.

The issue does not seem to happen when objects are created from .NET. E.g. this does not produce extra references:

var uri = new Uri("http://google.com").ToPython();
// uri.Refcount == 1
uri.Dispose();
// `tp_dealloc` called

Looks like we leak references when constructing new .NET objects from Python.

fixed leak in NewReference.Move

@lostmsu

@lostmsu Great work, thanks! And thank you @m-rossi for testing and reporting! Could you verify that the newest release branch indeed fixes the issue?

@lostmsu

Looks good, the gc sees no System.String-objects left and the memory consumption is better too:

Thanks @lostmsu !

fixed leak in NewReference.Move

Navigation Menu

Repository navigation

Deleted dotnet object does not get garbage collected #1872

Environment

Details

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Deleted dotnet object does not get garbage collected #1872

Description

Environment

Details

Activity

lostmsu commented on Jul 13, 2022

m-rossi commented on Jul 14, 2022

filmor commented on Jul 14, 2022

m-rossi commented on Jul 14, 2022

lostmsu commented on Jul 14, 2022

filmor commented on Jul 14, 2022

m-rossi commented on Jul 15, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions