Code objects can contain unmarshallable objects #89290

Activity

1. encukou commented on Sep 7, 2021

   

   encukou

   on Sep 7, 2021

   MemberAuthor

   More actions

   The replace method of code allows setting e.g.

   • co_filename to a subclass of str
   • co_consts to an arbitrary tuple
     and possibly more weird cases.

   This makes code objects unmarshallable.

   One way to create such a code object is to call compileall.compile_file with a str subclass as path. See the attached reproducers.

   This hit pip, see: pypa/pip#10358 (comment)
2. encukou commented on Sep 7, 2021

   

   encukou

   on Sep 7, 2021

   MemberAuthor

   More actions

   How to fix this? I guess:

   • co_filename should be converted to an actual str, or reject subclasses with an exception
   • for co_consts, the marshal docs could be updated to add code objects to "containers". The code docs already say co_consts "is a tuple containing the literals"; if someone's putting in non-literals they're voiding the warranty.

   And so on for all other fields of code objects.
3. encukou commented on Sep 7, 2021

   

   encukou

   on Sep 7, 2021

   Last edited by encukou

   MemberAuthor

   More actions

   See also gh-87005
4. ezio-melotti

   transferred this issue fromon Apr 10, 2022
5. iritkatriel

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Nov 27, 2023
6. vasiliyk commented on Sep 24, 2026

   

   vasiliyk

   on Sep 24, 2026

   Contributor

   More actions

   I want to propose to keep code objects unrestricted at creation and just improve the error at save time.

   Marshal already raises on dump, but the message does not say where the problem is. A code object can contain many values, and with nested code objects it is hard to tell what MyStr refers to.

   >>> import marshal
   >>> class MyStr(str): pass
   ... marshal.dumps(compile("", MyStr("foo.py"), "exec"))
   ...
   Traceback (most recent call last):
     File "<python-input-1>", line 2, in <module>
       marshal.dumps(compile("", MyStr("foo.py"), "exec"))
       ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   ValueError: cannot marshal MyStr objects

   The error could carry context to make it more helpful:

   Traceback (most recent call last):
     File "<python-input-1>", line 2, in <module>
       marshal.dumps(compile("", MyStr("foo.py"), "exec"))
       ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   ValueError: cannot marshal MyStr objects: 'foo.py' (subclasses of str are not supported)
   when marshalling code object field co_filename
   when marshalling code object '<module>' (foo.py, line 1)

   This covers every path that creates a code object.

   Why not reject such objects at creation?

   • Most code objects are never written to a .pyc (exec, bytecode tools, replace() in tests); restricting creation would break code that works today.
   • Nothing changes for code that is not saved, so there is no compatibility concern.
   • It seems Java/.Net use a similar model (allow runtime-only code to reference arbitrary objects and restrict what can be saved). Not sure how much weight 'others do it too' can carry :]
7. picnixz commented on Sep 24, 2026

   

   picnixz

   on Sep 24, 2026

   Member

   More actions

   @serhiy-storchaka
8. picnixz commented on Sep 24, 2026

   

   picnixz

   on Sep 24, 2026

   Member

   More actions

   FTR, if someone messes up with code.replace, it is at their own discretion and we do not want to validate their modifications. This is why we closes previous similar issues
9. picnixz

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Sep 24, 2026
10. serhiy-storchaka commented on Sep 24, 2026

    

    serhiy-storchaka

    on Sep 24, 2026

    Member

    More actions

    I agree with @vasiliyk. We should not restrict the use which does not involve marshalling.

    Adding notes to locate error like in pickle and json is also a good idea. It was impossibly to implement it before the changes which were made few days ago -- we did not have a raised exception, only an error code.
11. kevinprimeraquecreo-stack

    added this to Project_VKon Oct 18, 2026
12. github-project-automation

    moved this to Todo in Project_VKon Oct 18, 2026
13. kevinprimeraquecreo-stack

    added this to Project_VKon Oct 18, 2026
14. vasiliyk commented on Oct 21, 2026

    

    vasiliyk

    on Oct 21, 2026

    Contributor

    More actions

    I have implemented the notes support for marshalling, but I am at the open PR limit that was set on Oct 20 for the repo.

    • diff: main...vasiliyk:cpython:gh-89290-marshal-notes
    • branch: https://github.com/vasiliyk/cpython/tree/gh-89290-marshal-notes

    The notes follow pickle's convention: one note per enclosing level, innermost first, with the same wording for container items, dict values and set elements.

    3 differences with pickle:

    • a failing dict key gets a note, pickle leaves it unannotated;
    • a failing slice field is named directly (start, stop, step), pickle reports a tuple index inside the reconstructor arguments;
    • code object fields are identified by name, qualified name, file and line, pickle rejects code objects.

    current:

    >>> import marshal
    >>> class MyStr(str): pass
    ... marshal.dumps(compile("def f(): pass", MyStr("foo.py"), "exec"))
    ... 
    Traceback (most recent call last):
      File "<python-input-1>", line 2, in <module>
        marshal.dumps(compile("def f(): pass", MyStr("foo.py"), "exec"))
        ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ValueError: cannot marshal MyStr objects

    with notes:

    >>> import marshal
    >>> class MyStr(str): pass
    ... 
    >>> marshal.dumps(compile("def f(): pass", MyStr("foo.py"), "exec"))
    Traceback (most recent call last):
      File "<python-input-2>", line 1, in <module>
        marshal.dumps(compile("def f(): pass", MyStr("foo.py"), "exec"))
        ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ValueError: cannot marshal MyStr objects
    when marshalling co_filename of code object f (foo.py, line 1)
    when marshalling tuple item 0
    when marshalling co_consts of code object <module> (foo.py, line 1)

    New tests cover every container type, slice field and code object field.
    I will open the PR as soon as one of my open PRs is merged.