Multiprocessing spawn/forkserver fails to pass Queues #73151

Activity

1. duckythescientist commented on Dec 14, 2016

   

   duckythescientistmannequin

   on Dec 14, 2016

   MannequinAuthor

   More actions

   Python fails to pass a Queue when calling Process with multiprocessing.set_start_method set to "spawn" or "forkserver".

   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "/usr/lib/python3.5/multiprocessing/spawn.py", line 106, in spawn_main
       exitcode = _main(fd)
     File "/usr/lib/python3.5/multiprocessing/spawn.py", line 116, in _main
       self = pickle.load(from_parent)
     File "/usr/lib/python3.5/multiprocessing/synchronize.py", line 111, in __setstate__
       self._semlock = _multiprocessing.SemLock._rebuild(*state)
   FileNotFoundError: [Errno 2] No such file or directory

   Here is a minimized example:

   #!/usr/bin/env python3
   
   import multiprocessing
   
   def check_child(q):
       print("Queue", q)
   
   
   if __name__ == '__main__':
       multiprocessing.set_start_method('spawn')
       # multiprocessing.set_start_method('fork')
       # multiprocessing.set_start_method('forkserver')
   
       q = multiprocessing.Queue(-1)
       print("q", q)
   
       proc = multiprocessing.Process(target=check_child, args=(q,))
       proc.start()
   

   Also, this fails when the Queue is implicitly passed to the child.

   class Blerg():
       def __init__(self):
           self.q = multiprocessing.Queue(-1)
   
       def print_queue(self):
           print("Queue", self.q)
   
   
   if __name__ == '__main__':
       multiprocessing.set_start_method('spawn')
   
       blerg = Blerg()
   
       blerg.print_queue()
   
       proc = multiprocessing.Process(target=blerg.print_queue)
       proc.start()
   

   $ python3 --version
   Python 3.5.2

   Windows (which defaults to "spawn" style multiprocessing) does not seem to have this issue (at least in 2.7.12).
2. duckythescientist

   added

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Dec 14, 2016
3. applio

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   and removed

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Dec 14, 2016
4. piython commented on Dec 15, 2016

   

   piythonmannequin

   on Dec 15, 2016

   Mannequin

   More actions

   hi,

   few comments on this: [when tested with 'spawn']

   1. the document(3x, 17.2.1.2) clearly mentions that 'spawn' doesnt inherit the file-handles (hence FileNotFoundError).
   2. the document's examples has spawn() followed by join() where parent waits for the Child to finish and exits gracefully. The error "OSError: [Errno 9] Bad file descriptor" is seen for Child process.
   3. Isnt exception-handling for null file descriptors sufficient (with corresponding Document updates)

   am not a core developer yet and still trying to learn and contribute.

   Thanks
5. pitrou commented on Jul 22, 2017

   

   pitrou

   on Jul 22, 2017

   Member

   More actions

   Ok, this is really an error in your script. If you add p.join() to wait for the process to end, you'll see no error pops up. What really happens is that the parent process ends before the child is fully started, and the resource underlying the queue is cleaned up at this point.
6. pitrou commented on Jul 22, 2017

   

   pitrou

   on Jul 22, 2017

   Member

   More actions

   More exactly, since the child process is not "daemonic", the parent process waits for it to end, but after collecting the queue's resources.
7. pitrou commented on Jul 22, 2017

   

   pitrou

   on Jul 22, 2017

   Member

   More actions

   So, while it might be possible to improve on this, it would risk fragilizing other use cases, and besides, this is not a problem that is likely to happen in real usage. Therefore, I'd rather close this issue.
8. pitrou

   closed this as completedon Jul 22, 2017
9. mouse07410 commented on Mar 29, 2020

   

   mouse07410mannequin

   on Mar 29, 2020

   Mannequin

   More actions

   On MacOS Catalina 10.15.4, I still see this problem occasionally even with p.join() added. See https://bugs.python.org/msg365251 and subsequent messages.

   Also, see https://bugs.python.org/issue40106.
10. ezio-melotti

    transferred this issue fromon Apr 10, 2022
11. geryogam commented on May 8, 2022

    

    geryogam

    on May 8, 2022

    Contributor

    More actions

    Ok, this is really an error in your script. If you add p.join() to wait for the process to end, you'll see no error pops up. What really happens is that the parent process ends before the child is fully started, and the resource underlying the queue is cleaned up at this point.

    And if you don’t want to join the process explicitly like @pitrou suggested, here is an alternative: https://stackoverflow.com/a/72164506/2326961