← 输入框|github.com|刷新
Skip to content

Compileall improvements #82293

Closed
Closed
Compileall improvements#82293
Labels
topic-installation

Description

@encukou
encukou
opened on Sep 11, 2019
BPO 38112
Nosy @gpshead, @vstinner, @encukou, @frenzymadness, @pablogsal
PRs
  • bpo-38112: Compileall improvements #16012
  • bpo-38112: compileall: Skip long path path on Windows if the path can't be created #16414
  • bpo-38112: compileall tests: Use shorter name for long_path test #16419
  • bpo-38112: Document that compileall.compile_[dir,file] also accept multiple opt levels #20174
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-10-15.13:40:03.440>
created_at = <Date 2019-09-11.14:01:19.510>
labels = ['expert-installation']
title = 'Compileall improvements'
updated_at = <Date 2020-05-18.13:23:44.140>
user = 'https://github.com/encukou'

    bugs.python.org fields:

    activity = <Date 2020-05-18.13:23:44.140>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-15.13:40:03.440>
closer = 'vstinner'
components = ['Installation']
creation = <Date 2019-09-11.14:01:19.510>
creator = 'petr.viktorin'
dependencies = []
files = []
hgrepos = []
issue_num = 38112
keywords = ['patch']
message_count = 11.0
messages = ['351887', '353266', '353271', '353272', '353275', '353286', '353319', '354610', '354728', '362783', '369215']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'vstinner', 'petr.viktorin', 'frenzy', 'pablogsal']
pr_nums = ['16012', '16414', '16419', '20174']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38112'
versions = []

    Activity

    1. encukou commented on Sep 11, 2019

      @encukou
      encukou
      on Sep 11, 2019
      MemberAuthor

      Hello,
      In Fedora, we've long used a Bash script to compile modules to bytecode, as the compileall module was historically unsuitable.
      Recently, Lumír re-wrote our enhancements to compileall, and published the result as compileall2 on PyPI. We'd like to get the improvements back into the standard library so everyone can benefit.
      The improvements are:

      • Raise the default recursion limit
      • Control of the path baked into the *.pyc file (-s to strip a prefix; -p to prepend a new one) to support different build/install locations.
      • Compiling for multiple optimization levels at once (by specifying -o multiple times)

      A PR should follow shortly.

    2. encukou
      added
      topic-installation
      on Sep 11, 2019

    3. encukou commented on Sep 26, 2019

      @encukou
      encukou
      on Sep 26, 2019
      MemberAuthor

      New changeset 8e7bb99 by Petr Viktorin (Lumír 'Frenzy' Balhar) in branch 'master':
      bpo-38112: Compileall improvements (GH-16012)
      8e7bb99

    4. vstinner commented on Sep 26, 2019

      @vstinner
      vstinner
      on Sep 26, 2019
      Member

      test_compileall failed on AMD64 Windows8.1 Non-Debug 3.x:
      https://buildbot.python.org/all/#/builders/12/builds/3257

      Example:

      ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_compileall.py", line 48, in setUp
    os.makedirs(self.long_path)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 18 more times]
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [WinError 206] The filename or extension is too long: 'D:\\temp\\tmpcgsk1z3q\\long\\1\\2\\3\\4\\5\\6\\7\\8\\9\\10\\11\\12\\13\\14\\15\\16\\17\\18\\19\\20\\21\\22\\23\\24\\25\\26\\27\\28\\29\\30\\31\\32\\33\\34\\35\\36\\37\\38\\39\\40\\41\\42\\43\\44\\45\\46\\47\\48\\49\\50\\51\\52\\53\\54\\55\\56\\57\\58\\59\\60\\61\\62\\63\\64\\65\\66\\67\\68\\69\\70\\71\\72\\73\\74\\75\\76\\77\\78'

      Failures:

      ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      FAIL: test_no_args_respects_force_flag (test.test_compileall.CommandLineTestsWithSourceEpoch)
      Re-running failed tests in verbose mode
      Re-running test_tarfile in verbose mode
      Re-running test_compileall in verbose mode
      ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
      ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
      ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)

      pythoninfo says:

      sys.windowsversion: sys.getwindowsversion(major=6, minor=3, build=9600, platform=2, service_pack='')
      windows.RtlAreLongPathsEnabled: <function not available>

    5. vstinner commented on Sep 26, 2019

      @vstinner
      vstinner
      on Sep 26, 2019
      Member

      Similar errors on AMD64 Windows10 3.x:
      https://buildbot.python.org/all/#/builders/3/builds/3567

      pythoninfo:

      sys.windowsversion: sys.getwindowsversion(major=10, minor=0, build=17134, platform=2, service_pack='')
      windows.RtlAreLongPathsEnabled: False

      --

      And AMD64 Windows7 SP1 3.x:
      https://buildbot.python.org/all/#/builders/40/builds/3180

      pythoninfo:

      sys.windowsversion: sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1')
      windows.RtlAreLongPathsEnabled: <function not available>

    6. encukou commented on Sep 26, 2019

      @encukou
      encukou
      on Sep 26, 2019
      MemberAuthor

      Thanks!
      The buildbot complained in the pull request a while ago; I commented there that I'm looking into it.

      I'd appreciate a review for the fix; I don't have a good way to test it other than offer it to the buildbots.

    7. encukou commented on Sep 26, 2019

      @encukou
      encukou
      on Sep 26, 2019
      MemberAuthor

      New changeset 4267c98 by Petr Viktorin in branch 'master':
      bpo-38112: compileall: Skip long path path on Windows if the path can't be created (GH-16414)
      4267c98

    8. vstinner commented on Sep 26, 2019

      @vstinner
      vstinner
      on Sep 26, 2019
      Member

      FYI test_compileall.test_compile_dir_maxlevels() also fails on AMD64 FreeBSD CURRENT Shared 3.x:
      https://buildbot.python.org/all/#builders/168/builds/1529

    9. pablogsal commented on Oct 14, 2019

      @pablogsal
      pablogsal
      on Oct 14, 2019
      Member

      The AMD64 Windows7 SP1 3.x buildbot has been failing for a while since commit 4267c98, please take a look at https://bugs.python.org/issue38470.

    10. vstinner commented on Oct 15, 2019

      @vstinner
      vstinner
      on Oct 15, 2019
      Member

      Should be fixed by:

      New changeset eb1dda2 by Petr Viktorin (Victor Stinner) in branch 'master':
      bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)
      eb1dda2

    11. vstinner
      closed this as completedon Oct 15, 2019

    12. gpshead commented on Feb 27, 2020

      @gpshead
      gpshead
      on Feb 27, 2020
      Member

      https://bugs.python.org/issue39769 may be of interest. compileall's ddir= does not behave properly in recursive subdirectories, it did the right thing in 2.7 which is similar to a combination of compile_dir(dir, prependdir=ddir, stripdir=dir) after this issue's feature.

    13. vstinner commented on May 18, 2020

      @vstinner
      vstinner
      on May 18, 2020
      Member

      New changeset adc72bb by Lumír 'Frenzy' Balhar in branch 'master':
      bpo-38112: Document that compileall.compile_[dir,file] also accept multiple opt levels (GH-20174)
      adc72bb

    14. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      topic-installation

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions