Skip to content

2to3 Grammar pickles not created when upgrading to 3.3.0b2 #59850

Closed

Description

@stefanholek
BPO 15645
Nosy @terryjreedy, @ronaldoussoren, @benjaminp, @ned-deily, @regebro, @asvetlov, @hynek
Files
  • issue15645.txt
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2012-08-23.06:41:25.318>
    created_at = <Date 2012-08-14.08:16:19.821>
    labels = ['build', 'expert-2to3']
    title = '2to3 Grammar pickles not created when upgrading to 3.3.0b2'
    updated_at = <Date 2012-09-09.02:48:16.356>
    user = 'https://bugs.python.org/stefanholek'

    bugs.python.org fields:

    activity = <Date 2012-09-09.02:48:16.356>
    actor = 'ned.deily'
    assignee = 'none'
    closed = True
    closed_date = <Date 2012-08-23.06:41:25.318>
    closer = 'ned.deily'
    components = ['Build', '2to3 (2.x to 3.x conversion tool)']
    creation = <Date 2012-08-14.08:16:19.821>
    creator = 'stefanholek'
    dependencies = []
    files = ['26916']
    hgrepos = []
    issue_num = 15645
    keywords = ['patch', 'needs review']
    message_count = 19.0
    messages = ['168180', '168465', '168480', '168483', '168484', '168629', '168677', '168678', '168679', '168703', '168737', '168858', '168924', '168926', '169678', '169679', '169817', '170025', '170074']
    nosy_count = 10.0
    nosy_names = ['terry.reedy', 'ronaldoussoren', 'benjamin.peterson', 'ned.deily', 'lregebro', 'Arfrever', 'asvetlov', 'stefanholek', 'python-dev', 'hynek']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue15645'
    versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

    Activity

    1. stefanholek commented on Aug 14, 2012

      stefanholekmannequin
      MannequinAuthor

      After upgrading 3.3.0b1 to b2 (make install to same location) I noticed that the Grammar pickles for beta.2 were missing (the beta.1 pickles still being there).

      The result is permission errors and/or virtualenv SandboxViolations, when 2to3 is used for the first time. E.g.:

      SandboxViolation: open('/usr/local/python3.3/lib/python3.3/lib2to3/Grammar3.3.0.beta.2.pickle', 'wb') {}

      This is because users of Python (as opposed to root) may of course not write to the library dir.

      The pickles are created fine on a fresh install of Python 3.3.0b2 (make install to new location).

    2. terryjreedy commented on Aug 17, 2012

      @terryjreedy
      Member

      Your OS and distribution/version might be relevant information.

    3. stefanholek commented on Aug 17, 2012

      stefanholekmannequin
      MannequinAuthor

      Mac OS X 10.6.8 (Snow Leopard)
      Xcode Tools 3.2.6

      ./configure --prefix=/usr/local/python3.3
      make
      sudo make install

    4. terryjreedy commented on Aug 17, 2012

      @terryjreedy
      Member

      I believe there are known problems with 3.3 and mac, so I added the mac experts.

    5. ned-deily commented on Aug 17, 2012

      @ned-deily
      Member

      I took a quick look at the issue and I could reproduce by doing a "make install" of 3.3.0b1 then a build and "make install" of 3.3.0b2 in the same locations. Doing a clean 3.3.0b2 build and install produce the expected results. It appears to be a unix Makefile issue that shouldn't be unique to OS X. I don't have time to look further into it right at the moment, though. (And I'm not sure what "known problems with 3.3 and mac" means; the buildbots are happy.)

    6. ronaldoussoren commented on Aug 20, 2012

      @ronaldoussoren
      Contributor

      I'm pretty sure this is a generic Makefile bug. I could reproduce this problem with a clean install into a new prefix, after cleaning up the source tree: 'make clean' won't remove the generated file when srcdir != builddir.

      This line in the Makefile seems to generate the pickle file:

          -PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
                  $(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram,lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"
      

      This uses $(PYTHON_FOR_BUILD) to generate the cache file, and that which means the cache file is generated in the *source* tree, not in the installation tree. This command is at the end of the libinstall target, which means the generatd file is not copied into the installation prefix.

      Running 'make install' twice will therefore install the file.

      The attached patch seems to fix the issue by moving the command to the start of the libinstall target. The patch also fixes "make clean", by explicitly targetting the pickle files in $(srcdir)/Lib/lib2to3.

      BTW. What "known OSX problems"?

      BTW2. This should affect all platforms that use Makefiles (basicly anything but Windows)

    7. terryjreedy commented on Aug 20, 2012

      @terryjreedy
      Member

      I had the impression that there *were* problems with compiling 3.3 on the new OSX and Xcode or something. If that is wrong or all fixed, great. Sorry for the noise.

    8. ronaldoussoren commented on Aug 20, 2012

      @ronaldoussoren
      Contributor

      There were some problems building using llvm-gcc, those have been resolved by detecting falling back to clang when llvm-gcc is detected. That's the only real problem I know of.

    9. ronaldoussoren commented on Aug 20, 2012

      @ronaldoussoren
      Contributor

      BTW. I wouldn't mind a patch review, particularly by someone that can easily test on Linux. I'm not too keen on checking in this patch without a test on Linux, not this close to 3.3rc1.

    10. terryjreedy commented on Aug 20, 2012

      @terryjreedy
      Member

      If necessary, ask on pydev.

    11. ned-deily commented on Aug 21, 2012

      @ned-deily
      Member

      Patch LGTM and I verified that it does indeed install the pickles properly on both a vanilla Linux build/install and with an OS X framework build/install. (Note that, as expected, previously installed versioned pickles in the install library directory are not removed. They will harmlessly accumulate until you do a clean install or manually remove them.)

    12. regebro commented on Aug 22, 2012

      regebromannequin
      Mannequin

      I can confirm that this is an issue on Ubuntu 12.04 as well, and that removing the existing pre 3.3b2 install before installing solves the problem.

    13. python-dev commented on Aug 23, 2012

      python-devmannequin
      Mannequin

      New changeset 787ed9b03ef9 by Ned Deily in branch '2.7':
      Issue bpo-15645: Ensure 2to3 grammar pickles are properly installed.
      http://hg.python.org/cpython/rev/787ed9b03ef9

      New changeset a377a4298b4e by Ned Deily in branch '3.2':
      Issue bpo-15645: Ensure 2to3 grammar pickles are properly installed.
      http://hg.python.org/cpython/rev/a377a4298b4e

      New changeset b48cd7045909 by Ned Deily in branch 'default':
      issue bpo-15645: null merge
      http://hg.python.org/cpython/rev/b48cd7045909

      New changeset 995e58439b59 by Ned Deily in branch 'default':
      Issue bpo-15645: Ensure 2to3 grammar pickles are properly installed.
      http://hg.python.org/cpython/rev/995e58439b59

    14. ned-deily commented on Aug 23, 2012

      @ned-deily
      Member

      Fixes applied for release in 2.7.4, 3.2.4, and 3.3.0.

    15. removed
      buildThe build process and cross-build
      on Aug 23, 2012
    16. Arfrever commented on Sep 1, 2012

      Arfrevermannequin
      Mannequin

      libinstall target now contains:

      -PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
      	$(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"
      ...
      @for d in $(LIBSUBDIRS); \
      do \
      	a=$(srcdir)/Lib/$$d; \
      	if test ! -d $$a; then continue; else true; fi; \
      	if test `ls $$a | wc -l` -lt 1; then continue; fi; \
      	b=$(LIBDEST)/$$d; \
      	for i in $$a/*; \
      

      There is a small chance that with high parallelization the wildcard expansion in the last line will occur for $$a == lib2to3 before creation of pickles has finished, resulting in not installation of pickles.
      The solution would be to move creation of pickles to a separate target and make it a dependency of libinstall target.

    17. Arfrever commented on Sep 1, 2012

      Arfrevermannequin
      Mannequin

      I meant $$d == lib2to3 and $$a == $(srcdir)/Lib/lib2to3

    18. ronaldoussoren commented on Sep 4, 2012

      @ronaldoussoren
      Contributor

      Is it really possible that the pickle is created after calculation of the wildcard expansion? The GNU make manual says that make will run multiple recipes in parallel when the -j option is used (that is, multiple targets are build at the same time), not that the commands in a recipe are run at the same time. That latter would break a lot of assumptions.

    19. ronaldoussoren commented on Sep 8, 2012

      @ronaldoussoren
      Contributor

      See also bpo-15838

    20. ned-deily commented on Sep 9, 2012

      @ned-deily
      Member

      Is it really possible that the pickle is created after calculation of the wildcard expansion?

      With the current GNUmake, it seems it is not possible. GNUmake appears to serializes the individual steps in a target's recipe although I did not find an explicit guarantee in its documentation that that could never happen. (I did find at least one proprietary "make" replacement that does try to parallelize the individual steps in one target.)

      In any case, there were other problems with the approach implemented here and these changes have been superseded by those of bpo-15822.

    21. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      buildThe build process and cross-buildtopic-2to3

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions