Skip to content

make install tries to create lib2to3 grammar pickles in source directory #60042

Closed

Description

@ned-deily
BPO 15838
Nosy @ronaldoussoren, @benjaminp, @ned-deily, @tpn
Superseder
  • bpo-15822: installed python may fail incorrectly trying to rebuild lib2to3 grammar pickles
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2012-09-09.02:38:25.590>
    created_at = <Date 2012-09-01.11:26:44.135>
    labels = ['expert-2to3', 'build', 'expert-installation']
    title = 'make install tries to create lib2to3 grammar pickles in source directory'
    updated_at = <Date 2012-09-09.02:38:25.589>
    user = 'https://github.com/ned-deily'

    bugs.python.org fields:

    activity = <Date 2012-09-09.02:38:25.589>
    actor = 'ned.deily'
    assignee = 'none'
    closed = True
    closed_date = <Date 2012-09-09.02:38:25.590>
    closer = 'ned.deily'
    components = ['Build', 'Installation', '2to3 (2.x to 3.x conversion tool)']
    creation = <Date 2012-09-01.11:26:44.135>
    creator = 'ned.deily'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 15838
    keywords = []
    message_count = 3.0
    messages = ['169623', '169818', '170073']
    nosy_count = 5.0
    nosy_names = ['ronaldoussoren', 'benjamin.peterson', 'ned.deily', 'trent', 'Arfrever']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '15822'
    type = None
    url = 'https://bugs.python.org/issue15838'
    versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']

    Activity

    1. ned-deily commented on Sep 1, 2012

      @ned-deily
      MemberAuthor

      With the recent activity around making out-of-tree builds work, while investigating bpo-15822 I realized that the lib2to3 grammar pickle files are being built in the source directory by the libinstall target step of the main Makefile rather than in the build directory or just in the install location. The fixes for bpo-15645 moved the pickle building to the start of the target recipes:

      libinstall: build_all $(srcdir)/Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c
      -PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
      $(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"

      On reflection, I think the PYTHONPATH definition is bogus here since LIBDEST is populated immediately after this step. More importantly, lib2to3/pgen2/driver:load_grammar ends up writing the pickle files into the source directory. Unlike some other cases in the Makefile, if the source directory is read-only, the failure to create the pickle files is not an error during installation but it may likely cause problems for later users of the installed lib2to3 (like in the bpo-15822 scenario), i.e. when the missing pickles are attempted to be created in the read-only install directories.

    2. ronaldoussoren commented on Sep 4, 2012

      @ronaldoussoren
      Contributor

      FWIW the patch is bpo-15645 was a patch that fixed the issue with minimal changes, as it was created close to the first 3.3 release candidate I didn't want to change code when that could be avoided.

      A fix that writes the generated file into the build tree instead of the source tree requires changes to lib2to3.pgen.driver:load_grammar: it assumes it can derive the path of the pickle file from the path of the grammar file, which wouldn't be true when storing the pickle file in the build directory and the build directory is not the same as the source directory.

      Now that I look at this code again: the patch is bpo-15645 was probably make necessary by the introduction of $(PYTHON_FOR_BUILD), that expands to './$(BUILD_PYTHON) -E' on my machine, and that means the PYTHONPATH setting is ignored which is why the file is created in the source directory instead of the installation directory.

      This means it could be fairly easy to properly fix this issue:

      • Move the line back to the end of the libinstall recipe
      • Add "import sys, os; sys.path.insert(0, os.environ['PYTHONPATH']"
        to the start of the python scriptlet (before "import lib2to3....")

      I haven't tested yet if that actually works though.

    3. ned-deily commented on Sep 9, 2012

      @ned-deily
      MemberAuthor

      This problem is resolved by the fixes for bpo-15822 which implements a more explicit way of installing the grammar pickles.

    4. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions