You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
2to3 Grammar pickles not created when upgrading to 3.3.0b2 #59850
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=<Date2012-08-23.06:41:25.318>created_at=<Date2012-08-14.08:16:19.821>labels= ['build', 'expert-2to3']
title='2to3 Grammar pickles not created when upgrading to 3.3.0b2'updated_at=<Date2012-09-09.02:48:16.356>user='https://bugs.python.org/stefanholek'
After upgrading 3.3.0b1 to b2 (make install to same location) I noticed that the Grammar pickles for beta.2 were missing (the beta.1 pickles still being there).
The result is permission errors and/or virtualenv SandboxViolations, when 2to3 is used for the first time. E.g.:
I took a quick look at the issue and I could reproduce by doing a "make install" of 3.3.0b1 then a build and "make install" of 3.3.0b2 in the same locations. Doing a clean 3.3.0b2 build and install produce the expected results. It appears to be a unix Makefile issue that shouldn't be unique to OS X. I don't have time to look further into it right at the moment, though. (And I'm not sure what "known problems with 3.3 and mac" means; the buildbots are happy.)
I'm pretty sure this is a generic Makefile bug. I could reproduce this problem with a clean install into a new prefix, after cleaning up the source tree: 'make clean' won't remove the generated file when srcdir != builddir.
This line in the Makefile seems to generate the pickle file:
This uses $(PYTHON_FOR_BUILD) to generate the cache file, and that which means the cache file is generated in the *source* tree, not in the installation tree. This command is at the end of the libinstall target, which means the generatd file is not copied into the installation prefix.
Running 'make install' twice will therefore install the file.
The attached patch seems to fix the issue by moving the command to the start of the libinstall target. The patch also fixes "make clean", by explicitly targetting the pickle files in $(srcdir)/Lib/lib2to3.
BTW. What "known OSX problems"?
BTW2. This should affect all platforms that use Makefiles (basicly anything but Windows)
I had the impression that there *were* problems with compiling 3.3 on the new OSX and Xcode or something. If that is wrong or all fixed, great. Sorry for the noise.
There were some problems building using llvm-gcc, those have been resolved by detecting falling back to clang when llvm-gcc is detected. That's the only real problem I know of.
BTW. I wouldn't mind a patch review, particularly by someone that can easily test on Linux. I'm not too keen on checking in this patch without a test on Linux, not this close to 3.3rc1.
Patch LGTM and I verified that it does indeed install the pickles properly on both a vanilla Linux build/install and with an OS X framework build/install. (Note that, as expected, previously installed versioned pickles in the install library directory are not removed. They will harmlessly accumulate until you do a clean install or manually remove them.)
regebro commented on Aug 22, 2012
regebromannequin
Mannequin
More actions
I can confirm that this is an issue on Ubuntu 12.04 as well, and that removing the existing pre 3.3b2 install before installing solves the problem.
-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
$(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"
...
@for d in $(LIBSUBDIRS); \
do \
a=$(srcdir)/Lib/$$d; \
if test ! -d $$a; then continue; else true; fi; \
if test `ls $$a | wc -l` -lt 1; then continue; fi; \
b=$(LIBDEST)/$$d; \
for i in $$a/*; \
There is a small chance that with high parallelization the wildcard expansion in the last line will occur for $$a == lib2to3 before creation of pickles has finished, resulting in not installation of pickles.
The solution would be to move creation of pickles to a separate target and make it a dependency of libinstall target.
Arfrever commented on Sep 1, 2012
Arfrevermannequin
Mannequin
More actions
I meant $$d == lib2to3 and $$a == $(srcdir)/Lib/lib2to3
Is it really possible that the pickle is created after calculation of the wildcard expansion? The GNU make manual says that make will run multiple recipes in parallel when the -j option is used (that is, multiple targets are build at the same time), not that the commands in a recipe are run at the same time. That latter would break a lot of assumptions.
Is it really possible that the pickle is created after calculation of the wildcard expansion?
With the current GNUmake, it seems it is not possible. GNUmake appears to serializes the individual steps in a target's recipe although I did not find an explicit guarantee in its documentation that that could never happen. (I did find at least one proprietary "make" replacement that does try to parallelize the individual steps in one target.)
In any case, there were other problems with the approach implemented here and these changes have been superseded by those of bpo-15822.
Activity
stefanholek commentedon Aug 14, 2012
After upgrading 3.3.0b1 to b2 (make install to same location) I noticed that the Grammar pickles for beta.2 were missing (the beta.1 pickles still being there).
The result is permission errors and/or virtualenv SandboxViolations, when 2to3 is used for the first time. E.g.:
SandboxViolation: open('/usr/local/python3.3/lib/python3.3/lib2to3/Grammar3.3.0.beta.2.pickle', 'wb') {}
This is because users of Python (as opposed to root) may of course not write to the library dir.
The pickles are created fine on a fresh install of Python 3.3.0b2 (make install to new location).
terryjreedy commentedon Aug 17, 2012
Your OS and distribution/version might be relevant information.
stefanholek commentedon Aug 17, 2012
Mac OS X 10.6.8 (Snow Leopard)
Xcode Tools 3.2.6
./configure --prefix=/usr/local/python3.3
make
sudo make install
terryjreedy commentedon Aug 17, 2012
I believe there are known problems with 3.3 and mac, so I added the mac experts.
ned-deily commentedon Aug 17, 2012
I took a quick look at the issue and I could reproduce by doing a "make install" of 3.3.0b1 then a build and "make install" of 3.3.0b2 in the same locations. Doing a clean 3.3.0b2 build and install produce the expected results. It appears to be a unix Makefile issue that shouldn't be unique to OS X. I don't have time to look further into it right at the moment, though. (And I'm not sure what "known problems with 3.3 and mac" means; the buildbots are happy.)
ronaldoussoren commentedon Aug 20, 2012
I'm pretty sure this is a generic Makefile bug. I could reproduce this problem with a clean install into a new prefix, after cleaning up the source tree: 'make clean' won't remove the generated file when srcdir != builddir.
This line in the Makefile seems to generate the pickle file:
This uses $(PYTHON_FOR_BUILD) to generate the cache file, and that which means the cache file is generated in the *source* tree, not in the installation tree. This command is at the end of the libinstall target, which means the generatd file is not copied into the installation prefix.
Running 'make install' twice will therefore install the file.
The attached patch seems to fix the issue by moving the command to the start of the libinstall target. The patch also fixes "make clean", by explicitly targetting the pickle files in $(srcdir)/Lib/lib2to3.
BTW. What "known OSX problems"?
BTW2. This should affect all platforms that use Makefiles (basicly anything but Windows)
terryjreedy commentedon Aug 20, 2012
I had the impression that there *were* problems with compiling 3.3 on the new OSX and Xcode or something. If that is wrong or all fixed, great. Sorry for the noise.
ronaldoussoren commentedon Aug 20, 2012
There were some problems building using llvm-gcc, those have been resolved by detecting falling back to clang when llvm-gcc is detected. That's the only real problem I know of.
ronaldoussoren commentedon Aug 20, 2012
BTW. I wouldn't mind a patch review, particularly by someone that can easily test on Linux. I'm not too keen on checking in this patch without a test on Linux, not this close to 3.3rc1.
terryjreedy commentedon Aug 20, 2012
If necessary, ask on pydev.
ned-deily commentedon Aug 21, 2012
Patch LGTM and I verified that it does indeed install the pickles properly on both a vanilla Linux build/install and with an OS X framework build/install. (Note that, as expected, previously installed versioned pickles in the install library directory are not removed. They will harmlessly accumulate until you do a clean install or manually remove them.)
regebro commentedon Aug 22, 2012
I can confirm that this is an issue on Ubuntu 12.04 as well, and that removing the existing pre 3.3b2 install before installing solves the problem.
python-dev commentedon Aug 23, 2012
New changeset 787ed9b03ef9 by Ned Deily in branch '2.7':
Issue bpo-15645: Ensure 2to3 grammar pickles are properly installed.
http://hg.python.org/cpython/rev/787ed9b03ef9
New changeset a377a4298b4e by Ned Deily in branch '3.2':
Issue bpo-15645: Ensure 2to3 grammar pickles are properly installed.
http://hg.python.org/cpython/rev/a377a4298b4e
New changeset b48cd7045909 by Ned Deily in branch 'default':
issue bpo-15645: null merge
http://hg.python.org/cpython/rev/b48cd7045909
New changeset 995e58439b59 by Ned Deily in branch 'default':
Issue bpo-15645: Ensure 2to3 grammar pickles are properly installed.
http://hg.python.org/cpython/rev/995e58439b59
ned-deily commentedon Aug 23, 2012
Fixes applied for release in 2.7.4, 3.2.4, and 3.3.0.
Arfrever commentedon Sep 1, 2012
libinstall target now contains:
There is a small chance that with high parallelization the wildcard expansion in the last line will occur for $$a == lib2to3 before creation of pickles has finished, resulting in not installation of pickles.
The solution would be to move creation of pickles to a separate target and make it a dependency of libinstall target.
Arfrever commentedon Sep 1, 2012
I meant $$d == lib2to3 and $$a == $(srcdir)/Lib/lib2to3
ronaldoussoren commentedon Sep 4, 2012
Is it really possible that the pickle is created after calculation of the wildcard expansion? The GNU make manual says that make will run multiple recipes in parallel when the -j option is used (that is, multiple targets are build at the same time), not that the commands in a recipe are run at the same time. That latter would break a lot of assumptions.
ronaldoussoren commentedon Sep 8, 2012
See also bpo-15838
ned-deily commentedon Sep 9, 2012
With the current GNUmake, it seems it is not possible. GNUmake appears to serializes the individual steps in a target's recipe although I did not find an explicit guarantee in its documentation that that could never happen. (I did find at least one proprietary "make" replacement that does try to parallelize the individual steps in one target.)
In any case, there were other problems with the approach implemented here and these changes have been superseded by those of bpo-15822.