You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
configure options don't trickle down to distutils #48260
If you are fortunate enough to have all your third-party libraries in
a single quasi-standard location, say, /usr/local/lib, you will
probably have never encountered this problem, but setting environment
variables like LDFLAGS don't get translated into the relevant args for
the distutils build_ext command. The evidence for this is this output
at the end of the make process:
Failed to find the necessary bits to build these modules:
\_hashlib \_sqlite3 \_ssl
bsddb185 linuxaudiodev ossaudiodev
To find the necessary bits, look in setup.py in detect_modules() for
the module's name.
Failed to build these modules:
_curses _curses_panel _tkinter
gdbm readline
Here's the specific failure for readline:
>>> import readline_failed
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ld.so.1: python: fatal: libreadline.so.4: open failed: No
such file or directory
yet if you poke around in my config.status file you'll see that I set
the relevant -L and -R flags:
It's not hard to work around this problem, but it's tedious to build
the command line args from the massive list of directories I have to
educate distutils about.
Well, all of my modules are in a non-standard location and I have no
build issues on OS X 10.5. If you look at PyBuildExt.detect_modules()
you will see that the paths are at least added for the search path from
LDFLAGS and CPPFLAGS.
Brett> Well, all of my modules are in a non-standard location and I have
Brett> no build issues on OS X 10.5. If you look at
Brett> PyBuildExt.detect_modules() you will see that the paths are at
Brett> least added for the search path from LDFLAGS and CPPFLAGS.
Ah, there's the problem. It picks through LDFLAGS looking for -L but fails
to pay attention to -R. I'll see if I can work up a patch.
Also, I'm seeing this problem on Solaris. I suppose it's possible that
linkage on OS X records the location of the libraries it finds, sort of
implicitly doing a -R.
One of the problems that I see in that LDFLAGS is Makefile variable and
Makefile is part of distribution for posix build systems.
If you set specific LDFLAGS and you want to distribute own python build
user will get you specific settings.
One another environment variable is OPT (if compiler don't accept -R XXX
may be -Wl,-rpath,XXX ?) and we may use it. Also OPT environment
variable is also Makefile variable and user specific settings will go
into distribution too :( .
I quick look into distutils code show that sysconfig.py don't parse
LDFLAGS and append environment variable LDFLAGS as is to the ldshared
variable.
So the first question is why setup.py parse only -L flags instead to
pass variable value as is (aka sysconfig.py).
The second question is why LDFLAGS is a Makefile variable ?
For now I would like to know if you set on Solaris suitable
LD_RUN_PATH(for the build) and LD_LIBRARY_PATH (when run) did you
succeed to build and run python ?
One of the problems that I see in that LDFLAGS is Makefile variable and
Makefile is part of distribution for posix build systems.
If you set specific LDFLAGS and you want to distribute own python build
user will get you specific settings.
One another environment variable is OPT (if compiler don't accept -R XXX
may be -Wl,-rpath,XXX ?) and we may use it. Also OPT environment
variable is also Makefile variable and user specific settings will go
into distribution too :( .
I quick look into distutils code show that sysconfig.py don't parse
LDFLAGS and append environment variable LDFLAGS as is to the ldshared
variable.
So the first question is why setup.py parse only -L flags instead to
pass variable value as is (aka sysconfig.py).
Because that's the use case I was supporting when I added the support;
I just needed the -L flags for compiling the extensions. If someone
writes a patch that makes it more generic it has a chance of being
checked in.
The second question is why LDFLAGS is a Makefile variable ?
Because there is no guarantee that someone specified the envvar when
running setup.py, but if they were set for configure you know that
Python was built using those values.
rpetrov commented on Oct 5, 2008
rpetrovmannequin
Mannequin
More actions
Lets see method customize_compiler(from sysconfig.py)
The method search variables 'CPP', 'LDFLAGS' and 'CPPFLAGS' only in
environment. The variable CPP is not a Makefile variable. 'LDFLAGS' and
'CPPFLAGS' are makefile variables(macros). Usually makefile will export
to commands a macro only if exists environment variable with same name.
So to pass to distutils LDFLAGS is enough to to run as example
"LDFLAGS=fake make".
bpo-3678 is about LDFLAGS for python shared library and now command
start with $(LDSHARED) $(LDFLAGS) ...
If we modify Makefile to use "...LDFLAGS=$(LDFLAGS) ... setup.py"
customize_compiler will append LDFLAGS to LDSHARED.
So this issue is about to use LDFLAGS for python modules.
Another point is that configure script append LDFLAGS to LDSHARED only
on certain platforms (all BSD based) and ignore all others.
What is preferred patch:
a simple patch that just export LDFLAGS for setup.py;
a patch that export LDFLAGS for all python programs, i.e. in Makefile
to append to RUNSHARED LDFLAGS=@ldflags@ ;
a patch that append LDFLAGS to LDSHARED in configure for all platforms
and remove use of LDFLAGS in Makefile and setup.py ?
I checked in r66823 to add the -R flag extraction like Brett's -L
extraction. I'll leave the issue open while folks discuss Roumen's
proposal. I have no particular desire to delve deeper into
distutils.
rpetrov commented on Oct 10, 2008
rpetrovmannequin
Mannequin
More actions
I don't have rights to derive a new issue.
May be a new new title about distutils and makefile integration is more
appropriate.
Hi. We encountered this issue on a Solaris 10 while building python 2.6.2.
There's a problem with reptrov's patch from 2009-01-04. The LDFLAGS in the
Makefile.pre.in patch need to be quoted also. Like so:
LDFLAGS='$(LDFLAGS)'
This allows for multiple flags. Other than that, it works for us.
speno commented on Aug 21, 2009
spenomannequin
Mannequin
More actions
Furthermore, there's another bug in setup.py When extensions are built
and CPPFLAGS (or other arguments) has multiple arguments, the order of
the arguments are reversed. This is the wrong behavior. The specified
directories should be used in the given order.
For example, if CPPFLAGS="-I/one -I/two -I/three", then when an
extention is build, the order of the compile would be:
cc -I/three -I/two -I/one ...
There's this bit that seems reponsible in setup.py:
When I did the patch they were coming in reversed, so I reversed the reversal. Obviously something changed so now the manual reversal is not necessary anymore.
Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.
If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
Activity
smontanaro commentedon Oct 1, 2008
If you are fortunate enough to have all your third-party libraries in
a single quasi-standard location, say, /usr/local/lib, you will
probably have never encountered this problem, but setting environment
variables like LDFLAGS don't get translated into the relevant args for
the distutils build_ext command. The evidence for this is this output
at the end of the make process:
the module's name.
Here's the specific failure for readline:
>>> import readline_failed Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: ld.so.1: python: fatal: libreadline.so.4: open failed: No such file or directoryyet if you poke around in my config.status file you'll see that I set
the relevant -L and -R flags:
-R/opt/app/nonc++/ncurses-5.6/lib -L/opt/app/nonc++/gdbm-1.8/lib
-R/opt/app/nonc++/gdbm-1.8/lib -L/opt/app/nonc++/readline-4.3/lib
-R/opt/app/nonc++/readline-4.3/lib -L/opt/app/nonc++/tcl-8.4/lib
-R/opt/app/nonc++/tcl-8.4/lib -L/opt/app/nonc++/BerkleyDB-4.3/lib
-R/opt/app/nonc++/BerkleyDB-4.3/lib,g
The workaround is to run the build_ext command separately:
It's not hard to work around this problem, but it's tedious to build
the command line args from the massive list of directories I have to
educate distutils about.
brettcannon commentedon Oct 1, 2008
Well, all of my modules are in a non-standard location and I have no
build issues on OS X 10.5. If you look at PyBuildExt.detect_modules()
you will see that the paths are at least added for the search path from
LDFLAGS and CPPFLAGS.
smontanaro commentedon Oct 1, 2008
Brett> Well, all of my modules are in a non-standard location and I have
Brett> no build issues on OS X 10.5. If you look at
Brett> PyBuildExt.detect_modules() you will see that the paths are at
Brett> least added for the search path from LDFLAGS and CPPFLAGS.
Ah, there's the problem. It picks through LDFLAGS looking for -L but fails
to pay attention to -R. I'll see if I can work up a patch.
Also, I'm seeing this problem on Solaris. I suppose it's possible that
linkage on OS X records the location of the libraries it finds, sort of
implicitly doing a -R.
Skip
smontanaro commentedon Oct 1, 2008
Here's a patch. Works for me on Solaris 10. I'll try to check
it out on OS X 10.5. Would appreciate it if someone on Linux can
kick the tires too.
smontanaro commentedon Oct 2, 2008
Confirmed that nothing seems broken on my OS X 10.5 laptop (doing a
unix-style build, not a framework build).
smontanaro commentedon Oct 2, 2008
Seems to work for framework builds as well.
rpetrov commentedon Oct 2, 2008
One of the problems that I see in that LDFLAGS is Makefile variable and
Makefile is part of distribution for posix build systems.
If you set specific LDFLAGS and you want to distribute own python build
user will get you specific settings.
One another environment variable is OPT (if compiler don't accept -R XXX
may be -Wl,-rpath,XXX ?) and we may use it. Also OPT environment
variable is also Makefile variable and user specific settings will go
into distribution too :( .
I quick look into distutils code show that sysconfig.py don't parse
LDFLAGS and append environment variable LDFLAGS as is to the ldshared
variable.
So the first question is why setup.py parse only -L flags instead to
pass variable value as is (aka sysconfig.py).
The second question is why LDFLAGS is a Makefile variable ?
For now I would like to know if you set on Solaris suitable
LD_RUN_PATH(for the build) and LD_LIBRARY_PATH (when run) did you
succeed to build and run python ?
brettcannon commentedon Oct 2, 2008
On Thu, Oct 2, 2008 at 1:57 PM, Roumen Petrov <report@bugs.python.org> wrote:
Because that's the use case I was supporting when I added the support;
I just needed the -L flags for compiling the extensions. If someone
writes a patch that makes it more generic it has a chance of being
checked in.
Because there is no guarantee that someone specified the envvar when
running setup.py, but if they were set for configure you know that
Python was built using those values.
rpetrov commentedon Oct 5, 2008
Lets see method customize_compiler(from sysconfig.py)
The method search variables 'CPP', 'LDFLAGS' and 'CPPFLAGS' only in
environment. The variable CPP is not a Makefile variable. 'LDFLAGS' and
'CPPFLAGS' are makefile variables(macros). Usually makefile will export
to commands a macro only if exists environment variable with same name.
So to pass to distutils LDFLAGS is enough to to run as example
"LDFLAGS=fake make".
bpo-3678 is about LDFLAGS for python shared library and now command$(LDSHARED) $ (LDFLAGS) ...
start with
If we modify Makefile to use "...LDFLAGS=$(LDFLAGS) ... setup.py"
customize_compiler will append LDFLAGS to LDSHARED.
So this issue is about to use LDFLAGS for python modules.
Another point is that configure script append LDFLAGS to LDSHARED only
on certain platforms (all BSD based) and ignore all others.
What is preferred patch:
to append to RUNSHARED LDFLAGS=@ldflags@ ;
and remove use of LDFLAGS in Makefile and setup.py ?
smontanaro commentedon Oct 7, 2008
I checked in r66823 to add the -R flag extraction like Brett's -L
extraction. I'll leave the issue open while folks discuss Roumen's
proposal. I have no particular desire to delve deeper into
distutils.
rpetrov commentedon Oct 10, 2008
I don't have rights to derive a new issue.
May be a new new title about distutils and makefile integration is more
appropriate.
7 remaining items
akitada commentedon Feb 6, 2009
s/get_config_vars/get_config_var/
akitada commentedon Feb 28, 2009
Updated bpo-4010 to honor os.environ['CPPFLAGS'].
speno commentedon Aug 21, 2009
Hi. We encountered this issue on a Solaris 10 while building python 2.6.2.
There's a problem with reptrov's patch from 2009-01-04. The LDFLAGS in the
Makefile.pre.in patch need to be quoted also. Like so:
LDFLAGS='$(LDFLAGS)'
This allows for multiple flags. Other than that, it works for us.
speno commentedon Aug 21, 2009
Furthermore, there's another bug in setup.py When extensions are built
and CPPFLAGS (or other arguments) has multiple arguments, the order of
the arguments are reversed. This is the wrong behavior. The specified
directories should be used in the given order.
For example, if CPPFLAGS="-I/one -I/two -I/three", then when an
extention is build, the order of the compile would be:
cc -I/three -I/two -I/one ...
There's this bit that seems reponsible in setup.py:
merwok commentedon Nov 18, 2010
The reversed call was added by Brett in r60537. Brett, can you comment?
brettcannon commentedon Nov 18, 2010
When I did the patch they were coming in reversed, so I reversed the reversal. Obviously something changed so now the manual reversal is not necessary anymore.
merwok commentedon Nov 18, 2010
I don’t think there can be a unit test for that, so do we agree on just removing the reversed call?
brettcannon commentedon Nov 18, 2010
On Wed, Nov 17, 2010 at 17:19, Éric Araujo <report@bugs.python.org> wrote:
As long as the person removing it verifies it and keeps an eye on the
buildbots, sure.
zooba commentedon Feb 3, 2021
Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.
If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools