Skip to content

configure options don't trickle down to distutils #48260

Closed

Description

@smontanaro
BPO 4010
Nosy @jcea, @tarekziade, @merwok, @akitada, @zooba
Files
  • runtime.diff
  • py-issue-4010.patch: LDFLAGS as is for distutils
  • issue4010.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tarekziade'
    closed_at = <Date 2021-02-03.18:30:40.489>
    created_at = <Date 2008-10-01.16:43:52.654>
    labels = ['type-bug', 'library', 'build']
    title = "configure options don't trickle down to distutils"
    updated_at = <Date 2021-02-03.18:30:40.488>
    user = 'https://github.com/smontanaro'

    bugs.python.org fields:

    activity = <Date 2021-02-03.18:30:40.488>
    actor = 'steve.dower'
    assignee = 'tarek'
    closed = True
    closed_date = <Date 2021-02-03.18:30:40.489>
    closer = 'steve.dower'
    components = ['Build', 'Distutils']
    creation = <Date 2008-10-01.16:43:52.654>
    creator = 'skip.montanaro'
    dependencies = []
    files = ['11675', '12585', '13212']
    hgrepos = []
    issue_num = 4010
    keywords = ['patch']
    message_count = 25.0
    messages = ['74137', '74140', '74143', '74146', '74152', '74154', '74194', '74199', '74346', '74420', '74650', '76507', '77649', '79073', '79082', '80580', '81278', '82930', '91839', '91847', '121397', '121401', '121403', '121409', '386437']
    nosy_count = 7.0
    nosy_names = ['jcea', 'tarek', 'eric.araujo', 'rpetrov', 'akitada', 'speno', 'steve.dower']
    pr_nums = []
    priority = 'normal'
    resolution = 'out of date'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue4010'
    versions = ['Python 3.2']

    Activity

    1. smontanaro commented on Oct 1, 2008

      @smontanaro
      ContributorAuthor

      If you are fortunate enough to have all your third-party libraries in
      a single quasi-standard location, say, /usr/local/lib, you will
      probably have never encountered this problem, but setting environment
      variables like LDFLAGS don't get translated into the relevant args for
      the distutils build_ext command. The evidence for this is this output
      at the end of the make process:

      Failed to find the necessary bits to build these modules:
      \_hashlib           \_sqlite3           \_ssl
      bsddb185           linuxaudiodev      ossaudiodev
      To find the necessary bits, look in setup.py in detect_modules() for
      

      the module's name.

      Failed to build these modules:
      _curses            _curses_panel      _tkinter
      gdbm               readline
      

      Here's the specific failure for readline:

          >>> import readline_failed
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
      	ImportError: ld.so.1: python: fatal: libreadline.so.4: open failed: No
      such file or directory

      yet if you poke around in my config.status file you'll see that I set
      the relevant -L and -R flags:

      s,@LDFLAGS@,|#_!!_#|-L/opt/app/nonc++/ncurses-5.6/lib
      

      -R/opt/app/nonc++/ncurses-5.6/lib -L/opt/app/nonc++/gdbm-1.8/lib
      -R/opt/app/nonc++/gdbm-1.8/lib -L/opt/app/nonc++/readline-4.3/lib
      -R/opt/app/nonc++/readline-4.3/lib -L/opt/app/nonc++/tcl-8.4/lib
      -R/opt/app/nonc++/tcl-8.4/lib -L/opt/app/nonc++/BerkleyDB-4.3/lib
      -R/opt/app/nonc++/BerkleyDB-4.3/lib,g

      The workaround is to run the build_ext command separately:

      ./python ../setup.py build_ext --library-dirs=... --rpath=...
      

      It's not hard to work around this problem, but it's tedious to build
      the command line args from the massive list of directories I have to
      educate distutils about.

    2. added
      type-bugAn unexpected behavior, bug, or error
      buildThe build process and cross-build
      stdlibStandard Library Python modules in the Lib/ directory
      on Oct 1, 2008
    3. brettcannon commented on Oct 1, 2008

      @brettcannon
      Member

      Well, all of my modules are in a non-standard location and I have no
      build issues on OS X 10.5. If you look at PyBuildExt.detect_modules()
      you will see that the paths are at least added for the search path from
      LDFLAGS and CPPFLAGS.

    4. smontanaro commented on Oct 1, 2008

      @smontanaro
      ContributorAuthor

      Brett> Well, all of my modules are in a non-standard location and I have
      Brett> no build issues on OS X 10.5. If you look at
      Brett> PyBuildExt.detect_modules() you will see that the paths are at
      Brett> least added for the search path from LDFLAGS and CPPFLAGS.

      Ah, there's the problem. It picks through LDFLAGS looking for -L but fails
      to pay attention to -R. I'll see if I can work up a patch.

      Also, I'm seeing this problem on Solaris. I suppose it's possible that
      linkage on OS X records the location of the libraries it finds, sort of
      implicitly doing a -R.

      Skip

    5. smontanaro commented on Oct 1, 2008

      @smontanaro
      ContributorAuthor

      Here's a patch. Works for me on Solaris 10. I'll try to check
      it out on OS X 10.5. Would appreciate it if someone on Linux can
      kick the tires too.

    6. smontanaro commented on Oct 2, 2008

      @smontanaro
      ContributorAuthor

      Confirmed that nothing seems broken on my OS X 10.5 laptop (doing a
      unix-style build, not a framework build).

    7. smontanaro commented on Oct 2, 2008

      @smontanaro
      ContributorAuthor

      Seems to work for framework builds as well.

    8. rpetrov commented on Oct 2, 2008

      rpetrovmannequin
      Mannequin

      One of the problems that I see in that LDFLAGS is Makefile variable and
      Makefile is part of distribution for posix build systems.
      If you set specific LDFLAGS and you want to distribute own python build
      user will get you specific settings.
      One another environment variable is OPT (if compiler don't accept -R XXX
      may be -Wl,-rpath,XXX ?) and we may use it. Also OPT environment
      variable is also Makefile variable and user specific settings will go
      into distribution too :( .

      I quick look into distutils code show that sysconfig.py don't parse
      LDFLAGS and append environment variable LDFLAGS as is to the ldshared
      variable.
      So the first question is why setup.py parse only -L flags instead to
      pass variable value as is (aka sysconfig.py).
      The second question is why LDFLAGS is a Makefile variable ?

      For now I would like to know if you set on Solaris suitable
      LD_RUN_PATH(for the build) and LD_LIBRARY_PATH (when run) did you
      succeed to build and run python ?

    9. brettcannon commented on Oct 2, 2008

      @brettcannon
      Member

      On Thu, Oct 2, 2008 at 1:57 PM, Roumen Petrov <report@bugs.python.org> wrote:

      Roumen Petrov <bugtrack@roumenpetrov.info> added the comment:

      One of the problems that I see in that LDFLAGS is Makefile variable and
      Makefile is part of distribution for posix build systems.
      If you set specific LDFLAGS and you want to distribute own python build
      user will get you specific settings.
      One another environment variable is OPT (if compiler don't accept -R XXX
      may be -Wl,-rpath,XXX ?) and we may use it. Also OPT environment
      variable is also Makefile variable and user specific settings will go
      into distribution too :( .

      I quick look into distutils code show that sysconfig.py don't parse
      LDFLAGS and append environment variable LDFLAGS as is to the ldshared
      variable.
      So the first question is why setup.py parse only -L flags instead to
      pass variable value as is (aka sysconfig.py).

      Because that's the use case I was supporting when I added the support;
      I just needed the -L flags for compiling the extensions. If someone
      writes a patch that makes it more generic it has a chance of being
      checked in.

      The second question is why LDFLAGS is a Makefile variable ?

      Because there is no guarantee that someone specified the envvar when
      running setup.py, but if they were set for configure you know that
      Python was built using those values.

    10. rpetrov commented on Oct 5, 2008

      rpetrovmannequin
      Mannequin

      Lets see method customize_compiler(from sysconfig.py)
      The method search variables 'CPP', 'LDFLAGS' and 'CPPFLAGS' only in
      environment. The variable CPP is not a Makefile variable. 'LDFLAGS' and
      'CPPFLAGS' are makefile variables(macros). Usually makefile will export
      to commands a macro only if exists environment variable with same name.
      So to pass to distutils LDFLAGS is enough to to run as example
      "LDFLAGS=fake make".

      bpo-3678 is about LDFLAGS for python shared library and now command
      start with $(LDSHARED) $(LDFLAGS) ...

      If we modify Makefile to use "...LDFLAGS=$(LDFLAGS) ... setup.py"
      customize_compiler will append LDFLAGS to LDSHARED.
      So this issue is about to use LDFLAGS for python modules.

      Another point is that configure script append LDFLAGS to LDSHARED only
      on certain platforms (all BSD based) and ignore all others.

      What is preferred patch:

      • a simple patch that just export LDFLAGS for setup.py;
      • a patch that export LDFLAGS for all python programs, i.e. in Makefile
        to append to RUNSHARED LDFLAGS=@ldflags@ ;
      • a patch that append LDFLAGS to LDSHARED in configure for all platforms
        and remove use of LDFLAGS in Makefile and setup.py ?
    11. smontanaro commented on Oct 7, 2008

      @smontanaro
      ContributorAuthor

      I checked in r66823 to add the -R flag extraction like Brett's -L
      extraction. I'll leave the issue open while folks discuss Roumen's
      proposal. I have no particular desire to delve deeper into
      distutils.

    12. rpetrov commented on Oct 10, 2008

      rpetrovmannequin
      Mannequin

      I don't have rights to derive a new issue.
      May be a new new title about distutils and makefile integration is more
      appropriate.

    13. 7 remaining items

    14. self-assigned this
      on Jan 26, 2009
    15. akitada commented on Feb 6, 2009

      akitadamannequin
      Mannequin

      s/get_config_vars/get_config_var/

    16. akitada commented on Feb 28, 2009

      akitadamannequin
      Mannequin

      Updated bpo-4010 to honor os.environ['CPPFLAGS'].

    17. speno commented on Aug 21, 2009

      spenomannequin
      Mannequin

      Hi. We encountered this issue on a Solaris 10 while building python 2.6.2.

      There's a problem with reptrov's patch from 2009-01-04. The LDFLAGS in the
      Makefile.pre.in patch need to be quoted also. Like so:

      LDFLAGS='$(LDFLAGS)'

      This allows for multiple flags. Other than that, it works for us.

    18. speno commented on Aug 21, 2009

      spenomannequin
      Mannequin

      Furthermore, there's another bug in setup.py When extensions are built
      and CPPFLAGS (or other arguments) has multiple arguments, the order of
      the arguments are reversed. This is the wrong behavior. The specified
      directories should be used in the given order.

      For example, if CPPFLAGS="-I/one -I/two -I/three", then when an
      extention is build, the order of the compile would be:

      cc -I/three -I/two -I/one ...

      There's this bit that seems reponsible in setup.py:

                          for directory in reversed(options.dirs):
                              add_dir_to_list(dir_list, directory)
    19. added
      type-bugAn unexpected behavior, bug, or error
      and removed
      buildThe build process and cross-build
      on Jul 31, 2010
    20. merwok commented on Nov 18, 2010

      @merwok
      Member

      The reversed call was added by Brett in r60537. Brett, can you comment?

    21. brettcannon commented on Nov 18, 2010

      @brettcannon
      Member

      When I did the patch they were coming in reversed, so I reversed the reversal. Obviously something changed so now the manual reversal is not necessary anymore.

    22. merwok commented on Nov 18, 2010

      @merwok
      Member

      I don’t think there can be a unit test for that, so do we agree on just removing the reversed call?

    23. brettcannon commented on Nov 18, 2010

      @brettcannon
      Member

      On Wed, Nov 17, 2010 at 17:19, Éric Araujo <report@bugs.python.org> wrote:

      Éric Araujo <merwok@netwok.org> added the comment:

      I don’t think there can be a unit test for that, so do we agree on just removing the reversed call?

      As long as the person removing it verifies it and keeps an eye on the
      buildbots, sure.

    24. zooba commented on Feb 3, 2021

      @zooba
      Member

      Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

      If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools

    25. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    buildThe build process and cross-buildstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions