Skip to content

cross compilation of third-party extension modules #73019

Closed as not planned
Closed as not planned

Description

@xdegaye
mannequin
BPO 28833
Nosy @warsaw, @doko42, @vstinner, @merwok, @pmp-p, @vadmium, @zware, @dstufft, @moreati, @yan12125
PRs
  • bpo-28833: Fix cross-compilation of third-party extension modules #17420
  • Files
  • cross-build-extension.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2016-11-29.14:51:40.336>
    labels = ['type-bug', '3.7', 'build']
    title = 'cross compilation of third-party extension modules'
    updated_at = <Date 2019-12-10.08:01:23.697>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2019-12-10.08:01:23.697>
    actor = 'xdegaye'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Cross-Build']
    creation = <Date 2016-11-29.14:51:40.336>
    creator = 'xdegaye'
    dependencies = []
    files = ['45688']
    hgrepos = []
    issue_num = 28833
    keywords = ['patch']
    message_count = 12.0
    messages = ['281993', '281996', '281998', '282044', '282103', '282104', '282141', '282144', '282147', '282155', '282157', '357663']
    nosy_count = 10.0
    nosy_names = ['barry', 'doko', 'vstinner', 'eric.araujo', 'pmpp', 'martin.panter', 'zach.ware', 'dstufft', 'Alex.Willmer', 'yan12125']
    pr_nums = ['17420']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue28833'
    versions = ['Python 3.6', 'Python 3.7']

    Activity

    1. xdegaye commented on Nov 29, 2016

      xdegayemannequin
      MannequinAuthor

      With this patch, cross compiling a third-party extension module is done with the command:

      XBUILD_PYTHON_DIR=/path/to/python/dir python setup.py build

      where XBUILD_PYTHON_DIR is the location of the directory of the cross-compiled python executable.
      It may be:
      a) The build tree, which is the source tree when the cross compilation is not out of the source tree.
      b) '$DESTDIR/$exec_prefix/bin' when the cross built python has been installed with 'make DESTDIR=/some/path install'. In that case 'prefix' and 'exec_prefix' may be different.
      c) When the result of the cross compilation has been manually copied (for example to /some/path) and if 'prefix' and 'exec_prefix' are identical, this is /some/path/bin.
      In case b), one can use the 'install' setup.py command instead of the 'build' command, to build and install the extension module to '$DESTDIR/$exec_prefix/lib/python$VERSION/site-packages' with the appropriate 'egg-info' file.

      The patch uses the 'python-config' shell script (created at bpo-16235 [1]) that is initialized with the proper variables at the configure stage to provide the minimum information required by the sysconfig module to create (with the option --generate-posix-vars) the sysconfigdata file or to import the sysconfigdata module.
      The patch also fixes bpo-22724 [2] as sys.path is only modified during the time needed to import the sysconfigdata module.

      The patch fixes two minor problems:

      • '_PYTHON_HOST_PLATFORM' in Makefile.pre.in was not added to the sysconfig variables as intended, since those variables may not be prefixed with an underscore.
      • The sysconfigdata file name was terminated with a dangling underscore when 'multiarch' is not defined.
        Patch also tested with pyephem on an Android emulator.
        The patch misses the documentation.
        Please run autoconf after installing the patch.

      [1] bpo-16235: add python-config.sh for use during cross compilation
      http://bugs.python.org/issue16235
      [2] bpo-22724: byte-compile fails for cross-builds
      http://bugs.python.org/issue22724

    2. self-assigned this
      on Nov 29, 2016
    3. added
      buildThe build process and cross-build
      type-bugAn unexpected behavior, bug, or error
      on Nov 29, 2016
    4. doko42 commented on Nov 29, 2016

      @doko42
      Member

      This approach will not work with a "multiarch" enabled environment, and break cross builds on Debian and Ubuntu.

      Afaics, the proposal assumes that the python executable for the target architecture is installed (which it is not for the multiarch cross-build layout), and that each target architecture has to be identified by it's own directory tree (again, not in the multiarch environment, you can install multiple targets into the same path).

      I don't think that identifying the target by some path is the right way to go, and you should be able to identify the target by giving the target triplet as used by the configure parameters and then deduce the location from the target (or have an explicit location given).

      The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.

      Minor issue: please s/xbuild/cross_build/.

    5. doko42 commented on Nov 29, 2016

      @doko42
      Member
      • The sysconfigdata file name was terminated with a dangling
        underscore when 'multiarch' is not defined.

      That only solves part of the problem in that the kernel/os version gets encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of your runtime and build time kernel differ.

    6. xdegaye commented on Nov 29, 2016

      xdegayemannequin
      MannequinAuthor

      This approach will not work with a "multiarch" enabled environment, and break cross builds on Debian and Ubuntu.

      No, the patch does not break cross builds on Debian and Ubuntu, unless you can demonstrate it does.

      Afaics, the proposal assumes that the python executable for the target architecture is installed (which it is not for the multiarch cross-build layout), and that each target architecture has to be identified by it's own directory tree (again, not in the multiarch environment, you can install multiple targets into the same path).

      No, you are mistaken, the path name of the build tree may be used to cross build third-party extension modules as stated in case a) of msg281993, and this should also work with debian. BTW the same code is run to cross build a standard library extension module and a third-party extension module, and obviously python is not yet installed by the time the standard library extension modules are built ;-)

      The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.

      It is not clear why, because debian has this multiarch design, this should constrain our build system to follow their paradigm. After all, debian has already found some ways to cross-build the extension modules for their supported multiarch platforms since it is not possible, as of today, with upstream CPython.

      Minor issue: please s/xbuild/cross_build/.

      Agreed.

      > * The sysconfigdata file name was terminated with a dangling
      > underscore when 'multiarch' is not defined.

      That only solves part of the problem in that the kernel/os version gets encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of your runtime and build time kernel differ.

      No idea what is the problem you are refering to. It cannot be the "dangling underscore" problem since this is a cosmetic issue. Please explain.

    7. xdegaye commented on Nov 30, 2016

      xdegayemannequin
      MannequinAuthor

      I don't think that identifying the target by some path is the right way to go, and you should be able to identify the target by giving the target triplet as used by the configure parameters and then deduce the location from the target (or have an explicit location given).

      The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.

      Hum, you still need to provide the native python interpreter with the _path_ to the <target>-python-config executable that can be anywhere on the file system (its location is relative to the cross compiled build, not the native interpreter), so it seems that this contradicts your previous sentence saying "I don't think that identifying the target by some path is the right way to go".

    8. xdegaye commented on Nov 30, 2016

      xdegayemannequin
      MannequinAuthor

      So I suggest we start with this patch as it works for:

      • The standard library extension modules (for debian as well).
      • The third-party extension modules on platforms that have multiarch defined and on platforms that do not have multiarch defined (on debian systems one can only use the build directory for XBUILD_PYTHON_DIR, the other systems can use the three options described in msg281993).

      Then we can later extend the semantics of XBUILD_PYTHON_DIR to include the multiarch triplet if this is necessary for debian systems, unless the debian developers want to continue using the system they have developed and are currently using to cross compile third-party extension modules.

    9. yan12125 commented on Dec 1, 2016

      yan12125mannequin
      Mannequin

      Well, cross compiling extension modules already works fine...

      $ _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_aarch64-linux-android PYTHONHOME=~/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr python3.7 setup.py build_ext
      running build_ext
      building 'xx' extension
      creating build
      creating build/temp.linux-x86_64-3.7
      /opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -fPIE -fno-integrated-as -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -fPIE -fno-integrated-as -fPIC -I/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/include/python3.7m -c xxmodule.c -o build/temp.linux-x86_64-3.7/xxmodule.o
      creating build/lib.linux-x86_64-3.7
      /opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -shared -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -pie -L/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/lib -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -pie -L/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/lib build/temp.linux-x86_64-3.7/xxmodule.o -o build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so
      clang: warning: argument unused during compilation: '-pie'
      clang: warning: argument unused during compilation: '-pie'

      And it's running fine, too:

      $ adb push build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so /data/local/tmp
      [100%] /data/local/tmp/xx.cpython-37m-aarch64-linux-android.so
      $ adb shell
      shell@ASUS_Z00E_2:/ $ cd /data/local/tmp
      shell@ASUS_Z00E_2:/data/local/tmp $ . ./python3/tools/env.sh
      shell@ASUS_Z00E_2:/data/local/tmp $ python3.7m
      Python 3.7.0a0 (default:3d660ed2a60e+, Nov 23 2016, 20:22:14) 
      [GCC 4.2.1 Compatible Android Clang 3.8.256229 ] on linux
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import xx
      >>> xx
      <module 'xx' from '/data/local/tmp/xx.cpython-37m-aarch64-linux-android.so'>
      >>>

      Here's my xxmodule.c:

      #include "Python.h"

      PyDoc_STRVAR(module_doc,
      "This is a template module just for instruction.");

      static struct PyModuleDef xxmodule = {
          PyModuleDef_HEAD_INIT,
          "xx",
          module_doc,
          0,
          NULL,
          NULL,
          NULL,
          NULL,
          NULL
      };

      PyMODINIT_FUNC
      PyInit_xx(void)
      {
      return PyModuleDef_Init(&xxmodule);
      }

      And setup.py:

      from distutils.core import setup, Extension
      
      module1 = Extension('xx',
                          sources=['xxmodule.c'])
      
      setup(name='PackageName',
            version='1.0',
            description='This is a demo package',
            ext_modules=[module1])

      Both the host and target Python are built from a6e59a2e880e

      There's a little bug in the built filename: build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so, which should be build/lib.linux-aarch64-3.7/xx.cpython-37m-aarch64-linux-android.so.

      Also, I guess there may be quirks if NDK or the cross-built CPython is at different locations than built-time configurations. Those issues can be left to the future.

    10. xdegaye commented on Dec 1, 2016

      xdegayemannequin
      MannequinAuthor
      $ _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_aarch64-linux-android PYTHONHOME=~/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr python3.7 setup.py build_ext
      >

      Variables prefixed by an underscore such as '_PYTHON_SYSCONFIGDATA_NAME' are private, please do not use them, they may be changed or removed at any time.

    11. yan12125 commented on Dec 1, 2016

      yan12125mannequin
      Mannequin

      I know. I guess it can be determined without manually specifying but haven't investigated into details. Maybe some patches are necessary.

    12. doko42 commented on Dec 1, 2016

      @doko42
      Member

      again, I don't think relying on a specific target path for a cross target is a good idea. and now deciding that the last possibility to use a target id to identify is better is internal doesn't make it better. I'd appreciate if we could sit together where your work will lead to, and see how we accomplish everybody's goals.

    13. doko42 commented on Dec 1, 2016

      @doko42
      Member

      Hum, you still need to provide the native python interpreter
      with the _path_ to the <target>-python-config executable that
      can be anywhere on the file system

      No, it's found in the same path. No contradiction.

      So I suggest we start with this patch as it works for:

      Please do not.

      Then we can later extend the semantics of XBUILD_PYTHON_DIR

      Again, I think that's the wrong way to go rely on any path as the primary id for the target, then "extending" on it makes it worse.

    14. removed their assignment
      on Feb 3, 2017
    15. xdegaye commented on Nov 30, 2019

      xdegayemannequin
      MannequinAuthor

      PR 17420 fixes cross-compilation of third-party extension modules.

      The PYTHON_PROJECT_BASE environment variable is the path to the directory where Python has been cross-compiled. It is used by the native python interpreter to find the target sysconfigdata module.

      For example the following command builds a wheel file to be transfered and installed with pip on the target platform, provided the native python interpreter and the cross-compiled one both have the wheel package installed:

        $ PYTHON_PROJECT_BASE=/path/to/builddir python setup.py bdist_wheel
    16. transferred this issue fromon Apr 10, 2022
    17. encukou commented on May 14, 2026

      @encukou
      Member

      distutils was removed (PEP-632); compiling third-party extension modules is now the job of third-party tools -- setuptools, mason-python, etc.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of lifebuildThe build process and cross-buildtype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions