You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
cross compilation of third-party extension modules #73019
where XBUILD_PYTHON_DIR is the location of the directory of the cross-compiled python executable.
It may be:
a) The build tree, which is the source tree when the cross compilation is not out of the source tree.
b) '$DESTDIR/$exec_prefix/bin' when the cross built python has been installed with 'make DESTDIR=/some/path install'. In that case 'prefix' and 'exec_prefix' may be different.
c) When the result of the cross compilation has been manually copied (for example to /some/path) and if 'prefix' and 'exec_prefix' are identical, this is /some/path/bin.
In case b), one can use the 'install' setup.py command instead of the 'build' command, to build and install the extension module to '$DESTDIR/$exec_prefix/lib/python$VERSION/site-packages' with the appropriate 'egg-info' file.
The patch uses the 'python-config' shell script (created at bpo-16235 [1]) that is initialized with the proper variables at the configure stage to provide the minimum information required by the sysconfig module to create (with the option --generate-posix-vars) the sysconfigdata file or to import the sysconfigdata module.
The patch also fixes bpo-22724 [2] as sys.path is only modified during the time needed to import the sysconfigdata module.
The patch fixes two minor problems:
'_PYTHON_HOST_PLATFORM' in Makefile.pre.in was not added to the sysconfig variables as intended, since those variables may not be prefixed with an underscore.
The sysconfigdata file name was terminated with a dangling underscore when 'multiarch' is not defined.
Patch also tested with pyephem on an Android emulator.
The patch misses the documentation.
Please run autoconf after installing the patch.
This approach will not work with a "multiarch" enabled environment, and break cross builds on Debian and Ubuntu.
Afaics, the proposal assumes that the python executable for the target architecture is installed (which it is not for the multiarch cross-build layout), and that each target architecture has to be identified by it's own directory tree (again, not in the multiarch environment, you can install multiple targets into the same path).
I don't think that identifying the target by some path is the right way to go, and you should be able to identify the target by giving the target triplet as used by the configure parameters and then deduce the location from the target (or have an explicit location given).
The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.
The sysconfigdata file name was terminated with a dangling
underscore when 'multiarch' is not defined.
That only solves part of the problem in that the kernel/os version gets encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of your runtime and build time kernel differ.
xdegaye commented on Nov 29, 2016
xdegayemannequin
MannequinAuthor
More actions
This approach will not work with a "multiarch" enabled environment, and break cross builds on Debian and Ubuntu.
No, the patch does not break cross builds on Debian and Ubuntu, unless you can demonstrate it does.
Afaics, the proposal assumes that the python executable for the target architecture is installed (which it is not for the multiarch cross-build layout), and that each target architecture has to be identified by it's own directory tree (again, not in the multiarch environment, you can install multiple targets into the same path).
No, you are mistaken, the path name of the build tree may be used to cross build third-party extension modules as stated in case a) of msg281993, and this should also work with debian. BTW the same code is run to cross build a standard library extension module and a third-party extension module, and obviously python is not yet installed by the time the standard library extension modules are built ;-)
The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.
It is not clear why, because debian has this multiarch design, this should constrain our build system to follow their paradigm. After all, debian has already found some ways to cross-build the extension modules for their supported multiarch platforms since it is not possible, as of today, with upstream CPython.
Minor issue: please s/xbuild/cross_build/.
Agreed.
> * The sysconfigdata file name was terminated with a dangling
> underscore when 'multiarch' is not defined.
That only solves part of the problem in that the kernel/os version gets encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of your runtime and build time kernel differ.
No idea what is the problem you are refering to. It cannot be the "dangling underscore" problem since this is a cosmetic issue. Please explain.
xdegaye commented on Nov 30, 2016
xdegayemannequin
MannequinAuthor
More actions
I don't think that identifying the target by some path is the right way to go, and you should be able to identify the target by giving the target triplet as used by the configure parameters and then deduce the location from the target (or have an explicit location given).
The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.
Hum, you still need to provide the native python interpreter with the _path_ to the <target>-python-config executable that can be anywhere on the file system (its location is relative to the cross compiled build, not the native interpreter), so it seems that this contradicts your previous sentence saying "I don't think that identifying the target by some path is the right way to go".
xdegaye commented on Nov 30, 2016
xdegayemannequin
MannequinAuthor
More actions
So I suggest we start with this patch as it works for:
The standard library extension modules (for debian as well).
The third-party extension modules on platforms that have multiarch defined and on platforms that do not have multiarch defined (on debian systems one can only use the build directory for XBUILD_PYTHON_DIR, the other systems can use the three options described in msg281993).
Then we can later extend the semantics of XBUILD_PYTHON_DIR to include the multiarch triplet if this is necessary for debian systems, unless the debian developers want to continue using the system they have developed and are currently using to cross compile third-party extension modules.
yan12125 commented on Dec 1, 2016
yan12125mannequin
Mannequin
More actions
Well, cross compiling extension modules already works fine...
$ adb shell
shell@ASUS_Z00E_2:/ $ cd /data/local/tmp
shell@ASUS_Z00E_2:/data/local/tmp $ . ./python3/tools/env.sh
shell@ASUS_Z00E_2:/data/local/tmp $ python3.7m
Python 3.7.0a0 (default:3d660ed2a60e+, Nov 23 2016, 20:22:14)
[GCC 4.2.1 Compatible Android Clang 3.8.256229 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xx
>>> xx
<module 'xx' from '/data/local/tmp/xx.cpython-37m-aarch64-linux-android.so'>
>>>
Here's my xxmodule.c:
#include"Python.h"
PyDoc_STRVAR(module_doc,
"This is a template module just for instruction.");
fromdistutils.coreimportsetup, Extensionmodule1=Extension('xx',
sources=['xxmodule.c'])
setup(name='PackageName',
version='1.0',
description='This is a demo package',
ext_modules=[module1])
Both the host and target Python are built from a6e59a2e880e
There's a little bug in the built filename: build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so, which should be build/lib.linux-aarch64-3.7/xx.cpython-37m-aarch64-linux-android.so.
Also, I guess there may be quirks if NDK or the cross-built CPython is at different locations than built-time configurations. Those issues can be left to the future.
Variables prefixed by an underscore such as '_PYTHON_SYSCONFIGDATA_NAME' are private, please do not use them, they may be changed or removed at any time.
yan12125 commented on Dec 1, 2016
yan12125mannequin
Mannequin
More actions
I know. I guess it can be determined without manually specifying but haven't investigated into details. Maybe some patches are necessary.
again, I don't think relying on a specific target path for a cross target is a good idea. and now deciding that the last possibility to use a target id to identify is better is internal doesn't make it better. I'd appreciate if we could sit together where your work will lead to, and see how we accomplish everybody's goals.
Hum, you still need to provide the native python interpreter
with the _path_ to the <target>-python-config executable that
can be anywhere on the file system
No, it's found in the same path. No contradiction.
So I suggest we start with this patch as it works for:
Please do not.
Then we can later extend the semantics of XBUILD_PYTHON_DIR
Again, I think that's the wrong way to go rely on any path as the primary id for the target, then "extending" on it makes it worse.
PR 17420 fixes cross-compilation of third-party extension modules.
The PYTHON_PROJECT_BASE environment variable is the path to the directory where Python has been cross-compiled. It is used by the native python interpreter to find the target sysconfigdata module.
For example the following command builds a wheel file to be transfered and installed with pip on the target platform, provided the native python interpreter and the cross-compiled one both have the wheel package installed:
Activity
xdegaye commentedon Nov 29, 2016
With this patch, cross compiling a third-party extension module is done with the command:
XBUILD_PYTHON_DIR=/path/to/python/dir python setup.py build
where XBUILD_PYTHON_DIR is the location of the directory of the cross-compiled python executable.
It may be:
a) The build tree, which is the source tree when the cross compilation is not out of the source tree.
b) '$DESTDIR/$exec_prefix/bin' when the cross built python has been installed with 'make DESTDIR=/some/path install'. In that case 'prefix' and 'exec_prefix' may be different.
c) When the result of the cross compilation has been manually copied (for example to /some/path) and if 'prefix' and 'exec_prefix' are identical, this is /some/path/bin.
In case b), one can use the 'install' setup.py command instead of the 'build' command, to build and install the extension module to '$DESTDIR/$exec_prefix/lib/python$VERSION/site-packages' with the appropriate 'egg-info' file.
The patch uses the 'python-config' shell script (created at bpo-16235 [1]) that is initialized with the proper variables at the configure stage to provide the minimum information required by the sysconfig module to create (with the option --generate-posix-vars) the sysconfigdata file or to import the sysconfigdata module.
The patch also fixes bpo-22724 [2] as sys.path is only modified during the time needed to import the sysconfigdata module.
The patch fixes two minor problems:
Patch also tested with pyephem on an Android emulator.
The patch misses the documentation.
Please run autoconf after installing the patch.
[1] bpo-16235: add python-config.sh for use during cross compilation
http://bugs.python.org/issue16235
[2] bpo-22724: byte-compile fails for cross-builds
http://bugs.python.org/issue22724
doko42 commentedon Nov 29, 2016
This approach will not work with a "multiarch" enabled environment, and break cross builds on Debian and Ubuntu.
Afaics, the proposal assumes that the python executable for the target architecture is installed (which it is not for the multiarch cross-build layout), and that each target architecture has to be identified by it's own directory tree (again, not in the multiarch environment, you can install multiple targets into the same path).
I don't think that identifying the target by some path is the right way to go, and you should be able to identify the target by giving the target triplet as used by the configure parameters and then deduce the location from the target (or have an explicit location given).
The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.
Minor issue: please s/xbuild/cross_build/.
doko42 commentedon Nov 29, 2016
That only solves part of the problem in that the kernel/os version gets encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of your runtime and build time kernel differ.
xdegaye commentedon Nov 29, 2016
No, the patch does not break cross builds on Debian and Ubuntu, unless you can demonstrate it does.
No, you are mistaken, the path name of the build tree may be used to cross build third-party extension modules as stated in case a) of msg281993, and this should also work with debian. BTW the same code is run to cross build a standard library extension module and a third-party extension module, and obviously python is not yet installed by the time the standard library extension modules are built ;-)
It is not clear why, because debian has this multiarch design, this should constrain our build system to follow their paradigm. After all, debian has already found some ways to cross-build the extension modules for their supported multiarch platforms since it is not possible, as of today, with upstream CPython.
Agreed.
No idea what is the problem you are refering to. It cannot be the "dangling underscore" problem since this is a cosmetic issue. Please explain.
xdegaye commentedon Nov 30, 2016
Hum, you still need to provide the native python interpreter with the _path_ to the <target>-python-config executable that can be anywhere on the file system (its location is relative to the cross compiled build, not the native interpreter), so it seems that this contradicts your previous sentence saying "I don't think that identifying the target by some path is the right way to go".
xdegaye commentedon Nov 30, 2016
So I suggest we start with this patch as it works for:
Then we can later extend the semantics of XBUILD_PYTHON_DIR to include the multiarch triplet if this is necessary for debian systems, unless the debian developers want to continue using the system they have developed and are currently using to cross compile third-party extension modules.
yan12125 commentedon Dec 1, 2016
Well, cross compiling extension modules already works fine...
And it's running fine, too:
$ adb shell shell@ASUS_Z00E_2:/ $ cd /data/local/tmp shell@ASUS_Z00E_2:/data/local/tmp $ . ./python3/tools/env.sh shell@ASUS_Z00E_2:/data/local/tmp $ python3.7m Python 3.7.0a0 (default:3d660ed2a60e+, Nov 23 2016, 20:22:14) [GCC 4.2.1 Compatible Android Clang 3.8.256229 ] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import xx >>> xx <module 'xx' from '/data/local/tmp/xx.cpython-37m-aarch64-linux-android.so'> >>>Here's my xxmodule.c:
PyDoc_STRVAR(module_doc,
"This is a template module just for instruction.");
PyMODINIT_FUNC
PyInit_xx(void)
{
return PyModuleDef_Init(&xxmodule);
}
And setup.py:
Both the host and target Python are built from a6e59a2e880e
There's a little bug in the built filename: build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so, which should be build/lib.linux-aarch64-3.7/xx.cpython-37m-aarch64-linux-android.so.
Also, I guess there may be quirks if NDK or the cross-built CPython is at different locations than built-time configurations. Those issues can be left to the future.
xdegaye commentedon Dec 1, 2016
Variables prefixed by an underscore such as '_PYTHON_SYSCONFIGDATA_NAME' are private, please do not use them, they may be changed or removed at any time.
yan12125 commentedon Dec 1, 2016
I know. I guess it can be determined without manually specifying but haven't investigated into details. Maybe some patches are necessary.
doko42 commentedon Dec 1, 2016
again, I don't think relying on a specific target path for a cross target is a good idea. and now deciding that the last possibility to use a target id to identify is better is internal doesn't make it better. I'd appreciate if we could sit together where your work will lead to, and see how we accomplish everybody's goals.
doko42 commentedon Dec 1, 2016
No, it's found in the same path. No contradiction.
Please do not.
Again, I think that's the wrong way to go rely on any path as the primary id for the target, then "extending" on it makes it worse.
xdegaye commentedon Nov 30, 2019
PR 17420 fixes cross-compilation of third-party extension modules.
The PYTHON_PROJECT_BASE environment variable is the path to the directory where Python has been cross-compiled. It is used by the native python interpreter to find the target sysconfigdata module.
For example the following command builds a wheel file to be transfered and installed with pip on the target platform, provided the native python interpreter and the cross-compiled one both have the wheel package installed:
encukou commentedon May 14, 2026
distutilswas removed (PEP-632); compiling third-party extension modules is now the job of third-party tools -- setuptools, mason-python, etc.