Skip to content

Android's incomplete locale.h implementation prevents cross-compilation #64504

Closed

Description

@shiz
mannequin
BPO 20305
Nosy @malemburg, @loewis, @pitrou, @vstinner, @skrah, @xdegaye, @yan12125
Files
  • Python-3.4.0tip-workaround-android-locale-issues.patch: Patch to work around locale.h issues in Android.
  • Python-3.4-d51d6f1f9db8-workaround-android-locale-issues.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2016-04-25.19:09:35.571>
    created_at = <Date 2014-01-19.18:13:22.736>
    labels = ['build']
    title = "Android's incomplete locale.h implementation prevents cross-compilation"
    updated_at = <Date 2016-04-25.19:09:35.569>
    user = 'https://bugs.python.org/shiz'

    bugs.python.org fields:

    activity = <Date 2016-04-25.19:09:35.569>
    actor = 'skrah'
    assignee = 'none'
    closed = True
    closed_date = <Date 2016-04-25.19:09:35.571>
    closer = 'skrah'
    components = ['Cross-Build']
    creation = <Date 2014-01-19.18:13:22.736>
    creator = 'shiz'
    dependencies = []
    files = ['33549', '33569']
    hgrepos = []
    issue_num = 20305
    keywords = ['patch']
    message_count = 15.0
    messages = ['208486', '208489', '208541', '208542', '208561', '208583', '208603', '208660', '208661', '236888', '262074', '262075', '262088', '264162', '264194']
    nosy_count = 9.0
    nosy_names = ['lemburg', 'loewis', 'pitrou', 'vstinner', 'skrah', 'xdegaye', 'shiz', 'lizhenhua', 'yan12125']
    pr_nums = []
    priority = 'normal'
    resolution = 'out of date'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue20305'
    versions = ['Python 3.6']

    Activity

    1. shiz commented on Jan 19, 2014

      shizmannequin
      MannequinAuthor

      As a result of Android's relatively incomplete locale.h implementation1, some functions are not defined and some standard structs are lacking fields (e.g. decimal_point, thousand_sep).
      This prevents proper cross-compilation using the Android NDK2 from succeeding.

      Attached is a patch which works around the various issues posed by Android's locale.h implementation, mainly involving falling back to default locale values, causing compilation in that department for the author to succeed.

    2. added
      buildThe build process and cross-build
      on Jan 19, 2014
    3. changed the title [-]Android's incomplete locale.h implementation prevents compilation[/-] [+]Android's incomplete locale.h implementation prevents cross-compilation[/+] on Jan 19, 2014
    4. malemburg commented on Jan 19, 2014

      @malemburg
      Member

      I'd be +1 on such a patch if we were to officially support Android, but we'd need a volunteer to champion for this (which would be a good thing, IMO).

      Otherwise, such changes need to be maintained externally.

    5. skrah commented on Jan 20, 2014

      skrahmannequin
      Mannequin

      I agree with Marc-Andre. Also, we should have an Android buildbot.

    6. vstinner commented on Jan 20, 2014

      @vstinner
      Member

      Is there the only patch required to compile Python 3.4 on Android?

    7. shiz commented on Jan 20, 2014

      shizmannequin
      MannequinAuthor

      I managed to cross-compile Python 3.3.3 for arm-linux-androideabi (using the Android NDK r9c) with the patches in this issue, bpo-20306 and bpo-20307. It did require a small additional patch which addressed the fact that the host system can't run the generated parser generator, which I'll make an issue for as soon as I refactored the patch. The patch allows the user to specify a host parser generator that points to a compiled version of pgen that is runnable on the host through the HOSTPGEN environment variable in ./configure.

      Compiling the tip proved somewhat harder as the cross-compilation requires a host Python of the same version, which I didn't have. I formatted a patch which allows the user to supply a special host python path using the HOSTPYTHON environment variable to ./configure. I'll make an issue for that soon, and will bundle it with the HOSTPGEN patch if it's seen as a sufficient approach.

      As far as maintaining an Android port for CPython goes; I may be interested in this as I'd need to regularly use it anyway. Can anyone tell me what the possibilities are here?

      Since in the meanwhile the tip was updated to unconditionally include locale.h in Python/fileutils.c according to bpo-19036, here's an updated patch for this issue that applies cleanly against the tip.

    8. skrah commented on Jan 20, 2014

      skrahmannequin
      Mannequin

      Shiz <report@bugs.python.org> wrote:

      As far as maintaining an Android port for CPython goes; I may be interested
      in this as I'd need to regularly use it anyway. Can anyone tell me what the
      possibilities are here?

      There seem to be some external ports already. -- On the other hand, if we
      integrate your patches, we'd sort of make Android officially supported.

      The problems with this are:

      a) In the past proponents of non-mainstream platforms have not been
      very active after an initial enthusiasm.

      b) There were no build slaves (See http://www.python.org/dev/buildbot/)
      for these platforms.

      c) Many (or all) core committers did not have access to the platform
      in question.

      Wikipedia claims that "QEMU also powers the Android emulator which is part of
      the Android SDK." Can you run a build slave with network access and also run
      the test suite in this emulator?

      Running a build slave is rather easy, see:

      https://wiki.python.org/moin/BuildBot
      http://bugs.python.org/file24399/buildslave_install.txt

    9. shiz commented on Jan 21, 2014

      shizmannequin
      MannequinAuthor

      c) Many (or all) core committers did not have access to the platform
      in question.

      I'd say Android is quite a common platform these days, although I'd concur that it's not particularly easy to run Python OOTB on. :)

      Wikipedia claims that "QEMU also powers the Android emulator which is part of
      the Android SDK." Can you run a build slave with network access and also run
      the test suite in this emulator?

      I've been looking into this, and the only option provided by the Android NDK involves cross-compiling Python on a regular machine. This would require some custom commands before and in between the builds, to build a host Python first, then cross-compile the Android Python, and then start the emulator, transfer the binaries there and run the tests over adb. Can buildbot provide functionality so this can be configured on-slave, or would it require setting up a seperate master?

    10. skrah commented on Jan 21, 2014

      skrahmannequin
      Mannequin

      "I've been looking into this, and the only option provided by the Android NDK involves cross-compiling Python on a regular machine. This would require some custom commands before and in between the builds, to build a host Python first, then cross-compile the Android Python, and then start the emulator, transfer the binaries there and run the tests over adb. Can buildbot provide functionality so this can be configured on-slave, or would it require setting up a seperate master?"

      I'm not sure. Antoine, do you think this is possible?

    11. pitrou commented on Jan 21, 2014

      @pitrou
      Member

      "I've been looking into this, and the only option provided by the
      Android NDK involves cross-compiling Python on a regular machine. This
      would require some custom commands before and in between the builds,
      to build a host Python first, then cross-compile the Android Python,
      and then start the emulator, transfer the binaries there and run the
      tests over adb. Can buildbot provide functionality so this can be
      configured on-slave, or would it require setting up a seperate
      master?"

      The master can ask the slave to execute arbitrary commands. So, as long
      as you implement the necessary steps as well-known commands executable
      by the slave, you can then tell me how to insert them inside the build
      process.

      However, the only point of setting up an Android buildbot would be if:

      1. Python actually builds correctly on Android
      2. someone monitors the buildbot for failures and tries to act on them
    12. skrah commented on Feb 28, 2015

      skrahmannequin
      Mannequin

      Consider the libmpdec part rejected. It is too much work given that
      external libmpdecs need to be compatible and Android can use the Python
      version of decimal.

    13. yan12125 commented on Mar 20, 2016

      yan12125mannequin
      Mannequin

      Android NDK provides an "android-support" package [1]. There is no documentation other than README in its git repository, I guess it's a 'compatibility layer' between applications targetting a complete C library and Android's BioniC. With my patchset [2], Python builds and runs on my ASUS Zenfone 2 (Android 5.0.2). Patches proposed in this issue are no longer necessary.

      [1] https://android.googlesource.com/platform/ndk.git/+/master/sources/android/support/
      [2] https://github.com/yan12125/python3-android, see commit 82c971a96021e8f8006755b6963ac39f8a5fa412 and 73c2889c19c0bf259021fcc52a3baf5613b1e075

    14. skrah commented on Mar 20, 2016

      skrahmannequin
      Mannequin

      Thank you, this is excellent! Does that mean that all locale patches are no longer needed? Here is one left (perhaps accidentally):

      https://github.com/yan12125/python3-android/blob/cpython-hg/mk/ncurses/android-locale-fixes.patch

    15. yan12125 commented on Mar 20, 2016

      yan12125mannequin
      Mannequin

      curses was broken and now it's fixed. Yes - Android's locale patch is no longer necessary in _curses either. However, there are runtime errors:

      shell@ASUS_Z00E_2:/data/local/tmp $ python3.6 -c 'import curses; curses.initscr()'                           
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/data/local/tmp/python3/lib/python3.6/curses/__init__.py", line 30, in initscr
          fd=_sys.__stdout__.fileno())
      _curses.error: setupterm: could not find terminfo database
    16. xdegaye commented on Apr 25, 2016

      xdegayemannequin
      Mannequin

      This error:
      _curses.error: setupterm: could not find terminfo database

      does not occur for me after removing the '--disable-database --disable-home-terminfo' options from the configure options used to cross-compile ncurses, and after setting the TERMINFO environment variable to a location where the the compiled description of the vt100 terminal is set. See https://bitbucket.org/xdegaye/pyona.

      Also, with API level 21 (Android 5.0), the build of python3 with the official android toolchains (that is, without resorting to external libraries for wide character support) runs correctly and the python test suite is run with few errors, so IMHO this issue may be closed.

    17. skrah commented on Apr 25, 2016

      skrahmannequin
      Mannequin

      Thank you, closing this.

    18. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      buildThe build process and cross-build

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions