Skip to content

Argument Clinic for bisect.bisect_left #72940

Closed

Description

@JulienPalard
BPO 28754
Nosy @rhettinger, @vstinner, @larryhastings, @skrah, @vadmium, @serhiy-storchaka, @JulienPalard
PRs
  • bpo-28754: AC on bisect. #177
  • Files
  • issue28754.diff: Missing Modules/clinic/_bisectmodule.c.h
  • issue28754-2.diff
  • bisect_left.diff
  • bisect.diff
  • insort.diff
  • insort-left.diff
  • issue28754-3.diff
  • issue28754-4.diff
  • issue28754-5.diff
  • check_bisect_doc.py
  • issue28754-6.diff
  • issue28754-7.diff
  • issue28754-8.diff
  • issue28754-9.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/rhettinger'
    closed_at = <Date 2017-09-24.23:15:12.135>
    created_at = <Date 2016-11-20.16:51:59.575>
    labels = ['3.7', 'expert-argument-clinic']
    title = 'Argument Clinic for bisect.bisect_left'
    updated_at = <Date 2017-09-25.02:51:14.535>
    user = 'https://github.com/JulienPalard'

    bugs.python.org fields:

    activity = <Date 2017-09-25.02:51:14.535>
    actor = 'rhettinger'
    assignee = 'rhettinger'
    closed = True
    closed_date = <Date 2017-09-24.23:15:12.135>
    closer = 'rhettinger'
    components = ['Argument Clinic']
    creation = <Date 2016-11-20.16:51:59.575>
    creator = 'mdk'
    dependencies = []
    files = ['45562', '45563', '45568', '45569', '45570', '45571', '45572', '45626', '45628', '45642', '45650', '45693', '45832', '45836']
    hgrepos = []
    issue_num = 28754
    keywords = ['patch']
    message_count = 54.0
    messages = ['281282', '281283', '281285', '281299', '281302', '281303', '281652', '281669', '281685', '281712', '281713', '281717', '281734', '281761', '281762', '281763', '281764', '281765', '281768', '281772', '281779', '281791', '281792', '281832', '281834', '281878', '281880', '281888', '281892', '281899', '281901', '281926', '282028', '282030', '282038', '282041', '282049', '282063', '282308', '282350', '282842', '282854', '282864', '282875', '282877', '282882', '282888', '282889', '282891', '283100', '291160', '291162', '302894', '302908']
    nosy_count = 7.0
    nosy_names = ['rhettinger', 'vstinner', 'larry', 'skrah', 'martin.panter', 'serhiy.storchaka', 'mdk']
    pr_nums = ['177']
    priority = 'normal'
    resolution = 'rejected'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue28754'
    versions = ['Python 3.7']

    Activity

    1. JulienPalard commented on Nov 20, 2016

      @JulienPalard
      MemberAuthor

      Today I read https://docs.python.org/3.6/howto/clinic.html so I tried one: bisect.bisect_left.

      I was unable to do bisect_right, as it's an "alias" for bisect, and there's a unit-test checking self.assertEqual(self.module.bisect, self.module.bisect_right). Any hint welcome.

    2. larryhastings commented on Nov 20, 2016

      @larryhastings
      Contributor

      There's special syntax to handle aliases. From comments in clinic.py:

          # alternatively:
          #     modulename.fnname [as c_basename] = modulename.existing_fn_name
          # clones the parameters and return converter from that
          # function.  you can't modify them.  you must enter a
          # new docstring.
      

      Sorry the docs are out of date--patches welcome. And speaking of patches, thanks for the patch! I'll review it when you say it's ready.

    3. larryhastings commented on Nov 20, 2016

      @larryhastings
      Contributor

      Oh, and, if the code literally asserts they're the same function, that's just a sanity check based on the implementation. You could preserve that if you care to, or you could just write a new function and remove the assertion. Do what you think is best, then post your patch to the tracker and we'll all argue about it forever, how's that sound?

    4. JulienPalard commented on Nov 20, 2016

      @JulienPalard
      MemberAuthor

      I searched an occurrence of what I'm describing which is already using clinic and there is, at least, one in Modules/binascii.c line 1090:

      TL;DR: The idea is to use the modulename.fnname [as c_basename] = modulename.existing_fn_name clinic syntax, drop a "This function is also available as …" in the docstring, and check if they return the same thing in the tests.

      This way looks good to me, it uses the dedicated syntax which looks right, and I'll drop the self.assertEqual(self.module.bisect, self.module.bisect_right) which is an implementation detail, replacing it by tests checking that both have the same behavior.

      /*[clinic input]
      binascii.b2a_hex
      
          data: Py_buffer
          /
      
      Hexadecimal representation of binary data.
      
      The return value is a bytes object.  This function is also
      available as "hexlify()".
      [clinic start generated code]*/
      
      static PyObject *
      binascii_b2a_hex_impl(PyObject *module, Py_buffer *data)
      /*[clinic end generated code: output=92fec1a95c9897a0 input=96423cfa299ff3b1]*/
      {
          return _Py_strhex_bytes((const char *)data->buf, data->len);
      }
      
      /*[clinic input]
      binascii.hexlify = binascii.b2a_hex
      
      Hexadecimal representation of binary data.
      
      The return value is a bytes object.
      [clinic start generated code]*/
      
      static PyObject *
      binascii_hexlify_impl(PyObject *module, Py_buffer *data)
      /*[clinic end generated code: output=749e95e53c14880c input=2e3afae7f083f061]*/
      {
          return _Py_strhex_bytes((const char *)data->buf, data->len);
      }
      
    5. JulienPalard commented on Nov 20, 2016

      @JulienPalard
      MemberAuthor

      Here it is for the whole bisect module.

      I separated my work in commits, but I'm not sure how rietveld will eat that as they'll have unknown references, so I'll probably also upload a single patch with a known reference.

    6. JulienPalard commented on Nov 20, 2016

      @JulienPalard
      MemberAuthor

      The whole diff is reviewable in bpo-28754-3.diff.

    7. vstinner commented on Nov 24, 2016

      @vstinner
      Member

      Hum, I dislike your changes on test_bisect.py: I don't see why Argument Clinic would have to modify tests!?

      I created issue bpo-28792: "bisect: implement aliases in Python, remove C aliases".

    8. JulienPalard commented on Nov 24, 2016

      @JulienPalard
      MemberAuthor

      New simplier patch thanks to https://bugs.python.org/issue28792. Also corrected docstrings.

      Also ran a micro-benchmark of bisect.bisect(foo, "c") with foo = list("abcdef"):
      Median +- std dev: [before] 434 ns +- 17 ns -> [after] 369 ns +- 22 ns: 1.18x faster

      FTR: ./python -m perf timeit -s 'import bisect; foo = list("abdef")' 'bisect.bisect(foo, "c")' -o after.json --inherit=PYTHONPATH --affinity=2,3

    9. vstinner commented on Nov 25, 2016

      @vstinner
      Member

      bpo-28754-4.diff does 3 things:

      • convert C code to Argument Clinic
      • change docstrings of the C code
      • change unit tests

      bpo-28754-5.diff doesn't touch tests, but still does 2 things. Sadly, I dislike the changes on docstrings because it makes the C docstring and the Python docstring inconsistent.

      I suggest to copy/paste docstring between C and Python code.

      By the way, why do we have a docstring different from Doc/library/bisect.rst? Maybe we can simply use the same doc everywhere, but just add reST formatting in Doc/library/bisect.rst?

      Julien: I suggest to first restrict your change to Argument Clinic, and later write a new patch to update docstrings. I prefer to only do one thing per commit.

    10. vadmium commented on Nov 25, 2016

      @vadmium
      Member

      Victor, what changes to the doc strings are you talking about? Adding descriptions of the parameters, maybe? Other than that they look very similar to me.

      I haven’t looked closely, but the Python doc strings have been updated at least once more than the C doc strings: r54666 and bpo-1602378. So if you are copying anything, it might be better to copy from Py to C. But I would do that separately.

      The doc strings and reference documentation just seem to have been added independently: ce2d120c3903 (main documentation) vs 67b3ac439f64 (doc strings)

    11. vstinner commented on Nov 25, 2016

      @vstinner
      Member

      Martin: "Victor, what changes to the doc strings are you talking about?"

      See attached check_bisect_doc.py script.

      Before:
      ---
      === bisect_right ===
      bisect_right(a, x[, lo[, hi]]) -> index

      Return the index where to insert item x in list a, assuming a is sorted.

      The return value i is such that all e in a[:i] have e <= x, and all e in
      a[i:] have e > x. So if x already appears in the list, i points just
      beyond the rightmost x already there

      Optional args lo (default 0) and hi (default len(a)) bound the
      slice of a to be searched.

      === insort_right ===

      insort_right(a, x[, lo[, hi]])

      Insert item x in list a, and keep it sorted assuming a is sorted.

      If x is already in a, insert it to the right of the rightmost x.

      Optional args lo (default 0) and hi (default len(a)) bound the
      slice of a to be searched.

      === bisect_left ===
      bisect_left(a, x[, lo[, hi]]) -> index

      Return the index where to insert item x in list a, assuming a is sorted.

      The return value i is such that all e in a[:i] have e < x, and all e in
      a[i:] have e >= x. So if x already appears in the list, i points just
      before the leftmost x already there.

      Optional args lo (default 0) and hi (default len(a)) bound the
      slice of a to be searched.

      === insort_left ===

      insort_left(a, x[, lo[, hi]])

      Insert item x in list a, and keep it sorted assuming a is sorted.

      If x is already in a, insert it to the left of the leftmost x.

      Optional args lo (default 0) and hi (default len(a)) bound the
      slice of a to be searched.

      ---

      After:
      ---
      === bisect_right ===
      Return the index where to insert item x in list a, assuming a is sorted.

      a
      The list in which x is to be inserted.
      x
      The value to be inserted.
      lo
      Lower bound, defaults to 0.
      hi
      Upper bound, defaults to -1 (meaning len(a)).

      The return value i is such that all e in a[:i] have e <= x, and all e in
      a[i:] have e > x. So if x already appears in the list, i points just
      beyond the rightmost x already there.

      Optional args lo and hi bound the slice of a to be searched.
      === insort_right ===
      Insert item x in list a, and keep it sorted assuming a is sorted.

      a
      The list in which x will be inserted.
      x
      The value to insert.
      lo
      Lower bound, defaults to 0.
      hi
      Upper bound, defaults to -1 (meaning len(a)).

      If x is already in a, insert it to the right of the rightmost x.

      Optional args lo and hi bound the slice of a to be searched.
      === bisect_left ===
      Return the index where to insert item x in list a, assuming a is sorted.

      a
      The list in which x is to be inserted.
      x
      The value to be inserted.
      lo
      Lower bound, defaults to 0.
      hi
      Upper bound, defaults to -1 (meaning len(a)).

      The return value i is such that all e in a[:i] have e < x, and all e in
      a[i:] have e >= x. So if x already appears in the list, i points just
      before the leftmost x already there.

      Optional args lo and hi bound the slice of a to be searched.
      === insort_left ===
      Insert item x in list a, and keep it sorted assuming a is sorted.

      a
      The list in which x will be inserted.
      x
      The value to insert.
      lo
      Lower bound, defaults to 0.
      hi
      Upper bound, defaults to -1 (meaning len(a)).

      If x is already in a, insert it to the left of the leftmost x.

      Optional args lo and hi bound the slice of a to be searched.

      ---

      The output is different. I suggest to not touch the docstring in the patch upgrading _bisect.c to Argument Clinic to ease the review.

    12. vadmium commented on Nov 25, 2016

      @vadmium
      Member

      Taking the first function, bisect_right(), as an example, I see these differences:

      • bisect_right(a, x[, lo[, hi]]) -> index
        This signature is removed. I think removing it is reasonable, because pydoc can extract the proper signature from the Arg Clinic metadata.

      • Additional descriptions of each parameter. I tend to think these are redundant with the main text, so agree with removing them from the patch now.

      • Addition of full stop (.) at end of first paragraph. I suggested this as a minor cleanup, but it could be fixed later if you prefer.

      • Removal of default values in last paragraph. The first is redundant with the Arg Clinic signature, so I suggested to remove it. For the second, I would add it back if we remove the list of parameters, since it explains what the special value -1 means.

      Do you want to revert all these differences?

    13. JulienPalard commented on Nov 25, 2016

      @JulienPalard
      MemberAuthor

      Quoting @martin

      • bisect_right(a, x[, lo[, hi]]) -> index
        This signature is removed. I think removing it is reasonable, because pydoc can extract the proper signature from the Arg Clinic metadata.

      In fact, http://docs.python.org/howto/clinic.html, bullet ".4", told me do remove it:

      If the old docstring had a first line that looked like a function signature, throw that line away. (The docstring doesn’t need it anymore—when you use help() on your builtin in the future, the first line will be built automatically based on the function’s signature.)

      • Additional descriptions of each parameter. I tend to think these are redundant with the main text, so agree with removing them from the patch now.

      OK, will remove them.

      • Addition of full stop (.) at end of first paragraph. I suggested this as a minor cleanup, but it could be fixed later if you prefer.

      Can remove it to simplify verification, but as doctests will be different as doc told me to remove the signature, so is it work removing this dot ? I don't think so.

      • Removal of default values in last paragraph. The first is redundant with the Arg Clinic signature, so I suggested to remove it. For the second, I would add it back if we remove the list of parameters, since it explains what the special value -1 means.

      I'll revert it too as I'll remove the documentation in the parameter list.

      Will upload a patch later.

    14. 29 remaining items

    15. JulienPalard commented on Dec 10, 2016

      @JulienPalard
      MemberAuthor

      Added a test to ensure compatibility of both hi=None (introduced in original Python version) and hi=-1 (Introduced by the C version).

      Modified Python version to be compatible with the C-introduced hi=-1, so that the new test pass.

    16. JulienPalard commented on Dec 10, 2016

      @JulienPalard
      MemberAuthor

      Maybe wait for bpo-28933.

    17. JulienPalard commented on Dec 10, 2016

      @JulienPalard
      MemberAuthor

      Hi Larry,

      In any cases it looks like supporting hi=-1 and hi=None is mandatory:

      hi=None is the current implementation in the bisect (Python) module.
      hi=-1 is the current implementation in the _bisect (C) module.

      Both are currently living together, the C version takes over the Python version when available, but the Python version is older, so both -1 and None has been the "current implementation" during some time.

      Raymond legitimately fear that someone, somewhere, used -1 in some code. We should not break it (even if it had never been documented).

      So let's just accept both for the moment, and if deprecation of one or another should be discussed, let's do it later, in another issue. I'm _just_ trying to make AC move forward here, without breaking anything.

    18. larryhastings commented on Dec 10, 2016

      @larryhastings
      Contributor

      Okay, but do it with a converter function, not by changing the default behavior for Py_ssize_t. The vast majority of uses of Py_ssize_t should not accept None.

    19. rhettinger commented on Dec 11, 2016

      @rhettinger
      Contributor

      Why does this need a converter function? I there is any reason this can't use "O" with "hi" defaulting to None (matching the pure python API) and letting the C function handle both -1 and None in the implementation rather than in AC?

    20. JulienPalard commented on Dec 11, 2016

      @JulienPalard
      MemberAuthor

      Hi Raymond,

      I don't like having the converters in the C implementation too, that's why I'm working on bpo-28933 to clean this.

      letting the C function handle both -1 and None in the implementation rather than in AC?

      It works, yes. But I prefer to clearly split responsibilities: AC being responsible of adapting argument from PyObjects to C types, and C functions being responsible of ... doing their job.

      If the idea in bpo-28933 is accepted, we'll be able to declare hi as simply as:

      hi: Py_ssize_t(c_default="-1") = None
      

      meaning "C function will get a Py_ssize_t, default value for C code is -1, None is documented, and None can be passed to get the C default value", that's this last point "None can be passed to get the C default value" that I introduce in bpo-28933.

      With this syntax, both C converters and the python hi_parameter_converter can be dropped.

    21. larryhastings commented on Dec 11, 2016

      @larryhastings
      Contributor

      We don't *have* to use a converter function, no. But consider: someone somewhere will have to convert a PyObject * into a Py_ssize_t, also accepting None. Doing it in a converter function means we can easily share the code with bisect_right, which should presumably behave the same way. It also keeps all the argument parsing logic in the Argument Clinic domain instead of having some in AC and some in the implementation.

      Is there some drawback to using a converter function?

    22. JulienPalard commented on Dec 13, 2016

      @JulienPalard
      MemberAuthor

      As the pattern of this converter is not widely used, I'll let it in the code of _bisect for the moment, see: http://bugs.python.org/issue28933.

    23. serhiy-storchaka commented on Apr 5, 2017

      @serhiy-storchaka
      Member

      Now Py_ssize_t(accept={int, NoneType}) can be used instead of a local converter.

    24. rhettinger commented on Apr 5, 2017

      @rhettinger
      Contributor

      Wouldn't it be easily to let the OP apply AC to some other module. Right now, it isn't a very good fit and was this a tough one to start with.

    25. rhettinger commented on Sep 24, 2017

      @rhettinger
      Contributor

      After more thought, I'm going to close this one because the Argument Clinic and signature objects aren't yet a good fit for this API. The bisect module isn't broken (its API has been successfully living in the wild for a very long time). I do welcome Argument Clinic patches where there is a good fit but don't think this is the place. Each of the proposed patches makes the module worst in one way or another.

    26. rhettinger commented on Sep 25, 2017

      @rhettinger
      Contributor

      FWIW, the itertools module has many functions that can have the ArgumentClinic applied without any fundamental problems (pretty much all of the functions will work except for repeat(), accumulate(), and islice()).

    27. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions