➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
标题: Carry comments with the AST
类型: enhancement Stage: test needed
Components: Library (Lib) Versions: Python 3.5
process
状态: open Resolution:
Dependencies: 后续:
分配给: 抄送列表: BTaskaya, Mark.Shannon, brett.cannon, christian.heimes, gvanrossum, kernc, louielu, mbdevpl, pitrou, rhettinger, serhiy.storchaka
优先级: low 关键字:

brett.cannon 于 2015-05-03 15:21 创建。最近一次由 admin 于 2022-04-11 14:58 修改。

Messages (15)
msg242485 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) 日期: 2015-05-03 15:21
One thing about https://www.python.org/dev/peps/pep-0484/ is that it makes comments potentially semantically meaningful. Unfortunately the AST doesn't carry comments with it in any way, making it difficult to build a tool to implement a linter for PEP 484 using purely the ast module. Even if comments were carried along side-band and could do correlation by line number would be useful in this scenario.

I thought an issue had previously existed for this topic but I could find it.
msg242499 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) 日期: 2015-05-03 18:52
Comments don't belong on the AST. Where would you attach them?

The tokenizer module provides all information about comments. Tools can get the information quite easily if they need it.
msg242560 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) 日期: 2015-05-04 14:15
Normally I would agree comments don't belong there, but if we are going to start giving them semantic meaning then I don't think it's not so clear to me anymore.

As to where to attach, simple place is off of the Module node. Another is to have it be fundamental like lineno and only attach it when it is a line-trailing comment.

Yes, the tokenize module will give you the comments as well, but it is unfortunate you have to parse the code twice in order to get the comments and the AST.
msg242602 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) 日期: 2015-05-05 14:32
Another option is to provide a tool in 'tokenize' or 'ast' which will take the source and some comment regex and then attach the found comment metadata to the AST.
msg242603 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) 日期: 2015-05-05 14:39
Or a separate AST node -> comment mapping.
msg289643 - (view) Author: Mateusz Bysiek (mbdevpl) 日期: 2017-03-15 06:40
For some time now, there's an alternate ast implementation https://github.com/python/typed_ast that carries PEP 484 type comments with the AST as attributes of certain nodes.

Their approach is described here: https://github.com/python/typed_ast/blob/master/typed_ast/ast3.py#L5

If type comments become mainstream in Python, could this approach maybe be adopted as official Python AST at some point?
msg289685 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) 日期: 2017-03-15 18:21
The type annotation is already in the AST so there's nothing to carry over from typed_ast (we only care about the latest Python version while typed_ast tries to be version-agnostic).
msg292655 - (view) Author: Louie Lu (louielu) * 日期: 2017-05-01 08:51
Brett, which implement method will you prefer?

If we want to carry comment at builtin_compile_impl, it will need to change the grammar since tokenize just drop the comment when dealing with source code.

But if just using regex, will it be more easy with just combine the exists tokenize and ast module?
msg292661 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) 日期: 2017-05-01 13:16
After PEP 526, the need for this proposal may have evaporated.
msg292682 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) 日期: 2017-05-01 17:50
There's potentially some usefulness from other tools, but Raymond is right that the main motivation is definitely gone long-term. Dropping this down to "low" priority simply because others have asked for this kind of support before.
msg292683 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) 日期: 2017-05-01 17:57
Where would comment be attached in the following case?

a = ('start'         # comment 1
     'continuation') # comment 2
msg333540 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) 日期: 2019-01-13 04:18
Since the AST carries the module/lineno attributes isn't there already a way trace back into the token stream to recover comments?
msg333541 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) 日期: 2019-01-13 04:19
Possible superceder:  https://bugs.python.org/issue33337
msg334072 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) 日期: 2019-01-19 21:58
See also issue35766.
msg364061 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) 日期: 2020-03-13 01:04
I propose to close this issue, since (as of Python 3.8) we now have ast.parse(source, type_comments=True).
历史
日期 用户 动作 参数
2022-04-11 14:58:16admin修改github: 68307
2020-03-13 01:04:47gvanrossum修改消息: + msg364061
2020-03-12 20:11:39BTaskaya修改抄送: + BTaskaya
2019-01-19 21:58:29gvanrossum修改抄送: + gvanrossum
消息: + msg334072
2019-01-13 04:19:17rhettinger修改消息: + msg333541
2019-01-13 04:18:40rhettinger修改消息: + msg333540
2019-01-12 09:19:41kernc修改抄送: + kernc
2017-05-01 17:57:59serhiy.storchaka修改抄送: + serhiy.storchaka
消息: + msg292683
2017-05-01 17:50:33brett.cannon修改优先级: normal -> low

消息: + msg292682
2017-05-01 13:16:27rhettinger修改抄送: + rhettinger
消息: + msg292661
2017-05-01 08:51:37louielu修改抄送: + louielu
消息: + msg292655
2017-03-15 18:21:40brett.cannon修改消息: + msg289685
2017-03-15 06:40:23mbdevpl修改抄送: + mbdevpl
消息: + msg289643
2015-05-05 14:39:26pitrou修改抄送: + pitrou
消息: + msg242603
2015-05-05 14:32:17brett.cannon修改消息: + msg242602
2015-05-04 14:33:51christian.heimes修改抄送: + christian.heimes
2015-05-04 14:15:13brett.cannon修改消息: + msg242560
2015-05-03 18:52:45Mark.Shannon修改抄送: + Mark.Shannon
消息: + msg242499
2015-05-03 15:21:10brett.cannon创建