Repository navigation
Improve / Clear ASDL generator #84708
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-06-27.14:59:34.107>
created_at = <Date 2020-05-06.11:17:00.247>
labels = ['interpreter-core', '3.9']
title = 'Improve / Clear ASDL generator'
updated_at = <Date 2021-06-27.14:59:34.107>
user = 'https://github.com/isidentical'bugs.python.org fields:
activity = <Date 2021-06-27.14:59:34.107>
actor = 'BTaskaya'
assignee = 'none'
closed = True
closed_date = <Date 2021-06-27.14:59:34.107>
closer = 'BTaskaya'
components = ['Interpreter Core']
creation = <Date 2020-05-06.11:17:00.247>
creator = 'BTaskaya'
dependencies = []
files = []
hgrepos = []
issue_num = 40528
keywords = ['patch']
message_count = 17.0
messages = ['368235', '368238', '368239', '368240', '368245', '368247', '368248', '368250', '368251', '368252', '368257', '368365', '369281', '396353', '396477', '396583', '396584']
nosy_count = 3.0
nosy_names = ['serhiy.storchaka', 'pablogsal', 'BTaskaya']
pr_nums = ['19952', '19968', '20193', '26858', '26918']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40528'
versions = ['Python 3.9']
Activity
isidentical commentedon May 6, 2020
serhiy-storchaka commentedon May 6, 2020
You just read my thoughts. That's what I was going to do tonight. ;)
isidentical commentedon May 6, 2020
:) Awesome. Well I'll plan to continue this; @pablogsal and I was discussing the idea of extracting _ast.AST subclasses to python level (which would save a lot of time for new additions). So my primary goal is clearing bunch of old stuff / making it extendable and clean.
isidentical commentedon May 6, 2020
Currently working on implementing some items from this TODO https://github.com/eliben/asdl_parser/blob/master/TODO
pablogsal commentedon May 6, 2020
I was also thinking about getting rid of "string" nodes and just use Constant for those. Having the strings raw make the AST inconsistent because nodes that are 'strings' do not have line numbers and other metadata, making tools that need the source to report stuff more complex than it should.
What are your opinions on this @serhiy.storchaka and @BTaskaya?
isidentical commentedon May 6, 2020
I'm +1 on that (especially for finding import aliases, as we discussed earlier), if necessary precautions happen to ensure minimum breakage (from what I understand, it is something that will break everyone's code. Maybe we can implement custom Constant.__str__? and other different things to ensure it breaks small).
serhiy-storchaka commentedon May 6, 2020
"string" is only used for
For which of them a line number is meaningful and useful?
pablogsal commentedon May 6, 2020
Apologies, I was thinking of 'identifier', which is basically a PyObject representing a string. For instance, when parsing function calls like
the 'arg' attribute of the keyword is just the string 'x', without any metadata about it (like column offset and such).
isidentical commentedon May 6, 2020
Oh, I confuse with it identifier. IMHO replacing identifier with Constant would infer position in some cases.
isidentical commentedon May 6, 2020
A real world example would be tools like
unimport, that try to remove a certain part of import by looking start/end column offsets. Before (lib2to3), it was using tokens to manipulate source, and what I can tell is that having position information on multi line from imports would be a life saver for tool authors.pablogsal commentedon May 6, 2020
New changeset 091951a by Batuhan Taskaya in branch 'master':
bpo-40528: Improve and clear several aspects of the ASDL definition code for the AST (GH-19952)
091951a
isidentical commentedon May 7, 2020
Next is using asdl_int_seq for all simple sum types, and not for only cmpop. Looks like it is hardcoded to use it, and I plan to refactor this in a way that might save some time in python implementation of classes (also it will solve this bug). For not conflicting, I'll wait GH 19968.
pablogsal commentedon May 18, 2020
New changeset 63b8e0c by Batuhan Taskaya in branch 'master':
bpo-40528: Improve AST generation script to do builds simultaneously (GH-19968)
63b8e0c
isidentical commentedon Jun 22, 2021
New changeset 35ad425 by Batuhan Taskaya in branch 'main':
bpo-40528: Implement a metadata system for ASDL Generator (GH-20193)
35ad425
isidentical commentedon Jun 24, 2021
New changeset 6c76df2 by Batuhan Taskaya in branch 'main':
bpo-40528: move asdl identifier collection to the new metadata system (GH-26858)
6c76df2
isidentical commentedon Jun 27, 2021
New changeset 107a2c5 by Batuhan Taskaya in branch 'main':
bpo-40528: fix is_simple(sum)s behavior for attributes (GH-26918)
107a2c5
isidentical commentedon Jun 27, 2021
These were all the cleanups on my checklist, so can close the issue now. I'll probably create other issues if something additional comes up