delta/cpython-git.git/Parser/tokenizer.h, branch fix-cf-feature-versiob

closes bpo-39721: Fix constness of members of tok_state struct. (GH-18600)

2020-02-28T02:44:52+00:00

The function PyTokenizer_FromUTF8 from Parser/tokenizer.c had a comment:

    /* XXX: constify members. */

This patch addresses that.

In the tok_state struct:
    * end and start were non-const but could be made const
    * str and input were const but should have been non-const

Changes to support this include:
    * decode_str() now returns a char * since it is allocated.
    * PyTokenizer_FromString() and PyTokenizer_FromUTF8() each creates a
        new char * for an allocate string instead of reusing the input
        const char *.
    * PyTokenizer_Get() and tok_get() now take const char ** arguments.
    * Various local vars are const or non-const accordingly.

I was able to remove five casts that cast away constness.

bpo-36623: Clean parser headers and include files (GH-12253)

2019-04-13T16:05:14+00:00

After the removal of pgen, multiple header and function prototypes that lack implementation or are unused are still lying around.

bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)

2019-03-07T20:38:08+00:00

This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)



https://bugs.python.org/issue35975

bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)

2019-03-01T23:34:44+00:00

Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser.

This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.

bpo-35766: Merge typed_ast back into CPython (GH-11645)

2019-01-31T11:40:27+00:00

bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)

2019-01-13T04:05:13+00:00

bpo-33306: Improve SyntaxError messages for unbalanced parentheses. (GH-6516)

2018-12-17T15:34:14+00:00

tokenizer: Remove unused tabs options (#4422)

2017-11-17T09:25:47+00:00

Remove the following fields from tok_state structure which are now
used unused:

* altwarning: "Issue warning if alternate tabs don't match"
* alterror: "Issue error if alternate tabs don't match"
* alttabsize: "Alternate tab spacing"

Replace alttabsize variable with ALTTABSIZE define.

bpo-30406: Make async and await proper keywords (#1669)

2017-10-06T03:24:46+00:00

Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.

Remove obsolete declaration in tokenizer.h (#962)

2017-04-03T16:18:32+00:00