delta/cpython-git.git/Parser/tokenizer.c, branch berkerpeksag-update-codeowners

bpo-36623: Clean parser headers and include files (GH-12253)

2019-04-13T16:05:14+00:00

After the removal of pgen, multiple header and function prototypes that lack implementation or are unused are still lying around.

bpo-36459: Fix a possible double PyMem_FREE() due to tokenizer.c's tok_nextc() (12601)

2019-03-28T13:53:00+00:00

Remove the PyMem_FREE() call added in cb90c89.  The buffer will be
freed when PyTokenizer_Free() is called on the tokenizer state.

bpo-36367: Free buffer if realloc fails in tokenize.c (GH-12442)

2019-03-19T17:17:58+00:00

bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)

2019-03-07T20:38:08+00:00

This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)



https://bugs.python.org/issue35975

bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)

2019-03-01T23:34:44+00:00

Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser.

This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.

bpo-35766: Merge typed_ast back into CPython (GH-11645)

2019-01-31T11:40:27+00:00

bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)

2019-01-13T04:05:13+00:00

bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)

2018-12-22T09:18:40+00:00

"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.

"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.

Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".

The documentation contains now strings for operators and punctuation tokens.

bpo-33306: Improve SyntaxError messages for unbalanced parentheses. (GH-6516)

2018-12-17T15:34:14+00:00

bpo-35436: Add missing PyErr_NoMemory() calls and other minor bug fixes. (GH-11015)

2018-12-07T10:11:30+00:00

Set MemoryError when appropriate, add missing failure checks,
and fix some potential leaks.