delta/cpython-git.git/Parser/tokenizer.c, branch benjamin-interp-initialize

Remove unneeded PyUnicode_READY() in tokenizer.c (GH-9114)

2018-09-10T06:27:31+00:00

Fix Windows compiler warning in tokenize.c (GH-8359)

2018-07-21T01:36:06+00:00

Fix the following warning on Windows:

parser\tokenizer.c(1297): warning C4244: 'function': conversion from
'__int64' to 'int', possible loss of data.

bpo-33305: Improve SyntaxError for invalid numerical literals. (GH-6517)

2018-07-09T12:09:35+00:00

tokenizer: Remove unused tabs options (#4422)

2017-11-17T09:25:47+00:00

Remove the following fields from tok_state structure which are now
used unused:

* altwarning: "Issue warning if alternate tabs don't match"
* alterror: "Issue error if alternate tabs don't match"
* alttabsize: "Alternate tab spacing"

Replace alttabsize variable with ALTTABSIZE define.

bpo-30406: Make async and await proper keywords (#1669)

2017-10-06T03:24:46+00:00

Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.

bpo-25324: add missing comma in Parser/tokenizer.c (GH-1910)

2017-06-01T20:51:27+00:00

bpo-25324: copy tok_name before changing it (#1608)

2017-05-31T14:00:21+00:00

* add test to check if were modifying token

* copy list so import tokenize doesnt have side effects on token

* shorten line

* add tokenize tokens to token.h to get them to show up in token

* move ERRORTOKEN back to its previous location, and fix nitpick

* copy comments from token.h automatically

* fix whitespace and make more pythonic

* change to fix comments from @haypo

* update token.rst and Misc/NEWS

* change wording

* some more wording changes

Issue #28489: Merge from 3.6

2017-02-05T01:33:11+00:00

Issue #28489: Fix comment in tokenizer.c

2017-02-05T01:32:39+00:00

Patch by Ryan Gonzalez.

Use _PyObject_CallNoArg()

2016-12-06T17:45:50+00:00

Replace:
    PyObject_CallObject(callable, NULL)
with:
    _PyObject_CallNoArg(callable)