summaryrefslogtreecommitdiff
path: root/Parser/tokenizer.c
Commit message (Collapse)AuthorAgeFilesLines
* bpo-36878: Only allow text after `# type: ignore` if first character ASCII ↵Michael J. Sullivan2019-05-221-2/+3
| | | | | | | | | | | (GH-13504) This disallows things like `# type: ignoreé`, which seems wrong. Also switch to using Py_ISALNUM for the alnum check, for consistency with other code (and maybe correctness re: locale issues?). https://bugs.python.org/issue36878
* bpo-36878: Track extra text added to 'type: ignore' in the AST (GH-13479)Michael J. Sullivan2019-05-221-2/+6
| | | | | GH-13238 made extra text after a # type: ignore accepted by the parser. This finishes the job and actually plumbs the extra text through the parser and makes it available in the AST.
* bpo-2180: Treat line continuation at EOF as a `SyntaxError` (GH-13401)Anthony Sottile2019-05-181-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This makes the parser consistent with the tokenize module (already the case in `pypy`). sample ------ ```python x = 5\ ``` before ------ ```console $ python3 t.py $ python3 -mtokenize t.py t.py:2:0: error: EOF in multi-line statement ``` after ----- ```console $ ./python t.py File "t.py", line 3 x = 5\ ^ SyntaxError: unexpected EOF while parsing $ ./python -m tokenize t.py t.py:2:0: error: EOF in multi-line statement ``` https://bugs.python.org/issue2180
* bpo-36878: Allow extra text after `# type: ignore` comments (GH-13238)Michael J. Sullivan2019-05-111-8/+5
| | | | | | | In the parser, when using the type_comments=True option, recognize a TYPE_IGNORE as anything containing `# type: ignore` followed by a non-alphanumeric character. This is to allow ignores such as `# type: ignore[E1000]`.
* bpo-36623: Clean parser headers and include files (GH-12253)Pablo Galindo2019-04-131-1/+0
| | | After the removal of pgen, multiple header and function prototypes that lack implementation or are unused are still lying around.
* bpo-36459: Fix a possible double PyMem_FREE() due to tokenizer.c's ↵Zackery Spytz2019-03-281-1/+0
| | | | | | tok_nextc() (12601) Remove the PyMem_FREE() call added in cb90c89. The buffer will be freed when PyTokenizer_Free() is called on the tokenizer state.
* bpo-36367: Free buffer if realloc fails in tokenize.c (GH-12442)Pablo Galindo2019-03-191-2/+8
|
* bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)Guido van Rossum2019-03-071-0/+79
| | | | | | | This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.) https://bugs.python.org/issue35975
* bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)Pablo Galindo2019-03-011-56/+0
| | | | | Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser. This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.
* bpo-35766: Merge typed_ast back into CPython (GH-11645)Guido van Rossum2019-01-311-1/+56
|
* bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)Anthony Sottile2019-01-131-0/+7
|
* bpo-30455: Generate all token related code and docs from Grammar/Tokens. ↵Serhiy Storchaka2018-12-221-237/+0
| | | | | | | | | | | | | | | | | | | (GH-10370) "Include/token.h", "Lib/token.py" (containing now some data moved from "Lib/tokenize.py") and new files "Parser/token.c" (containing the code moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by "Tools/scripts/generate_token.py". The script overwrites files only if needed and can be used on the read-only sources tree. "Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py" instead of been executable itself. Added new make targets "regen-token" and "regen-symbol" which are now dependencies of "regen-all". The documentation contains now strings for operators and punctuation tokens.
* bpo-33306: Improve SyntaxError messages for unbalanced parentheses. (GH-6516)Serhiy Storchaka2018-12-171-0/+32
|
* bpo-35436: Add missing PyErr_NoMemory() calls and other minor bug fixes. ↵Zackery Spytz2018-12-071-0/+5
| | | | | | (GH-11015) Set MemoryError when appropriate, add missing failure checks, and fix some potential leaks.
* Remove unneeded PyUnicode_READY() in tokenizer.c (GH-9114)Zackery Spytz2018-09-101-1/+1
|
* Fix Windows compiler warning in tokenize.c (GH-8359)Victor Stinner2018-07-211-1/+1
| | | | | | Fix the following warning on Windows: parser\tokenizer.c(1297): warning C4244: 'function': conversion from '__int64' to 'int', possible loss of data.
* bpo-33305: Improve SyntaxError for invalid numerical literals. (GH-6517)Serhiy Storchaka2018-07-091-13/+52
|
* tokenizer: Remove unused tabs options (#4422)Victor Stinner2017-11-171-31/+11
| | | | | | | | | | Remove the following fields from tok_state structure which are now used unused: * altwarning: "Issue warning if alternate tabs don't match" * alterror: "Issue error if alternate tabs don't match" * alttabsize: "Alternate tab spacing" Replace alttabsize variable with ALTTABSIZE define.
* bpo-30406: Make async and await proper keywords (#1669)Jelle Zijlstra2017-10-051-63/+0
| | | Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
* bpo-25324: add missing comma in Parser/tokenizer.c (GH-1910)Albert-Jan Nijburg2017-06-011-1/+1
|
* bpo-25324: copy tok_name before changing it (#1608)Albert-Jan Nijburg2017-05-311-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | * add test to check if were modifying token * copy list so import tokenize doesnt have side effects on token * shorten line * add tokenize tokens to token.h to get them to show up in token * move ERRORTOKEN back to its previous location, and fix nitpick * copy comments from token.h automatically * fix whitespace and make more pythonic * change to fix comments from @haypo * update token.rst and Misc/NEWS * change wording * some more wording changes
* Issue #28489: Merge from 3.6Berker Peksag2017-02-051-1/+1
|\
| * Issue #28489: Fix comment in tokenizer.cBerker Peksag2017-02-051-1/+1
| | | | | | | | Patch by Ryan Gonzalez.
* | Use _PyObject_CallNoArg()Victor Stinner2016-12-061-3/+3
|/ | | | | | | Replace: PyObject_CallObject(callable, NULL) with: _PyObject_CallNoArg(callable)
* Replaced outdated macros _PyUnicode_AsString and _PyUnicode_AsStringAndSizeSerhiy Storchaka2016-11-201-1/+1
| | | | with PyUnicode_AsUTF8 and PyUnicode_AsUTF8AndSize.
* merge 3.5 (#24022)Benjamin Peterson2016-09-181-1/+1
|\
| * merge 3.4Benjamin Peterson2016-09-181-1/+1
| |\
| | * properly handle the single null-byte file (closes #24022)Benjamin Peterson2016-09-181-1/+1
| | |
* | | merge 3.5 (#27981)Benjamin Peterson2016-09-121-15/+17
|\ \ \ | |/ /
| * | restructure fp_setreadl so as to avoid refleaks (closes #27981)Benjamin Peterson2016-09-121-15/+17
| | |
* | | Issue #26331: Implement the parsing part of PEP 515.Brett Cannon2016-09-091-68/+162
| | | | | | | | | | | | Thanks to Georg Brandl for the patch.
* | | Skip unused value in tokenizer codeChristian Heimes2016-09-091-1/+1
| | | | | | | | | | | | | | | | | | | | | In the case of an escape character, c is never read. tok_next() is used to advance the pointer. CID 1225097
* | | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
|\ \ \ | |/ /
| * | Issue #22570: Renamed Py_SETREF to Py_XSETREF.Serhiy Storchaka2016-04-061-1/+1
| | |
* | | remove duplicated check for fractions and complex numbers (closes #26076)Benjamin Peterson2016-03-241-4/+0
| | | | | | | | | | | | Patch by Oren Milman.
* | | Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-0/+1
|\ \ \ | |/ /
| * | Issue #26581: Use the first coding cookie on a line, not the last one.Serhiy Storchaka2016-03-201-0/+1
| | |
* | | Issue #25923: Added more const qualifiers to signatures of static and ↵Serhiy Storchaka2015-12-251-3/+3
| | | | | | | | | | | | private functions.
* | | Issue #20440: Massive replacing unsafe attribute setting code with specialSerhiy Storchaka2015-12-241-2/+1
|\ \ \ | |/ / | | | | | | macro Py_SETREF.
| * | Issue #20440: Massive replacing unsafe attribute setting code with specialSerhiy Storchaka2015-12-241-2/+1
| | | | | | | | | | | | macro Py_SETREF.
* | | Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
|\ \ \ | |/ / | | | | | | with a null byte.
| * | Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
| |\ \ | | |/ | | | | | | with a null byte.
| | * Issue #25388: Fixed tokenizer crash when processing undecodable source codeSerhiy Storchaka2015-11-141-8/+6
| | | | | | | | | | | | with a null byte.
* | | Issue #24965: Implement PEP 498 "Literal String Interpolation". ↵Eric V. Smith2015-09-191-3/+5
| | | | | | | | | | | | Documentation is still needed, I'll open an issue for that.
* | | Fixed indentation.Eric V. Smith2015-09-121-1/+1
|/ /
* | Issue #24619: Simplify async/await tokenization.Yury Selivanov2015-07-231-87/+39
| | | | | | | | | | | | | | | | | | | | This commit simplifies async/await tokenization in tokenizer.c, tokenize.py & lib2to3/tokenize.py. Previous solution was to keep a stack of async-def & def blocks, whereas the new approach is just to remember position of the outermost async-def block. This change won't bring any parsing performance improvements, but it makes the code much easier to read and validate.
* | Issue #24619: New approach for tokenizing async/await.Yury Selivanov2015-07-221-35/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes how one-line async-defs and defs are tracked by tokenizer. It allows to correctly parse invalid code such as: >>> async def f(): ... def g(): pass ... async = 10 and valid code such as: >>> async def f(): ... async def g(): pass ... await z As a consequence, is is now possible to have one-line 'async def foo(): await ..' functions: >>> async def foo(): return await bar()
* | Issue 24226: Fix parsing of many sequential one-line 'def' statements.Yury Selivanov2015-05-181-8/+17
| |
* | PEP 0492 -- Coroutines with async and await syntax. Issue #24017.Yury Selivanov2015-05-111-0/+68
| |
* | merge 3.4 (#24022)Benjamin Peterson2015-04-211-4/+3
|\ \ | |/
View Lorry Controller Status