summaryrefslogtreecommitdiff
path: root/Tools/scripts/generate_token.py
Commit message (Collapse)AuthorAgeFilesLines
* bpo-40267: Fix message when last input character produces a SyntaxError ↵Lysandros Nikolaou2020-04-151-0/+4
| | | | | | | | (GH-19521) When there is a SyntaxError after reading the last input character from the tokenizer and if no newline follows it, the error message used to be `unexpected EOF while parsing`, which is wrong.
* Mark files as executable that are meant as scripts. (GH-15354)Greg Price2019-09-091-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | This is the converse of GH-15353 -- in addition to plenty of scripts in the tree that are marked with the executable bit (and so can be directly executed), there are a few that have a leading `#!` which could let them be executed, but it doesn't do anything because they don't have the executable bit set. Here's a command which finds such files and marks them. The first line finds files in the tree with a `#!` line *anywhere*; the next-to-last step checks that the *first* line is actually of that form. In between we filter out files that already have the bit set, and some files that are meant as fragments to be consumed by one or another kind of preprocessor. $ git grep -l '^#!' \ | grep -vxFf <( \ git ls-files --stage \ | perl -lane 'print $F[3] if (!/^100644/)' \ ) \ | grep -ve '\.in$' -e '^Doc/includes/' \ | while read f; do head -c2 "$f" | grep -qxF '#!' \ && chmod a+x "$f"; \ done
* bpo-30455: Generate all token related code and docs from Grammar/Tokens. ↵Serhiy Storchaka2018-12-221-0/+268
(GH-10370) "Include/token.h", "Lib/token.py" (containing now some data moved from "Lib/tokenize.py") and new files "Parser/token.c" (containing the code moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by "Tools/scripts/generate_token.py". The script overwrites files only if needed and can be used on the read-only sources tree. "Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py" instead of been executable itself. Added new make targets "regen-token" and "regen-symbol" which are now dependencies of "regen-all". The documentation contains now strings for operators and punctuation tokens.