diff options
| author | Serhiy Storchaka <storchaka@gmail.com> | 2018-12-22 11:18:40 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2018-12-22 11:18:40 +0200 |
| commit | 8ac658114dec4964479baecfbc439fceb40eaa79 (patch) | |
| tree | e66c4c3beda293a6fdf01763306697d15d0af157 /Doc/library/token.rst | |
| parent | c1b4b0f6160e1919394586f44b12538505fed300 (diff) | |
| download | cpython-git-8ac658114dec4964479baecfbc439fceb40eaa79.tar.gz | |
bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.
"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.
Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".
The documentation contains now strings for operators and punctuation tokens.
Diffstat (limited to 'Doc/library/token.rst')
| -rw-r--r-- | Doc/library/token.rst | 59 |
1 files changed, 1 insertions, 58 deletions
diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 373991027e..5358eb5a29 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -44,64 +44,7 @@ functions. The functions mirror definitions in the Python C header files. The token constants are: -.. data:: ENDMARKER - NAME - NUMBER - STRING - NEWLINE - INDENT - DEDENT - LPAR - RPAR - LSQB - RSQB - COLON - COMMA - SEMI - PLUS - MINUS - STAR - SLASH - VBAR - AMPER - LESS - GREATER - EQUAL - DOT - PERCENT - LBRACE - RBRACE - EQEQUAL - NOTEQUAL - LESSEQUAL - GREATEREQUAL - TILDE - CIRCUMFLEX - LEFTSHIFT - RIGHTSHIFT - DOUBLESTAR - PLUSEQUAL - MINEQUAL - STAREQUAL - SLASHEQUAL - PERCENTEQUAL - AMPEREQUAL - VBAREQUAL - CIRCUMFLEXEQUAL - LEFTSHIFTEQUAL - RIGHTSHIFTEQUAL - DOUBLESTAREQUAL - DOUBLESLASH - DOUBLESLASHEQUAL - AT - ATEQUAL - RARROW - ELLIPSIS - OP - ERRORTOKEN - N_TOKENS - NT_OFFSET - +.. include:: token-list.inc The following token type values aren't used by the C tokenizer but are needed for the :mod:`tokenize` module. |