diff options
| author | Ezio Melotti <ezio.melotti@gmail.com> | 2009-12-20 12:24:57 +0000 |
|---|---|---|
| committer | Ezio Melotti <ezio.melotti@gmail.com> | 2009-12-20 12:24:57 +0000 |
| commit | a8f6f1e2d63b90549278b1162d264bb279459007 (patch) | |
| tree | 875477b66eb7659659be382131beaffa54ab5e32 /Doc/library/tokenize.rst | |
| parent | 7e5229c3abd6195a7dbca8a0da30dbd203212205 (diff) | |
| download | cpython-git-a8f6f1e2d63b90549278b1162d264bb279459007.tar.gz | |
Fixed capitalization and markup; added imports in the example
Diffstat (limited to 'Doc/library/tokenize.rst')
| -rw-r--r-- | Doc/library/tokenize.rst | 9 |
1 files changed, 6 insertions, 3 deletions
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index deb7826393..d66b07c62d 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -94,16 +94,19 @@ function it uses to do this is available: (as a string) and a list of any lines (not decoded from bytes) it has read in. - It detects the encoding from the presence of a utf-8 bom or an encoding - cookie as specified in pep-0263. If both a bom and a cookie are present, + It detects the encoding from the presence of a UTF-8 BOM or an encoding + cookie as specified in :pep:`263`. If both a BOM and a cookie are present, but disagree, a SyntaxError will be raised. - If no encoding is specified, then the default of 'utf-8' will be returned. + If no encoding is specified, then the default of ``'utf-8'`` will be returned. Example of a script re-writer that transforms float literals into Decimal objects:: + from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP + from io import BytesIO + def decistmt(s): """Substitute Decimals for floats in a string of statements. |
