diff options
30 files changed, 635 insertions, 197 deletions
diff --git a/Doc/Makefile b/Doc/Makefile index c131ad7291..82f5bef0fa 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -49,7 +49,7 @@ checkout: fi @if [ ! -d tools/jinja2 ]; then \ echo "Checking out Jinja..."; \ - svn checkout $(SVNROOT)/external/Jinja2-2.7.2/jinja2 tools/jinja2; \ + svn checkout $(SVNROOT)/external/Jinja-2.3.1/jinja2 tools/jinja2; \ fi @if [ ! -d tools/pygments ]; then \ echo "Checking out Pygments..."; \ diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 2324142c64..9c558bc503 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -109,6 +109,13 @@ convert a function to work with it. Let's dive in! support all of these scenarios. But these are advanced topics--let's do something simpler for your first function. + Also, if the function has multiple calls to :c:func:`PyArg_ParseTuple` + or :c:func:`PyArg_ParseTupleAndKeywords` where it supports different + types for the same argument, or if the function uses something besides + PyArg_Parse functions to parse its arguments, it probably + isn't suitable for conversion to Argument Clinic. Argument Clinic + doesn't support generic functions or polymorphic parameters. + 3. Add the following boilerplate above the function, creating our block:: /*[clinic input] @@ -170,6 +177,11 @@ convert a function to work with it. Let's dive in! Write a pickled representation of obj to the open file. [clinic start generated code]*/ + The name of the class and module should be the same as the one + seen by Python. Check the name defined in the :c:type:`PyModuleDef` + or :c:type:`PyTypeObject` as appropriate. + + 8. Declare each of the parameters to the function. Each parameter should get its own line. All the parameter lines should be @@ -455,6 +467,28 @@ convert a function to work with it. Let's dive in! Advanced Topics =============== +Now that you've had some experience working with Argument Clinic, it's time +for some advanced topics. + + +Symbolic default values +----------------------- + +The default value you provide for a parameter can't be any arbitrary +expression. Currently the following are explicitly supported: + +* Numeric constants (integer and float) +* String constants +* ``True``, ``False``, and ``None`` +* Simple symbolic constants like ``sys.maxsize``, which must + start with the name of the module + +In case you're curious, this is implemented in ``from_builtin()`` +in ``Lib/inspect.py``. + +(In the future, this may need to get even more elaborate, +to allow full expressions like ``CONSTANT - 1``.) + Renaming the C functions generated by Argument Clinic ----------------------------------------------------- @@ -479,6 +513,29 @@ The base function would now be named ``pickler_dumper()``, and the impl function would now be named ``pickler_dumper_impl()``. +The NULL default value +---------------------- + +For string and object parameters, you can set them to ``None`` to indicate +that there is no default. However, that means the C variable will be +initialized to ``Py_None``. For convenience's sakes, there's a special +value called ``NULL`` for just this case: from Python's perspective it +behaves like a default value of ``None``, but the C variable is initialized +with ``NULL``. + + +Converting functions using PyArg_UnpackTuple +-------------------------------------------- + +To convert a function parsing its arguments with :c:func:`PyArg_UnpackTuple`, +simply write out all the arguments, specifying each as an ``object``. You +may specify the ``type`` argument to cast the type as appropriate. All +arguments should be marked positional-only (add a ``/`` on a line by itself +after the last argument). + +Currently the generated code will use :c:func:`PyArg_ParseTuple`, but this +will change soon. + Optional Groups --------------- @@ -570,8 +627,8 @@ To save time, and to minimize how much you need to learn to achieve your first port to Argument Clinic, the walkthrough above tells you to use "legacy converters". "Legacy converters" are a convenience, designed explicitly to make porting existing code to Argument Clinic -easier. And to be clear, their use is entirely acceptable when porting -code for Python 3.4. +easier. And to be clear, their use is acceptable when porting code for +Python 3.4. However, in the long term we probably want all our blocks to use Argument Clinic's real syntax for converters. Why? A couple @@ -585,8 +642,8 @@ reasons: restricted to what :c:func:`PyArg_ParseTuple` supports; this flexibility won't be available to parameters using legacy converters. -Therefore, if you don't mind a little extra effort, you should consider -using normal converters instead of legacy converters. +Therefore, if you don't mind a little extra effort, please use the normal +converters instead of legacy converters. In a nutshell, the syntax for Argument Clinic (non-legacy) converters looks like a Python function call. However, if there are no explicit @@ -597,12 +654,19 @@ the same converters. All arguments to Argument Clinic converters are keyword-only. All Argument Clinic converters accept the following arguments: -``doc_default`` - If the parameter takes a default value, normally this value is also - provided in the ``inspect.Signature`` metadata, and displayed in the - docstring. ``doc_default`` lets you override the value used in these - two places: pass in a string representing the Python value you wish - to use in these two contexts. +``py_default`` + The default value for this parameter when defined in Python. + Specifically, the value that will be used in the ``inspect.Signature`` + string. + If a default value is specified for the parameter, defaults to + ``repr(default)``, else defaults to ``None``. + Specified as a string. + +``c_default`` + The default value for this parameter when defined in C. + Specifically, this will be the initializer for the variable declared + in the "parse function". + Specified as a string. ``required`` If a parameter takes a default value, Argument Clinic infers that the @@ -612,6 +676,9 @@ All Argument Clinic converters accept the following arguments: Clinic that this parameter is not optional, even if it has a default value. + (The need for ``required`` may be obviated by ``c_default``, which is + newer but arguably a better solution.) + ``annotation`` The annotation value for this parameter. Not currently supported, because PEP 8 mandates that the Python library may not use @@ -634,10 +701,11 @@ on the right is the text you'd replace it with. ``'et'`` ``str(encoding='name_of_encoding', types='bytes bytearray str')`` ``'f'`` ``float`` ``'h'`` ``short`` -``'H'`` ``unsigned_short`` +``'H'`` ``unsigned_short(bitwise=True)`` ``'i'`` ``int`` -``'I'`` ``unsigned_int`` -``'K'`` ``unsigned_PY_LONG_LONG`` +``'I'`` ``unsigned_int(bitwise=True)`` +``'k'`` ``unsigned_long(bitwise=True)`` +``'K'`` ``unsigned_PY_LONG_LONG(bitwise=True)`` ``'L'`` ``PY_LONG_LONG`` ``'n'`` ``Py_ssize_t`` ``'O!'`` ``object(subclass_of='&PySomething_Type')`` @@ -681,6 +749,15 @@ available. For each converter it'll show you all the parameters it accepts, along with the default value for each parameter. Just run ``Tools/clinic/clinic.py --converters`` to see the full list. +Py_buffer +--------- + +When using the ``Py_buffer`` converter +(or the ``'s*'``, ``'w*'``, ``'*y'``, or ``'z*'`` legacy converters), +you *must* not call :c:func:`PyBuffer_Release` on the provided buffer. +Argument Clinic generates code that does it for you (in the parsing function). + + Advanced converters ------------------- @@ -745,15 +822,26 @@ encode the value you return like normal. Currently Argument Clinic supports only a few return converters:: + bool int + unsigned int long + unsigned int + size_t Py_ssize_t + float + double DecodeFSDefault None of these take parameters. For the first three, return -1 to indicate error. For ``DecodeFSDefault``, the return type is ``char *``; return a NULL pointer to indicate an error. +(There's also an experimental ``NoneType`` converter, which lets you +return ``Py_None`` on success or ``NULL`` on failure, without having +to increment the reference count on ``Py_None``. I'm not sure it adds +enough clarity to be worth using.) + To see all the return converters Argument Clinic supports, along with their parameters (if any), just run ``Tools/clinic/clinic.py --converters`` for the full list. @@ -873,13 +961,6 @@ to specify in your subclass. Here's the current list: The Python default value for this parameter, as a Python value. Or the magic value ``unspecified`` if there is no default. -``doc_default`` - ``default`` as it should appear in the documentation, - as a string. - Or ``None`` if there is no default. - This string, when run through ``eval()``, should produce - a Python value. - ``py_default`` ``default`` as it should appear in Python code, as a string. @@ -951,6 +1032,26 @@ write your own return converter, please read ``Tools/clinic/clinic.py``, specifically the implementation of ``CReturnConverter`` and all its subclasses. +METH_O and METH_NOARGS +---------------------------------------------- + +To convert a function using ``METH_O``, make sure the function's +single argument is using the ``object`` converter, and mark the +arguments as positional-only:: + + /*[clinic input] + meth_o_sample + + argument: object + / + [clinic start generated code]*/ + + +To convert a function using ``METH_NOARGS``, just don't specify +any arguments. + +You can still use a self converter, a return converter, and specify +a ``type`` argument to the object converter for ``METH_O``. Using Argument Clinic in Python files ------------------------------------- diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 994835537f..f5fe12a739 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -1085,7 +1085,7 @@ to work with because the underlying list is accessible as an attribute. A real :class:`list` object used to store the contents of the :class:`UserList` class. -**Subclassing requirements:** Subclasses of :class:`UserList` are expect to +**Subclassing requirements:** Subclasses of :class:`UserList` are expected to offer a constructor which can be called with either no arguments or one argument. List operations which return a new sequence attempt to create an instance of the actual implementation class. To do so, it assumes that the diff --git a/Doc/library/xml.rst b/Doc/library/xml.rst index c0863e55ec..c3a26f3c1d 100644 --- a/Doc/library/xml.rst +++ b/Doc/library/xml.rst @@ -95,7 +95,7 @@ external entity expansion content into the XML document. DTD retrieval - Some XML libraries like Python's mod:'xml.dom.pulldom' retrieve document type + Some XML libraries like Python's :mod:`xml.dom.pulldom` retrieve document type definitions from remote or local locations. The feature has similar implications as the external entity expansion issue. diff --git a/Doc/library/xmlrpc.server.rst b/Doc/library/xmlrpc.server.rst index 18fee2fc1d..aca4f366bb 100644 --- a/Doc/library/xmlrpc.server.rst +++ b/Doc/library/xmlrpc.server.rst @@ -184,6 +184,70 @@ server:: # Print list of available methods print(s.system.listMethods()) +The following example included in `Lib/xmlrpc/server.py` module shows a server +allowing dotted names and registering a multicall function. + +.. warning:: + + Enabling the *allow_dotted_names* option allows intruders to access your + module's global variables and may allow intruders to execute arbitrary code on + your machine. Only use this example only within a secure, closed network. + +:: + + import datetime + + class ExampleService: + def getData(self): + return '42' + + class currentTime: + @staticmethod + def getCurrentTime(): + return datetime.datetime.now() + + server = SimpleXMLRPCServer(("localhost", 8000)) + server.register_function(pow) + server.register_function(lambda x,y: x+y, 'add') + server.register_instance(ExampleService(), allow_dotted_names=True) + server.register_multicall_functions() + print('Serving XML-RPC on localhost port 8000') + try: + server.serve_forever() + except KeyboardInterrupt: + print("\nKeyboard interrupt received, exiting.") + server.server_close() + sys.exit(0) + +This ExampleService demo can be invoked from the command line:: + + python -m xmlrpc.server + + +The client that interacts with the above server is included in +`Lib/xmlrpc/client.py`:: + + server = ServerProxy("http://localhost:8000") + + try: + print(server.currentTime.getCurrentTime()) + except Error as v: + print("ERROR", v) + + multi = MultiCall(server) + multi.getData() + multi.pow(2,9) + multi.add(1,2) + try: + for response in multi(): + print(response) + except Error as v: + print("ERROR", v) + +This client which interacts with the demo XMLRPC server can be invoked as:: + + python -m xmlrpc.client + CGIXMLRPCRequestHandler ----------------------- diff --git a/Doc/make.bat b/Doc/make.bat index 675e79300b..d6f70744e2 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -34,10 +34,10 @@ echo. goto end :checkout -svn co %SVNROOT%/external/Sphinx-1.0.7/sphinx tools/sphinx -svn co %SVNROOT%/external/docutils-0.6/docutils tools/docutils +svn co %SVNROOT%/external/Sphinx-1.2/sphinx tools/sphinx +svn co %SVNROOT%/external/docutils-0.11/docutils tools/docutils svn co %SVNROOT%/external/Jinja-2.3.1/jinja2 tools/jinja2 -svn co %SVNROOT%/external/Pygments-1.5dev-20120930/pygments tools/pygments +svn co %SVNROOT%/external/Pygments-1.6/pygments tools/pygments goto end :update diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst index 83f151be25..d54397378c 100644 --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -733,7 +733,7 @@ http ---- :meth:`~http.server.BaseHTTPRequestHandler.send_error` now accepts an -optional additional *exaplain* parameter which can be used to provide an +optional additional *explain* parameter which can be used to provide an extended error description, overriding the hardcoded default if there is one. This extended error description will be formatted using the :attr:`~http.server.HTTP.error_message_format` attribute and sent as the body diff --git a/Lib/email/quoprimime.py b/Lib/email/quoprimime.py index bc02281b1d..30bf916201 100644 --- a/Lib/email/quoprimime.py +++ b/Lib/email/quoprimime.py @@ -53,8 +53,9 @@ EMPTYSTRING = '' # space-wise. Remember that headers and bodies have different sets of safe # characters. Initialize both maps with the full expansion, and then override # the safe bytes with the more compact form. -_QUOPRI_HEADER_MAP = dict((c, '=%02X' % c) for c in range(256)) -_QUOPRI_BODY_MAP = _QUOPRI_HEADER_MAP.copy() +_QUOPRI_MAP = ['=%02X' % c for c in range(256)] +_QUOPRI_HEADER_MAP = _QUOPRI_MAP[:] +_QUOPRI_BODY_MAP = _QUOPRI_MAP[:] # Safe header bytes which need no encoding. for c in b'-!*+/' + ascii_letters.encode('ascii') + digits.encode('ascii'): @@ -121,8 +122,7 @@ def unquote(s): def quote(c): - return '=%02X' % ord(c) - + return _QUOPRI_MAP[ord(c)] def header_encode(header_bytes, charset='iso-8859-1'): @@ -140,68 +140,16 @@ def header_encode(header_bytes, charset='iso-8859-1'): if not header_bytes: return '' # Iterate over every byte, encoding if necessary. - encoded = [] - for octet in header_bytes: - encoded.append(_QUOPRI_HEADER_MAP[octet]) + encoded = header_bytes.decode('latin1').translate(_QUOPRI_HEADER_MAP) # Now add the RFC chrome to each encoded chunk and glue the chunks # together. - return '=?%s?q?%s?=' % (charset, EMPTYSTRING.join(encoded)) - - -class _body_accumulator(io.StringIO): - - def __init__(self, maxlinelen, eol, *args, **kw): - super().__init__(*args, **kw) - self.eol = eol - self.maxlinelen = self.room = maxlinelen - - def write_str(self, s): - """Add string s to the accumulated body.""" - self.write(s) - self.room -= len(s) - - def newline(self): - """Write eol, then start new line.""" - self.write_str(self.eol) - self.room = self.maxlinelen - - def write_soft_break(self): - """Write a soft break, then start a new line.""" - self.write_str('=') - self.newline() - - def write_wrapped(self, s, extra_room=0): - """Add a soft line break if needed, then write s.""" - if self.room < len(s) + extra_room: - self.write_soft_break() - self.write_str(s) - - def write_char(self, c, is_last_char): - if not is_last_char: - # Another character follows on this line, so we must leave - # extra room, either for it or a soft break, and whitespace - # need not be quoted. - self.write_wrapped(c, extra_room=1) - elif c not in ' \t': - # For this and remaining cases, no more characters follow, - # so there is no need to reserve extra room (since a hard - # break will immediately follow). - self.write_wrapped(c) - elif self.room >= 3: - # It's a whitespace character at end-of-line, and we have room - # for the three-character quoted encoding. - self.write(quote(c)) - elif self.room == 2: - # There's room for the whitespace character and a soft break. - self.write(c) - self.write_soft_break() - else: - # There's room only for a soft break. The quoted whitespace - # will be the only content on the subsequent line. - self.write_soft_break() - self.write(quote(c)) + return '=?%s?q?%s?=' % (charset, encoded) +_QUOPRI_BODY_ENCODE_MAP = _QUOPRI_BODY_MAP[:] +for c in b'\r\n': + _QUOPRI_BODY_ENCODE_MAP[c] = chr(c) + def body_encode(body, maxlinelen=76, eol=NL): """Encode with quoted-printable, wrapping at maxlinelen characters. @@ -226,26 +174,56 @@ def body_encode(body, maxlinelen=76, eol=NL): if not body: return body - # The last line may or may not end in eol, but all other lines do. - last_has_eol = (body[-1] in '\r\n') - - # This accumulator will make it easier to build the encoded body. - encoded_body = _body_accumulator(maxlinelen, eol) - - lines = body.splitlines() - last_line_no = len(lines) - 1 - for line_no, line in enumerate(lines): - last_char_index = len(line) - 1 - for i, c in enumerate(line): - if body_check(ord(c)): - c = quote(c) - encoded_body.write_char(c, i==last_char_index) - # Add an eol if input line had eol. All input lines have eol except - # possibly the last one. - if line_no < last_line_no or last_has_eol: - encoded_body.newline() - - return encoded_body.getvalue() + # quote speacial characters + body = body.translate(_QUOPRI_BODY_ENCODE_MAP) + + soft_break = '=' + eol + # leave space for the '=' at the end of a line + maxlinelen1 = maxlinelen - 1 + + encoded_body = [] + append = encoded_body.append + + for line in body.splitlines(): + # break up the line into pieces no longer than maxlinelen - 1 + start = 0 + laststart = len(line) - 1 - maxlinelen + while start <= laststart: + stop = start + maxlinelen1 + # make sure we don't break up an escape sequence + if line[stop - 2] == '=': + append(line[start:stop - 1]) + start = stop - 2 + elif line[stop - 1] == '=': + append(line[start:stop]) + start = stop - 1 + else: + append(line[start:stop] + '=') + start = stop + + # handle rest of line, special case if line ends in whitespace + if line and line[-1] in ' \t': + room = start - laststart + if room >= 3: + # It's a whitespace character at end-of-line, and we have room + # for the three-character quoted encoding. + q = quote(line[-1]) + elif room == 2: + # There's room for the whitespace character and a soft break. + q = line[-1] + soft_break + else: + # There's room only for a soft break. The quoted whitespace + # will be the only content on the subsequent line. + q = soft_break + quote(line[-1]) + append(line[start:-1] + q) + else: + append(line[start:]) + + # add back final newline if present + if body[-1] in CRLF: + append('') + + return eol.join(encoded_body) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py index 9b41b9d277..bd9b994556 100644 --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -390,10 +390,12 @@ class StringTemplateStyle(PercentStyle): def format(self, record): return self._tpl.substitute(**record.__dict__) +BASIC_FORMAT = "%(levelname)s:%(name)s:%(message)s" + _STYLES = { - '%': PercentStyle, - '{': StrFormatStyle, - '$': StringTemplateStyle + '%': (PercentStyle, BASIC_FORMAT), + '{': (StrFormatStyle, '{levelname}:{name}:{message}'), + '$': (StringTemplateStyle, '${levelname}:${name}:${message}'), } class Formatter(object): @@ -458,7 +460,7 @@ class Formatter(object): if style not in _STYLES: raise ValueError('Style must be one of: %s' % ','.join( _STYLES.keys())) - self._style = _STYLES[style](fmt) + self._style = _STYLES[style][0](fmt) self._fmt = self._style._fmt self.datefmt = datefmt @@ -1643,8 +1645,6 @@ Logger.manager = Manager(Logger.root) # Configuration classes and functions #--------------------------------------------------------------------------- -BASIC_FORMAT = "%(levelname)s:%(name)s:%(message)s" - def basicConfig(**kwargs): """ Do basic configuration for the logging system. @@ -1718,9 +1718,12 @@ def basicConfig(**kwargs): stream = kwargs.get("stream") h = StreamHandler(stream) handlers = [h] - fs = kwargs.get("format", BASIC_FORMAT) dfs = kwargs.get("datefmt", None) style = kwargs.get("style", '%') + if style not in _STYLES: + raise ValueError('Style must be one of: %s' % ','.join( + _STYLES.keys())) + fs = kwargs.get("format", _STYLES[style][1]) fmt = Formatter(fs, dfs, style) for h in handlers: if h.formatter is None: diff --git a/Lib/platform.py b/Lib/platform.py index 48b81ac7e5..11b501e7d7 100755 --- a/Lib/platform.py +++ b/Lib/platform.py @@ -649,7 +649,8 @@ def _mac_ver_xml(): except ImportError: return None - pl = plistlib.readPlist(fn) + with open(fn, 'rb') as f: + pl = plistlib.load(f) release = pl['ProductVersion'] versioninfo = ('', '', '') machine = os.uname().machine diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index ded356249f..acefb39504 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -4246,6 +4246,11 @@ class TestQuopri(unittest.TestCase): def test_encode_one_line_eol(self): self._test_encode('hello\n', 'hello\r\n', eol='\r\n') + def test_encode_one_line_eol_after_non_ascii(self): + # issue 20206; see changeset 0cf700464177 for why the encode/decode. + self._test_encode('hello\u03c5\n'.encode('utf-8').decode('latin1'), + 'hello=CF=85\r\n', eol='\r\n') + def test_encode_one_space(self): self._test_encode(' ', '=20') diff --git a/Lib/test/test_format.py b/Lib/test/test_format.py index 4b1fdf9171..29330f9171 100644 --- a/Lib/test/test_format.py +++ b/Lib/test/test_format.py @@ -142,6 +142,7 @@ class FormatTest(unittest.TestCase): testformat("%#+027.23X", big, "+0X0001234567890ABCDEF12345") # same, except no 0 flag testformat("%#+27.23X", big, " +0X001234567890ABCDEF12345") + testformat("%x", float(big), "123456_______________", 6) big = 0o12345670123456701234567012345670 # 32 octal digits testformat("%o", big, "12345670123456701234567012345670") testformat("%o", -big, "-12345670123456701234567012345670") @@ -181,6 +182,7 @@ class FormatTest(unittest.TestCase): testformat("%034.33o", big, "0012345670123456701234567012345670") # base marker shouldn't change that testformat("%0#34.33o", big, "0o012345670123456701234567012345670") + testformat("%o", float(big), "123456__________________________", 6) # Some small ints, in both Python int and flavors). testformat("%d", 42, "42") testformat("%d", -42, "-42") @@ -191,6 +193,7 @@ class FormatTest(unittest.TestCase): testformat("%#x", 1, "0x1") testformat("%#X", 1, "0X1") testformat("%#X", 1, "0X1") + testformat("%#x", 1.0, "0x1") testformat("%#o", 1, "0o1") testformat("%#o", 1, "0o1") testformat("%#o", 0, "0o0") @@ -207,10 +210,12 @@ class FormatTest(unittest.TestCase): testformat("%x", -0x42, "-42") testformat("%x", 0x42, "42") testformat("%x", -0x42, "-42") + testformat("%x", float(0x42), "42") testformat("%o", 0o42, "42") testformat("%o", -0o42, "-42") testformat("%o", 0o42, "42") testformat("%o", -0o42, "-42") + testformat("%o", float(0o42), "42") testformat("%r", "\u0378", "'\\u0378'") # non printable testformat("%a", "\u0378", "'\\u0378'") # non printable testformat("%r", "\u0374", "'\u0374'") # printable diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 3765f36aa6..3fcc77acb4 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -3517,6 +3517,22 @@ class BasicConfigTest(unittest.TestCase): # level is not explicitly set self.assertEqual(logging.root.level, self.original_logging_level) + def test_strformatstyle(self): + with captured_stdout() as output: + logging.basicConfig(stream=sys.stdout, style="{") + logging.error("Log an error") + sys.stdout.seek(0) + self.assertEqual(output.getvalue().strip(), + "ERROR:root:Log an error") + + def test_stringtemplatestyle(self): + with captured_stdout() as output: + logging.basicConfig(stream=sys.stdout, style="$") + logging.error("Log an error") + sys.stdout.seek(0) + self.assertEqual(output.getvalue().strip(), + "ERROR:root:Log an error") + def test_filename(self): def cleanup(h1, h2, fn): diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py index bf4243212e..14e7e1518e 100644 --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -4683,6 +4683,21 @@ class BufferIOTest(SocketConnectedTest): _testRecvFromIntoMemoryview = _testRecvFromIntoArray + def testRecvFromIntoSmallBuffer(self): + # See issue #20246. + buf = bytearray(8) + self.assertRaises(ValueError, self.cli_conn.recvfrom_into, buf, 1024) + + def _testRecvFromIntoSmallBuffer(self): + self.serv_conn.send(MSG*2048) + + def testRecvFromIntoEmptyBuffer(self): + buf = bytearray() + self.cli_conn.recvfrom_into(buf) + self.cli_conn.recvfrom_into(buf, 0) + + _testRecvFromIntoEmptyBuffer = _testRecvFromIntoArray + TIPC_STYPE = 2000 TIPC_LOWER = 200 diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py index 9d318100fa..5bae9f474f 100644 --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -234,6 +234,12 @@ class CommonReadTest(ReadTest): finally: tar.close() + def test_non_existent_tarfile(self): + # Test for issue11513: prevent non-existent gzipped tarfiles raising + # multiple exceptions. + with self.assertRaisesRegex(FileNotFoundError, "xxx"): + tarfile.open("xxx", self.mode) + def test_null_tarfile(self): # Test for issue6123: Allow opening empty archives. # This test guarantees that tarfile.open() does not treat an empty @@ -446,11 +452,7 @@ class MiscReadTest(MiscReadTestBase, unittest.TestCase): test_fail_comp = None class GzipMiscReadTest(GzipTest, MiscReadTestBase, unittest.TestCase): - def test_non_existent_targz_file(self): - # Test for issue11513: prevent non-existent gzipped tarfiles raising - # multiple exceptions. - with self.assertRaisesRegex(FileNotFoundError, "xxx"): - tarfile.open("xxx", self.mode) + pass class Bz2MiscReadTest(Bz2Test, MiscReadTestBase, unittest.TestCase): def test_no_name_argument(self): diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py index 63883a591c..8175fee65f 100644 --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1149,11 +1149,11 @@ class UnicodeTest(string_tests.CommonTest, self.assertEqual('%X' % letter_m, '6D') self.assertEqual('%o' % letter_m, '155') self.assertEqual('%c' % letter_m, 'm') - self.assertRaises(TypeError, '%x'.__mod__, pi) - self.assertRaises(TypeError, '%x'.__mod__, 3.14) - self.assertRaises(TypeError, '%X'.__mod__, 2.11) - self.assertRaises(TypeError, '%o'.__mod__, 1.79) - self.assertRaises(TypeError, '%c'.__mod__, pi) + self.assertWarns(DeprecationWarning, '%x'.__mod__, pi), + self.assertWarns(DeprecationWarning, '%x'.__mod__, 3.14), + self.assertWarns(DeprecationWarning, '%X'.__mod__, 2.11), + self.assertWarns(DeprecationWarning, '%o'.__mod__, 1.79), + self.assertWarns(DeprecationWarning, '%c'.__mod__, pi), def test_formatting_with_enum(self): # issue18780 diff --git a/Lib/test/test_wsgiref.py b/Lib/test/test_wsgiref.py index 05c1f4fc23..4107664429 100644 --- a/Lib/test/test_wsgiref.py +++ b/Lib/test/test_wsgiref.py @@ -286,7 +286,7 @@ class UtilityTests(TestCase): def testAppURIs(self): self.checkAppURI("http://127.0.0.1/") self.checkAppURI("http://127.0.0.1/spam", SCRIPT_NAME="/spam") - self.checkAppURI("http://127.0.0.1/sp%C3%A4m", SCRIPT_NAME="/späm") + self.checkAppURI("http://127.0.0.1/sp%E4m", SCRIPT_NAME="/sp\xe4m") self.checkAppURI("http://spam.example.com:2071/", HTTP_HOST="spam.example.com:2071", SERVER_PORT="2071") self.checkAppURI("http://spam.example.com/", @@ -300,15 +300,19 @@ class UtilityTests(TestCase): def testReqURIs(self): self.checkReqURI("http://127.0.0.1/") self.checkReqURI("http://127.0.0.1/spam", SCRIPT_NAME="/spam") - self.checkReqURI("http://127.0.0.1/sp%C3%A4m", SCRIPT_NAME="/späm") + self.checkReqURI("http://127.0.0.1/sp%E4m", SCRIPT_NAME="/sp\xe4m") self.checkReqURI("http://127.0.0.1/spammity/spam", SCRIPT_NAME="/spammity", PATH_INFO="/spam") + self.checkReqURI("http://127.0.0.1/spammity/sp%E4m", + SCRIPT_NAME="/spammity", PATH_INFO="/sp\xe4m") self.checkReqURI("http://127.0.0.1/spammity/spam;ham", SCRIPT_NAME="/spammity", PATH_INFO="/spam;ham") self.checkReqURI("http://127.0.0.1/spammity/spam;cookie=1234,5678", SCRIPT_NAME="/spammity", PATH_INFO="/spam;cookie=1234,5678") self.checkReqURI("http://127.0.0.1/spammity/spam?say=ni", SCRIPT_NAME="/spammity", PATH_INFO="/spam",QUERY_STRING="say=ni") + self.checkReqURI("http://127.0.0.1/spammity/spam?s%E4y=ni", + SCRIPT_NAME="/spammity", PATH_INFO="/spam",QUERY_STRING="s%E4y=ni") self.checkReqURI("http://127.0.0.1/spammity/spam", 0, SCRIPT_NAME="/spammity", PATH_INFO="/spam",QUERY_STRING="say=ni") diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index efc3e725c6..3ac14be6d5 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -576,7 +576,7 @@ class PanedWindowTest(AbstractWidgetTest, unittest.TestCase): widget = self.create() self.assertEqual(str(widget['orient']), 'vertical') errmsg='attempt to change read-only option' - if get_tk_patchlevel() < (8, 6, 0): # actually this was changen in 8.6b3 + if get_tk_patchlevel() < (8, 6, 0): # actually this was changed in 8.6b3 errmsg='Attempt to change read-only option' self.checkInvalidParam(widget, 'orient', 'horizontal', errmsg=errmsg) diff --git a/Lib/wsgiref/util.py b/Lib/wsgiref/util.py index 1f1e6cce17..516fe898d0 100644 --- a/Lib/wsgiref/util.py +++ b/Lib/wsgiref/util.py @@ -57,14 +57,14 @@ def application_uri(environ): if environ['SERVER_PORT'] != '80': url += ':' + environ['SERVER_PORT'] - url += quote(environ.get('SCRIPT_NAME') or '/') + url += quote(environ.get('SCRIPT_NAME') or '/', encoding='latin1') return url def request_uri(environ, include_query=True): """Return the full request URI, optionally including the query string""" url = application_uri(environ) from urllib.parse import quote - path_info = quote(environ.get('PATH_INFO',''),safe='/;=,') + path_info = quote(environ.get('PATH_INFO',''), safe='/;=,', encoding='latin1') if not environ.get('SCRIPT_NAME'): url += path_info[1:] else: diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py index 9e0f0691bf..1238b9bf37 100644 --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -1461,18 +1461,18 @@ if __name__ == "__main__": # simple test program (from the XML-RPC specification) - # server = ServerProxy("http://localhost:8000") # local server - server = ServerProxy("http://time.xmlrpc.com/RPC2") + # local server, available from Lib/xmlrpc/server.py + server = ServerProxy("http://localhost:8000") try: print(server.currentTime.getCurrentTime()) except Error as v: print("ERROR", v) - # The server at xmlrpc.com doesn't seem to support multicall anymore. multi = MultiCall(server) - multi.currentTime.getCurrentTime() - multi.currentTime.getCurrentTime() + multi.getData() + multi.pow(2,9) + multi.add(1,2) try: for response in multi(): print(response) diff --git a/Lib/xmlrpc/server.py b/Lib/xmlrpc/server.py index d914282917..304e218c00 100644 --- a/Lib/xmlrpc/server.py +++ b/Lib/xmlrpc/server.py @@ -960,10 +960,24 @@ class DocCGIXMLRPCRequestHandler( CGIXMLRPCRequestHandler, if __name__ == '__main__': + import datetime + + class ExampleService: + def getData(self): + return '42' + + class currentTime: + @staticmethod + def getCurrentTime(): + return datetime.datetime.now() + server = SimpleXMLRPCServer(("localhost", 8000)) server.register_function(pow) server.register_function(lambda x,y: x+y, 'add') + server.register_instance(ExampleService(), allow_dotted_names=True) + server.register_multicall_functions() print('Serving XML-RPC on localhost port 8000') + print('It is advisable to run this example server within a secure, closed network.') try: server.serve_forever() except KeyboardInterrupt: @@ -1218,6 +1218,7 @@ Eric V. Smith Gregory P. Smith Mark Smith Roy Smith +Ryan Smith-Roberts Rafal Smotrzyk Eric Snow Dirk Soede @@ -25,14 +25,25 @@ Core and Builtins Library ------- -<<<<<<< local -- Issue #20152: Ported Python/import.c over to Argument Clinic. -======= +- Issue #20242: Fixed basicConfig() format strings for the alternative + formatting styles. Thanks to kespindler for the bug report and patch. + +- Issue #20246: Fix buffer overflow in socket.recvfrom_into. + +- Issues #20206 and #5803: Fix edge case in email.quoprimime.encode where it + truncated lines ending in a character needing encoding but no newline by + using a more efficient algorithm that doesn't have the bug. + +- Issue #19082: Working xmlrpc.server and xmlrpc.client examples. Both in + modules and in documentation. Initial patch contributed by Vajrasky Kok. + +- Issue #20138: The wsgiref.application_uri() and wsgiref.request_uri() + functions now conform to PEP 3333 when handle non-ASCII URLs. + - Issue #19097: Raise the correct Exception when cgi.FieldStorage is given an Invalid fileobj. -- Issue #20217: Fix build in SCHED_SPORADIC is defined. ->>>>>>> other +- Issue #20152: Ported Python/import.c over to Argument Clinic. - Issue #13107: argparse and optparse no longer raises an exception when output a help on environment with too small COLUMNS. Based on patch by @@ -59,6 +70,8 @@ Library - Issue #20072: Fixed multiple errors in tkinter with wantobjects is False. +- Issue #20229: Avoid plistlib deprecation warning in platform.mac_ver(). + IDLE ---- @@ -73,6 +86,16 @@ Tests Tools/Demos ----------- +- Issue #20228: Argument Clinic now has special support for class special + methods. + +- Issue #20214: Fixed a number of small issues and documentation errors in + Argument Clinic (see issue for details). + +- Issue #20196: Fixed a bug where Argument Clinic did not generate correct + parsing code for functions with positional-only parameters where all arguments + are optional. + - Issue #18960: 2to3 and the findnocoding.py script now ignore the source encoding declaration on the second line if the first line contains anything except a comment. @@ -117,8 +140,8 @@ Core and Builtins - Issue #19969: PyBytes_FromFormatV() now raises an OverflowError if "%c" argument is not in range [0; 255]. -- Issue #19995: %c, %o, %x, and %X now raise TypeError on non-integer input; - reworded docs to clarify that an integer type should define both __int__ +- Issue #19995: %c, %o, %x, and %X now issue a DeprecationWarning on non-integer + input; reworded docs to clarify that an integer type should define both __int__ and __index__. - Issue #19787: PyThread_set_key_value() now always set the value. In Python diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 310e67207e..c1aa9a30bc 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -1699,11 +1699,13 @@ defdict_init(PyObject *self, PyObject *args, PyObject *kwds) } PyDoc_STRVAR(defdict_doc, -"defaultdict(default_factory) --> dict with default factory\n\ +"defaultdict(default_factory[, ...]) --> dict with default factory\n\ \n\ The default factory is called without arguments to produce\n\ a new value when a key is not present, in __getitem__ only.\n\ A defaultdict compares equal to a dict with the same items.\n\ +All remaining arguments are treated the same as if they were\n\ +passed to the dict constructor, including keyword arguments.\n\ "); /* See comment in xxsubtype.c */ diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 917dd45f36..e687a1eb0e 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4068,9 +4068,6 @@ PyDoc_STRVAR(_pickle_Pickler___init____doc__, "to map the new Python 3 names to the old module names used in Python\n" "2, so that the pickle data stream is readable with Python 2."); -#define _PICKLE_PICKLER___INIT___METHODDEF \ - {"__init__", (PyCFunction)_pickle_Pickler___init__, METH_VARARGS|METH_KEYWORDS, _pickle_Pickler___init____doc__}, - static PyObject * _pickle_Pickler___init___impl(PicklerObject *self, PyObject *file, PyObject *protocol, int fix_imports); @@ -4095,7 +4092,7 @@ exit: static PyObject * _pickle_Pickler___init___impl(PicklerObject *self, PyObject *file, PyObject *protocol, int fix_imports) -/*[clinic end generated code: checksum=2b5ce6452544600478cf9f4b701ab9d9b5efbab9]*/ +/*[clinic end generated code: checksum=defa3d9e9f8b51fb257d4fdfca99db503db0e6df]*/ { _Py_IDENTIFIER(persistent_id); _Py_IDENTIFIER(dispatch_table); @@ -6637,9 +6634,6 @@ PyDoc_STRVAR(_pickle_Unpickler___init____doc__, "respectively. The *encoding* can be \'bytes\' to read these 8-bit\n" "string instances as bytes objects."); -#define _PICKLE_UNPICKLER___INIT___METHODDEF \ - {"__init__", (PyCFunction)_pickle_Unpickler___init__, METH_VARARGS|METH_KEYWORDS, _pickle_Unpickler___init____doc__}, - static PyObject * _pickle_Unpickler___init___impl(UnpicklerObject *self, PyObject *file, int fix_imports, const char *encoding, const char *errors); @@ -6665,7 +6659,7 @@ exit: static PyObject * _pickle_Unpickler___init___impl(UnpicklerObject *self, PyObject *file, int fix_imports, const char *encoding, const char *errors) -/*[clinic end generated code: checksum=9ce6783224e220573d42a94fe1bb7199d6f1c5a6]*/ +/*[clinic end generated code: checksum=26c1d4a06841a8e51d29a0c244ba7f4607ff358a]*/ { _Py_IDENTIFIER(persistent_load); diff --git a/Modules/audioop.c b/Modules/audioop.c index bae4f2687e..f8bb18acf9 100644 --- a/Modules/audioop.c +++ b/Modules/audioop.c @@ -1505,7 +1505,7 @@ audioop_lin2adpcm(PyObject *self, PyObject *args) Py_ssize_t i; int size, step, valpred, delta, index, sign, vpdiff, diff; - PyObject *rv = NULL, *state, *str; + PyObject *rv = NULL, *state, *str = NULL; int outputbuffer = 0, bufferstep; if (!PyArg_ParseTuple(args, "y*iO:lin2adpcm", @@ -1605,8 +1605,8 @@ audioop_lin2adpcm(PyObject *self, PyObject *args) bufferstep = !bufferstep; } rv = Py_BuildValue("(O(ii))", str, valpred, index); - Py_DECREF(str); exit: + Py_XDECREF(str); PyBuffer_Release(&view); return rv; } diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 6c229bcc10..7add6b6651 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -2864,7 +2864,6 @@ sock_recvfrom_into(PySocketSockObject *s, PyObject *args, PyObject* kwds) return NULL; buf = pbuf.buf; buflen = pbuf.len; - assert(buf != 0 && buflen > 0); if (recvlen < 0) { PyBuffer_Release(&pbuf); @@ -2875,6 +2874,11 @@ sock_recvfrom_into(PySocketSockObject *s, PyObject *args, PyObject* kwds) if (recvlen == 0) { /* If nbytes was not specified, use the buffer's length */ recvlen = buflen; + } else if (recvlen > buflen) { + PyBuffer_Release(&pbuf); + PyErr_SetString(PyExc_ValueError, + "nbytes is greater than the length of the buffer"); + return NULL; } readlen = sock_recvfrom_guts(s, buf, recvlen, flags, &addr); diff --git a/Modules/zlibmodule.c b/Modules/zlibmodule.c index 5139888ffb..575dbd2d59 100644 --- a/Modules/zlibmodule.c +++ b/Modules/zlibmodule.c @@ -198,7 +198,7 @@ static PyObject * zlib_compress(PyModuleDef *module, PyObject *args) { PyObject *return_value = NULL; - Py_buffer bytes = {NULL, NULL, 0, 0, 0, 0, NULL, NULL, NULL, NULL, NULL}; + Py_buffer bytes = {NULL, NULL}; int group_right_1 = 0; int level = 0; @@ -219,7 +219,7 @@ zlib_compress(PyModuleDef *module, PyObject *args) return_value = zlib_compress_impl(module, &bytes, group_right_1, level); /* Cleanup for bytes */ - if (bytes.buf) + if (bytes.obj) PyBuffer_Release(&bytes); return return_value; @@ -227,7 +227,7 @@ zlib_compress(PyModuleDef *module, PyObject *args) static PyObject * zlib_compress_impl(PyModuleDef *module, Py_buffer *bytes, int group_right_1, int level) -/*[clinic end generated code: checksum=9f055a396620bc1a8a13d74c3496249528b32b0d]*/ +/*[clinic end generated code: checksum=2c59af563a4595c5ecea4011701f482ae350aa5f]*/ { PyObject *ReturnVal = NULL; Byte *input, *output = NULL; @@ -789,7 +789,7 @@ static PyObject * zlib_Decompress_decompress(PyObject *self, PyObject *args) { PyObject *return_value = NULL; - Py_buffer data = {NULL, NULL, 0, 0, 0, 0, NULL, NULL, NULL, NULL, NULL}; + Py_buffer data = {NULL, NULL}; unsigned int max_length = 0; if (!PyArg_ParseTuple(args, @@ -800,7 +800,7 @@ zlib_Decompress_decompress(PyObject *self, PyObject *args) exit: /* Cleanup for data */ - if (data.buf) + if (data.obj) PyBuffer_Release(&data); return return_value; @@ -808,7 +808,7 @@ exit: static PyObject * zlib_Decompress_decompress_impl(compobject *self, Py_buffer *data, unsigned int max_length) -/*[clinic end generated code: checksum=5b1e4f9f1ef8eca55fff78356f9df0c81232ed3b]*/ +/*[clinic end generated code: checksum=e0058024c4a97b411d2e2197791b89fde175f76f]*/ { int err; unsigned int old_length, length = DEFAULTALLOC; diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 48eccf7cbb..1ce10cfd90 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -14004,11 +14004,24 @@ mainformatlong(PyObject *v, if (!PyNumber_Check(v)) goto wrongtype; + /* make sure number is a type of integer */ + /* if not, issue deprecation warning for now */ if (!PyLong_Check(v)) { if (type == 'o' || type == 'x' || type == 'X') { iobj = PyNumber_Index(v); if (iobj == NULL) { - return -1; + PyErr_Clear(); + if (PyErr_WarnEx(PyExc_DeprecationWarning, + "automatic int conversions have been deprecated", + 1)) { + return -1; + } + iobj = PyNumber_Long(v); + if (iobj == NULL ) { + if (PyErr_ExceptionMatches(PyExc_TypeError)) + goto wrongtype; + return -1; + } } } else { @@ -14090,10 +14103,22 @@ formatchar(PyObject *v) PyObject *iobj; long x; /* make sure number is a type of integer */ + /* if not, issue deprecation warning for now */ if (!PyLong_Check(v)) { iobj = PyNumber_Index(v); if (iobj == NULL) { - goto onError; + PyErr_Clear(); + if (PyErr_WarnEx(PyExc_DeprecationWarning, + "automatic int conversions have been deprecated", + 1)) { + return -1; + } + iobj = PyNumber_Long(v); + if (iobj == NULL ) { + if (PyErr_ExceptionMatches(PyExc_TypeError)) + goto onError; + return -1; + } } v = iobj; Py_DECREF(iobj); diff --git a/Tools/clinic/clinic.py b/Tools/clinic/clinic.py index 421b9e3d39..86df329a72 100755 --- a/Tools/clinic/clinic.py +++ b/Tools/clinic/clinic.py @@ -38,6 +38,7 @@ version = '1' _empty = inspect._empty _void = inspect._void +NoneType = type(None) class Unspecified: def __repr__(self): @@ -110,13 +111,16 @@ is_legal_c_identifier = re.compile('^[A-Za-z_][A-Za-z0-9_]*$').match def is_legal_py_identifier(s): return all(is_legal_c_identifier(field) for field in s.split('.')) -# added "module", "self", "cls", and "null" just to be safe -# (clinic will generate variables with these names) +# though it's called c_keywords, really it's a list of parameter names +# that are okay in Python but aren't a good idea in C. so if they're used +# Argument Clinic will add "_value" to the end of the name in C. +# (We added "args", "type", "module", "self", "cls", and "null" +# just to be safe, even though they're not C keywords.) c_keywords = set(""" -asm auto break case char cls const continue default do double +args asm auto break case char cls const continue default do double else enum extern float for goto if inline int long module null register return self short signed sizeof static struct switch -typedef typeof union unsigned void volatile while +type typedef typeof union unsigned void volatile while """.strip().split()) def ensure_legal_c_identifier(s): @@ -391,11 +395,17 @@ class CLanguage(Language): @staticmethod def template_base(*args): - flags = '|'.join(f for f in args if f) - return """ + # HACK suppress methoddef define for METHOD_NEW and METHOD_INIT + base = """ PyDoc_STRVAR({c_basename}__doc__, {docstring}); +""" + + if args[-1] == None: + return base + flags = '|'.join(f for f in args if f) + return base + """ #define {methoddef_name} \\ {{"{name}", (PyCFunction){c_basename}, {methoddef_flags}, {c_basename}__doc__}}, """.replace('{methoddef_flags}', flags) @@ -591,6 +601,12 @@ static {impl_return_type} count_min = min(count_min, count) count_max = max(count_max, count) + if count == 0: + add(""" case 0: + break; +""") + continue + group_ids = {p.group for p in subset} # eliminate duplicates d = {} d['count'] = count @@ -643,7 +659,13 @@ static {impl_return_type} name = full_name.rpartition('.')[2] template_dict['name'] = name - c_basename = f.c_basename or full_name.replace(".", "_") + if f.c_basename: + c_basename = f.c_basename + else: + fields = full_name.split(".") + if fields[-1] == '__new__': + fields.pop() + c_basename = "_".join(fields) template_dict['c_basename'] = c_basename methoddef_name = "{}_METHODDEF".format(c_basename.upper()) @@ -672,8 +694,8 @@ static {impl_return_type} c.render(p, data) positional = parameters[-1].kind == inspect.Parameter.POSITIONAL_ONLY - if has_option_groups: - assert positional + if has_option_groups and (not positional): + fail("You cannot use optional groups ('[' and ']')\nunless all parameters are positional-only ('/').") # now insert our "self" (or whatever) parameters # (we deliberately don't call render on self converters) @@ -1164,8 +1186,81 @@ class Class: def __repr__(self): return "<clinic.Class " + repr(self.name) + " at " + str(id(self)) + ">" +unsupported_special_methods = set(""" + +__abs__ +__add__ +__and__ +__bytes__ +__call__ +__complex__ +__delitem__ +__divmod__ +__eq__ +__float__ +__floordiv__ +__ge__ +__getattr__ +__getattribute__ +__getitem__ +__gt__ +__hash__ +__iadd__ +__iand__ +__idivmod__ +__ifloordiv__ +__ilshift__ +__imod__ +__imul__ +__index__ +__int__ +__invert__ +__ior__ +__ipow__ +__irshift__ +__isub__ +__iter__ +__itruediv__ +__ixor__ +__le__ +__len__ +__lshift__ +__lt__ +__mod__ +__mul__ +__neg__ +__new__ +__next__ +__or__ +__pos__ +__pow__ +__radd__ +__rand__ +__rdivmod__ +__repr__ +__rfloordiv__ +__rlshift__ +__rmod__ +__rmul__ +__ror__ +__round__ +__rpow__ +__rrshift__ +__rshift__ +__rsub__ +__rtruediv__ +__rxor__ +__setattr__ +__setitem__ +__str__ +__sub__ +__truediv__ +__xor__ + +""".strip().split()) + -DATA, CALLABLE, METHOD, STATIC_METHOD, CLASS_METHOD = range(5) +INVALID, CALLABLE, STATIC_METHOD, CLASS_METHOD, METHOD_INIT, METHOD_NEW = range(6) class Function: """ @@ -1200,6 +1295,8 @@ class Function: @property def methoddef_flags(self): + if self.kind in (METHOD_INIT, METHOD_NEW): + return None flags = [] if self.kind == CLASS_METHOD: flags.append('METH_CLASS') @@ -1315,6 +1412,10 @@ class CConverter(metaclass=CConverterAutoRegister): # Or the magic value "unspecified" if there is no default. default = unspecified + # If not None, default must be isinstance() of this type. + # (You can also specify a tuple of types.) + default_type = None + # "default" as it should appear in the documentation, as a string. # Or None if there is no default. doc_default = None @@ -1381,12 +1482,21 @@ class CConverter(metaclass=CConverterAutoRegister): self.name = name if default is not unspecified: + if self.default_type and not isinstance(default, self.default_type): + if isinstance(self.default_type, type): + types_str = self.default_type.__name__ + else: + types_str = ', '.join((cls.__name__ for cls in self.default_type)) + fail("{}: default value {!r} for field {} is not of type {}".format( + self.__class__.__name__, default, name, types_str)) self.default = default self.py_default = py_default if py_default is not None else py_repr(default) self.doc_default = doc_default if doc_default is not None else self.py_default self.c_default = c_default if c_default is not None else c_repr(default) - elif doc_default is not None: - fail(function.fullname + " argument " + name + " specified a 'doc_default' without having a 'default'") + else: + self.py_default = py_default + self.doc_default = doc_default + self.c_default = c_default if annotation != unspecified: fail("The 'annotation' parameter is not currently permitted.") self.required = required @@ -1532,6 +1642,7 @@ class CConverter(metaclass=CConverterAutoRegister): class bool_converter(CConverter): type = 'int' + default_type = bool format_unit = 'p' c_ignored_default = '0' @@ -1541,12 +1652,19 @@ class bool_converter(CConverter): class char_converter(CConverter): type = 'char' + default_type = str format_unit = 'c' c_ignored_default = "'\0'" + def converter_init(self): + if len(self.default) != 1: + fail("char_converter: illegal default value " + repr(self.default)) + + @add_legacy_c_converter('B', bitwise=True) class byte_converter(CConverter): type = 'byte' + default_type = int format_unit = 'b' c_ignored_default = "'\0'" @@ -1556,11 +1674,13 @@ class byte_converter(CConverter): class short_converter(CConverter): type = 'short' + default_type = int format_unit = 'h' c_ignored_default = "0" class unsigned_short_converter(CConverter): type = 'unsigned short' + default_type = int format_unit = 'H' c_ignored_default = "0" @@ -1571,6 +1691,7 @@ class unsigned_short_converter(CConverter): @add_legacy_c_converter('C', types='str') class int_converter(CConverter): type = 'int' + default_type = int format_unit = 'i' c_ignored_default = "0" @@ -1582,6 +1703,7 @@ class int_converter(CConverter): class unsigned_int_converter(CConverter): type = 'unsigned int' + default_type = int format_unit = 'I' c_ignored_default = "0" @@ -1591,11 +1713,13 @@ class unsigned_int_converter(CConverter): class long_converter(CConverter): type = 'long' + default_type = int format_unit = 'l' c_ignored_default = "0" class unsigned_long_converter(CConverter): type = 'unsigned long' + default_type = int format_unit = 'k' c_ignored_default = "0" @@ -1605,11 +1729,13 @@ class unsigned_long_converter(CConverter): class PY_LONG_LONG_converter(CConverter): type = 'PY_LONG_LONG' + default_type = int format_unit = 'L' c_ignored_default = "0" class unsigned_PY_LONG_LONG_converter(CConverter): type = 'unsigned PY_LONG_LONG' + default_type = int format_unit = 'K' c_ignored_default = "0" @@ -1619,23 +1745,27 @@ class unsigned_PY_LONG_LONG_converter(CConverter): class Py_ssize_t_converter(CConverter): type = 'Py_ssize_t' + default_type = int format_unit = 'n' c_ignored_default = "0" class float_converter(CConverter): type = 'float' + default_type = float format_unit = 'f' c_ignored_default = "0.0" class double_converter(CConverter): type = 'double' + default_type = float format_unit = 'd' c_ignored_default = "0.0" class Py_complex_converter(CConverter): type = 'Py_complex' + default_type = complex format_unit = 'D' c_ignored_default = "{0.0, 0.0}" @@ -1644,10 +1774,16 @@ class object_converter(CConverter): type = 'PyObject *' format_unit = 'O' - def converter_init(self, *, type=None, subclass_of=None): - if subclass_of: + def converter_init(self, *, converter=None, type=None, subclass_of=None): + if converter: + if subclass_of: + fail("object: Cannot pass in both 'converter' and 'subclass_of'") + self.format_unit = 'O&' + self.converter = converter + elif subclass_of: self.format_unit = 'O!' self.subclass_of = subclass_of + if type is not None: self.type = type @@ -1659,6 +1795,7 @@ class object_converter(CConverter): @add_legacy_c_converter('z#', nullable=True, length=True) class str_converter(CConverter): type = 'const char *' + default_type = (str, Null, NoneType) format_unit = 's' def converter_init(self, *, encoding=None, types="str", @@ -1725,6 +1862,7 @@ class PyByteArrayObject_converter(CConverter): class unicode_converter(CConverter): type = 'PyObject *' + default_type = (str, Null, NoneType) format_unit = 'U' @add_legacy_c_converter('u#', length=True) @@ -1732,6 +1870,7 @@ class unicode_converter(CConverter): @add_legacy_c_converter('Z#', nullable=True, length=True) class Py_UNICODE_converter(CConverter): type = 'Py_UNICODE *' + default_type = (str, Null, NoneType) format_unit = 'u' def converter_init(self, *, nullable=False, length=False): @@ -1754,11 +1893,11 @@ class Py_buffer_converter(CConverter): type = 'Py_buffer' format_unit = 'y*' impl_by_reference = True - c_ignored_default = "{NULL, NULL, 0, 0, 0, 0, NULL, NULL, NULL, NULL, NULL}" + c_ignored_default = "{NULL, NULL}" def converter_init(self, *, types='bytes bytearray buffer', nullable=False): - if self.default != unspecified: - fail("There is no legal default value for Py_buffer ") + if self.default not in (unspecified, None): + fail("The only legal default value for Py_buffer is None.") self.c_default = self.c_ignored_default types = set(types.strip().split()) bytes_type = set(('bytes',)) @@ -1777,7 +1916,7 @@ class Py_buffer_converter(CConverter): fail('Py_buffer_converter: illegal combination of arguments (nullable=True)') elif types == (bytes_bytearray_buffer_type): format_unit = 'y*' - elif types == (bytearray_type | rwuffer_type): + elif types == (bytearray_type | rwbuffer_type): format_unit = 'w*' if not format_unit: fail("Py_buffer_converter: illegal combination of arguments") @@ -1786,7 +1925,7 @@ class Py_buffer_converter(CConverter): def cleanup(self): name = ensure_legal_c_identifier(self.name) - return "".join(["if (", name, ".buf)\n PyBuffer_Release(&", name, ");\n"]) + return "".join(["if (", name, ".obj)\n PyBuffer_Release(&", name, ");\n"]) class self_converter(CConverter): @@ -1797,7 +1936,7 @@ class self_converter(CConverter): type = "PyObject *" def converter_init(self, *, type=None): f = self.function - if f.kind == CALLABLE: + if f.kind in (CALLABLE, METHOD_INIT): if f.cls: self.name = "self" else: @@ -1809,6 +1948,9 @@ class self_converter(CConverter): elif f.kind == CLASS_METHOD: self.name = "cls" self.type = "PyTypeObject *" + elif f.kind == METHOD_NEW: + self.name = "type" + self.type = "PyTypeObject *" if type: self.type = type @@ -1889,34 +2031,59 @@ return_value = Py_None; Py_INCREF(Py_None); '''.strip()) -class int_return_converter(CReturnConverter): +class bool_return_converter(CReturnConverter): type = 'int' def render(self, function, data): self.declare(data) self.err_occurred_if("_return_value == -1", data) - data.return_conversion.append( - 'return_value = PyLong_FromLong((long)_return_value);\n') - + data.return_conversion.append('return_value = PyBool_FromLong((long)_return_value);\n') class long_return_converter(CReturnConverter): type = 'long' + conversion_fn = 'PyLong_FromLong' + cast = '' def render(self, function, data): self.declare(data) self.err_occurred_if("_return_value == -1", data) data.return_conversion.append( - 'return_value = PyLong_FromLong(_return_value);\n') + ''.join(('return_value = ', self.conversion_fn, '(', self.cast, '_return_value);\n'))) + +class int_return_converter(long_return_converter): + type = 'int' + cast = '(long)' +class unsigned_long_return_converter(long_return_converter): + type = 'unsigned long' + conversion_fn = 'PyLong_FromUnsignedLong' + +class unsigned_int_return_converter(unsigned_long_return_converter): + type = 'unsigned int' + cast = '(unsigned long)' -class Py_ssize_t_return_converter(CReturnConverter): +class Py_ssize_t_return_converter(long_return_converter): type = 'Py_ssize_t' + conversion_fn = 'PyLong_FromSsize_t' + +class size_t_return_converter(long_return_converter): + type = 'size_t' + conversion_fn = 'PyLong_FromSize_t' + + +class double_return_converter(CReturnConverter): + type = 'double' + cast = '' def render(self, function, data): self.declare(data) - self.err_occurred_if("_return_value == -1", data) + self.err_occurred_if("_return_value == -1.0", data) data.return_conversion.append( - 'return_value = PyLong_FromSsize_t(_return_value);\n') + 'return_value = PyFloat_FromDouble(' + self.cast + '_return_value);\n') + +class float_return_converter(double_return_converter): + type = 'float' + cast = '(double)' class DecodeFSDefault_return_converter(CReturnConverter): @@ -2184,6 +2351,18 @@ class DSLParser: function_name = fields.pop() module, cls = self.clinic._module_and_class(fields) + fields = full_name.split('.') + if fields[-1] == '__new__': + if (self.kind != CLASS_METHOD) or (not cls): + fail("__new__ must be a class method!") + self.kind = METHOD_NEW + elif fields[-1] == '__init__': + if (self.kind != CALLABLE) or (not cls): + fail("__init__ must be a normal method, not a class or static method!") + self.kind = METHOD_INIT + elif fields[-1] in unsupported_special_methods: + fail(fields[-1] + " should not be converted to Argument Clinic! (Yet.)") + if not module: fail("Undefined module used in declaration of " + repr(full_name.strip()) + ".") self.function = Function(name=function_name, full_name=full_name, module=module, cls=cls, c_basename=c_basename, @@ -2335,6 +2514,10 @@ class DSLParser: if isinstance(expr, ast.Name) and expr.id == 'NULL': value = NULL elif isinstance(expr, ast.Attribute): + c_default = kwargs.get("c_default") + if not (isinstance(c_default, str) and c_default): + fail("When you specify a named constant (" + repr(py_default) + ") as your default value,\nyou MUST specify a valid c_default.") + a = [] n = expr while isinstance(n, ast.Attribute): @@ -2344,11 +2527,8 @@ class DSLParser: fail("Malformed default value (looked like a Python constant)") a.append(n.id) py_default = ".".join(reversed(a)) - value = None - c_default = kwargs.get("c_default") - if not (isinstance(c_default, str) and c_default): - fail("When you specify a named constant (" + repr(py_default) + ") as your default value,\nyou MUST specify a valid c_default.") kwargs["py_default"] = py_default + value = eval(py_default) else: value = ast.literal_eval(expr) else: @@ -2382,7 +2562,8 @@ class DSLParser: if isinstance(annotation, ast.Name): return annotation.id, False, {} - assert isinstance(annotation, ast.Call) + if not isinstance(annotation, ast.Call): + fail("Annotations must be either a name, a function call, or a string.") name = annotation.func.id kwargs = {node.arg: ast.literal_eval(node.value) for node in annotation.keywords} |