summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
* Fix send to controllerJeff Widman2018-11-181-44/+89
| | | | | | | | | The controller send error handling was completely broken. It also pinned the metadata version unnecessarily. Additionally, several of the methods were sending to the controller but either that was unnecessary, or just plain wrong. So updated following the pattern of the Java Admin client.
* Add group coordinator lookupJeff Widman2018-11-181-0/+40
| | | | | | | | | | | | | | | | | | | | | | | We need a way to send a request to the group coordinator. I spent a day and a half trying to implement a `_send_request_to_group_coordinator()` that included: 1. caching the value of the group coordinator so that it wouldn't have to be repeatedly looked up on every call. This is particularly important because the `list_consumer_groups()`, `list_consumer_group_offsets()`, and `describe_consumer_groups()` will frequently be used by monitoring scripts. I know across the production clusters that I support, using a cached value will save ~1M calls per day. 2. clean and consistent error handling. This is difficult because the responses are inconsistent about error codes. Some have a top-level error code, some bury it within the description of the actual item. 3. Avoiding tight coupling between this method and the request/response classes... the custom parsing logic for errors etc, given that it's non-standard, should live in the callers, not here. So finally I gave up and just went with this simpler solution and made it so the callers can optionally bypass this if they somehow already know the group coordinator.
* Remove support for api versions as strings from KafkaAdminJeff Widman2018-11-181-10/+0
| | | | | This is a new class, so let's not support the old version strings and saddle ourselves with tech debt right from the get-go.
* Be explicit with tuples for %s formattingJeff Widman2018-11-1826-45/+45
| | | | Fix #1633
* Stop using broker-errors for client-side problemsJeff Widman2018-11-183-38/+44
| | | | | | | | | | | | | `UnsupportedVersionError` is intended to indicate a server-side error: https://github.com/dpkp/kafka-python/blob/ba7372e44ffa1ee49fb4d5efbd67534393e944db/kafka/errors.py#L375-L378 So we should not be raising it for client-side errors. I realize that semantically this seems like the appropriate error to raise. However, this is confusing when debugging... for a real-life example, see https://github.com/Parsely/pykafka/issues/697. So I strongly feel that server-side errors should be kept separate from client-side errors, even if all the client is doing is proactively protecting against hitting a situation where the broker would return this error.
* Update requirements-dev to latest versions used on travis runsDana Powers2018-11-171-11/+11
|
* Pin pytest on 3.x -- we arent ready for pytest 4.0 yetDana Powers2018-11-171-1/+1
|
* Use TypeError for invalid typeJeff Widman2018-11-171-1/+1
|
* Update changelogDana Powers2018-11-122-0/+106
|
* Migrate from `Unittest` to `pytest` (#1620)Jeff Widman2018-11-107-73/+34
|
* raising logging level on messages signalling data loss (#1553)Alexander Sibiryakov2018-11-101-2/+3
|
* set socket timeout for the wake_w (#1577)flaneur2018-11-102-0/+6
|
* (Attempt to) Fix deadlock between consumer and heartbeat (#1628)Dana Powers2018-11-102-4/+2
|
* Fix typoJeff Widman2018-11-071-1/+1
|
* Pre-compile pack/unpack function callsbillyevans2018-10-291-13/+29
| | | | | | | | | | | | | | | | I noticed that pack/unpack functions from https://github.com/dpkp/kafka-python/blob/master/kafka/protocol/types.py might be slightly improved. I made pre-compilation for them. It gives about 10% better performance compared to the current implementation. Consumption of 100msg: ``` 239884 0.187 0.000 0.287 0.000 types.py:18(_unpack) # new version 239884 0.192 0.000 0.323 0.000 types.py:17(_unpack) ``` I also made some profiling for producers/consumers. It gives about 1-1.5% time savings.
* Document KafkaAdmin classJeff Widman2018-10-293-0/+13
|
* Cleanup fixture importsJeff Widman2018-10-277-15/+10
| | | | | | | | | | | | | | | `random_string` now comes from `test.fixtures` and was being transparently imported via `test.testutil` so this bypasses the pointless indirect import. Similarly, `kafka_version` was transparently imported by `test.testutil` from `test.fixtures`. Also removed `random_port()` in `test.testutil` because its unused as its been replaced by the one in `test.fixtures`. This is part of the pytest migration that was started back in a1869c4be5f47b4f6433610249aaf29af4ec95e5.
* Minor aesthetic cleanup of partitioner testsJeff Widman2018-10-271-11/+10
|
* Remove unused ivy_root variableJeff Widman2018-10-271-1/+0
| | | | This is no longer used anywhere in the codebase
* Minor cleanup of testing docJeff Widman2018-10-271-13/+17
| | | | | | | Removed some of the hardcoded values as they are now outdated, and just pointed to where to find the current value in the code. Also some minor wordsmithing.
* Stop using deprecated log.warn()Jeff Widman2018-10-261-3/+3
| | | | I missed this in my previous cleanup back in 9221fcf83528b5c3657e43636cb84c1d18025acd.
* Stop pinning `pylint`Jeff Widman2018-10-261-1/+1
| | | | | | | | | We have many deprecation warnings in the travis logs for things that are fixed in newer versions of `pylint` or `pylint`'s dependencies. Note that `pylint` >= 2.0 does not support python 2, so this will result in different versions of pylint running for python 2 vs python 3. Personally, I am just fine with this.
* Add temp workaround for upstream pylint bugJeff Widman2018-10-261-1/+3
| | | | Temporarily workaround https://github.com/PyCQA/pylint/issues/2571 so that we can stop pinning `pylint`.
* Add KafkaAdmin classRichard Lee2018-10-2410-0/+684
| | | | | | Requires cluster version > 0.10.0.0, and uses new wire protocol classes to do many things via broker connection that previously needed to be done directly in zookeeper.
* Fix sphinx urlJeff Widman2018-10-242-2/+2
| | | | | | | When I was fixing urls the other day, I noticed that sphinx hadn't added https but there was an open ticket: https://github.com/sphinx-doc/sphinx/issues/5522 Now that that is resolved, I'm updating it here.
* Prevent `pylint` import errors on `six.moves`Jeff Widman2018-10-241-0/+1
| | | | | | | | | | | | | | | | | | | | | `six.moves` is a dynamically-created namespace that doesn't actually exist and therefore `pylint` can't statically analyze it. By default, `pylint` is smart enough to realize that and ignore the import errors. However, because we vendor it, the location changes to `kafka.vendor.six.moves` so `pylint` doesn't realize it should be ignored. So this explicitly ignores it. `pylint` documentation of this feature: http://pylint.pycqa.org/en/1.9/technical_reference/features.html?highlight=ignored-modules#id34 More background: * https://github.com/PyCQA/pylint/issues/1640 * https://github.com/PyCQA/pylint/issues/223
* Vendor enum34Jeff Widman2018-10-221-0/+841
| | | | | | | | | This is needed for https://github.com/dpkp/kafka-python/pull/1540 While the usage there is trivial and could probably be worked around, I'd rather vendor it so that future code can use enums... since `enum` is already available in the python 3 stdlib, this will be easy enough to eventually stop vendoring whenever we finally drop python 2 support.
* Vendor `six` consistentlyJeff Widman2018-10-2215-30/+34
| | | | Use vendored `six`, and also `six.moves.range` rather than `xrange`
* Update remote urls: snappy, https, etcJeff Widman2018-10-225-10/+10
| | | | | Snappy URL was outdated. Similarly, many of these sites now support https.
* Bump vendored `six` to `1.11.0`Jeff Widman2018-10-221-16/+42
| | | | | | | | | Bump `six` to `1.11.0`. Most changes do not affect us, but it's good to stay up to date. Also, we will likely start vendoring `enum34` in which case https://github.com/benjaminp/six/pull/178 is needed. Note that this preserves the `kafka-python` customization from https://github.com/dpkp/kafka-python/pull/979 which has been submitted upstream as https://github.com/benjaminp/six/pull/176 but not yet merged.
* Fix typo in file nameJeff Widman2018-10-221-0/+0
|
* Support produce with Kafka record headersHeikki Nousiainen2018-09-275-18/+40
|
* Expose record headers in ConsumerRecordsHeikki Nousiainen2018-09-273-6/+13
|
* Add positive tests for headers in record encode/decodeHeikki Nousiainen2018-09-272-3/+18
|
* Expose ConsumerRebalanceListener in allBen Harack2018-09-271-1/+1
| | | | | | This solves a warning in linters like PyCharm, which warns that a line like: from kafka import ConsumerRebalanceListener is actually accessing a protected member of a class or module. Adding it to __all__ should solve this.
* Remove ConsumerTimeoutJonathan Emord2018-09-271-4/+0
|
* Clear the metrics dictionary on close. (#1569)Kishore Nallan2018-08-311-0/+2
|
* Return future from commit_offsets_async (#1560)Mike Lang2018-08-311-1/+6
|
* add kerberos domain name config for gssapi sasl mechanism handshake (#1542)the-sea2018-08-314-4/+17
|
* add support for smaller topic metadata fetch during bootstrap (#1541)Ning Xie2018-08-313-6/+14
|
* Document connections_max_idle_msJeff Widman2018-06-263-0/+15
| | | | | This was added in #1068 but never documented. Fix #1497
* Don't use `kafka.common` internally1.3.5Jeff Widman2018-06-0516-36/+32
| | | | This finishes the split from `kafka.common` to `kafka.errors`/`kafka.structs`.
* Bump version for developmentDana Powers2018-05-261-1/+1
|
* Release 1.4.31.4.3Dana Powers2018-05-263-1/+85
|
* Retain but deprecate kafka.errors.ConnectionError for compatibilityDana Powers2018-05-261-0/+4
|
* Improve connection handling when bootstrap list is invalid (#1507)Dana Powers2018-05-261-6/+3
| | | | * only perform single dns lookup for connect_blocking() * fix blocking timeout in check_version()
* Ignore MetadataResponses with empty broker list (#1506)Dana Powers2018-05-263-1/+28
|
* Document methods that return NoneJeff Widman2018-05-231-0/+3
| | | | | If a valid broker in the cluster has no partitions, it will return None rather than an empty set. Similarly updated a few other methods.
* Stop shadowing `ConnectionError`Jeff Widman2018-05-238-31/+30
| | | | | | | | | | In Python3, `ConnectionError` is a native exception. So rename our custom one to `KafkaConnectionError` to prevent accidentally shadowing the native one. Note that there are still valid uses of `ConnectionError` in this code. They already expect a native Python3 `ConnectionError`, and also already handle the Python2 compatibility issues.
* Stop using deprecated log.warn()Jeff Widman2018-05-101-3/+3
|
View Lorry Controller Status