| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
KafkaClient.send()
|
| | |
|
| |
|
| |
In an effort to reduce the surface area of lock coordination, and thereby hopefully reduce lock contention, I think we can remove locking from the read-only KafkaClient methods: connected, is_disconnected, in_flight_request_count, and least_loaded_node . Given that the read data could change after the lock is released but before the caller uses it, the value of acquiring a lock here does not seem high to me.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Since 1.3b1 (released Oct 10, 2014) Sphinx has support for NumPy and
Google style docstring support via sphinx.ext.napoleon extension.
The latter is already used, but sphinxcontrib-napoleon requirement
still presents.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The problem is that the type of required operation result is
"long".
```
>>> type(278251978 & 0xffffffff)
<type 'long'>
```
However, by default "format" method uses __format__():
```
>>> (278251978 & 0xffffffff).__format__('')
'278251978'
```
So, let's compare things using the same engine:
```
>>> "{!r}".format(278251978 & 0xffffffff)
'278251978L'
```
Fixes: https://github.com/dpkp/kafka-python/issues/1717
Signed-off-by: Stanislav Levin <slev@altlinux.org>
|
| | |
|
| |
|
|
| |
stable group (#1695)
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
This `skip_double_compressed_messages` flag was added in https://github.com/dpkp/kafka-python/pull/755 in
order to fix https://github.com/dpkp/kafka-python/issues/718.
However, grep'ing through the code, it looks like it this is no longer
used anywhere and doesn't do anything.
So removing it.
|
| | |
|
| |
|
|
|
|
|
| |
`getattr(object, 'x', object.y)` will evaluate the default argument
`object.y` regardless of whether `'x'` exists.
For details see: https://stackoverflow.com/q/31443989/770425
|
| |
|
|
|
| |
Popen objects may deadlock when using stdout=PIPE or stderr=PIPE
with Popen.wait(). Using Popen.communicate() avoids the issue.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Previously we weren't accounting for when the response tuple also has a
`error_message` value.
Note that in Java, the error fieldname is inconsistent:
- `CreateTopicsResponse` / `CreatePartitionsResponse` uses `topic_errors`
- `DeleteTopicsResponse` uses `topic_error_codes`
So this updates the `CreateTopicsResponse` classes to match.
The fix is a little brittle, but should suffice for now.
|
| | |
|
| |
|
| |
`isinstance()` won't work here, as the types require identity comparison.
|
| | |
|
| | |
|
| |
|
|
| |
Cleanup the formatting, remove parens, extraneous spaces, etc.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This was completely broken previously because it didn't lookup the group
coordinator of the consumer group. Also added basic error
handling/raising.
Note:
I added the `group_coordinator_id` as an optional kwarg. As best I
can tell, the Java client doesn't include this and instead looks it up
every time. However, if we add this, it allows the caller the
flexibility to bypass the network round trip of the lookup if for some
reason they already know the `group_coordinator_id`.
|
| |
|
|
|
|
|
| |
Set a clear default value for `validate_only` / `include_synonyms`
Previously the kwarg defaulted to `None`, but then sent a `False` so this
makes it more explicit and reduces ambiguity.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, this only queried the controller. In actuality, the Kafka
protocol requires that the client query all brokers in order to get the
full list of consumer groups.
Note: The Java code (as best I can tell) doesn't allow limiting this to
specific brokers. And on the surface, this makes sense... you typically
don't care about specific brokers.
However, the inverse is true... consumer groups care about knowing their
group coordinator so they don't have to repeatedly query to find it.
In fact, a Kafka broker will only return the groups that it's a
coordinator for. While this is an implementation detail that is not
guaranteed by the upstream broker code, and technically should not be
relied upon, I think it very unlikely to change.
So monitoring scripts that fetch the offsets or describe the consumers
groups of all groups in the cluster can simply issue one call per broker
to identify all the coordinators, rather than having to issue one call
per consumer group. For an ad-hoc script this doesn't matter, but for
a monitoring script that runs every couple of minutes, this can be a big
deal. I know in the situations where I will use this, this matters more
to me than the risk of the interface unexpectedly breaking.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Support fetching the offsets of a consumer group.
Note: As far as I can tell (the Java code is a little inscrutable), the
Java AdminClient doesn't allow specifying the `coordinator_id` or the
`partitions`.
But I decided to include them because they provide a lot of additional
flexibility:
1. allowing users to specify the partitions allows this method to be used even for
older brokers that don't support the OffsetFetchRequest_v2
2. allowing users to specify the coordinator ID gives them a way to
bypass a network round trip. This method will frequently be used for
monitoring, and if you've got 1,000 consumer groups that are being
monitored once a minute, that's ~1.5M requests a day that are
unnecessarily duplicated as the coordinator doesn't change unless
there's an error.
|
| |
|
|
|
|
|
|
|
| |
The controller send error handling was completely broken.
It also pinned the metadata version unnecessarily.
Additionally, several of the methods were sending to the controller
but either that was unnecessary, or just plain wrong. So updated
following the pattern of the Java Admin client.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We need a way to send a request to the group coordinator.
I spent a day and a half trying to implement a `_send_request_to_group_coordinator()`
that included:
1. caching the value of the group coordinator so that it wouldn't have
to be repeatedly looked up on every call. This is particularly important
because the `list_consumer_groups()`, `list_consumer_group_offsets()`,
and `describe_consumer_groups()` will frequently be used by monitoring
scripts. I know across the production clusters that I support, using a
cached value will save ~1M calls per day.
2. clean and consistent error handling. This is difficult because the
responses are inconsistent about error codes. Some have a top-level
error code, some bury it within the description of the actual item.
3. Avoiding tight coupling between this method and the request/response
classes... the custom parsing logic for errors etc, given that it's
non-standard, should live in the callers, not here.
So finally I gave up and just went with this simpler solution and made
it so the callers can optionally bypass this if they somehow already
know the group coordinator.
|
| |
|
|
|
| |
This is a new class, so let's not support the old version strings and
saddle ourselves with tech debt right from the get-go.
|
| |
|
|
| |
Fix #1633
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
`UnsupportedVersionError` is intended to indicate a server-side error:
https://github.com/dpkp/kafka-python/blob/ba7372e44ffa1ee49fb4d5efbd67534393e944db/kafka/errors.py#L375-L378
So we should not be raising it for client-side errors. I realize that
semantically this seems like the appropriate error to raise. However,
this is confusing when debugging... for a real-life example, see
https://github.com/Parsely/pykafka/issues/697. So I strongly feel that
server-side errors should be kept separate from client-side errors,
even if all the client is doing is proactively protecting against
hitting a situation where the broker would return this error.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
