diff options
author | Carlo Cabanilla <carlo@datadoghq.com> | 2014-02-28 12:53:18 -0500 |
---|---|---|
committer | Carlo Cabanilla <carlo@datadoghq.com> | 2014-02-28 19:28:49 +0000 |
commit | a798c1e32b91325f4311461362b510e720b925e3 (patch) | |
tree | b9472122c8cce1251013012b87086ab4e05581ad /kafka/protocol.py | |
parent | 4abf7ee1fbbdc47c8cb7b35f2600e58f1f95e6bb (diff) | |
download | kafka-python-a798c1e32b91325f4311461362b510e720b925e3.tar.gz |
fix consumer retry logic (fixes #135)
Fixes bug in the follow condition:
* Starting buffer size is 1024, max buffer size is 2048, both set on an instance level
* Fetch from p0, p1 and received response
* p0 has more than 1024 bytes, consumer doubles buffer size to 2048 and marks p0 for retry
* p1 has more than 1024 bytes, consumer tries to double buffer size, but sees that it's at
the max and raises ConsumerFetchSizeTooSmall
The fix changes the logic to the following:
* Starting buffer size is 1024 set on a per-partition level, max buffer size is 2048 set on an instance level
* Fetch from p0, p1 and received response
* p0 has more than 1024 bytes, consumer doubles buffer size to 2048 for p0 and marks p0 for retry
* p1 has more than 1024 bytes, consumer double buffer size to 2048 for p1 and marks p1 for retry
* Consumer sees that there's partitions to retry, repeats parsing loop
* p0 sent all the bytes this time, consumer yields these messages
* p1 sent all the bytes this time, consumer yields these messages
Diffstat (limited to 'kafka/protocol.py')
0 files changed, 0 insertions, 0 deletions