summaryrefslogtreecommitdiff
path: root/kafka
Commit message (Collapse)AuthorAgeFilesLines
...
* | | Improve async producer code: logic and style fixesViktor Shlapakov2015-06-034-43/+55
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - send_producer_request with fail_on_error=False to retry failed reqs only - using an internal dict with with namedtuple keys for retry counters - refresh metadata on refresh_error irrespective to retries options - removed infinite retries (retry_options.limit=None) as an over-feature - separate producer init args for retries options (limit,backoff,on_timeouts) - AsyncProducerQueueFull returns a list of failed messages - producer tests improved thanks to @rogaha and @toli
* | | Change backoff message log levelViktor Shlapakov2015-06-031-1/+1
| | |
* | | Fix async producer queue put argumentsViktor Shlapakov2015-06-031-2/+4
| | |
* | | Fix names for async retries opts, add timeout for putViktor Shlapakov2015-06-034-20/+34
| | |
* | | Simplification of retry logicViktor Shlapakov2015-06-032-31/+29
| | |
* | | Fix small issues with names/testsViktor Shlapakov2015-06-032-9/+11
| | |
* | | async queue: refactored code; add one more testEduard Iskandarov2015-06-031-2/+7
| | |
* | | add producer send batch queue overfilled testИскандаров Эдуард2015-06-031-1/+7
| | |
* | | Disable retry on timeouts by default (old behaviour)Viktor Shlapakov2015-06-031-1/+1
| | |
* | | Clean and simplify retry logicViktor Shlapakov2015-06-032-60/+49
| | |
* | | Arg fixes for base/keyed producersViktor Shlapakov2015-06-032-4/+2
| | |
* | | Improved retry logicViktor Shlapakov2015-06-034-31/+73
| | |
* | | Fixed tests and other issues after rebaseViktor Shlapakov2015-06-032-2/+1
| | |
* | | Returned default behaviour with no retriesViktor Shlapakov2015-06-031-1/+1
| | |
* | | Fix: check failed reqs to retry only for positive limitViktor Shlapakov2015-06-031-3/+3
| | |
* | | Fixed compatible issues with testsViktor Shlapakov2015-06-031-3/+5
| | |
* | | Producer _send_upstream fixes, added tests for retriesViktor Shlapakov2015-06-031-6/+20
| | |
* | | Fixed base producer importsViktor Shlapakov2015-06-031-1/+2
| | |
* | | Retries for async batchingViktor Shlapakov2015-06-034-15/+50
| |/ |/|
* | Check vs None directly.William Ting2015-05-171-4/+4
|/
* Fix calling of _offsetsArturo Filastò2015-05-141-1/+1
| | | | | | | | Previously you would see this error: ``` self.offsets._fetch[(topic, partition)], AttributeError: 'function' object has no attribute '_fetch' ```
* Merge pull request #367 from dpkp/clean_metadata_refreshDana Powers2015-04-121-13/+14
|\ | | | | Clear local metadata cache before refresh in client.load_metadata_for_topics()
| * Update debug log messages in client.load_metadata_for_topicsDana Powers2015-04-121-2/+2
| |
| * clean local metadata before requesting refresh in load_metadata_for_topics()Dana Powers2015-04-121-2/+7
| |
| * Simplify client.reset_topic_metadataDana Powers2015-04-121-9/+5
| |
* | Merge pull request #366 from dpkp/failed_payloadsDana Powers2015-04-121-79/+77
|\ \ | |/ |/| Refactor client request/response handling to support better retries
| * Improve send_produce_request docstringDana Powers2015-04-061-6/+17
| |
| * Add debug logging for all broker-aware requests/responsesDana Powers2015-04-061-0/+3
| |
| * Refactor KafkaClient._send_broker_aware_request to return a list of responsesDana Powers2015-04-061-73/+57
| | | | | | | | | | | | and include individual (unraised) FailedPayloadsError instances rather than always raising a FailedPayloadsError. This should allow producers to determine which payloads succeeded and which failed, specifically.
* | Use kafka_bytestring when decoding message topics in ↵Dana Powers2015-04-041-18/+22
| | | | | | | | KafkaConsumer.fetch_messages
* | Use list comprehension on _topics in KafkaConsumer.fetch_messagesDana Powers2015-04-041-8/+8
| |
* | call _consume_topic_partition() before storing offsetDana Powers2015-04-041-1/+1
| | | | | | | | in KafkaConsumer.set_topic_partitions((topic, partition, offset))
* | Fix python3 str/bytes bug in KafkaConsumer.set_topic_partitions({(topic, ↵Dana Powers2015-04-041-1/+1
|/ | | | partition): offset, })
* Merge pull request #360 from dpkp/atexit_cleanupDana Powers2015-04-042-4/+51
|\ | | | | Register atexit handlers for consumer and producer thread/multiprocess cleanup
| * Register atexit handlers for consumer and producer thread/multiprocess ↵Dana Powers2015-03-312-4/+51
| | | | | | | | cleanup (not __del__)
* | Make external API consistently support python3 strings for topic.Space2015-04-035-4/+17
|/
* Merge pull request #356 from dpkp/always_fetch_offsetsDana Powers2015-03-303-12/+30
|\ | | | | fetch commit offsets in base consumer unless group is None
| * Bulk fetch offset partitions in base consumer -- suggested by ecanzonieriDana Powers2015-03-301-8/+9
| |
| * Fetch previously committed offsets in base consumer class so long asDana Powers2015-03-303-12/+29
| | | | | | | | | | | | | | | | a group is configured (but document that group must be None for old servers). This fixes multiprocessor consumer issue that prevented access to commit offsets if auto_commit is disabled. Also refactor fetch_last_known_offsets based on KafkaConsumer While still setting unknown offsets to 0
* | Merge pull request #355 from dpkp/correlation_id_moduloDana Powers2015-03-301-7/+7
|\ \ | |/ |/| correlation_id modulo
| * Rollover KafkaClient correlation ids at 2**31 to keep within int32 protocol ↵Dana Powers2015-03-291-7/+7
| | | | | | | | encoding
* | Merge pull request #357 from dpkp/del_consumer_stopDana Powers2015-03-301-0/+3
|\ \ | | | | | | Stop consumers on delete
| * | Stop consumers on deleteDana Powers2015-03-301-0/+3
| |/
* | fixup kafka consumer docstringDana Powers2015-03-291-1/+1
| |
* | Improve KafkaConsumer docstringsDana Powers2015-03-291-134/+94
|/
* Merge pull request #340 from dpkp/deprecate_metadata_broker_listDana Powers2015-03-291-8/+26
|\ | | | | Deprecate metadata_broker_list in favor of bootstrap_servers
| * Deprecate KafkaConsumer config metadata_broker_list in favor of ↵Dana Powers2015-03-231-8/+26
| | | | | | | | bootstrap_servers
* | Merge pull request #336 from scrapinghub/feature-mp-consumer-paramsDana Powers2015-03-292-35/+51
|\ \ | | | | | | Using additional params for MP consumer child process
| * | Manageable queue.put() operation for MPConsumer processesViktor Shlapakov2015-03-252-4/+12
| | |
| * | Using mp.manager to solve the issue with join for MPConsumerViktor Shlapakov2015-03-241-6/+7
| | |
View Lorry Controller Status