summaryrefslogtreecommitdiff
path: root/kafka/consumer
Commit message (Collapse)AuthorAgeFilesLines
* Fix _mp_consume queue variable name conflictDana Powers2015-12-091-2/+2
|
* Merge pull request #426 from ucarion/ucarion-kafkaconsumer-closeDana Powers2015-12-061-0/+4
|\ | | | | Add KafkaConsumer#close
| * Add KafkaConsumer#closeUlysse Carion2015-07-071-0/+4
| |
* | new pylint disables for pylint 1.5.1Zack Dever2015-12-022-2/+2
| | | | | | | | | | tested locally with `pylint --rcfile=pylint.rc -E kafka test`. should pass travis builds.
* | Fix python3 / python2 comments re queue/QueueDana Powers2015-12-022-10/+10
| |
* | Merge pull request #457 from saaros/block-for-number-of-messagesDana Powers2015-12-022-11/+18
|\ \ | | | | | | Consumers get_messages: allow blocking until some messages are received
| * | Consumers get_messages: allow blocking until some messages are receivedOskari Saarenmaa2015-09-182-11/+18
| |/ | | | | | | | | | | | | | | | | Modified MultiProcessConsumer's and SimpleConsumer's `block` argument to allow integer value which defines the number of messages to block for. This allows callers to ask for a high number of messages and block only until some of them are received. Otherwise callers would have to request messages one by one or block for some time.
* | Merge pull request #446 from zackdever/consumer-class-configsDana Powers2015-12-021-2/+3
|\ \ | | | | | | Set default config on consumer class to support subclass config changes.
| * | set default config on consumer class to support subclass config changes.Zack Dever2015-08-131-2/+3
| |/
* | Merge pull request #456 from barricadeio/mp-retryDana Powers2015-12-022-46/+59
|\ \ | | | | | | Support retry semantics in MultiProcessConsumer
| * | Support retry semantics in MultiProcessConsumerRoss Duggan2015-09-142-46/+59
| |/
* | allow to retrieve partition info in mp consumerMartin Olveyra2015-06-293-8/+10
|/
* Merge pull request #412 from haosdent/seek_absolute_offsetDana Powers2015-06-201-17/+46
|\ | | | | fix #410 SimpleConsumer cannot seek to an absolute offset.
| * fix #410 SimpleConsumer cannot seek to an absolute offset.haosdent2015-06-201-17/+46
| |
* | Lower logging level on replica not available and commitEnrico Canzonieri2015-06-191-1/+1
|/
* Refactor SimpleConsumer get_messages and _fetchDana Powers2015-06-101-29/+33
| | | | | - attempt to make these easier to read - add marginal retry logic to internal _fetch loop
* Merge pull request #405 from dpkp/log_error_typeDana Powers2015-06-101-5/+8
|\ | | | | Log response error types in consumer and producer logs
| * Add error type to SimpleConsumer error loggingDana Powers2015-06-101-5/+8
| |
* | Change SimpleConsumer.reset_partition_offset to return offset / None on ↵Dana Powers2015-06-101-5/+19
| | | | | | | | failure (dont raise exception)
* | Change Consumer commit() to return True/False and log error; dont raise ↵Dana Powers2015-06-101-12/+18
|/ | | | client exceptions
* KafkaConsumer style nitsDana Powers2015-06-081-26/+48
|
* Update KafkaConsumer to handle request-specific FailedPayloadsErrorsDana Powers2015-06-081-11/+13
|
* Cleanup imports in kafka/client and kafka/consumerDana Powers2015-06-082-22/+20
|
* Merge pull request #389 from dpkp/task_done_key_errorDana Powers2015-06-081-2/+10
|\ | | | | KafkaConsumer.task_done: warn and skip unrecognized topic-partitions
| * KafkaConsumer.task_done - warn and skip messages from unrecognized ↵Dana Powers2015-06-061-2/+10
| | | | | | | | topic-partitions
* | SimpleConsumer should refresh metadata and retry on leadership changes; only ↵Dana Powers2015-06-081-1/+4
| | | | | | | | raise on UnknownTopicOrPartition
* | Merge pull request #380 from dpkp/multiprocess_consumer_partitionsDana Powers2015-06-081-3/+7
|\ \ | | | | | | Support optional partitions kwarg in MultiProcessConsumer
| * | Support optional partitions kwarg in MultiProcessConsumerDana Powers2015-05-171-3/+7
| |/
* | try to fix uncaught FailedPayloadsErrorreAsOn20102015-06-091-1/+8
| |
* | Use separate module loggers instead of a single 'kafka' loggerDana Powers2015-06-063-3/+8
|/
* Check vs None directly.William Ting2015-05-171-4/+4
|
* Fix calling of _offsetsArturo Filastò2015-05-141-1/+1
| | | | | | | | Previously you would see this error: ``` self.offsets._fetch[(topic, partition)], AttributeError: 'function' object has no attribute '_fetch' ```
* Use kafka_bytestring when decoding message topics in ↵Dana Powers2015-04-041-18/+22
| | | | KafkaConsumer.fetch_messages
* Use list comprehension on _topics in KafkaConsumer.fetch_messagesDana Powers2015-04-041-8/+8
|
* call _consume_topic_partition() before storing offsetDana Powers2015-04-041-1/+1
| | | | in KafkaConsumer.set_topic_partitions((topic, partition, offset))
* Fix python3 str/bytes bug in KafkaConsumer.set_topic_partitions({(topic, ↵Dana Powers2015-04-041-1/+1
| | | | partition): offset, })
* Merge pull request #360 from dpkp/atexit_cleanupDana Powers2015-04-041-3/+25
|\ | | | | Register atexit handlers for consumer and producer thread/multiprocess cleanup
| * Register atexit handlers for consumer and producer thread/multiprocess ↵Dana Powers2015-03-311-3/+25
| | | | | | | | cleanup (not __del__)
* | Make external API consistently support python3 strings for topic.Space2015-04-032-4/+4
|/
* Bulk fetch offset partitions in base consumer -- suggested by ecanzonieriDana Powers2015-03-301-8/+9
|
* Fetch previously committed offsets in base consumer class so long asDana Powers2015-03-303-12/+29
| | | | | | | | a group is configured (but document that group must be None for old servers). This fixes multiprocessor consumer issue that prevented access to commit offsets if auto_commit is disabled. Also refactor fetch_last_known_offsets based on KafkaConsumer While still setting unknown offsets to 0
* Merge pull request #357 from dpkp/del_consumer_stopDana Powers2015-03-301-0/+3
|\ | | | | Stop consumers on delete
| * Stop consumers on deleteDana Powers2015-03-301-0/+3
| |
* | fixup kafka consumer docstringDana Powers2015-03-291-1/+1
| |
* | Improve KafkaConsumer docstringsDana Powers2015-03-291-134/+94
|/
* Merge pull request #340 from dpkp/deprecate_metadata_broker_listDana Powers2015-03-291-8/+26
|\ | | | | Deprecate metadata_broker_list in favor of bootstrap_servers
| * Deprecate KafkaConsumer config metadata_broker_list in favor of ↵Dana Powers2015-03-231-8/+26
| | | | | | | | bootstrap_servers
* | Merge pull request #336 from scrapinghub/feature-mp-consumer-paramsDana Powers2015-03-292-35/+51
|\ \ | | | | | | Using additional params for MP consumer child process
| * | Manageable queue.put() operation for MPConsumer processesViktor Shlapakov2015-03-252-4/+12
| | |
| * | Using mp.manager to solve the issue with join for MPConsumerViktor Shlapakov2015-03-241-6/+7
| | |
View Lorry Controller Status