summaryrefslogtreecommitdiff
path: root/kafka/producer.py
Commit message (Collapse)AuthorAgeFilesLines
* Separate consumers/producers/partitionersDana Powers2014-09-101-319/+0
|
* Merge pull request #223 from dpkp/metadata_refactorDana Powers2014-09-081-10/+9
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Metadata Refactor * add MetadataRequest and MetadataResponse namedtuples * add TopicMetadata namedtuple * add error codes to Topic and Partition Metadata * add KafkaClient.send_metadata_request() method * KafkaProtocol.decode_metadata_response changed to return a MetadataResponse object so that it is consistent with server api: [broker_list, topic_list] * raise server exceptions in load_metadata_for_topics(*topics) unless topics is null (full refresh) * Replace non-standard exceptions (LeaderUnavailable, PartitionUnavailable) with server standard exceptions (LeaderNotAvailableError, UnknownTopicOrPartitionError) Conflicts: kafka/client.py test/test_client.py test/test_producer_integration.py test/test_protocol.py
| * Refactor internal metadata dicts in KafkaClientDana Powers2014-09-011-10/+9
| | | | | | | | | | | | | | | | - use helper methods not direct access - add get_partition_ids_for_topic - check for topic and partition errors during load_metadata_for_topics - raise LeaderNotAvailableError when topic is being auto-created or UnknownTopicOrPartitionError if auto-creation off
* | Merge pull request #227 from wizzat-feature/py3Dana Powers2014-09-071-5/+11
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Python 3 Support Conflicts: kafka/producer.py test/test_client.py test/test_client_integration.py test/test_codec.py test/test_consumer.py test/test_consumer_integration.py test/test_failover_integration.py test/test_producer.py test/test_producer_integration.py test/test_protocol.py test/test_util.py
| * | Bytes in self.msg()Bruno Renié2014-09-031-1/+1
| | |
| * | Use built-in next()Bruno Renié2014-09-031-2/+2
| | |
| * | Make all unit tests pass on py3.3/3.4Bruno Renié2014-09-031-4/+10
| |/
* | Change message type requirement from str to bytes for clarity and prep for ↵Dana Powers2014-09-041-5/+5
|/ | | | python3
* Add warnings to README, docstring, and logging that async producer does not ↵Dana Powers2014-08-261-0/+6
| | | | retry failed messages
* Raise TypeError in kafka.producer.send_messages if any msg is not a str (or ↵Dana Powers2014-08-261-0/+21
| | | | subclass); document
* Handle New Topic CreationMark Roberts2014-05-221-2/+5
| | | | | Adds ensure_topic_exists to KafkaClient, redirects test case to use that. Fixes #113 and fixes #150.
* Improve error handling and tests w.r.t. codecsPatrick Lucas2014-05-071-20/+9
| | | | | | | | | | Add function kafka.protocol.create_message_set() that takes a list of payloads and a codec and returns a message set with the desired encoding. Introduce kafka.common.UnsupportedCodecError, raised if an unknown codec is specified. Include a test for the new function.
* Merge branch 'teach_producers_about_compression' into producer_compressionMark Roberts2014-05-071-9/+33
|\ | | | | | | | | | | Conflicts: servers/0.8.0/kafka-src test/test_unit.py
| * Add 'codec' parameter to ProducerPatrick Lucas2014-05-031-9/+33
| | | | | | | | | | Adds a codec parameter to Producer.__init__ that lets the user choose a compression codec to use for all messages sent by it.
* | added random_start param to SimpleProducer to enable/disable randomization ↵Alex Couture-Beil2014-04-011-4/+11
| | | | | | | | of the initial partition messages are published to
* | Changed randomization to simply randomize the initial starting partition of ↵Alex Couture-Beil2014-04-011-3/+7
| | | | | | | | the sorted list of partition rather than completely randomizing the initial ordering before round-robin cycling the partitions
* | Modified SimpleProducer to randomize the initial round robin orderingAlex Couture-Beil2014-03-111-1/+4
|/ | | | | of partitions to prevent the first message from always being published to partition 0.
* Use TopicAndPartition when producing async messagesOmar Ghishan2014-01-271-8/+11
|
* Make producers take a topic argument at send rather than init timeOmar Ghishan2014-01-231-34/+40
| | | | This allows a single producer to be used to send to multiple topics. See https://github.com/mumrah/kafka-python/issues/110
* Merge branch 'repr' of https://github.com/mahendra/kafka-python into ↵mrtheb2014-01-141-0/+6
|\ | | | | | | | | | | | | | | mahendra-repr Conflicts: kafka/client.py kafka/consumer.py
| * Add proper string representations for each classMahendra M2013-10-081-0/+6
| |
* | Exception hierarchy, invalidate more md on errorsThomas Dimson2014-01-131-8/+7
| |
* | Merge pull request #88 from rdiomar/rdiomar_changesOmar2014-01-131-3/+3
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Various changes/fixes, including: * Allow customizing socket timeouts * Read the correct number of bytes from kafka * Guarantee reading the expected number of bytes from the socket every time * Remove bufsize from client and conn * SimpleConsumer flow changes * Fix some error handling * Add optional upper limit to consumer fetch buffer size * Add and fix unit and integration tests
| * | Change log.error() back to log.exception()Omar Ghishan2014-01-081-5/+5
| | |
| * | Raise a ConnectionError when a socket.error is raised when receiving dataOmar Ghishan2014-01-061-3/+3
| | | | | | | | | | | | Also, log.exception() is unhelpfully noisy. Use log.error() with some error details in the message instead.
* | | Merge pull request #66 from jcrobak/fix-import-collisionDavid Arthur2014-01-081-0/+2
|\ \ \ | |/ / |/| | Enable absolute imports for modules using Queue.
| * | Enable absolute imports for modules using Queue.Joe Crobak2013-10-211-0/+2
| |/ | | | | | | | | | | | | When running on Linux with code on a case-insensitive file system, imports of the `Queue` module fail because python resolves the wrong file (It is trying to use a relative import of `queue.py` in the kafka directory). This change forces absolute imports via PEP328.
* | Replaced _send_upstream datetime logic with simpler time().Niek Sanders2013-12-251-3/+3
|/
* Merge branch 'master' into prod-windowsMahendra M2013-10-081-3/+8
|\ | | | | | | | | Conflicts: kafka/producer.py
| * make changes to be more fault tolerant: clean up connections, brokers, ↵Jim Lim2013-10-041-3/+8
| | | | | | | | | | | | | | | | failed_messages - add integration tests for sync producer - add integration tests for async producer w. leadership election - use log.exception
* | Ensure that async producer works in windows. Fixes #46Mahendra M2013-10-071-49/+63
|/ | | | | | | | | | | | | | | As per the multiprocessing module's documentation, the objects passed to the Process() class must be pickle-able in Windows. So, the Async producer did not work in windows. To fix this we have to ensure that code which uses multiprocessing has to follow certain rules * The target=func should not be a member function * We cannot pass objects like socket() to multiprocessing This ticket fixes these issues. For KafkaClient and KafkaConnection objects, we make copies of the object and reinit() them inside the child processes.
* remove unused exception instance variablesVetoshkin Nikita2013-10-031-1/+1
|
* Test cases for new producerMahendra M2013-06-271-1/+1
|
* Optimize sending of batch messagesMahendra M2013-06-261-6/+9
|
* Got batched mode to work properlyMahendra M2013-06-261-59/+48
|
* Update README with examplesMahendra M2013-06-261-1/+1
|
* Add support for batched message sendMahendra M2013-06-261-9/+100
| | | | | Also improve on the logic for stopping the async Processor instance. Ensure that unsend messages are sent before it is stopped.
* Make the default case as 'ack on local write'Mahendra M2013-06-201-3/+3
| | | | | | | Also, ensure that the case of 'no-acks' works fine In conn.send(), do not wait for the response. Wait for it only on conn.recv(). This behaviour is fine now since the connection is not shared among consumer threads etc.
* Fix import errorsMahendra M2013-06-201-3/+5
|
* Add ack support for synchronous producerMahendra M2013-06-201-10/+39
| | | | | Add support for two options in the producer - req_acks and ack_timeout The acks, if any, are passed to the caller directly
* Merge branch 'master' into asyncproducerMahendra M2013-06-131-0/+34
|\ | | | | | | | | Conflicts: kafka/producer.py
| * Fix bugs and testingMahendra M2013-06-131-2/+2
| | | | | | | | | | | | * Ensure that round-robin partitioner works fine * _load_metadata_for_topics() would cause duplicate and stale entries in self.topic_partitions. Fix this
| * Implement support for Keyed producerMahendra M2013-06-131-0/+32
| | | | | | | | | | | | Provides support for two partitioners * Round robin * Hashed (default as per kafka clients)
* | Support for async producerMahendra M2013-06-131-8/+57
|/ | | | | | | | | | The Java/Scala Kafka client supports a mechanism for sending messages asynchronously by using a queue and a thread. Messages are put on the queue and the worker thread keeps sending it to the broker. This ticket implements this feature in python We use multiprocessing instead of threads to send the messages
* PEP8-ify most of the filesMahendra M2013-05-291-1/+3
| | | | consumer.py and conn.py will be done later after pending merges
* Bring acks and timeout down to the clientDavid Arthur2013-04-021-2/+2
|
* Refactoring a bit, cleanup for 0.8David Arthur2013-04-021-2/+3
| | | | Marking some stuff as not compatible for 0.8 (will be added in 0.8.1)
* Big code re-orgDavid Arthur2013-04-021-0/+22
View Lorry Controller Status