summaryrefslogtreecommitdiff
path: root/test
Commit message (Collapse)AuthorAgeFilesLines
...
* | | Convert more tests to struct.packMark Roberts2014-04-081-29/+75
| | |
* | | Convert serveral tests to struct.packMark Roberts2014-04-081-81/+93
| | |
* | | Reinstate test_integrate, make test_protocol more explicit, create testutilMark Roberts2014-04-084-953/+974
| | |
* | | Explicit testing of protocol errors. Make tests more explicit, and start ↵Mark Roberts2014-04-081-13/+34
| | | | | | | | | | | | working on intermittent failures in test_encode_fetch_request and test_encode_produc_request
* | | Comment out all of test_integration because it currently does not workMark Roberts2014-04-081-936/+936
| | |
* | | Split test files, modify test_protocolMark Roberts2014-04-085-376/+396
| | |
* | | Remove test support for py26 since it's brokenMark Roberts2014-04-073-0/+115
|/ /
* | Merge pull request #109 from mrtheb/developDana Powers2014-03-211-19/+174
|\ \ | | | | | | TopicAndPartition fix when partition has no leader = -1
| * | Fix py26 compatibility issue, add mock to toxMark Roberts2014-03-181-17/+15
| | |
| * | Merge branch 'master' into developmrtheb2014-03-172-32/+153
| |\ \ | | |/ | | | | | | | | | Conflicts: test/test_unit.py
| * | fixing _get_leader_for_partition unittestsmrtheb2014-02-151-6/+6
| | |
| * | Changes based on comments by @rdiomar, plus added LeaderUnavailableError for ↵mrtheb2014-02-151-4/+4
| | | | | | | | | | | | clarity
| * | check for broker None in send_broker_aware_request (added test for it)mrtheb2014-01-311-0/+28
| | |
| * | Merge branch 'master' into developmrtheb2014-01-311-41/+41
| |\ \
| * | | Handle cases for partition with leader=-1 (not defined)Marc Labbe2014-01-311-48/+65
| | | |
| * | | removed commented out linesmrtheb2014-01-191-3/+0
| | | |
| * | | added mockmrtheb2014-01-181-6/+129
| | | |
* | | | Check against basestring instead of str in collect.hosts.Saulius Zemaitaitis2014-03-171-0/+10
| |_|/ |/| |
* | | Merge pull request #122 from mrtheb/multihostsOmar2014-02-262-16/+102
|\ \ \ | | | | | | | | Support for multiple hosts on KafkaClient boostrap (improves on #70)
| * | | clean up after comments from @rdiomarmrtheb2014-02-151-130/+6
| | | |
| * | | Support list (or comma-separated) of hosts (replaces host and port arguments)mrtheb2014-02-092-21/+45
| | | |
| * | | Merge branch 'master' into multihostsmrtheb2014-01-313-308/+529
| |\ \ \ | | | |/ | | |/| | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: kafka/client.py kafka/conn.py setup.py test/test_integration.py test/test_unit.py
| * | | Allow KafkaClient to take in a list of brokers for bootstrappingMarc Labbe2013-11-142-10/+200
| | | |
* | | | Make it possible to read and write xerial snappyGreg Bowyer2014-02-191-0/+43
| |/ / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes mumrah/kafka-python#126 TL;DR ===== This makes it possible to read and write snappy compressed streams that are compatible with the java and scala kafka clients (the xerial blocking format)) Xerial Details ============== Kafka supports transparent compression of data (both in transit and at rest) of messages, one of the allowable compression algorithms is Google's snappy, an algorithm which has excellent performance at the cost of efficiency. The specific implementation of snappy used in kafka is the xerial-snappy implementation, this is a readily available java library for snappy. As part of this implementation, there is a specialised blocking format that is somewhat none standard in the snappy world. Xerial Format ------------- The blocking mode of the xerial snappy library is fairly simple, using a magic header to identify itself and then a size + block scheme, unless otherwise noted all items in xerials blocking format are assumed to be big-endian. A block size (```xerial_blocksize``` in implementation) controls how frequent the blocking occurs 32k is the default in the xerial library, this blocking controls the size of the uncompressed chunks that will be fed to snappy to be compressed. The format winds up being | Header | Block1 len | Block1 data | Blockn len | Blockn data | | ----------- | ---------- | ------------ | ---------- | ------------ | | 16 bytes | BE int32 | snappy bytes | BE int32 | snappy bytes | It is important to not that the blocksize is the amount of uncompressed data presented to snappy at each block, whereas the blocklen is the number of bytes that will be present in the stream, that is the length will always be <= blocksize. Xerial blocking header ---------------------- Marker | Magic String | Null / Pad | Version | Compat ------ | ------------ | ---------- | -------- | -------- byte | c-string | byte | int32 | int32 ------ | ------------ | ---------- | -------- | -------- -126 | 'SNAPPY' | \0 | variable | variable The pad appears to be to ensure that SNAPPY is a valid cstring, and to align the header on a word boundary. The version is the version of this format as written by xerial, in the wild this is currently 1 as such we only support v1. Compat is there to claim the minimum supported version that can read a xerial block stream, presently in the wild this is 1. Implementation specific details =============================== The implementation presented here follows the Xerial implementation as of its v1 blocking format, no attempts are made to check for future versions. Since none-xerial aware clients might have persisted snappy compressed messages to kafka brokers we allow clients to turn on xerial compatibility for message sending, and perform header sniffing to detect xerial vs plain snappy payloads.
* | | Make producers take a topic argument at send rather than init timeOmar Ghishan2014-01-231-41/+41
| |/ |/| | | | | This allows a single producer to be used to send to multiple topics. See https://github.com/mumrah/kafka-python/issues/110
* | Make integration test warming more intuitiveThomas Dimson2014-01-141-91/+86
| |
* | Add a few 9s test_multi_process_consumer, make clear what setUp is doingThomas Dimson2014-01-131-13/+17
| |
* | Exception hierarchy, invalidate more md on errorsThomas Dimson2014-01-131-27/+45
| |
* | Some cleanup and easier to read test fixture outputOmar Ghishan2014-01-072-5/+5
| |
* | Skip snappy/gzip tests if they're not availableOmar Ghishan2014-01-071-4/+7
| |
* | Add object type and ID to message prefix in fixtures output for easier debuggingOmar Ghishan2014-01-061-15/+21
| |
* | Make kafka brokers per-test in failover integration testsOmar Ghishan2014-01-061-13/+11
| | | | | | | | | | This is better since the tests stop/start brokers, and if something goes wrong they can affect eachother.
* | Add tests for limited and unlimited consumer max_buffer_sizeOmar Ghishan2014-01-061-1/+25
| |
* | Fix client and consumer params in integration testsOmar Ghishan2014-01-061-6/+12
| |
* | Handle starting/stopping Kafka brokers that are already started/stopped in ↵Omar Ghishan2014-01-061-0/+11
| | | | | | | | | | | | | | integration tests If some of the tests stop brokers then error out, the teardown method will try to close the same brokers and fail. This change allows it to continue.
* | Style fix for importsOmar Ghishan2014-01-061-6/+2
| |
* | Fix unit tests.Omar Ghishan2014-01-061-150/+314
|/ | | | | | This is pretty much a rewrite. The tests that involve offset requests/responses are not implemented since that API is not supported in kafka 0.8 yet. Only kafka.codec and kafka.protocol are currently tested, so there is more work to be done here.
* Set host and port in ExternalService test fixtureDavid Arthur2013-11-111-0/+2
| | | | | So we can run tests against an already running Kafka instance KAFKA_URI=tcp://localhost:9092 python -m test.test_integration
* make changes to be more fault tolerant: clean up connections, brokers, ↵Jim Lim2013-10-043-11/+124
| | | | | | | | failed_messages - add integration tests for sync producer - add integration tests for async producer w. leadership election - use log.exception
* Test fixes after flake8 runmrtheb2013-10-031-2/+2
|
* Revert "Disable unit tests for 2.6, close #57"David Arthur2013-10-031-5/+0
| | | | This reverts commit e39e05f8a50b7528a22fed99dc67d561cbd79c41.
* Disable unit tests for 2.6, close #57David Arthur2013-10-031-0/+5
|
* Cherry-pick mrtheb/kafka-python 8b9c7e51mrtheb2013-10-033-27/+30
| | | | | | | | | Sync tests and fixtures with kafka 0.8.0-beta1 tag Conflicts: README.md kafka-src
* Auto-adjusting consumer fetch sizeDavid Arthur2013-09-091-12/+6
| | | | | | | | | | | | | | | Related to #42 Adds new ConsumerFetchSizeTooSmall exception that is thrown when `_decode_message_set_iter` gets a BufferUnderflowError but has not yet yielded a message In this event, SimpleConsumer will increase the fetch size by 1.5 and continue the fetching loop while _not_ increasing the offset (basically just retries the request with a larger fetch size) Once the consumer fetch size has been increased, it will remain increased while SimpleConsumer fetches from that partition
* Fixed #42, make fetch size configurableDavid Arthur2013-09-081-0/+34
| | | | | | | | | | Was hard coded to 1024 bytes which meant that larger messages were unconsumable since they would always get split causing the consumer to stop. It would probably be best to automatically retry truncated messages with a larger request size so you don't have to know your max message size ahead of time
* Merge branch 'issue-35'David Arthur2013-07-261-9/+120
|\ | | | | | | | | | | | | Conflicts: kafka/__init__.py kafka/consumer.py test/test_integration.py
| * Test cases for multi-process consumer and blocking APIsMahendra M2013-06-271-9/+120
| |
* | Test cases for new producerMahendra M2013-06-271-3/+274
|/
* Use 127.0.0.1 instead of localhostIvan Pouzyrevsky2013-06-071-1/+1
| | | | This alleviates IPv4 -vs- IPv6 issues in ZK and Kafka.
* toxify all the tests and use xfail marksIvan Pouzyrevsky2013-06-072-28/+69
|
View Lorry Controller Status