summaryrefslogtreecommitdiff
path: root/taskflow/engines/worker_based/proxy.py
Commit message (Collapse)AuthorAgeFilesLines
* Remove sixTakashi Kajinami2022-05-181-3/+2
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* python3.0 has deprecated LOG.warnxhzhf2017-03-071-3/+3
| | | | | | | | python3.0 has deprecated LOG.warn https://docs.python.org/3/library/logging.html#logging.warning Closes-Bug: #1650843 Change-Id: Ib8473cbdb84547f385e619fe08ed723b1ee7bcdb
* Remove 2.6 classifier + 2.6 compatibility codeJoshua Harlow2015-06-211-2/+2
| | | | | | | | Fixes bug 1445827 Depends-On: I02e3c9aacef0b295a2f823a5cbaf11768a90cb82 Change-Id: I1db681803598ac1bc917fd74a99458bc61edf3f1
* Abstract out the worker finding from the WBE engine0.7.0Joshua Harlow2015-01-311-7/+12
| | | | | | | | | | | | | | To be able to easily plug-in future types of ways to get which topics (and tasks) workers exist on (and can perform) and to identify and keep this information up-to date refactor the functionality that currently does this using periodic messages into a finder type and a periodic function that exists on it (that will be periodically activated by an updated and improved periodic worker). Part of blueprint wbe-worker-info Change-Id: Ib3ae29758af3d244b4ac4624ac380caf88b159fd
* Add and use a nicer kombu message formatterJoshua Harlow2015-01-291-1/+2
| | | | | | | | | | | Since the kombu message object that is recieved has no useful __str__ or __repr__ and on reception and processing we should the message (and the delivery tag) it is nice to have a more useful formatting of that message for debugging and such (where more detail about the messages are quite useful to see). Change-Id: I6730b10122a5de1a0a074525931f312fbd97b8c0
* Use kombu socket.timeout alias instead of socket.timeoutJoshua Harlow2015-01-291-2/+2
| | | | | | | | Use the kombu timeout error alias to socket.timeout to be more resilient against kombu ever changing this exception type to something different. Change-Id: I53c01db67742c65ee49264fd0d8d36417cec3ec1
* Merge "Link WBE docs together better (especially around arguments)"Jenkins2015-01-291-7/+8
|\
| * Link WBE docs together better (especially around arguments)Joshua Harlow2015-01-261-7/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Link the engine/worker 'retry_options' to the proxy which has linkage to what the defaults are and to where more information can be gathered about the option (which really gets sent in to kombu) so that users can more easily understand it. Also removes showing the docs about the following as its more of an internal API/class not meant for public consumption: * Module: taskflow.engines.worker_based.executor Change-Id: Ib16ee10765ec6b9a0af320bd6818d9649c2485b1
* | Merge "Emit a warning when no routing keys provided on publish()"Jenkins2015-01-291-0/+8
|\ \
| * | Emit a warning when no routing keys provided on publish()Joshua Harlow2015-01-261-0/+8
| |/ | | | | | | | | | | | | | | | | When no routing keys are provided the message will no be published anywhere; so when this happens emit a warning to tell the library user that something is likely misconfigured and/or wrong. Change-Id: Id27e8ab73c1ae8a02cb4252958d9dc896a5df1f0
* | Add docstring to wbe proxy to denote not for public useJoshua Harlow2015-01-281-1/+4
|/ | | | Change-Id: I9bd59ae5d58ef3ace960509903403252fbed6277
* Use explicit WBE object arguments (instead of kwargs)Joshua Harlow2015-01-221-14/+11
| | | | | | | | | | | | | | Instead of passing around kwargs to root WBE classes and to contained classes prefer to use explicitnamed arguments that are passed around. This makes the code more obvious as to what the intended arguments are and makes it easier for error validation when other unknown arguments are passed (as well as for docs). Also moves the docs for the worker engine to be a sub-TOC under the main engine document so that it can be more easily explored and managed/found... Change-Id: I9413fad187c330fee494f0d4536cc27d9a90f0fb
* Increase robustness of WBE producer/consumersJoshua Harlow2015-01-201-19/+84
| | | | | | | | | Use the kombu provided ensure() decorator/wrapper along with sensible default settings to ensure that retries are attempted when kombu detects recoverable connection or recoverable channel errors have occurred. Change-Id: If47f72d02561d0b5d556ac386869a6122c8b871d
* Add and use a new simple helper logging moduleJoshua Harlow2014-12-081-1/+1
| | | | | | | | | Add a new logging BLATHER level to easily allow its usage for messages that are below the normal DEBUG level such as compilation information and scope lookup info which can be very verbose in logs if always enabled. Change-Id: I828211403bd02bfd6777b10cdcfe58fb0637a52c
* Be explicit about publish keyword argumentsJoshua Harlow2014-12-041-11/+11
| | | | | | | | | Instead of allowing for arbitrary keyword arguments which makes the API hard to change in the future prefer to have explicit arguments (and keyword arguments) instead of allowing **kwargs to be passed. Change-Id: I374db6b19ef76c2f9ee04771f5d928c79b7cf049
* Remove attrdict and just use existing typesJoshua Harlow2014-10-181-7/+22
| | | | | | | | | | | | In order to make it simpler (and less code) just prefer to use object types that already exist instead of trying to make dictionaries also behave like objects. For those that really need this kind of functionality: https://pypi.python.org/pypi/attrdict Change-Id: Ib7ddfa517f0500082fafac2c3e53fd6a158a6ddf
* Use and verify event and latch wait() return using timeoutsJoshua Harlow2014-10-181-2/+2
| | | | | | | | | | | | | Instead of blocking up the whole test suite when a latch or event was not decremented to its desired value (or not set for an event) we should use a reasonably high value that we use when waiting for those actions to occur and verify that when those wait() functions return that we have reached the desired state and if not either raise an exception or stop further testing. Fixes bug 1363739 Change-Id: I8b40282ac2db9cabd48b0b65c8a2a49610d77c4f
* Increase/adjust the logging of the WBE response/send activitiesJoshua Harlow2014-09-221-1/+1
| | | | Change-Id: I1d8309ce87114a0890dfc93a0a2c4b68f80ef828
* Make the WBE worker banner information more meaningfulJoshua Harlow2014-08-211-9/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add in more details that are displayed in the LOG when a WBE worker is started up that is useful to show to be able to help in debugging, or other informational and operational purposes. Example of the new output is the following: TaskFlow v0.3.21.62 WBE worker. Connection details: Driver = py-amqp v1.4.5 Exchange = test Topic = test Transport = amqp Uri = amqp://guest@localhost:5672// Powered by: Executor = concurrent.futures.thread.ThreadPoolExecutor Thread count = 3 Supported endpoints: - taskflow.tests.utils.NastyTask ... - taskflow.tests.utils.TaskMultiArgOneReturn System details: Hostname = lappy.gateway.net Pid = 28364 Platform = Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty Python = 2.7.6 (default, Mar 22 2014, 22:59:56) Thread id = 139875992315712 Change-Id: I6d7dba3406007ddc80cce96cfdbbfd25935a12ae
* Use a common message dispatcherJoshua Harlow2014-07-251-4/+10
| | | | | | | | | | | | | | | | Instead of recreating a dispatcher in the server and executor objects use a common dispatcher that is shared between them. It will dispatch based on the message type received into a provided dict of dispatch handler callbacks. It also can generically requeue messages and can reject messages if they are missing key required message properties ('type' in the current case). Part of blueprint wbe-message-validation Change-Id: I8320f4707183f36e6a69f0552cf62f99a5467b7e
* Enabled hacking checks H305 and H307Christian Berendt2014-07-171-1/+1
| | | | | | | * H305 imports not grouped correctly * H307 like imports should be grouped together Change-Id: If1dd9c89f65ede6959865a885777cb08c263eca0
* Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-5/+3
| | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Run worker-based engine tests fasterIvan A. Melnikov2014-03-261-1/+7
| | | | | | | | | This change reduces polling intervals for transport and proxy from default 1 second to 0.01 second. On my machine this reduces time needed to run WorkerBasedEngineTest almost 5 times, which, in turn, makes testr almost twice faster. Change-Id: I8dfe08c06234f33e838059b2f8d9d6a7c7819e06
* Improve logging for proxy publishStanislav Kudriashev2014-03-201-0/+1
| | | | | | Added message type logging during proxy publishing. Change-Id: Iddbfab0e4dd6e9f5b6d1c3ea5c86029700a2d66d
* Improve proxy publish methodStanislav Kudriashev2014-03-201-7/+14
| | | | | | | Added possibility to send a message to multiple routing keys without having to create new connection for each send. Change-Id: Ie23cfd130996b693c091e164043d80f35f04d694
* Introduce message types for WBE protocolStanislav Kudriashev2014-03-141-1/+2
| | | | | | | | | | | | * Abstract message class added - all messages types are derived from it now and have to implement the `to_dict` method, so it can be serialized and transferred with broker; * Implemented the `Response` message type, that restores failures from dictionary on creation; * Corrected and improved unit tests; Change-Id: I10e017a613f0422420d0244b9f8786f988863107
* Rework proxy publish functionalityStanislav Kudriashev2014-03-121-3/+1
| | | | | | | * Task uuid is not a required parameter now; * Exchange declaration does not require a connection object. Change-Id: I9a77c9afe7d9fad007902d77d24a0b1a96c6c519
* Worker-based engine clean-upsStanislav Kudriashev2014-03-071-32/+11
| | | | | | | | * Auto-delete queues, redundant proxy logic removed; * Updated unit tests; * Small fixes in comments. Change-Id: I8d6fdbf06625daad7a886ff87607b31f369062b8
* A few worker-engine cleanupsJoshua Harlow2014-03-061-7/+7
| | | | | | | | | | | | | | | - Remove usage of % with logs, just let the underlying LOG do the interpolation for us if the log level passes. - Use LOG.exception instead of LOG.warn and then including the exception as a interpolated variable. Just let the LOG.exception output the exception trace and info. - Capture the failure object before doing other work using a new context manager that can yield back a failure object before further code is called (to avoid a context switch happening in eventlet or elsewhere that would clear the exception state). Change-Id: Ic1a504ba69c56ba226b1a067b5ade4590c245dd8
* Rename uuid to topicJoshua Harlow2014-02-251-3/+3
| | | | | | This appears to be the topic and not a uuid. Change-Id: I191fcef543aa0879913fd965ebef0496b0163336
* Remove extraneous vim configuration commentsStanislav Kudriashev2014-02-211-2/+0
| | | | Change-Id: I06eada275398adc3523c02a99d9e4650e2dd1637
* Message-oriented worker-based flow with kombuStanislav Kudriashev2014-02-191-0/+126
* Implemented Worker to be started on remote host for handling tasks request. * Implemented WorkerTaskExecutor that proxies tasks requests to remote workers. * Implemented Proxy that is used for consuming and publishing messages by Worker and Executor. * Added worker-based engine and worker task executor. * Added kombu dependency to requirements. * Added worker-based flow example. * Added unit-tests for worker-based flow components. Implements: blueprint worker-based-engine Change-Id: I8c6859ba4a1a56c2592e3d67cdfb8968b13ee99c
View Lorry Controller Status