summaryrefslogtreecommitdiff
path: root/taskflow/engines/worker_based/worker.py
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Spice up WBE banner and add simple worker __main__ entrypoint"Jenkins2016-04-281-71/+49
|\
| * Spice up WBE banner and add simple worker __main__ entrypointJoshua Harlow2016-02-081-71/+49
| | | | | | | | Change-Id: Ifdf275a623352aa3e42fbf0aa9a4394b64b54337
* | Refactor Atom/BaseTask/Task/Retry class hierarchyGreg Hill2016-02-181-1/+1
|/ | | | | | | | * Moved common argument mapping logic to Atom * Removed BaseTask * Removed duplicated logic from subclasses Change-Id: I275745adb80cecb0c35c1230eac76436bf3b0157
* Integrate futurist (and **remove** taskflow originating code)Joshua Harlow2015-07-091-2/+2
| | | | Change-Id: If89baa042695f19e42b6368034f3ccf22c2cf0aa
* Just let the future executors handle the max workersJoshua Harlow2015-03-181-8/+5
| | | | | | | | | | | | Instead of providing and retaining a thread count in the worker and action engine executors and checking it and handling the none case, we can just let the future types handle this already (which they already do). And when displaying this information in the worker banner use a new future executor attribute that is the maximum number of workers that will be ever created. Change-Id: I765c22936b53cdbb8a90195a764d4c67bcc3f34b
* Make the worker banner template part of the worker classJoshua Harlow2015-03-031-33/+38
| | | | | | | | | Instead of having the banner template be part of the worker module, have it be part of the worker class so that anyone that desires to subclass and create new workers can alter the banner template more easily (if they so choose). Change-Id: Id56815ffd726f8952a74f9ab866b1bd006dcfcad
* Link WBE docs together better (especially around arguments)Joshua Harlow2015-01-261-3/+5
| | | | | | | | | | | | | | Link the engine/worker 'retry_options' to the proxy which has linkage to what the defaults are and to where more information can be gathered about the option (which really gets sent in to kombu) so that users can more easily understand it. Also removes showing the docs about the following as its more of an internal API/class not meant for public consumption: * Module: taskflow.engines.worker_based.executor Change-Id: Ib16ee10765ec6b9a0af320bd6818d9649c2485b1
* Use explicit WBE worker object arguments (instead of kwargs)Joshua Harlow2015-01-231-8/+14
| | | | | | | | Removes the kwargs usage that is now uniform across the other WBE components from the workers module so that the usage of kwargs for setting up these objects no longer is valid. Change-Id: I4e25b88c5d2f7e2d7933ff270e2782cebe227025
* Increase robustness of WBE producer/consumersJoshua Harlow2015-01-201-0/+2
| | | | | | | | | Use the kombu provided ensure() decorator/wrapper along with sensible default settings to ensure that retries are attempted when kombu detects recoverable connection or recoverable channel errors have occurred. Change-Id: If47f72d02561d0b5d556ac386869a6122c8b871d
* Switch to using 'oslo_utils' vs 'oslo.utils'Joshua Harlow2015-01-141-1/+1
| | | | | | | | Prefer the non-deprecated 'oslo_utils' instead of the namespaced 'oslo.utils' wherever it was previously used. Change-Id: I9a78150ef5266e1ff22147278162fe3cfe1b2e3f
* Use platform neutral line separator(s)Joshua Harlow2015-01-121-2/+7
| | | | | | | | To at least try to support things like windows it's better if we can make an attempt to use the platform neutral characters for line separator(s) where appropriate. Change-Id: Icc533ed4d4c94f461b7f19600b74146221f17b18
* Move over to using oslo.utils [reflection, uuidutils]Joshua Harlow2014-12-181-2/+3
| | | | | | | | | | | | | | The reflection module is now part of oslo.utils so we should remove our local version and use that version instead; this also goes for the uuidutils module which is now part of oslo.utils as well so we no longer need our local version copied from the incubator... Note that one reflection method `find_subclasses` which was to specific to taskflow is now moved to the misc utility module instead of its prior home in the reflection module. Change-Id: I069881c80b0b2916cc0c414992b80171f7eeb79f
* Add and use a new simple helper logging moduleJoshua Harlow2014-12-081-1/+1
| | | | | | | | | Add a new logging BLATHER level to easily allow its usage for messages that are below the normal DEBUG level such as compilation information and scope lookup info which can be very verbose in logs if always enabled. Change-Id: I828211403bd02bfd6777b10cdcfe58fb0637a52c
* Allow the worker banner to be written to an arbitrary locationJoshua Harlow2014-10-081-3/+7
| | | | | | | | | | | Instead of always writing the worker startup banner to LOG.info make it so that a callback can be provided that will receive the worker banner and can write it to a desired location that the callback specifies (or drop it completely). This is useful for those who use workers but have an alternate location they desire the banner to go (for example some arbitrary stream). Change-Id: I61a4b7e9dd33ee06137caaed33153f5b3fbeb661
* Documentation cleanups and tweaksJoshua Harlow2014-09-231-28/+10
| | | | | | | | | Apply some adjustments on the docs by rewording certain statements, linking to the correct classes and methods and linking to the correct exceptions to make it easier for users to follow the docs and there associated types and links. Change-Id: I03aac77c814fc4c376003f09a45559a0995b3c6c
* Make the WBE worker banner information more meaningfulJoshua Harlow2014-08-211-10/+78
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add in more details that are displayed in the LOG when a WBE worker is started up that is useful to show to be able to help in debugging, or other informational and operational purposes. Example of the new output is the following: TaskFlow v0.3.21.62 WBE worker. Connection details: Driver = py-amqp v1.4.5 Exchange = test Topic = test Transport = amqp Uri = amqp://guest@localhost:5672// Powered by: Executor = concurrent.futures.thread.ThreadPoolExecutor Thread count = 3 Supported endpoints: - taskflow.tests.utils.NastyTask ... - taskflow.tests.utils.TaskMultiArgOneReturn System details: Hostname = lappy.gateway.net Pid = 28364 Platform = Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty Python = 2.7.6 (default, Mar 22 2014, 22:59:56) Thread id = 139875992315712 Change-Id: I6d7dba3406007ddc80cce96cfdbbfd25935a12ae
* Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-2/+2
| | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Merge "Avoid shutting down of a passed executor"Jenkins2014-03-241-0/+4
|\
| * Avoid shutting down of a passed executorJoshua Harlow2014-03-201-0/+4
| | | | | | | | | | | | | | | | | | | | For the case where an executor is being provided we should not shutdown the executor on-behalf of the provider since they would likely expect that they can still use the executor elsewhere. If the executor is shutdown then further usage is not possible. Change-Id: Ia83d8c2f1df06200e32e9b0c7340f32e74fd5d85
* | Merge "Move endpoint subclass finding to reflection util"Jenkins2014-03-131-33/+1
|\ \
| * | Move endpoint subclass finding to reflection utilJoshua Harlow2014-03-071-33/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | The endpoint finding uses a utility method that can be generalized into finding subclasses of a given list of items, so instead of having this logic be in the worker just move it to reflection instead. Change-Id: Id79efd9514f7b2f7713a1426a7e6ed0861ecee70
* | | Worker-based engine clean-upsStanislav Kudriashev2014-03-071-2/+2
| |/ |/| | | | | | | | | | | | | * Auto-delete queues, redundant proxy logic removed; * Updated unit tests; * Small fixes in comments. Change-Id: I8d6fdbf06625daad7a886ff87607b31f369062b8
* | Fixups for threads_count usage and loggingJoshua Harlow2014-02-251-5/+14
| | | | | | | | | | | | | | | | | | | | In the cases where threads_count is not applicable (when a custom executor is provided) we should not log a message that says the number of threads being used (since it will likely not be an accurate message of what the provided executor is really using). Change-Id: I092c69716f300b3ac95cc320c96243966eb982ba
* | Remove extraneous vim configuration commentsStanislav Kudriashev2014-02-211-2/+0
|/ | | | Change-Id: I06eada275398adc3523c02a99d9e4650e2dd1637
* Message-oriented worker-based flow with kombuStanislav Kudriashev2014-02-191-0/+132
* Implemented Worker to be started on remote host for handling tasks request. * Implemented WorkerTaskExecutor that proxies tasks requests to remote workers. * Implemented Proxy that is used for consuming and publishing messages by Worker and Executor. * Added worker-based engine and worker task executor. * Added kombu dependency to requirements. * Added worker-based flow example. * Added unit-tests for worker-based flow components. Implements: blueprint worker-based-engine Change-Id: I8c6859ba4a1a56c2592e3d67cdfb8968b13ee99c
View Lorry Controller Status