summaryrefslogtreecommitdiff
path: root/taskflow/engines/worker_based/server.py
Commit message (Collapse)AuthorAgeFilesLines
* Use LOG.warning instead of deprecated LOG.warnTakashi Kajinami2021-11-291-2/+2
| | | | | | | | | The LOG.warn method is deprecated[1] and the LOG.warning method should be used instead. [1] https://docs.python.org/3/library/logging.html#logging.warning Change-Id: I4321a489c56eb1aa650e776ee35d8f88d4d8910c
* Merge "python3.0 has deprecated LOG.warn"Jenkins2017-05-031-19/+21
|\
| * python3.0 has deprecated LOG.warnxhzhf2017-03-071-19/+21
| | | | | | | | | | | | | | | | python3.0 has deprecated LOG.warn https://docs.python.org/3/library/logging.html#logging.warning Closes-Bug: #1650843 Change-Id: Ib8473cbdb84547f385e619fe08ed723b1ee7bcdb
* | Fix py35 test failureChangBo Guo(gcb)2017-05-031-1/+1
|/ | | | | | | | dict.keys() returns object dict_keys on Python 3, can't work well with kombu recently, convert it to list before futhur processing. Closes-Bug: #1687855 Change-Id: I9dcfadd8f16eccce06f4cde3b2a6669f28a3f8bc
* Handle cases where exc_args can't be serialized as JSON in the WBEGreg Hill2016-01-281-5/+5
| | | | | | | | | First try to get them, but if they aren't able to be serialized, just return the failure without them. This only affects the WBE that serializes failures as part of the JSON response that is sent over the wire from the worker process to the WBE engine. Change-Id: I86e3255b612bc15097aabe63a684cf8b8808e61b
* Move all internal blather usage/calls to trace usage/callsJoshua Harlow2015-12-171-4/+4
| | | | Change-Id: I415a81d3b6b15b17a9a91cc2a0681c159172a4e1
* Provide a deprecated alias for the now removed stop watch classJoshua Harlow2015-07-121-2/+2
| | | | | | | | | | | | | This allows those who were using it to still continue using it until 2.0 where it will be removed; this makes it possible for those users to get off that code in a way that will be easily do-able (without totally breaking there code-bases, until we do that in the 2.0 release). This also removes all internal usage of that stop watch class so that the library itself will not reference it anymore. Change-Id: If313d8e7b9bdc8741db2e2e1dfb381aa3260b971
* Use a collections.namedtuple for the request work unitJoshua Harlow2015-03-131-15/+16
| | | | | | | | | | Instead of returning a tuple with hard to understand and read contents a namedtuple can help understand what the tuple is composed of and how it can be used/what it is, so we should prefer to use it when we can for the wbe requested work to perform. Change-Id: I8147814449d04ba9a03730547ac113e4ef7b272a
* Add + use failure json schema validationJoshua Harlow2015-03-111-29/+1
| | | | Change-Id: Ie3aa386c831459a028ba494570bafd53b998126e
* Make the dispatcher handler be an actual typeJoshua Harlow2015-02-151-6/+6
| | | | | | | | | | | | | Instead of having the dispatcher target be a tuple or a single callback, have it be an actual type with comments as to what the types fields are and how they are used. This makes it more obvious as to what those fields are and how they are used so that it is easier to understand how the WBE engine and components work. Change-Id: I5541ccd5a7aa6ae73ed9adceeac2c0524e2a1dc9
* Remove delayed decorator and replace with nicer methodJoshua Harlow2015-02-131-20/+38
| | | | | | | | Instead of having a delayed decorator that is in the module have one that is attached onto the instance and that can log how long it took for the target method to finally be ran. Change-Id: Ie6e51b0a2a60a063554dfba7fcc321ecd6661949
* Abstract out the worker finding from the WBE engine0.7.0Joshua Harlow2015-01-311-1/+2
| | | | | | | | | | | | | | To be able to easily plug-in future types of ways to get which topics (and tasks) workers exist on (and can perform) and to identify and keep this information up-to date refactor the functionality that currently does this using periodic messages into a finder type and a periodic function that exists on it (that will be periodically activated by an updated and improved periodic worker). Part of blueprint wbe-worker-info Change-Id: Ib3ae29758af3d244b4ac4624ac380caf88b159fd
* Add and use a nicer kombu message formatterJoshua Harlow2015-01-291-18/+20
| | | | | | | | | | | Since the kombu message object that is recieved has no useful __str__ or __repr__ and on reception and processing we should the message (and the delivery tag) it is nice to have a more useful formatting of that message for debugging and such (where more detail about the messages are quite useful to see). Change-Id: I6730b10122a5de1a0a074525931f312fbd97b8c0
* Use explicit WBE object arguments (instead of kwargs)Joshua Harlow2015-01-221-4/+8
| | | | | | | | | | | | | | Instead of passing around kwargs to root WBE classes and to contained classes prefer to use explicitnamed arguments that are passed around. This makes the code more obvious as to what the intended arguments are and makes it easier for error validation when other unknown arguments are passed (as well as for docs). Also moves the docs for the worker engine to be a sub-TOC under the main engine document so that it can be more easily explored and managed/found... Change-Id: I9413fad187c330fee494f0d4536cc27d9a90f0fb
* Use the notifier type in the task class/module directlyJoshua Harlow2014-12-131-21/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having code that is some what like the notifier code we already have, but is duplicated and is slightly different in the task class just move the code that was in the task class (and doing similar actions) to instead now use a notifier that is directly contained in the task base class for internal task triggering of internal task events. Breaking change: alters the capabilities of the task to process notifications itself, most actions now must go through the task notifier property and instead use that (update_progress still exists as a common utility method, since it's likely the most common type of notification that will be used). Removes the following methods from task base class (as they are no longer needed with a notifier attribute): - trigger (replaced with notifier.notify) - autobind (removed, not replaced, can be created by the user of taskflow in a simple manner, without requiring functionality in taskflow) - bind (replaced with notifier.register) - unbind (replaced with notifier.unregister) - listeners_iter (replaced with notifier.listeners_iter) Due to this change we can now also correctly proxy back events from remote tasks to the engine for correct proxying back to the original task. Fixes bug 1370766 Change-Id: Ic9dfef516d72e6e32e71dda30a1cb3522c9e0be6
* Add and use a new simple helper logging moduleJoshua Harlow2014-12-081-1/+1
| | | | | | | | | Add a new logging BLATHER level to easily allow its usage for messages that are below the normal DEBUG level such as compilation information and scope lookup info which can be very verbose in logs if always enabled. Change-Id: I828211403bd02bfd6777b10cdcfe58fb0637a52c
* Remove direct usage of the deprecated failure locationJoshua Harlow2014-10-211-6/+7
| | | | | | | | Internally we should be using the new location and not the deprecated location wherever possible. This avoids emitting warnings messages on our own code, which is a dirty habit. Change-Id: Idac5a772eca7529d92542ada3be1cea092880e25
* Increase robustness of WBE message and request processingJoshua Harlow2014-09-271-15/+43
| | | | | | | | | | | | | | When a notification request/response can't be processed ensure we log an error message at the same level as the other function that sends back responses. Also adds in a return boolean from the _reply message (which is used for the X number of replies to a servers task request) function and use this boolean to determine if the worker should attempt to perform the final handler call that activates the desired task. Change-Id: I7f3914c126f39c56d0d2e3dfe02f3b112391ff43
* Adjust the WBE log levelsJoshua Harlow2014-09-271-13/+24
| | | | | | | | | | | | To conform better with the logging level standards move away from using LOG.exception when the level is more appropriately a warning/warn. Also changes how a message that can not be sent is really a critical error and should be treated as such (since such an error affects the overall execution model). Change-Id: I7cebd882b655958d539be36ce3b4deb75ac4a0b7
* Make the WBE worker banner information more meaningfulJoshua Harlow2014-08-211-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add in more details that are displayed in the LOG when a WBE worker is started up that is useful to show to be able to help in debugging, or other informational and operational purposes. Example of the new output is the following: TaskFlow v0.3.21.62 WBE worker. Connection details: Driver = py-amqp v1.4.5 Exchange = test Topic = test Transport = amqp Uri = amqp://guest@localhost:5672// Powered by: Executor = concurrent.futures.thread.ThreadPoolExecutor Thread count = 3 Supported endpoints: - taskflow.tests.utils.NastyTask ... - taskflow.tests.utils.TaskMultiArgOneReturn System details: Hostname = lappy.gateway.net Pid = 28364 Platform = Linux-3.13.0-30-generic-x86_64-with-Ubuntu-14.04-trusty Python = 2.7.6 (default, Mar 22 2014, 22:59:56) Thread id = 139875992315712 Change-Id: I6d7dba3406007ddc80cce96cfdbbfd25935a12ae
* WBE request message validationJoshua Harlow2014-07-251-1/+4
| | | | | | | | | | | Add receive validation of the request message sent by executors to workers to initiate work requests so that we can be more robust around invalid message formats being sent and received. Part of blueprint wbe-message-validation Change-Id: If7fc4c870252b6ef29e626a874e42c82d3879512
* WBE notification message validationJoshua Harlow2014-07-251-1/+4
| | | | | | | | | | | Add send and receive validation of the notify message that is sent between executors and workers to be more robust around invalid message formats being sent and received. Part of blueprint wbe-message-validation Change-Id: I7300d6f2d00e48c4f989c7f958a028bdff4afdd4
* Use a common message dispatcherJoshua Harlow2014-07-251-40/+21
| | | | | | | | | | | | | | | | Instead of recreating a dispatcher in the server and executor objects use a common dispatcher that is shared between them. It will dispatch based on the message type received into a provided dict of dispatch handler callbacks. It also can generically requeue messages and can reject messages if they are missing key required message properties ('type' in the current case). Part of blueprint wbe-message-validation Change-Id: I8320f4707183f36e6a69f0552cf62f99a5467b7e
* Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-6/+8
| | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Revert "Move taskflow.utils.misc.Failure to its own module"Jeremy Stanley2014-03-311-14/+13
| | | | | | | | | | | | This reverts commit 42ca240e8157b840c298d14fbf478ae570376633 which was a breaking change in a library consumed by other OpenStack projects with no deprecation or backwards compatibility considerations. It was able to merge because openstack/taskflow is apparently not yet part of the integrated gate via the proposed I202f4809afd689155e2cc4a00fc704fd772a0e92 change. Change-Id: I96cf36dc317499df91e43502efc85221f8177395 Closes-Bug: #1300161
* Move taskflow.utils.misc.Failure to its own moduleIvan A. Melnikov2014-03-311-13/+14
| | | | | | | | | | Failure class is important part of TaskFlow API, so it should be more visible and accessible. Breaking change: any client that used taskflow.utils.misc.Failure should be updated. Change-Id: Ib30000c9246bbcb227b34dfb0aba4d0b950bf926
* Persistence cleanup part oneJoshua Harlow2014-03-261-7/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | - Convert the various functions that take a task detail into ones that take atom details (since this is now the generic type they should take). - Don't expose the detail type strings as part of the atom detail api, leave those as private hidden strings and provide conversion functions from string<->class instead. - Have the logbook objects contain the following new methods to reduce the dependence on persistence_utils to do the same. - to_dict() which converts the current object into a dict - from_dict() which converts the provided dict into a object - merge() which merges a incoming objects data with the current objects - Have the persistence backends + storage + action engine use these new methods instead of there current usage. - Don't compare to logbook.RETRY_DETAIL or logbook.TASK_DETAIL since python has the isinstance function just use it (ideally we should fix the code so that this isn't even needed, usage of isinstance means something is not designed/structured right). - In storage tests we can't assume that failures will be non-lossy since under certain backends when a failure is stored information about the internally held exc_info is lost, so take this into account when testing by using matches() where applicable. Change-Id: Ie8a274cfd4cb4e64e87c355dc99d466d74a4e82c
* Merge "Avoid shutting down of a passed executor"Jenkins2014-03-241-1/+0
|\
| * Avoid shutting down of a passed executorJoshua Harlow2014-03-201-1/+0
| | | | | | | | | | | | | | | | | | | | For the case where an executor is being provided we should not shutdown the executor on-behalf of the provider since they would likely expect that they can still use the executor elsewhere. If the executor is shutdown then further usage is not possible. Change-Id: Ia83d8c2f1df06200e32e9b0c7340f32e74fd5d85
* | Improve logging for proxy publishStanislav Kudriashev2014-03-201-1/+0
| | | | | | | | | | | | Added message type logging during proxy publishing. Change-Id: Iddbfab0e4dd6e9f5b6d1c3ea5c86029700a2d66d
* | [WBE] Collect information from workersStanislav Kudriashev2014-03-181-6/+24
| | | | | | | | | | | | | | | | | | | | | | * Added the `Notify` message type. Used to notify workers about executor has been started and to receive workers information back (topic and tasks list); * Implemented `WorkersCache` to interact with workers information easly; * Got rid of the engine `workers_info` parameter. Change-Id: I4a810b1ddb0b04c11d12b47afc5f9cdf77d070be
* | Introduce message types for WBE protocolStanislav Kudriashev2014-03-141-9/+18
| | | | | | | | | | | | | | | | | | | | | | | | * Abstract message class added - all messages types are derived from it now and have to implement the `to_dict` method, so it can be serialized and transferred with broker; * Implemented the `Response` message type, that restores failures from dictionary on creation; * Corrected and improved unit tests; Change-Id: I10e017a613f0422420d0244b9f8786f988863107
* | Rename remote task to requestStanislav Kudriashev2014-03-121-5/+6
| | | | | | | | | | | | | | | | | | * Renamed remote task -> request and moved to protocol.py; * Used `to_dict` method instead of `request` property for request objects; * Renamed `name` request property to `task_cls`; * Corrected unit tests. Change-Id: I6133748ab5064391480f031971c38a56cb7f4f9f
* | Rework proxy publish functionalityStanislav Kudriashev2014-03-121-1/+1
| | | | | | | | | | | | | | * Task uuid is not a required parameter now; * Exchange declaration does not require a connection object. Change-Id: I9a77c9afe7d9fad007902d77d24a0b1a96c6c519
* | Use message.requeue instead of message.rejectStanislav Kudriashev2014-03-111-7/+7
| | | | | | | | | | | | | | Message.requeue does exactly the same as message.reject with requeue=True, so why not to use it directly. Change-Id: I42462172c2eb5d5c534e3c55ce7dfcc1a7169e31
* | A few worker-engine cleanupsJoshua Harlow2014-03-061-25/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Remove usage of % with logs, just let the underlying LOG do the interpolation for us if the log level passes. - Use LOG.exception instead of LOG.warn and then including the exception as a interpolated variable. Just let the LOG.exception output the exception trace and info. - Capture the failure object before doing other work using a new context manager that can yield back a failure object before further code is called (to avoid a context switch happening in eventlet or elsewhere that would clear the exception state). Change-Id: Ic1a504ba69c56ba226b1a067b5ade4590c245dd8
* | Rename uuid to topicJoshua Harlow2014-02-251-2/+2
|/ | | | | | This appears to be the topic and not a uuid. Change-Id: I191fcef543aa0879913fd965ebef0496b0163336
* Remove extraneous vim configuration commentsStanislav Kudriashev2014-02-211-2/+0
| | | | Change-Id: I06eada275398adc3523c02a99d9e4650e2dd1637
* Message-oriented worker-based flow with kombuStanislav Kudriashev2014-02-191-0/+178
* Implemented Worker to be started on remote host for handling tasks request. * Implemented WorkerTaskExecutor that proxies tasks requests to remote workers. * Implemented Proxy that is used for consuming and publishing messages by Worker and Executor. * Added worker-based engine and worker task executor. * Added kombu dependency to requirements. * Added worker-based flow example. * Added unit-tests for worker-based flow components. Implements: blueprint worker-based-engine Change-Id: I8c6859ba4a1a56c2592e3d67cdfb8968b13ee99c
View Lorry Controller Status