summaryrefslogtreecommitdiff
path: root/taskflow/engines/worker_based/protocol.py
Commit message (Collapse)AuthorAgeFilesLines
* Remove sixTakashi Kajinami2022-05-181-7/+5
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Use a automaton machine for WBE request state machineJoshua Harlow2016-02-211-37/+89
| | | | Change-Id: Ib26ff4418ab1128a578519be964c4d39cbd1d2f4
* Add WBE worker expiryJoshua Harlow2016-02-141-0/+4
| | | | | | | | | When a worker hasn't responded to a notification request for a given amount of time remove it from being a useable worker that we can match task submissions to. Change-Id: I596bccc1c42f83ee79136dd27bc87039154ff7b1
* Some WBE protocol/executor cleanupsJoshua Harlow2016-02-141-59/+31
| | | | | | | | | | | | | | | | | | | Remove some of the usage of @property as none of these objects are publicly exposed (or have docstrings on them) to save some space/lines of code that aren't really adding any benefit. Use less **kwargs when we know exactly what the keyword arguments will or will not be. Being explicit makes it easier to understand these functions (vs not knowing what the arguments can or can't be). Removes base worker finder because right now we only have one implementation (at some point we will have two) but we can just wait to add a base class until then. Change-Id: I7107ff6b77a355b4c5d301948355fb6386605388
* Fix for WBE sporadic timeout of tasksJoshua Harlow2016-02-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes the sporadic of tasks that would happen under certain circumstances. What happened was that a new worker notification would be sent to a callback while at the same time a task submission would come in and there would be a small race period where the task would insert itself into the requests cache while the callback was processing. So to work around this the whole concept of a requests cache was revamped and now the WBE executor just maintains its own local dictionary of ongoing requests and accesses it safely. During the on_wait function that is periodically called by kombu the previous expiry of work happens but now any requests that are pending are matched to any new workers that may have appeared. This avoids the race (and ensures that even if a new worker is found but a submission is in progress that the duration until that submission happens will only be until the next on_wait call happens). Related-Bug: #1431097 Change-Id: I98b0caeedc77ab2f7214847763ae1eb0433d4a78
* Handle cases where exc_args can't be serialized as JSON in the WBEGreg Hill2016-01-281-2/+15
| | | | | | | | | First try to get them, but if they aren't able to be serialized, just return the failure without them. This only affects the WBE that serializes failures as part of the JSON response that is sent over the wire from the worker process to the WBE engine. Change-Id: I86e3255b612bc15097aabe63a684cf8b8808e61b
* Make more of the WBE logging and '__repr__' message more usefulJoshua Harlow2015-10-091-3/+12
| | | | | | | | | When running the examples, especially when running them in TRACE/BLATHER/DEBUG logging level these updates make it more clear = what is being processed, the messages being sent/acked/received and what there contents are. Change-Id: I94a497c9064df30197454ae480fe3d471ba1dc7d
* Provide a deprecated alias for the now removed stop watch classJoshua Harlow2015-07-121-2/+1
| | | | | | | | | | | | | This allows those who were using it to still continue using it until 2.0 where it will be removed; this makes it possible for those users to get off that code in a way that will be easily do-able (without totally breaking there code-bases, until we do that in the 2.0 release). This also removes all internal usage of that stop watch class so that the library itself will not reference it anymore. Change-Id: If313d8e7b9bdc8741db2e2e1dfb381aa3260b971
* Integrate futurist (and **remove** taskflow originating code)Joshua Harlow2015-07-091-2/+2
| | | | Change-Id: If89baa042695f19e42b6368034f3ccf22c2cf0aa
* Remove all 'lock_utils' now that fasteners provides equivalentsJoshua Harlow2015-06-041-2/+2
| | | | | | Depends-On: I442249783da4a6ae10c78b95e0b279409c95d2e6 Change-Id: I877928c858e8d6176d3f01ad9de2765104acf5c3
* Use the ability to chain exceptions correctlyJoshua Harlow2015-04-031-12/+20
| | | | | | | | | | | In the zookeeper jobboard (and persistence backends) we are associating the cause of a new exception when raising a exception. Using the new exception helper we can make this work better on py2.x and py3.x so that the py3.x version has the chain setup correctly (while the py2.x version just uses the class 'cause' attribute instead). Change-Id: Ieeac2f70e1834d4612556565762ffd3be3e5b5a1
* Use the class name instead of the TYPE property in __str__Joshua Harlow2015-03-171-1/+2
| | | | Change-Id: Ie6fc31b0c89b9eaf9fabc1feb17d697edc5ff522
* Just use the class name instead of TYPE constantJoshua Harlow2015-03-151-4/+7
| | | | | | | | | | The TYPE constant is more of an internal protocol detail and is not the best constant to use in exceptions that tell the received data is invalid, instead just prefer and use the class name instead which is more relevant than the TYPE constant. Change-Id: I60652a149eb1a3c3d8523544c355044bbd6fec94
* Use a collections.namedtuple for the request work unitJoshua Harlow2015-03-131-2/+7
| | | | | | | | | | Instead of returning a tuple with hard to understand and read contents a namedtuple can help understand what the tuple is composed of and how it can be used/what it is, so we should prefer to use it when we can for the wbe requested work to perform. Change-Id: I8147814449d04ba9a03730547ac113e4ef7b272a
* Add + use failure json schema validationJoshua Harlow2015-03-111-14/+57
| | | | Change-Id: Ie3aa386c831459a028ba494570bafd53b998126e
* Merge "WBE documentation tweaks/adjustments"Jenkins2015-01-231-8/+21
|\
| * WBE documentation tweaks/adjustmentsJoshua Harlow2015-01-231-8/+21
| | | | | | | | | | | | | | | | | | Include the protocol code/classes (and the json schema that is used for validation) in the docs; adjust the examples to be after the protocol definition/classes. Some other small wording tweaks as well. Change-Id: Ic2987c6f4393a5398065de883bdb15ffec923451
* | Tidy up the WBE cache (now WBE types) moduleJoshua Harlow2015-01-221-6/+9
|/ | | | | | | | | | | | | | | Instead of using the expiring cache type as a means to store worker information just avoid using that type since we don't support expiry in the first place on worker information and use a worker container and a worker object that we can later extend as needed. Also add on clear methods to the cache type that will be used when the WBE executor stop occurs. This ensures we clear out the worker information and any unfinished requests. Change-Id: I6a520376eff1e8a6edcef0a59f2d8b9c0eb15752
* Switch to using 'oslo_utils' vs 'oslo.utils'Joshua Harlow2015-01-141-2/+2
| | | | | | | | Prefer the non-deprecated 'oslo_utils' instead of the namespaced 'oslo.utils' wherever it was previously used. Change-Id: I9a78150ef5266e1ff22147278162fe3cfe1b2e3f
* Move over to using oslo.utils [reflection, uuidutils]Joshua Harlow2014-12-181-1/+1
| | | | | | | | | | | | | | The reflection module is now part of oslo.utils so we should remove our local version and use that version instead; this also goes for the uuidutils module which is now part of oslo.utils as well so we no longer need our local version copied from the incubator... Note that one reflection method `find_subclasses` which was to specific to taskflow is now moved to the misc utility module instead of its prior home in the reflection module. Change-Id: I069881c80b0b2916cc0c414992b80171f7eeb79f
* Use the notifier type in the task class/module directlyJoshua Harlow2014-12-131-16/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having code that is some what like the notifier code we already have, but is duplicated and is slightly different in the task class just move the code that was in the task class (and doing similar actions) to instead now use a notifier that is directly contained in the task base class for internal task triggering of internal task events. Breaking change: alters the capabilities of the task to process notifications itself, most actions now must go through the task notifier property and instead use that (update_progress still exists as a common utility method, since it's likely the most common type of notification that will be used). Removes the following methods from task base class (as they are no longer needed with a notifier attribute): - trigger (replaced with notifier.notify) - autobind (removed, not replaced, can be created by the user of taskflow in a simple manner, without requiring functionality in taskflow) - bind (replaced with notifier.register) - unbind (replaced with notifier.unregister) - listeners_iter (replaced with notifier.listeners_iter) Due to this change we can now also correctly proxy back events from remote tasks to the engine for correct proxying back to the original task. Fixes bug 1370766 Change-Id: Ic9dfef516d72e6e32e71dda30a1cb3522c9e0be6
* Merge "Add and use a new simple helper logging module"Jenkins2014-12-111-1/+1
|\
| * Add and use a new simple helper logging moduleJoshua Harlow2014-12-081-1/+1
| | | | | | | | | | | | | | | | | | Add a new logging BLATHER level to easily allow its usage for messages that are below the normal DEBUG level such as compilation information and scope lookup info which can be very verbose in logs if always enabled. Change-Id: I828211403bd02bfd6777b10cdcfe58fb0637a52c
* | Stop returning atoms from execute/revert methodsJoshua Harlow2014-12-061-1/+2
|/ | | | | | | | | | | | | | | | It is not needed to return the atom that was executed from the futures result() method, since we can just as easily set an attribute on the future and reference it from there when using it later. This is also required for a process based executor since it is not typically possible to send back a raw task object (and is not desireable to require this); even if it was possible the task would be pickled and unpickled multiple times so when this happens it can not be guaranteed to even be the same object (in fact it is not). Part of blueprint process-executor Change-Id: I4a05ea5dcdef97218312e3a88ed4a1dfdf1b1edf
* Remove direct usage of the deprecated failure locationJoshua Harlow2014-10-211-5/+5
| | | | | | | | Internally we should be using the new location and not the deprecated location wherever possible. This avoids emitting warnings messages on our own code, which is a dirty habit. Change-Id: Idac5a772eca7529d92542ada3be1cea092880e25
* Switch to using oslo.utils and oslo.serializationJoshua Harlow2014-09-201-1/+1
| | | | | | | | | | | | Instead of copying modules from the incubator into taskflow we can now directly use these same modules from supported libraries instead so this moves the usage of everything except uuidutils which wasn't moved over to using those newly published libraries. Part of blueprint integrate-and-use-oslo-utils-serialization Change-Id: I1183bda96e1ddb062d9cab91990186f0f56f0a0e
* Update transitioning function name to be more understandableJoshua Harlow2014-09-051-2/+11
| | | | | | | | To make it more obvious what this function does adjust the naming to be more clear and add a docstring that explains this in more depth. Change-Id: Ib39afd28e0bdeb94d50e85f2a39e210d3cbd7b01
* Use explicit WBE request state transitionsJoshua Harlow2014-09-041-12/+86
| | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having an implicit state machine for a requests lifecycle move toward an explicit state model and transition set that is validated and transitioned in a more easy to understand/reason about manner. This also fixes a bug that was found due to a response validation not taking into account a transition that was found due to this stricter transition checking. Includes a few tiny related/affected commits: * Remove testing of request repr() and str() as these types of tests are not useful and we removed the repr() version of the request message as the base classes is good enough. * Raise and capture a better exception and save its associated failure object when a request has expired (this gives expired requests better failure objects and associated details). Fixes bug 1356658 Partially fixes bug 1357117 Change-Id: Ie1386cca13a2da7265e22447b4c111a0a0074201
* Avoid naming time type module the same as a builtinJoshua Harlow2014-08-121-2/+2
| | | | | | | | | In order to avoid naming conflicts which can easily occur if we name our time module as a python builtin just pick a name that doesn't conflict (it still retains the same meaning). Change-Id: Ia19f3776689d2b1f31f7cbfaa15e53f99a3ee900
* WBE request message validationJoshua Harlow2014-07-251-2/+54
| | | | | | | | | | | Add receive validation of the request message sent by executors to workers to initiate work requests so that we can be more robust around invalid message formats being sent and received. Part of blueprint wbe-message-validation Change-Id: If7fc4c870252b6ef29e626a874e42c82d3879512
* WBE response message validationJoshua Harlow2014-07-251-1/+60
| | | | | | | | | | | Add receive validation of the response message sent by workers for progress updates and for result completion to be more robust around invalid message formats being sent and received. Part of blueprint wbe-message-validation Change-Id: I583b9d6344224724d147f2a5b10ddbd23d6a5fdc
* WBE notification message validationJoshua Harlow2014-07-251-0/+45
| | | | | | | | | | | Add send and receive validation of the notify message that is sent between executors and workers to be more robust around invalid message formats being sent and received. Part of blueprint wbe-message-validation Change-Id: I7300d6f2d00e48c4f989c7f958a028bdff4afdd4
* Enabled hacking checks H305 and H307Christian Berendt2014-07-171-2/+1
| | | | | | | * H305 imports not grouped correctly * H307 like imports should be grouped together Change-Id: If1dd9c89f65ede6959865a885777cb08c263eca0
* Create a top level time typeJoshua Harlow2014-06-201-1/+2
| | | | | | | | | | | | Create a time module under types and place the stop watch class and timeout classes there so that we can have a single location where generic types that do things with time can be located. Part of blueprint top-level-types Change-Id: I8eb7e897c0354b20a7fa0d061434006e775d5e94
* Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-4/+9
| | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Revert "Move taskflow.utils.misc.Failure to its own module"Jeremy Stanley2014-03-311-6/+5
| | | | | | | | | | | | This reverts commit 42ca240e8157b840c298d14fbf478ae570376633 which was a breaking change in a library consumed by other OpenStack projects with no deprecation or backwards compatibility considerations. It was able to merge because openstack/taskflow is apparently not yet part of the integrated gate via the proposed I202f4809afd689155e2cc4a00fc704fd772a0e92 change. Change-Id: I96cf36dc317499df91e43502efc85221f8177395 Closes-Bug: #1300161
* Move taskflow.utils.misc.Failure to its own moduleIvan A. Melnikov2014-03-311-5/+6
| | | | | | | | | | Failure class is important part of TaskFlow API, so it should be more visible and accessible. Breaking change: any client that used taskflow.utils.misc.Failure should be updated. Change-Id: Ib30000c9246bbcb227b34dfb0aba4d0b950bf926
* Persistence cleanup part oneJoshua Harlow2014-03-261-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | - Convert the various functions that take a task detail into ones that take atom details (since this is now the generic type they should take). - Don't expose the detail type strings as part of the atom detail api, leave those as private hidden strings and provide conversion functions from string<->class instead. - Have the logbook objects contain the following new methods to reduce the dependence on persistence_utils to do the same. - to_dict() which converts the current object into a dict - from_dict() which converts the provided dict into a object - merge() which merges a incoming objects data with the current objects - Have the persistence backends + storage + action engine use these new methods instead of there current usage. - Don't compare to logbook.RETRY_DETAIL or logbook.TASK_DETAIL since python has the isinstance function just use it (ideally we should fix the code so that this isn't even needed, usage of isinstance means something is not designed/structured right). - In storage tests we can't assume that failures will be non-lossy since under certain backends when a failure is stored information about the internally held exc_info is lost, so take this into account when testing by using matches() where applicable. Change-Id: Ie8a274cfd4cb4e64e87c355dc99d466d74a4e82c
* Improve logging for proxy publishStanislav Kudriashev2014-03-201-1/+1
| | | | | | Added message type logging during proxy publishing. Change-Id: Iddbfab0e4dd6e9f5b6d1c3ea5c86029700a2d66d
* [WBE] Collect information from workersStanislav Kudriashev2014-03-181-7/+30
| | | | | | | | | | | * Added the `Notify` message type. Used to notify workers about executor has been started and to receive workers information back (topic and tasks list); * Implemented `WorkersCache` to interact with workers information easly; * Got rid of the engine `workers_info` parameter. Change-Id: I4a810b1ddb0b04c11d12b47afc5f9cdf77d070be
* Merge "Introduce message types for WBE protocol"Jenkins2014-03-171-2/+51
|\
| * Introduce message types for WBE protocolStanislav Kudriashev2014-03-141-2/+51
| | | | | | | | | | | | | | | | | | | | | | | | * Abstract message class added - all messages types are derived from it now and have to implement the `to_dict` method, so it can be serialized and transferred with broker; * Implemented the `Response` message type, that restores failures from dictionary on creation; * Corrected and improved unit tests; Change-Id: I10e017a613f0422420d0244b9f8786f988863107
* | Update request `expired` property docstingStanislav Kudriashev2014-03-141-3/+2
|/ | | | Change-Id: If5a9881227e3fe5484df63b3117e078321365699
* Rename remote task to requestStanislav Kudriashev2014-03-121-0/+82
| | | | | | | | | * Renamed remote task -> request and moved to protocol.py; * Used `to_dict` method instead of `request` property for request objects; * Renamed `name` request property to `task_cls`; * Corrected unit tests. Change-Id: I6133748ab5064391480f031971c38a56cb7f4f9f
* Worker-based engine clean-upsStanislav Kudriashev2014-03-071-2/+2
| | | | | | | | * Auto-delete queues, redundant proxy logic removed; * Updated unit tests; * Small fixes in comments. Change-Id: I8d6fdbf06625daad7a886ff87607b31f369062b8
* Remove extraneous vim configuration commentsStanislav Kudriashev2014-02-211-2/+0
| | | | Change-Id: I06eada275398adc3523c02a99d9e4650e2dd1637
* Message-oriented worker-based flow with kombuStanislav Kudriashev2014-02-191-0/+47
* Implemented Worker to be started on remote host for handling tasks request. * Implemented WorkerTaskExecutor that proxies tasks requests to remote workers. * Implemented Proxy that is used for consuming and publishing messages by Worker and Executor. * Added worker-based engine and worker task executor. * Added kombu dependency to requirements. * Added worker-based flow example. * Added unit-tests for worker-based flow components. Implements: blueprint worker-based-engine Change-Id: I8c6859ba4a1a56c2592e3d67cdfb8968b13ee99c
View Lorry Controller Status