summaryrefslogtreecommitdiff
path: root/taskflow/engines/action_engine/executor.py
Commit message (Collapse)AuthorAgeFilesLines
* Remove sixTakashi Kajinami2022-05-181-3/+1
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* remove unused LOGji-xuepeng2016-07-101-3/+0
| | | | | | This is to remove unused LOG to keep code clean. Change-Id: Ie44659f3b7af9612f127f7d46bb2c1481b6a634f
* Instead of a multiprocessing queue use sockets via asyncoreJoshua Harlow2016-05-241-406/+5
| | | | | | | | | | | | | | | | | | | | | | For a local process based executor usage currently to ensure that task emitted notifications are proxied we use the multi processing library and use its queue concept. This sadly creates a proxy process that gets associated, and this proxy process handles the queue and messages sent to and from it. Instead of doing this we can instead just create a temporary local socket using a random socket and have tasks (which are running in different processes) use that to communicate back any emitted notifications instead (and we can use the asyncore module to handle the emitted notifications since it handles the lower level socket reading, polling and dispatching). To ensure that the socket created is somewhat secure we use a similar process as the multi-processing library uses where we sign all messages with a hmac that uses a one time key that only the main process and the child process know about (and reject any messages that do not validate using this key). Change-Id: Iff9180054bf14495e5667af00ae2fafbdbc23791
* Allow for specifying green threaded to parallel engineJoshua Harlow2016-05-031-0/+18
| | | | | | | | | | | | | Currently when a string is passed to the parallel engine it will only know how to create a process or a native thread based executor. The futurist library also supports making a green thread based executor, so support creating it. This will save glance some code that they have to create a executor based on different options (one of those is a green option). Change-Id: I15c164a38b4445d28eb6062aed6c56cce0e0364b
* Move all internal blather usage/calls to trace usage/callsJoshua Harlow2015-12-171-14/+14
| | | | Change-Id: I415a81d3b6b15b17a9a91cc2a0681c159172a4e1
* Merge "Convert executor proxied engine options into their correct type"Jenkins2015-11-241-4/+19
|\
| * Convert executor proxied engine options into their correct typeJoshua Harlow2015-11-181-4/+19
| | | | | | | | Change-Id: If0f213a0304bc336e95d72157cc2d4d402ebdd3f
* | Relabel internal engine 'event' -> 'outcome'Joshua Harlow2015-10-051-1/+1
|/ | | | | | | | | | | | Instead of calling the variable name 'event' it is more appropriate to call these variable names 'outcome' (or 'outcomes') since they represent the 'outcome' (EXECUTED, or REVERTED) of a atom and are not event types. To avoid the confusion just relabel these to be outcome(s) instead. Change-Id: Ia9caebe279145e4887491151d40a4f59650d40d4
* Create and use a serial retry executorJoshua Harlow2015-07-211-5/+43
| | | | | | | | | | | | To make it easily possible to change the retry atom execution from being in the engine thread this creates a retry executor (which is similar to the task executor) and provide that a serial executor (which it will use to execute with). This makes the retry and task actions closer to being the same and makes the surrounding code that much similar (which makes understanding it easier). Change-Id: I993e938280df3bd97f8076293183ef21989e2dba
* Provide a deprecated alias for the now removed stop watch classJoshua Harlow2015-07-121-3/+2
| | | | | | | | | | | | | This allows those who were using it to still continue using it until 2.0 where it will be removed; this makes it possible for those users to get off that code in a way that will be easily do-able (without totally breaking there code-bases, until we do that in the 2.0 release). This also removes all internal usage of that stop watch class so that the library itself will not reference it anymore. Change-Id: If313d8e7b9bdc8741db2e2e1dfb381aa3260b971
* Integrate futurist (and **remove** taskflow originating code)Joshua Harlow2015-07-091-4/+4
| | | | Change-Id: If89baa042695f19e42b6368034f3ccf22c2cf0aa
* Remove 2.6 classifier + 2.6 compatibility codeJoshua Harlow2015-06-211-1/+2
| | | | | | | | Fixes bug 1445827 Depends-On: I02e3c9aacef0b295a2f823a5cbaf11768a90cb82 Change-Id: I1db681803598ac1bc917fd74a99458bc61edf3f1
* Remove a couple more useless 'pass' keywords foundJoshua Harlow2015-05-291-2/+0
| | | | Change-Id: I01e444a766836ef6867e3312c18043cae6ac6c4b
* Just let the future executors handle the max workersJoshua Harlow2015-03-181-5/+2
| | | | | | | | | | | | Instead of providing and retaining a thread count in the worker and action engine executors and checking it and handling the none case, we can just let the future types handle this already (which they already do). And when displaying this information in the worker banner use a new future executor attribute that is the maximum number of workers that will be ever created. Change-Id: I765c22936b53cdbb8a90195a764d4c67bcc3f34b
* Stopwatch usage cleanup/tweakJoshua Harlow2015-01-281-7/+8
| | | | | | | | | | | | | Instead of optionally creating a stopwatch when a provided timeout is not none (to avoid the stopwatch leftover() method raising a error) just allow the stopwatch leftover() method to not raise when no duration is provided to avoid these repeated styles of usage/checks in the first place. By default the leftover() method still raises an error (a new keyword argument is now accepted to turn off this behavior). Change-Id: If934ee6e6855adbb6975cd6ea41e273d40e73dac
* Merge "Have the serial task executor shutdown/restart its executor"Jenkins2015-01-211-0/+6
|\
| * Have the serial task executor shutdown/restart its executorJoshua Harlow2015-01-161-0/+6
| | | | | | | | | | | | | | | | | | | | When the serial task executor is stopped, have it match what the other executors do and shutdown its executor, and on start have it restart/start it to be come functional again. This matches what the other executors do and makes it easier to understand the common pattern that is applied/used. Change-Id: Id62b588b05262aa9e334a64f53e4c4a0d5b66159
* | Switch to using 'oslo_utils' vs 'oslo.utils'Joshua Harlow2015-01-141-4/+4
|/ | | | | | | | Prefer the non-deprecated 'oslo_utils' instead of the namespaced 'oslo.utils' wherever it was previously used. Change-Id: I9a78150ef5266e1ff22147278162fe3cfe1b2e3f
* Merge "Create and use a multiprocessing sync manager subclass"Jenkins2015-01-131-8/+15
|\
| * Create and use a multiprocessing sync manager subclassJoshua Harlow2015-01-121-8/+15
| | | | | | | | | | | | | | | | | | Instead of accessing private variables of the manager base class just create a subclass and more nicely expose methods that can be used to introspect the managers state and perform actions based on that state. Change-Id: Ied570a25e52de94370b59d844ecdcc6d3551fd3d
* | Merge "Include docstrings for parallel engine types/strings supported"Jenkins2015-01-131-0/+2
|\ \ | |/ |/|
| * Include docstrings for parallel engine types/strings supportedJoshua Harlow2015-01-101-0/+2
| | | | | | | | | | | | | | | | | | | | | | When a 'executor' option is passed to a action engine requested to run in parallel mode that option will internally be examined and it will affect the internally used execution model that the engine will use; to make it understandable what the valid options are include a docstring + table(s) that describes the options and there valid values. Change-Id: I9a1852427bae22a01f5993862617e384f10ec005
* | Use a single senderJoshua Harlow2015-01-121-2/+4
|/ | | | | | | | | Instead of register many different senders (one for each needed proxy notification type) just use a single sender for all the different types to save space and time when the senders are pickled across to the worker(s). Change-Id: I1e1ab6c708c855e5868061bc338d399a38ad954e
* Merge "Allow specifying the engine 'executor' as a string"Jenkins2015-01-051-0/+4
|\
| * Allow specifying the engine 'executor' as a stringJoshua Harlow2015-01-011-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To enable a parallel process executor to be used without having to pass in a futures executor allow for the executor option to be a string 'processes' (or similar) that will cause the default parallel process executor to be used automatically. Also allow for a 'threads' string that ensure a parallel thread executor is used to match the ability to uses processes. This also adjusts the WBE engine to have a similar executor fetching function (which in the WBE case now validates a provided executor to be of the desired type). Change-Id: I54a82584c32c697922507b4f6e01ea7b8acc73c6
* | Disallowing starting the executor when worker runningJoshua Harlow2014-12-311-6/+8
|/ | | | | | | | To avoid consistency/threading/runtime issues stop the action engine executor from being started if it is already running with a valid worker thread. Change-Id: I39925e55e7b171f289152d941ebdf390552f880c
* Merge "Use a single shared queue for an executors lifecycle"Jenkins2015-01-011-158/+186
|\
| * Use a single shared queue for an executors lifecycleJoshua Harlow2014-12-311-158/+186
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having many queues (one per task) just have a shared one and use sender identifiers to know which originating task to proxy back to. This scales better and avoids the needless polling of many queues for a potential message to emit (and can now instead just poll one queue); when now we can just poll a single queue for any messages (this does though remove the ability to throttle messages from a given sender). Change-Id: I3566a5ab20ad15a80a4c6969f48b076ddde1d7ac
* | Merge "Register with 'ANY' in the cloned process"Jenkins2014-12-311-4/+0
|\ \ | |/ |/|
| * Register with 'ANY' in the cloned processJoshua Harlow2014-12-251-4/+0
| | | | | | | | | | | | | | | | | | Seems like we should have registered with 'ANY' since otherwise if a cloned task emits a unknown notification then the 'ANY' event should be triggered instead of not being emitted. Change-Id: Iedd3c0eb034043ba8e5b9e9a02a6e49c451e17b3
* | Ensure manager started/shutdown/joined and resetJoshua Harlow2014-12-211-6/+23
|/ | | | | | | | | | | | | Ensure that when the task executor is started that we correctly create a new multiprocessing manager (if needed) and that on stop we correctly shut that manager down and join it. Also does a tiny adjustment to the joinable work item to move the finish logic into its own method and ensures that we have no targets on reset of the dispatcher. Change-Id: I688df323fb24a7e228f4fa237f2fa772d9c0dc62
* Get event/notification sending working correctlyJoshua Harlow2014-12-201-10/+322
| | | | | | | | | | | | | | | | | | | In order to support tasks notifications and progress updates we need to establish a channel & proxy by which those events can be sent from the process executing and producing those events and the originating process that requested that task to be executed. This review adds on such a proxy and adjusts a cloned tasks notification callbacks to place messages on a queue that will be picked up by a thread in the originating process for dispatch to the original callbacks that were registered with the non-cloned task (therefore making the original callbacks appear to be called as they are supposed to be). Part of blueprint process-executor Change-Id: I01c83f13186e4be9fa28c32e34e907bb133e8fb3
* Get the basics of a process executor workingJoshua Harlow2014-12-191-14/+52
| | | | | | | | | | | | | | | | | Since we support various executors (threaded and distributed) the next best executor when a threaded executor will not perform and a distributed one requires to much setup is a local process based one so it would be great to support this where we can. Things that are currently (likely never) not going to work: * Non-pickleable/non-copyable tasks * Tasks that return non-pickleable/non-copyable results * Tasks that use non-pickleable/non-copyable args/kwargs Part of blueprint process-executor Change-Id: I966ae01d390c7217b858db3feb2db949ce5c08d1
* Base task executor should provide 'wait_for_any'Joshua Harlow2014-12-151-7/+1
| | | | | | | | | | | Instead of having each task executor reproduce the same code for 'wait_for_any' we can just have the base task implementation provide the function that everyone is replicating instead; making common code common and save the headache caused by the same code being in multiple places (which is bad for multiple reasons). Change-Id: Icea4b7e3df605ab11b17c248d05acb3f9c02a1ca
* Replace autobind with a notifier module helper functionJoshua Harlow2014-12-141-18/+7
| | | | | | | | | Instead of having the executor method provide a bind and unbind function just have the notifier module provide this, which allows those who were using the task autobind to use this helper as well as the execution code itself. Change-Id: If61c8c1669d7c0d66e2daab5fd773b8c7756f202
* Use the notifier type in the task class/module directlyJoshua Harlow2014-12-131-17/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having code that is some what like the notifier code we already have, but is duplicated and is slightly different in the task class just move the code that was in the task class (and doing similar actions) to instead now use a notifier that is directly contained in the task base class for internal task triggering of internal task events. Breaking change: alters the capabilities of the task to process notifications itself, most actions now must go through the task notifier property and instead use that (update_progress still exists as a common utility method, since it's likely the most common type of notification that will be used). Removes the following methods from task base class (as they are no longer needed with a notifier attribute): - trigger (replaced with notifier.notify) - autobind (removed, not replaced, can be created by the user of taskflow in a simple manner, without requiring functionality in taskflow) - bind (replaced with notifier.register) - unbind (replaced with notifier.unregister) - listeners_iter (replaced with notifier.listeners_iter) Due to this change we can now also correctly proxy back events from remote tasks to the engine for correct proxying back to the original task. Fixes bug 1370766 Change-Id: Ic9dfef516d72e6e32e71dda30a1cb3522c9e0be6
* Remove the base postfix from the internal task executorJoshua Harlow2014-12-121-4/+4
| | | | | | | | | | There is no need to have a postfix of 'base' in the task executor as it is obvious by having a metaclass of abc.ABCMeta and the fact that it has no methods that the task executor base is a base class and should be used as such without a 'base' postfix to also signify the same information. Change-Id: I1f5cbcd29f1453d68e774f9f9f733eb873efc7cb
* Stop returning atoms from execute/revert methodsJoshua Harlow2014-12-061-11/+18
| | | | | | | | | | | | | | | | It is not needed to return the atom that was executed from the futures result() method, since we can just as easily set an attribute on the future and reference it from there when using it later. This is also required for a process based executor since it is not typically possible to send back a raw task object (and is not desireable to require this); even if it was possible the task would be pickled and unpickled multiple times so when this happens it can not be guaranteed to even be the same object (in fact it is not). Part of blueprint process-executor Change-Id: I4a05ea5dcdef97218312e3a88ed4a1dfdf1b1edf
* Rework pieces of the task callback capabilityJoshua Harlow2014-12-061-2/+2
| | | | | | | | | | | | | | | | | | | Unifies the bind, unbind, autobind parameters. Also to make it easier to introspect what are a tasks associated callbacks and events are provide a listeners_iter() method that can be used to introspect the registered (event, callbacks) pairs that are registered with a task. Also adds more useful docstrings to the various callback associated binding, unbinding functions to make it more understandable how they are used and what they are provided. Also makes the currently only default provided event 'update_progress' a constant that can be referenced from the task module, which allows others to easily find it and use it. Change-Id: I14181a150b74fbd97f6ea976723f37c0ba4cec36
* Remove direct usage of the deprecated failure locationJoshua Harlow2014-10-211-3/+3
| | | | | | | | Internally we should be using the new location and not the deprecated location wherever possible. This avoids emitting warnings messages on our own code, which is a dirty habit. Change-Id: Idac5a772eca7529d92542ada3be1cea092880e25
* Add a futures type that can unify our future functionalityJoshua Harlow2014-10-181-8/+9
| | | | | | | | | | | | | | | | | Move the currently existing green future executor and associated code to a new futures types module so that it can be accessed from this new location (TODO: deprecate the old location and link the old to the new for one release so that we can remove the old link in N + 1 release). This unifies the API that the existing pool (thread or process) future executors and the green thread pool future executor, and the newly added synchronous executor (replacing the previous `make_completed_future` function) provide so there usage is as seamless as possible. Part of blueprint top-level-types Change-Id: Ie5500eaa7f4425edb604b2dd13a15f82909a673b
* Use constants for revert automatically provided kwargsJoshua Harlow2014-10-171-2/+3
| | | | | | | | | | Instead of using strings use module level constants for the automatically provided keyword arguments to the tasks revert function. This makes it easier for users of taskflow to associate these constants with the actual keywords, without having to resort to using raw strings directly. Change-Id: I12726812615052d8405071d46272ef2b8286cfe2
* Allow worker count to be specified when no executor providedJoshua Harlow2014-08-171-7/+11
| | | | | | | | | When a multi-threaded engine is used it is nice to be able to inform the engine how many workers should be created when it creates its own executor. To allow this to be possible accept a new keywork argument that can be used to set this value. Change-Id: I0095d548249372440abbcf9b5c3b8fa841ca0ea9
* add pre/post execute/retry callbacks to tasksGreg Hill2014-08-061-0/+6
| | | | | | | | This enables us to execute code to set up or tear down global state in running tasks. Change-Id: Ib1e5d03ab46b3ce1d03fa83b91bf437fa950b758 Implements: blueprint task-callbacks
* Use futures wait() when possibleJoshua Harlow2014-05-141-1/+1
| | | | | | | | | Instead of always using our custom future wait functionality, only use that functionality if there are green futures and in other cases just use the future wait() function instead. Change-Id: I1eadcf53eb4b5f47b9543965610bfe04fec52e70
* Revert "Move taskflow.utils.misc.Failure to its own module"Jeremy Stanley2014-03-311-3/+3
| | | | | | | | | | | | This reverts commit 42ca240e8157b840c298d14fbf478ae570376633 which was a breaking change in a library consumed by other OpenStack projects with no deprecation or backwards compatibility considerations. It was able to merge because openstack/taskflow is apparently not yet part of the integrated gate via the proposed I202f4809afd689155e2cc4a00fc704fd772a0e92 change. Change-Id: I96cf36dc317499df91e43502efc85221f8177395 Closes-Bug: #1300161
* Move taskflow.utils.misc.Failure to its own moduleIvan A. Melnikov2014-03-311-3/+3
| | | | | | | | | | Failure class is important part of TaskFlow API, so it should be more visible and accessible. Breaking change: any client that used taskflow.utils.misc.Failure should be updated. Change-Id: Ib30000c9246bbcb227b34dfb0aba4d0b950bf926
* Merge "Message-oriented worker-based flow with kombu"Jenkins2014-02-201-6/+6
|\
| * Message-oriented worker-based flow with kombuStanislav Kudriashev2014-02-191-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Implemented Worker to be started on remote host for handling tasks request. * Implemented WorkerTaskExecutor that proxies tasks requests to remote workers. * Implemented Proxy that is used for consuming and publishing messages by Worker and Executor. * Added worker-based engine and worker task executor. * Added kombu dependency to requirements. * Added worker-based flow example. * Added unit-tests for worker-based flow components. Implements: blueprint worker-based-engine Change-Id: I8c6859ba4a1a56c2592e3d67cdfb8968b13ee99c
* | Remove extraneous vim configuration commentsyangxurong2014-02-141-2/+0
|/ | | | | | | | | Remove line containing comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4 Change-Id: I7581cc88b8de433d5609ed06c6570b0b45c13573 Closes-Bug:#1229324
View Lorry Controller Status