summaryrefslogtreecommitdiff
path: root/taskflow/engines/action_engine/runtime.py
Commit message (Collapse)AuthorAgeFilesLines
* Update TaskFlow for networkx 2.xMichael Johnson2019-10-181-5/+5
| | | | | | | | | | | The networkx 2.x series has been out for two years now and supports python 3.6 and greater[1]. This patch updates TaskFlow to require a minimum of networkx 2.1. It also updates the code to support recent deprecation expiration introduced in the 2.4 release. [1] https://networkx.github.io/documentation/stable/news.html Change-Id: Ife31d353ba80824ebc63c8b21ee90943badc8da3
* Rename engine analyzer to be named selectorJoshua Harlow2017-01-031-4/+46
| | | | | | | | | | | | | This moves out the engine next to run (or revert) selection process to a single class that just does this and moves out the common functions the analyzer class provided to the runtime object (which all components can access). This makes it easier to adjust the selection algorithm in different ways. Change-Id: I091c69297a7bff60729791d3ca6c3fae14d6eea5
* Add logging around metadata, ignore tallying + historyJoshua Harlow2016-08-011-0/+5
| | | | | | | | This can be useful to look at for debugging purposes when trying to understand how (and if) a atom is being ignored by a set of deciders. Change-Id: I550a2f7f361e516c255683d725b2f3a8130a0a86
* Allow for alterations in decider 'area of influence'Joshua Harlow2016-01-091-14/+31
| | | | | | | | Christmas came early. Closes-Bug: #1479466 Change-Id: I931d826690c925f022dbfffe9afb7bf41345b1d0
* Cache atom name -> actions and provide accessor functionJoshua Harlow2015-12-301-0/+11
| | | | | | | | | | Instead of doing repeated type checks in the completer engine internal component/object just use the runtime compilation routine to cache the mapping of atom names to actions and then use an exposed function to fetch the needed action in the completer object as needed. Change-Id: I07161e7956d039cf89d057b8082e12b82adcd82f
* Merge "Change name of misc.ensure_dict to misc.safe_copy_dict"Jenkins2015-12-151-1/+1
|\
| * Change name of misc.ensure_dict to misc.safe_copy_dictJoshua Harlow2015-12-141-1/+1
| | | | | | | | Change-Id: I2ff3b9bb4d0b8143163e4bf370f10c74e79dbd31
* | Use small helper routine to fetch atom metadata entriesJoshua Harlow2015-12-051-11/+10
| | | | | | | | Change-Id: Ibcffbdd9f499e91f2a4218133271cb10d16078a8
* | Merge "Use batch 'get_atoms_states' where we can"Jenkins2015-11-181-2/+3
|\ \ | |/ |/|
| * Use batch 'get_atoms_states' where we canJoshua Harlow2015-10-131-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | Instead of doing many single calls to 'get_atom_state' when we are iterating over many atoms instead prefer to use the bulk call via 'get_atoms_states' that can gather a large amount of needed states in a single call, optimizing more of the reading that taskflow does with regards to states. This also removes the proxy method 'get_state' from the engine internal code, since it is no longer needed after this change. Change-Id: I90eb43e754a7e5efb657468361d67dbe69d31844
* | Add optional 'defer_reverts' behaviorGreg Hill2015-11-171-1/+6
| | | | | | | | | | | | | | | | | | | | | | This makes it possible to REVERT a subflow and have it also revert the parent flow if the parent flow doesn't have its own retry strategy. We will probably want to make this new behavior the default or only behavior in a future release. Change-Id: Iea5ac366380ba7396a87d0185703549fb0c2f825
* | Correctly apply deciders across flow boundariesJoshua Harlow2015-11-131-10/+31
|/ | | | | | | | | | | | | | When a flow is added to another flow and there is a decider placed on that link we need to make sure that we retain that decider. This also ensures that any predecessor of that flow that has a decider that affects the flow nodes execution also gets used in the decision process. Closes-Bug: #1515748 Change-Id: Ifdf20378b26cdd13e0a3ff87cec8990fe89c0661
* Simplify flow action engine compilation1.22.0Joshua Harlow2015-10-011-43/+35
| | | | | | | | | | | | | | Instead of the added complexity of discarding flow nodes we can simplify the compilation process by just retaining them and jumping over them in further iteration and graph and tree runtime usage. This change moves toward a model that does just this, which makes it also easier to in the future use the newly added flow graph nodes to do meaningful things (like use them as a point to change which flow_detail is used). Change-Id: Icb1695f4b995a0392f940837514774768f222db4
* Remove extra runner layer and just use use machine in engineJoshua Harlow2015-07-281-5/+5
| | | | | | | | | Just directly use the built machine in the action engine and avoid having another layer of abstraction that does not provide that much value. This makes the code cleaner, and more easy to understand (and so-on). Change-Id: Iae1279098112254338258c1941c15889f1ad1a79
* Bump futurist and remove waiting code in taskflowJoshua Harlow2015-07-251-2/+3
| | | | Change-Id: Ifc9780aa129a4a2804cead301a519895c2bfc0b5
* Create and use a serial retry executorJoshua Harlow2015-07-211-3/+7
| | | | | | | | | | | | To make it easily possible to change the retry atom execution from being in the engine thread this creates a retry executor (which is similar to the task executor) and provide that a serial executor (which it will use to execute with). This makes the retry and task actions closer to being the same and makes the surrounding code that much similar (which makes understanding it easier). Change-Id: I993e938280df3bd97f8076293183ef21989e2dba
* Compile lists of retry/task atoms at runtime compile timeJoshua Harlow2015-07-091-0/+16
| | | | | | | | | Instead of recompiling and rebuilding this list every iteration of the ``iterate_retries`` function we can just locally cache this information in the runtime compile function and later just use it directly. Change-Id: I70e8409391d655730da61413300db05b25843350
* Add support for conditional executionJoshua Harlow2015-07-011-0/+20
| | | | | | | | | | | | | | | To make it possible to alter the runtime flow via a simple conditional like structure make it possible to have the graph flow link function take a decider that is expected to be some callable that will decide (via a boolean return) whether the edge should actually be traversed when running. When a decider returns false; the affected + successors will be set into the IGNORE state and they will be exempt from future runtime and scheduling decisions. Part of blueprint taskflow-conditional-execution Change-Id: Iab0ee46f86d6b8e747911174d54a7295b3fa404d
* Add docstrings on runtime objects methods and link to them in docsJoshua Harlow2015-06-221-3/+20
| | | | Change-Id: Ia85fe4b7849068271a5452d622fbed163dc9847d
* Cache the individual atom schedulers at compile timeJoshua Harlow2015-06-181-0/+23
| | | | | | These do not need to be re-fetched/re-examined each time. Change-Id: Ie48100caa12575c725530911ad3d1dc9046e9d26
* Split-off the additional retry states from the task statesJoshua Harlow2015-06-181-33/+55
| | | | | | | | | | | | | | Split the states that are not task states (but are retry states) into there own additional set and then use that set and a new function to validate the transition at other locations in the code-base. This makes the transitions that are valid for tasks/retries easily viewable, more easy to read and understand, and more correct (instead of being a mix of task + retry atom transitions and states). Change-Id: I9515c19daf59a21e581f51e757ece2050f348214
* Have reset state handlers go through a shared listJoshua Harlow2015-04-151-8/+13
| | | | | | | | | To make it hopefully easier to understand the commonality of this function and its handlers have the handlers be defined at the top of the function and use the common list in the node iteration routine when resetting/matching. Change-Id: I288eb548172b2f6f35e2bd90683f2065d3aa02dc
* Make resolution/retry strategies more clear and betterJoshua Harlow2015-04-091-4/+8
| | | | | | | | | | | | | | | This adds a separate method which is used to locate the action and handler callback that will be used to resolve the failure. This also adjusts the consultation of the parent retry (if any) to ensure that we handle the no parent retry case correctly. Once a decision has been made; return it and add logging that shows what is being activated and how many nodes were affected by the resolution strategy (which can be useful to know during debugging). Change-Id: I28101765fce000dd7c56b7c3a1fbcf1a4315799b
* Merge "Always return scope walker instances from `fetch_scopes_for`"Jenkins2015-03-191-6/+6
|\
| * Always return scope walker instances from `fetch_scopes_for`Joshua Harlow2015-03-181-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of returning tuples with fully expanded scopes return walker instances that internally know how to avoid recomputing the visible scopes (they do this by caching each visibility level and looking in the local cache before computing the scope and storing it in the cache). This makes the usage more uniform and avoids returning different items depending on what is found; making the code easier to follow and understand. Also makes the scope walker call to '_extract_atoms' to go via a static method so that if it is ever desired to alter what '_extract_atoms' does it can be more easily done (using standard inheritance). Change-Id: I5916838163e6be843429fe7b89a0b5622e9c2f36
* | Give the GC a breakJoshua Harlow2015-03-171-1/+1
|/ | | | | | | | | | | Instead of retaining cycles which the GC will have a harder time breaking, give it a break and use the knowledge that the associated objects will always exist while the runtime object does to just use a weakref proxy so that the GC will have a easier time breaking this cycle. Change-Id: I6241b2f33354fa58565835a5f08e5766aa601704
* Allow the storage unit to use the right scoping strategyJoshua Harlow2015-03-111-9/+16
| | | | | | | | | | | | | | | Instead of having the fetch arguments functions need to be provided a scope walker to correctly find the right arguments, which only the internals of the action engine know about provide a default scope walker (that is the same one the action engine internal uses) to the storage unit and have it be the default strategy used so that users need not know how to pass it in (which they should not care about). This allows for users to fetch the same mapped arguments as the internals of the engine will fetch. Change-Id: I1beca532b2b7c7ad98b09265a0c4477658052d16
* Allow injected atom args to be persistedJoshua Harlow2015-03-111-3/+3
| | | | | | | | | | | | | | | | | | | | | Instead of only storing injected atom arguments in memory allow for specifying those to be persisted; so that users who desire this feature can persist them (it defaults to being transient to retain the old API behavior). This also reworks the validating of engine dependencies to be more correct. It removes the validation of these dependencies from the prepare() method and moves them to a new engine validate() method; this allows users to prepare() the engine, then inject there atom non-transient arguments and then validate(); the validation would fail prior to this at preparation time since no injected arguments would exist and the user would not have the ability to inject any that target a specific atom, since the atom detail would not have been created yet (since that is populated in the prepartion method). Change-Id: I2846d0334db32a115592f850d85b206d9e6a3f07
* Remove 'SaveOrderTask' and test state in class variablesJoshua Harlow2015-01-151-2/+3
| | | | | | | | | | | | | | | | | | | Instead of saving task state in a class variable that is later introspected by further test code just remove that concept (which doesn't work in multiprocessing or worker engines which can not have access those types of shared/globally available concepts due to how they run) and use a specialized listener that can gather the same information in a more decoupled manner (and it will work in multiprocessing and worker engines correctly). This allows our engine test cases to work in those engine types which increases those engines test coverage (and future coverage and engine tests that are added). Fixes a bunch of occurrences of bug 1357117 as well that were removed during this cleanup and adjustment process... Change-Id: Ic9901de2902ac28ec255bef146be5846d18f9bfb
* Move atom action handlers to there own subfolder/submoduleJoshua Harlow2014-12-051-7/+6
| | | | | | | | | | | | | | Since action handlers are atom specific it seems appropriate to place the handlers under a specific submodule so that the action engine module/folder layout is more obvious and makes more sense. This also adds on a handles(atom) method to those handlers so that the runtime module/class can use those methods to determine which handler should handle resetting an atom to its initial state. Change-Id: I7067d91347b41613fba1492d81d68a590f22c467
* Cache immutable visible scopes in the runtime componentJoshua Harlow2014-12-021-7/+14
| | | | | | | | | | Instead of recalculating/rewalking over the visible scopes of atoms just calculate the scope once and cache it (as it currently does not change at runtime) and return the cached tuple instead to avoid the needless recreation whenever a scope is requested for a given atom. Change-Id: I47d24054c63e8620d26e7ade4baa239295daed0a
* Merge "Move scheduler and completer classes to there own modules"Jenkins2014-12-021-178/+12
|\
| * Move scheduler and completer classes to there own modulesJoshua Harlow2014-11-241-178/+12
| | | | | | | | | | | | | | | | | | To allow other components to more easily import these modules without causing circular imports split these two classes into there own modules so that they can be referenced as/where needed in the future. Change-Id: I9cf1fd1dd133be58ef521a8911d1740f1daa9950
* | Mark 'task_notifier' as renamed to 'atom_notifier'Joshua Harlow2014-11-201-4/+4
|/ | | | | | | | | | | | | | Deprecate the usage of an engines 'task_notifier' in favor of the more appropriately named 'atom_notifier' and mark the 'task_notifier' property as subject to removal in a future version. This makes the usage of this notifier more clear since it is not only used for task notification but also for retry notification (which is why naming it atom notifier is more suited to its actual usage). Change-Id: I79d58a08fd8e6d7c8990e70bdfaae415202aa929
* Revert wrapt usage until further noticeJoshua Harlow2014-11-201-4/+4
| | | | | | | | | | | Currently its usage is failing on stable/icehouse which is being fixed, but that branch is also currently broken due to a tempest failure; so until this situation is resolved take out the usage of wrapt. Part of fixes for bug 1394647 Change-Id: Ibfe21944b6e6882f19f7cf4359e8356a64200278
* Mark 'task_notifier' as renamed to 'atom_notifier'Joshua Harlow2014-11-191-4/+4
| | | | | | | | | | | | | | Deprecate the usage of an engines 'task_notifier' in favor of the more appropriately named 'atom_notifier' and mark the 'task_notifier' property as subject to removal in a future version. This makes the usage of this notifier more clear since it is not only used for task notification but also for retry notification (which is why naming it atom notifier is more suited to its actual usage). Change-Id: Idfa31a6665e43dcc9360f0aed40e27817d9e6737
* Remove direct usage of the deprecated failure locationJoshua Harlow2014-10-211-2/+3
| | | | | | | | Internally we should be using the new location and not the deprecated location wherever possible. This avoids emitting warnings messages on our own code, which is a dirty habit. Change-Id: Idac5a772eca7529d92542ada3be1cea092880e25
* Revamp the symbol lookup mechanismJoshua Harlow2014-09-081-5/+12
| | | | | | | | | | | | | | | | | | To complement the future changes in patterns we also want to allow the execution of patterns to be affected in a similar manner so that symbol lookup is no longer as confined as it was. This change adds in the following: - Symbol lookup by walking through an atoms contained scope/s. - Better error messaging when symbols are not found. - Adjusted & new tests (existing ones work). - Better logging of the symbol lookup mechanism (helpful during debugging, although it is very verbose...) Part of blueprint taskflow-improved-scoping Change-Id: Id921a4abd9bf2b7b5c5a762337f8e90e8f1fe194
* Enabled hacking checks H305 and H307Christian Berendt2014-07-171-6/+5
| | | | | | | * H305 imports not grouped correctly * H307 like imports should be grouped together Change-Id: If1dd9c89f65ede6959865a885777cb08c263eca0
* Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-9/+23
| | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Make the runner a runtime provided propertyJoshua Harlow2014-06-031-0/+5
| | | | | | | | | The runner should be a component of a runtime system and as such should be part of the runtime object as a provided property instead of something that is constructed outside of the runtime object. Change-Id: I431a377e2dc4274102a60b6502a2d0d6e08c9556
* Rename the graph analyzer to analyzerJoshua Harlow2014-06-031-8/+7
| | | | | | | | | | | | Adjust the graph analyzer to be a more generic compilation analyzer which analyzes compilation objects (which are now changed to be an object and not a named tuple) and provides utility functions ontop of that object. This helps it become possible to provide other useful analysis functions that are not directly tied to the execution graph component but can be provided ontop of other compilation components. Change-Id: I2ab08db4f566d5f329d7e79b1c50ed65aad9e4b3
* Finish factoring apart the graph_action moduleJoshua Harlow2014-05-291-0/+250
Factor out the scheduling, running and completion components of graph_action so that we can allow this to be plugged in with other types of scheduling, running and completion strategies. The newly added components are the following: - A runtime container class (serves as a holder of some small utility functions) and all the other runtime components. - A runner class that acts as the action engines run loop. - A scheduler class that schedules nodes using a provided executor and returns futures that can be used to introspect there results as they complete. - A completer class that completes nodes and futures that the scheduler started, persisting there results and doing any further post-execution analysis. Part of blueprint plug-engine Change-Id: I1dbf46654377fc34e9d90eeabf7b0062020bdc5e
View Lorry Controller Status