summaryrefslogtreecommitdiff
path: root/taskflow/patterns/graph_flow.py
Commit message (Collapse)AuthorAgeFilesLines
* Remove sixTakashi Kajinami2022-05-181-4/+3
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Update TaskFlow for networkx 2.xMichael Johnson2019-10-181-5/+6
| | | | | | | | | | | The networkx 2.x series has been out for two years now and supports python 3.6 and greater[1]. This patch updates TaskFlow to require a minimum of networkx 2.1. It also updates the code to support recent deprecation expiration introduced in the 2.4 release. [1] https://networkx.github.io/documentation/stable/news.html Change-Id: Ife31d353ba80824ebc63c8b21ee90943badc8da3
* Fix code to support networkx > 1.0Michal Arbet2018-07-111-1/+1
| | | | | | | | | | | | | With the release of NetworkX 2.0 the reporting API was moved to view/iterator model. Many methods were moved from reporting lists or dicts to iterating over the information. Methods that used to return containers now return views and methods that returned iterators have been removed in networkx. Because of this change in NetworkX 2.0 , taskflow code have to be changed also to support networkx > 2.0 Change-Id: I23c226f37bd85c1e38039fbcb302a2d0de49f333 Closes-Bug: #1778115
* Fix some misspellings in the function name and descriptionsPablo Iranzo Gómez2016-06-301-1/+1
| | | | Change-Id: I7e3451feb94b1f25b00c5e7b197bb6b527548306
* Allow for alterations in decider 'area of influence'Joshua Harlow2016-01-091-3/+25
| | | | | | | | Christmas came early. Closes-Bug: #1479466 Change-Id: I931d826690c925f022dbfffe9afb7bf41345b1d0
* Use graphs as the underlying structure of patternsJoshua Harlow2015-09-181-1/+1
| | | | | | | | | This unifies all the patterns to be graph based so that they are more uniform and there underlying constraints are more easy to understand (taskflow basically processes graphs). Change-Id: Ib2ab07c1c87165cf40a06508128010887f658391
* iter_nodes method added to flowsTimofey Durakov2015-08-281-3/+7
| | | | | | | New method allows to iterate over flow node and get access to node metadata during iteration. Change-Id: Ib0fc77f0597961602fbc3b49ba09e4df815d8230
* Improve docstrings in graph flow to denote exceptions raisedJoshua Harlow2015-08-051-4/+27
| | | | | | | | | | | Link the existing exceptions mentioned to there sphinx doc about them and also adds a section about when the 'DependencyFailure' exception is raised. Also tweaks the class docstring a little, to make it easier to understand. Change-Id: Ie4b989444c5ad73660cc61c0c3b2b1702b669087
* Add docs for u, v, decider on graph flow link methodJoshua Harlow2015-07-101-13/+28
| | | | | | | | | | | | | | The docstring is now quite important to know what the params are (they are no longer as obvious due to the decider addition) so add a useful docstring to that method. This also fixes how the targeted flow add was not passing allow **kwargs as it should have been and removes the duplicated docstrings that are in the child class (the parent class docstrings are just fine). Change-Id: Idacb7ee9f652fab4bdc762c1b49d7523e46e9a7b
* Add support for conditional executionJoshua Harlow2015-07-011-5/+13
| | | | | | | | | | | | | | | To make it possible to alter the runtime flow via a simple conditional like structure make it possible to have the graph flow link function take a decider that is expected to be some callable that will decide (via a boolean return) whether the edge should actually be traversed when running. When a decider returns false; the affected + successors will be set into the IGNORE state and they will be exempt from future runtime and scheduling decisions. Part of blueprint taskflow-conditional-execution Change-Id: Iab0ee46f86d6b8e747911174d54a7295b3fa404d
* Use 'node' terminology instead of 'item' terminologyJoshua Harlow2015-03-301-44/+47
| | | | | | | | | Since graphs are composed of 'nodes' it seems more appropriate to use that terminology (which is known to people with graph experience) instead of the non descriptive and/or standard 'item' terminology. Change-Id: I04a4521386d3cdf7e58fb9fa8cf26c00443c2cf6
* Make the graph '_unsatisfied_requires' be a staticmethodJoshua Harlow2015-03-251-6/+9
| | | | | | | | | | To make it possible for users to subclass this flow and replace this method (if they so choose to) make it a static method that is directed to the current module level function and call into the staticmethod when used (users can override staticmethods and change them in subclasses). Change-Id: I747c45636596ca5a8ad81dabcbba12ab55ce77d7
* Use ordered[set/dict] to retain orderingJoshua Harlow2015-03-111-2/+11
| | | | | | | | | | | | | | Instead of using always using a set/dict which do not retain use a ordered set and a ordered dict for requires, optional, and provides and rebind mappings types so that the ordering of these containers is maintained later when they are used. These ordering can be useful depending on the atom type (such as in a map and reduce tasks). Partial-Bug: 1357117 Change-Id: I365d11bbba4aa221bc36ca15441acecf199b4d56
* Use constants for link metadata keysJoshua Harlow2014-10-181-4/+4
| | | | | | | | | | | | Instead of using strings (which can be easy to mistype and get wrong), provide a set of constants that can be used to attach and use these keys in flows and at compilation. This also helps make it more clear what the keys do and where they are used. Change-Id: I5283b27617961136a4582bbcfce4617f05e8dd1d
* Relax the graph flow symbol constraintsJoshua Harlow2014-09-081-59/+97
| | | | | | | | | | In order to make it possible to have a symbol tree we need to relax the constraints that are being imposed by the graph flow. Part of blueprint taskflow-improved-scoping Change-Id: I2e14de2131c3ba4e3e4eb3108477583d0f02dae2
* Revamp the symbol lookup mechanismJoshua Harlow2014-09-081-1/+1
| | | | | | | | | | | | | | | | | | To complement the future changes in patterns we also want to allow the execution of patterns to be affected in a similar manner so that symbol lookup is no longer as confined as it was. This change adds in the following: - Symbol lookup by walking through an atoms contained scope/s. - Better error messaging when symbols are not found. - Adjusted & new tests (existing ones work). - Better logging of the symbol lookup mechanism (helpful during debugging, although it is very verbose...) Part of blueprint taskflow-improved-scoping Change-Id: Id921a4abd9bf2b7b5c5a762337f8e90e8f1fe194
* Be smarter about required flow symbolsJoshua Harlow2014-09-081-4/+21
| | | | | | | | | | | | | | Instead of blindly assuming all the symbols that are provided automatically work for all flows even if the flow has ordering constraints we should set the base flow class requires property to be abstract and provide flow specific properties that can do the appropriate analysis to determine what the flows unsatisfied symbol requirements actually are. Part of blueprint taskflow-improved-scoping Change-Id: Ie149c05b3305c5bfff9d9f2c05e7e064c3a6d0c7
* Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-3/+6
| | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Put provides and requires code to basic FlowIvan A. Melnikov2014-05-071-19/+3
| | | | | | | | | | Code that calculates provides and requires for flow is almost identical for all patterns, so this change makes it completely identical and puts it to the base class. Other patterns are still allowed to override these properties for sake of customization or optimization. Change-Id: I6e875e863047b5287ec727fc9a491f252f144ecf
* Add a directed graph type (new types module)Joshua Harlow2014-04-201-10/+12
| | | | | | | | | | | | | | | | Most of the utility graph functions we have can be connected to a directed graph class that itself derives (and adds on to) the networkx base class. Doing this allows for functionality that isn't exposed in networkx to be exposed in our subclass (which is a useful pattern to have). It also makes it possible (if ever needed) to replace the networkx usage in taskflow with something else if this ever becomes a major request. Change-Id: I0a825d5637236d7b5dbdbda0d426adb0183d5ba3
* Adjust the exception hierachyJoshua Harlow2014-03-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Group the exceptions into the following groups * Storage * Jobs * Execution * Other (wrapped failure here) This grouping makes it easier to understand where one type of exception should be used vs using another type of exception. Backwards incompatible changes: * StorageError -> StorageFailure * AlreadyExists -> Duplicate * WrappedFailure now inherits from Exception and not directly from TaskFlowException since it wraps arbitrary other exceptions and is not specific to taskflow. Cleanups: * JobNotFound -> NotFound * EmptyFlow -> Empty * JobAlreadyExists -> AlreadyExists * InvariantViolation (X) * ConnectionFailure (X) Change-Id: I0e1e81b513fbbc7adb8bfaa1244993e345ab70d3
* Rework graph flow unit testsIvan A. Melnikov2014-03-221-11/+5
| | | | | | | | This commit adds unit tests that check graph flow methods without executing or flattening it. Now-redundant tests from other test suites are deleted. Change-Id: I8dafe0f9b295428831eddb3f9fd48f042d2f1ffc
* Iteration over links in flow interfaceIvan A. Melnikov2014-03-211-8/+15
| | | | | | | | | In addition to iteration over its children (atoms or subflows) each pattern now provides iter_links() method that iterates over dependency links. This allows engines to treat all patterns in the same way instead knowing what structure each pattern expresses. Change-Id: I52cb5b0b501eefc8eb56a9ef5303aeb318013e11
* Add retry to Flow patternsAnastasia Karpinska2014-03-141-2/+18
| | | | | | | | Implement dependency checks when adding items to flows: - retry can't be depended on tasks from it's subflow. - retry can't provide same values as tasks or other retries Change-Id: Iffa8e673fd2de39407ae22cd38ad523d484cbba7
* Remove extraneous vim configuration commentsyangxurong2014-02-141-2/+0
| | | | | | | | | Remove line containing comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4 Change-Id: I7581cc88b8de433d5609ed06c6570b0b45c13573 Closes-Bug:#1229324
* Doc strings and comments clean-upskudriashev2014-01-261-1/+1
| | | | | | | | * Added missing period for doc strings * Correct syntax errors * Remove H402 from flake8 ignore list Change-Id: Ia8592bf99378e3658d6cca2ceb148bf9eb0b5de8
* Don't forget to return selfJoshua Harlow2013-12-061-0/+2
| | | | | | | | To retain the functionality where we can chain function calls to add, link we need to return the object we are acting on (instead of none). Change-Id: Iadf0f9d3a21b6d079c1edd29063196180f5222ed
* Targeted graph flow patternIvan A. Melnikov2013-12-021-2/+63
| | | | | | | | | This change adds graph flow like pattern that is able to ignore graph nodes (not even add them to flow) if they are not needed to run certain task. Implements blueprint targeted-graph-flow Change-Id: I57d5a1c0598bd032d77e2a262b2d9644418ce3f6
* Remove uuid from taskflow.flow.FlowIvan A. Melnikov2013-10-151-2/+2
| | | | | | | In a way our resumption works it may not correspond uuid from flow details, and so it is hardly useful. Change-Id: I090d017e2f0f8475594af94a2430a34e6ed1ce70
* Pattern comment additions/adjustmentsJoshua Harlow2013-10-121-2/+4
| | | | Change-Id: I3154d1c228474d8699f3ae4d0be2fb46406a2f41
* Add reasons as to why the edges were createdJoshua Harlow2013-10-071-13/+28
| | | | | | | | | Reasons are useful for later analysis when running as to why the edge between two nodes was created so when linking items in the graph it would be nice to assign a reason. Change-Id: I2185cf5fb3c2c07c0f5536d3b210080c6f61d5dd
* Don't allow mutating operations on the underlying graphJoshua Harlow2013-09-301-39/+62
| | | | | | | | | | Instead of allowing a direct graph return of the underlying graph_flow graph we should return a frozen version instead so that users of the returned value can not mutate the graph without going through the graph_flow pattern (which could have undesired and harmful side-effects if this occurs). Change-Id: I38b35119d6e7bd7387b8ab467eba53aee5500629
* Graph flow, sequential graph actionAnastasia Karpinska2013-09-181-115/+91
| | | | Change-Id: I07cc820aa2f37d0f9599f34efab07b28cf47ca48
* Engine, task, linear_flow unificationJoshua Harlow2013-09-051-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In order to move away from the existing flows having their own implementation of running, start moving the existing flows to be patterns that only structure tasks (and impose constraints about how the group of tasks can run) in useful ways. Let the concept of running those patterns be handled by an engine instead of being handled by the flow itself. This will allow for varying engines to be able to run flows in whichever way the engine chooses (as long as the constraints set up by the flow are observed). Currently threaded flow and graph flow are broken by this commit, since they have not been converted to being a structure of tasks + constraints. The existing engine has not yet been modified to run those structures either, work is underway to remediate this. Part of: blueprint patterns-and-engines Followup bugs that must be addressed: Bug: 1221448 Bug: 1221505 Change-Id: I3a8b96179f336d1defe269728ebae0caa3d832d7
* Split utils moduleIvan A. Melnikov2013-09-031-3/+3
| | | | | | | | | | | | | | | In this commit we split utils module into several parts: - flow_utils, with code used in running flows; - threading_utils, with code that helps in working with threads; - reflection, with code that inspects python objects metadata; - misc, with all the other code that used to live in utils.py. We also move graph_utils into taskflow.utils package. This commit just moves code around. It should not change any logic (with exception of complex_graph example). Change-Id: Iebfe45395f0ff502bc00fc7ae14829130b2c6abe
* Rename Task.__call__ to Task.executeIvan A. Melnikov2013-09-031-2/+0
| | | | | | | | | We'd like library interface to be nice and simple, which implies that it does not involve too many underscores. Also 'execute' matches with 'revert' nicely. Suggested-by: Anastasia Karpinska <akarpinska@griddynamics.com> Change-Id: Iea72e6230a32665b4236a1e7d1dcc41bd56d0418
* Add a locally running threaded flowJoshua Harlow2013-08-261-1/+1
| | | | | | | | | | | | | | | | | | | Propose a new graph flow that will run every task in the graph at the same time but will use a count down latch concept to ensure that a tasks dependents are provided before the task itself can run. This allows for tasks to run in parallel (if they have no dependents or are placed in disjoint parts of graph). Note: although this flow uses threads it is typically expected that the underlying threads would be greenthreads since python native threading is still suboptimal (for various reasons). Implements: blueprint locally-run-many-at-once Change-Id: If434abd77758aa12fc99da395a2559995305a853
* Adjust a bunch of hacking violations.Joshua Harlow2013-08-011-3/+6
| | | | | | | - Fix new hacking violations! - Add more comments! Change-Id: Ida0c1b6c98a8f45a15985b33529a1f479e595db5
* Provide the length of the flows.Joshua Harlow2013-07-101-0/+3
| | | | | | | | | Its useful to be able to check if a flow contains anything by just inspecting the len() of the flow. This can be used to avoid running a flow in the first place if nothing is in said flow. Change-Id: Ibfea7f10fc67d5dd3ddfa6f6fe6c08805562998a
* Allow graph dependencies to be manually provided.Joshua Harlow2013-07-091-39/+38
| | | | | | | | | Instead of always automatically finding dependencies of tasks it would be nice to allow developers to manually provide them if they want to. Adjust how dependencies are located due to this alteration + a few test cases for it. Change-Id: I61027be7c458cd26b163ca58e703dc413735396c
* Merge "Use the runner interface as the best task lookup."Jenkins2013-07-101-2/+2
|\
| * Use the runner interface as the best task lookup.Joshua Harlow2013-07-061-2/+2
| | | | | | | | | | | | | | | | | | | | Instead of having to examine tasks for requires or provides or optional properties/attributes, have the runner class do this instead and then have the flows which use the runner class just access the known to exist attributes directly. Change-Id: I2db4ce5296bdbd50e882f5940572a4c513e73d20
* | Merge "Unify creation/usage of uuids."Jenkins2013-07-101-2/+4
|\ \
| * | Unify creation/usage of uuids.Joshua Harlow2013-07-061-2/+4
| |/ | | | | | | | | | | | | | | | | | | | | Instead of using the name it is likely better to give the flows a uuid and use that for tracking purposes rather than using the flow name (which maybe duplicated). Also unify the usage of the job tracking id -> uuid as well as adjust the visibility of said uuids (to be read-only). Change-Id: I592800bd9e08e3a7bde33ff250a454588324f052
* | Add helper reset internals function.Joshua Harlow2013-07-081-6/+8
|/ | | | | | | | Use a little helper function to reset the internal state of the flow when needed instead of duplicating code that does this. Change-Id: I51d83538a2920c7d387ffd1756e8d99413f4077e
* Merge "Clear out before connecting."Jenkins2013-07-061-16/+23
|\
| * Clear out before connecting.Joshua Harlow2013-07-061-16/+23
| | | | | | | | | | | | | | | | Since the node set may have been altered since the last connect we want to make sure clear out the edges and the providers of items before connecting. Change-Id: I73e6b3bc5d83a82d2b13dbb6b06c66e794d66e9c
* | Ensure we document and complete correct removal.Joshua Harlow2013-07-061-10/+11
| | | | | | | | Change-Id: I156a7d53c3640f6be64532a5c6b74f71ed651d14
* | Move how resuming is done to be disconnected from jobs/flows.Joshua Harlow2013-07-061-6/+7
|/ | | | | | | | | | | Instead of having resuming tied to a job allow a workflow to have a resumption strategy object that will split its initial work order into 2 segments. One that has finished previously and one that has not finished previously. Refactor the code that previously tied a single resumption strategy to the job class and move it to a more generic resumption module folder. Change-Id: I8709cd6cb7a9deecefe8d2927be517a00acb422d
* Make connection/validation of tasks be after they are added.Joshua Harlow2013-07-061-0/+13
| | | | | | | | | Instead of having validation be immediate when tasks are added to the linear flow, follow the same paradigm as the graph_flow where only upon connection (or determination of the order) will tasks be connected/validated. Change-Id: Ia8275ec88b0229f0793819249ae59fad0a2e9935
View Lorry Controller Status