summaryrefslogtreecommitdiff
path: root/taskflow/patterns/linear_flow.py
Commit message (Collapse)AuthorAgeFilesLines
* Update TaskFlow for networkx 2.xMichael Johnson2019-10-181-3/+3
| | | | | | | | | | | The networkx 2.x series has been out for two years now and supports python 3.6 and greater[1]. This patch updates TaskFlow to require a minimum of networkx 2.1. It also updates the code to support recent deprecation expiration introduced in the 2.4 release. [1] https://networkx.github.io/documentation/stable/news.html Change-Id: Ife31d353ba80824ebc63c8b21ee90943badc8da3
* Use graphs as the underlying structure of patternsJoshua Harlow2015-09-181-15/+28
| | | | | | | | | This unifies all the patterns to be graph based so that they are more uniform and there underlying constraints are more easy to understand (taskflow basically processes graphs). Change-Id: Ib2ab07c1c87165cf40a06508128010887f658391
* iter_nodes method added to flowsTimofey Durakov2015-08-281-0/+4
| | | | | | | New method allows to iterate over flow node and get access to node metadata during iteration. Change-Id: Ib0fc77f0597961602fbc3b49ba09e4df815d8230
* Use constants for link metadata keysJoshua Harlow2014-10-181-1/+1
| | | | | | | | | | | | Instead of using strings (which can be easy to mistype and get wrong), provide a set of constants that can be used to attach and use these keys in flows and at compilation. This also helps make it more clear what the keys do and where they are used. Change-Id: I5283b27617961136a4582bbcfce4617f05e8dd1d
* Relax the linear flow symbol constraintsJoshua Harlow2014-09-081-31/+2
| | | | | | | | | | | | In order to make it possible to have a symbol tree we need to relax and remove the constraints that are being imposed by the linear constraints and later move those constraint checks and validations into the engines compilation stage. Part of blueprint taskflow-improved-scoping Change-Id: I6efdc821ff991e83572d89f56be5c678d007f9f8
* Be smarter about required flow symbolsJoshua Harlow2014-09-081-2/+13
| | | | | | | | | | | | | | Instead of blindly assuming all the symbols that are provided automatically work for all flows even if the flow has ordering constraints we should set the base flow class requires property to be abstract and provide flow specific properties that can do the appropriate analysis to determine what the flows unsatisfied symbol requirements actually are. Part of blueprint taskflow-improved-scoping Change-Id: Ie149c05b3305c5bfff9d9f2c05e7e064c3a6d0c7
* Put provides and requires code to basic FlowIvan A. Melnikov2014-05-071-19/+0
| | | | | | | | | | Code that calculates provides and requires for flow is almost identical for all patterns, so this change makes it completely identical and puts it to the base class. Other patterns are still allowed to override these properties for sake of customization or optimization. Change-Id: I6e875e863047b5287ec727fc9a491f252f144ecf
* Adjust the exception hierachyJoshua Harlow2014-03-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Group the exceptions into the following groups * Storage * Jobs * Execution * Other (wrapped failure here) This grouping makes it easier to understand where one type of exception should be used vs using another type of exception. Backwards incompatible changes: * StorageError -> StorageFailure * AlreadyExists -> Duplicate * WrappedFailure now inherits from Exception and not directly from TaskFlowException since it wraps arbitrary other exceptions and is not specific to taskflow. Cleanups: * JobNotFound -> NotFound * EmptyFlow -> Empty * JobAlreadyExists -> AlreadyExists * InvariantViolation (X) * ConnectionFailure (X) Change-Id: I0e1e81b513fbbc7adb8bfaa1244993e345ab70d3
* Merge "Linear flow: mark links and rework unit tests"Jenkins2014-03-241-2/+1
|\
| * Linear flow: mark links and rework unit testsIvan A. Melnikov2014-03-211-2/+1
| | | | | | | | | | | | | | | | | | | | - add 'invariant': True to all links produced by linear flow; - write new unit tests for basic functionality of linear flow pattern; - remove some simple now-redundant unit tests from test_flow_dependencies. Change-Id: I1f4fa8a1dfc61485555a10e8d0629a67aab1997f
* | Drop indexing operator from linear flowIvan A. Melnikov2014-03-211-3/+0
|/ | | | | | | | It is not needed. Also, using it complicates transitioning to another pattern (e.g. graph flow) and encourages bad programming practices. Change-Id: I9d0a4646a27df1a11ed436f26a0cb074b717647e
* Iteration over links in flow interfaceIvan A. Melnikov2014-03-211-0/+9
| | | | | | | | | In addition to iteration over its children (atoms or subflows) each pattern now provides iter_links() method that iterates over dependency links. This allows engines to treat all patterns in the same way instead knowing what structure each pattern expresses. Change-Id: I52cb5b0b501eefc8eb56a9ef5303aeb318013e11
* Add retry to Flow patternsAnastasia Karpinska2014-03-141-2/+5
| | | | | | | | Implement dependency checks when adding items to flows: - retry can't be depended on tasks from it's subflow. - retry can't provide same values as tasks or other retries Change-Id: Iffa8e673fd2de39407ae22cd38ad523d484cbba7
* Remove extra quote symbolIvan A. Melnikov2014-03-131-1/+1
| | | | | | Three quotes is just enough to open multiline string literal. Change-Id: I15d489f1a76be81a05b2378ef35120ee84d5eb04
* Remove extraneous vim configuration commentsyangxurong2014-02-141-2/+0
| | | | | | | | | Remove line containing comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4 Change-Id: I7581cc88b8de433d5609ed06c6570b0b45c13573 Closes-Bug:#1229324
* Doc strings and comments clean-upskudriashev2014-01-261-1/+1
| | | | | | | | * Added missing period for doc strings * Correct syntax errors * Remove H402 from flake8 ignore list Change-Id: Ia8592bf99378e3658d6cca2ceb148bf9eb0b5de8
* Merge "Skip invariant checking and adding when nothing provided"Jenkins2014-01-241-0/+3
|\
| * Skip invariant checking and adding when nothing providedJoshua Harlow2013-12-311-0/+3
| | | | | | | | Change-Id: Icc3dca675820d331fb1d6e8f3a5a4169d1b2d113
* | Exceptions cleanupIvan A. Melnikov2014-01-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Most TaskFlow exception don't have 'Exception' suffix in their names, so we drop it from the few that have it, for sake of consistency and more concise and expressive code. MissingDependencies parent object is changed from InvalidState to InvariantViolation, which is more appropriate. Unused ClosedException is dropped. Breaking change: any client code that used renamed exceptions, ClosedException or relied on catching MissingDependencies as InvariantViolationException is affected. Change-Id: I649b7760d382ccb6df333bdffd667ba6b67a431e
* | Check tasks should not provide same valuesAnastasia Karpinska2014-01-021-0/+10
|/ | | | | | | | | | Check is added to linear_flow and unordered_flow patterns that there are no two child items provide same values. Unit tests added. Change-Id: I7a293d58a962de37ba2fd1dd989bf0a218417430 Closes-Bug: #1264995
* Remove uuid from taskflow.flow.FlowIvan A. Melnikov2013-10-151-2/+2
| | | | | | | In a way our resumption works it may not correspond uuid from flow details, and so it is hardly useful. Change-Id: I090d017e2f0f8475594af94a2430a34e6ed1ce70
* Pattern comment additions/adjustmentsJoshua Harlow2013-10-121-2/+2
| | | | Change-Id: I3154d1c228474d8699f3ae4d0be2fb46406a2f41
* Allow access by indexJoshua Harlow2013-10-041-0/+3
| | | | | | | | | Since a linear flow is a linear set of tasks or subflows we can easily add in an accessor that can fetch these items by index which is useful for indexing for any post-add operations. Change-Id: Ib1690584a08e24fae29285225e6c519b06ce2ddd
* Task decorator was removed and examples updatedAnastasia Karpinska2013-09-231-2/+0
| | | | Change-Id: Ie49fe6c2f48a18130d1fd2a3aa5485cd8cee4ed4
* Verify provides and requiresIvan A. Melnikov2013-09-181-3/+19
| | | | | | | | | | | | | | When flow is being constructed we ensure that it is actually valid by checking its invariants: - for linear flow, no item should depend on items that are executed after it; - for unordered flow, items should be independent. We also add check that all necessary provides were injected to storage before actually running flow. Implements: blueprint flow-verification Change-Id: I499898f543505b7dd6f82716ae8f4011cb08f601
* Add provides and requires properties to FlowAnastasia Karpinska2013-09-111-0/+16
| | | | | | | | | | | Provides and requires properties are used to browse all required and provided values for the whole Flow. The same properties were added to the Task. Appropriate Task properties were renamed to rebind and save_as. Change-Id: I02eb02303a9701a13f1a54f06f20bbf9aebd1d04
* Engine, task, linear_flow unificationJoshua Harlow2013-09-051-252/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In order to move away from the existing flows having their own implementation of running, start moving the existing flows to be patterns that only structure tasks (and impose constraints about how the group of tasks can run) in useful ways. Let the concept of running those patterns be handled by an engine instead of being handled by the flow itself. This will allow for varying engines to be able to run flows in whichever way the engine chooses (as long as the constraints set up by the flow are observed). Currently threaded flow and graph flow are broken by this commit, since they have not been converted to being a structure of tasks + constraints. The existing engine has not yet been modified to run those structures either, work is underway to remediate this. Part of: blueprint patterns-and-engines Followup bugs that must be addressed: Bug: 1221448 Bug: 1221505 Change-Id: I3a8b96179f336d1defe269728ebae0caa3d832d7
* Split utils moduleIvan A. Melnikov2013-09-031-5/+7
| | | | | | | | | | | | | | | In this commit we split utils module into several parts: - flow_utils, with code used in running flows; - threading_utils, with code that helps in working with threads; - reflection, with code that inspects python objects metadata; - misc, with all the other code that used to live in utils.py. We also move graph_utils into taskflow.utils package. This commit just moves code around. It should not change any logic (with exception of complex_graph example). Change-Id: Iebfe45395f0ff502bc00fc7ae14829130b2c6abe
* Rename Task.__call__ to Task.executeIvan A. Melnikov2013-09-031-2/+0
| | | | | | | | | We'd like library interface to be nice and simple, which implies that it does not involve too many underscores. Also 'execute' matches with 'revert' nicely. Suggested-by: Anastasia Karpinska <akarpinska@griddynamics.com> Change-Id: Iea72e6230a32665b4236a1e7d1dcc41bd56d0418
* Notify on task reversionJoshua Harlow2013-08-291-8/+2
| | | | | | | | | When a runner and its associated task are reverted we should fire off notifications that this is occuring so that others can perform various activities when this occurs. Change-Id: I3fc797255e5cacd0d5cff9cb6ec444ccb9e7ac2e
* Have runner keep the exceptionJoshua Harlow2013-08-291-4/+5
| | | | | | | | | If the runner when executing caused an exception to be thrown then it is nice to keep the exc_info() associated with that exception for later usage in stack-traces, rethrowing or similar... Change-Id: I32080eae06977acdfb367756693069245a8db5d7
* Add a locally running threaded flowJoshua Harlow2013-08-261-8/+23
| | | | | | | | | | | | | | | | | | | Propose a new graph flow that will run every task in the graph at the same time but will use a count down latch concept to ensure that a tasks dependents are provided before the task itself can run. This allows for tasks to run in parallel (if they have no dependents or are placed in disjoint parts of graph). Note: although this flow uses threads it is typically expected that the underlying threads would be greenthreads since python native threading is still suboptimal (for various reasons). Implements: blueprint locally-run-many-at-once Change-Id: If434abd77758aa12fc99da395a2559995305a853
* File movementsJoshua Harlow2013-08-171-1/+1
| | | | | | | | | | | | In order to rework some of the persistence layer I would like to move around some of the files first, keeping job specifics in a jobs folder. Having some of the items which are root level taskflow items (flow, task) be at the root of the hiearchy. Also for now until the celery work is commited move that, since it doesn't make sense in backends anyway. Change-Id: If6c149710b40f70d4ec69ee8e523defe8f5e766d
* Added Memory Persistence API and Generic Datatypeskchenweijie2013-08-091-2/+2
| | | | | | | | | | Added a generic persistence API for JobBoards, Jobs, LogBooks, Workflows, FlowDetails, Tasks, and TaskDetails. The currently implemented backends are in-memory. Also implemented generic return types (listed above) for the generic API. Change-Id: I6d09860ee08a900faf2c213a230429bf9e0dec01
* Merge "Quiet the provider linking."Jenkins2013-08-021-2/+0
|\
| * Quiet the provider linking.Joshua Harlow2013-08-021-2/+0
| | | | | | | | | | | | | | | | | | The log output actually seems to be less useful when this is integrated into cinder than I thought it would be so lets turn it off. Change-Id: I2832f6bc9153153fcf6e76e65d8430ba536539e8
* | Merge "Deep-copy not always possible."Jenkins2013-08-021-11/+1
|\ \
| * | Deep-copy not always possible.Joshua Harlow2013-08-021-11/+1
| |/ | | | | | | | | | | | | | | | | | | | | In working with cinder it appears that at last cinder is using objects which can not be deep copied (sqlalchemy objects) and which actually cause the deepcopy routine to fail. To get around this avoid the deepcopy until proven its really needed. Change-Id: Ia7b7c49e82c2151351cc0f78ecb1531c35540652
* | Adjust a bunch of hacking violations.Joshua Harlow2013-08-011-4/+5
|/ | | | | | | - Fix new hacking violations! - Add more comments! Change-Id: Ida0c1b6c98a8f45a15985b33529a1f479e595db5
* Provide the length of the flows.Joshua Harlow2013-07-101-0/+3
| | | | | | | | | Its useful to be able to check if a flow contains anything by just inspecting the len() of the flow. This can be used to avoid running a flow in the first place if nothing is in said flow. Change-Id: Ibfea7f10fc67d5dd3ddfa6f6fe6c08805562998a
* Allow graph dependencies to be manually provided.Joshua Harlow2013-07-091-7/+0
| | | | | | | | | Instead of always automatically finding dependencies of tasks it would be nice to allow developers to manually provide them if they want to. Adjust how dependencies are located due to this alteration + a few test cases for it. Change-Id: I61027be7c458cd26b163ca58e703dc413735396c
* Merge "Use the runner interface as the best task lookup."Jenkins2013-07-101-2/+2
|\
| * Use the runner interface as the best task lookup.Joshua Harlow2013-07-061-2/+2
| | | | | | | | | | | | | | | | | | | | Instead of having to examine tasks for requires or provides or optional properties/attributes, have the runner class do this instead and then have the flows which use the runner class just access the known to exist attributes directly. Change-Id: I2db4ce5296bdbd50e882f5940572a4c513e73d20
* | Merge "Unify creation/usage of uuids."Jenkins2013-07-101-6/+7
|\ \
| * | Unify creation/usage of uuids.Joshua Harlow2013-07-061-6/+7
| |/ | | | | | | | | | | | | | | | | | | | | Instead of using the name it is likely better to give the flows a uuid and use that for tracking purposes rather than using the flow name (which maybe duplicated). Also unify the usage of the job tracking id -> uuid as well as adjust the visibility of said uuids (to be read-only). Change-Id: I592800bd9e08e3a7bde33ff250a454588324f052
* | Add helper reset internals function.Joshua Harlow2013-07-081-7/+7
|/ | | | | | | | Use a little helper function to reset the internal state of the flow when needed instead of duplicating code that does this. Change-Id: I51d83538a2920c7d387ffd1756e8d99413f4077e
* Merge "Ensure we document and complete correct removal."Jenkins2013-07-061-9/+18
|\
| * Ensure we document and complete correct removal.Joshua Harlow2013-07-061-9/+18
| | | | | | | | Change-Id: I156a7d53c3640f6be64532a5c6b74f71ed651d14
* | Pass runners instead of task objects/uuids.Joshua Harlow2013-07-061-8/+4
|/ | | | | | | | Runners contain tasks and provide the identifying characteristics for the tasks, pass these along to the underlying components instead of tasks. Change-Id: I795a52d3a218e508c38ea209d757750ae6a936ea
* Move how resuming is done to be disconnected from jobs/flows.Joshua Harlow2013-07-061-52/+41
| | | | | | | | | | | Instead of having resuming tied to a job allow a workflow to have a resumption strategy object that will split its initial work order into 2 segments. One that has finished previously and one that has not finished previously. Refactor the code that previously tied a single resumption strategy to the job class and move it to a more generic resumption module folder. Change-Id: I8709cd6cb7a9deecefe8d2927be517a00acb422d
View Lorry Controller Status