summaryrefslogtreecommitdiff
path: root/taskflow/patterns
Commit message (Collapse)AuthorAgeFilesLines
...
* Graph flow, sequential graph actionAnastasia Karpinska2013-09-181-115/+91
| | | | Change-Id: I07cc820aa2f37d0f9599f34efab07b28cf47ca48
* Verify provides and requiresIvan A. Melnikov2013-09-182-17/+53
| | | | | | | | | | | | | | When flow is being constructed we ensure that it is actually valid by checking its invariants: - for linear flow, no item should depend on items that are executed after it; - for unordered flow, items should be independent. We also add check that all necessary provides were injected to storage before actually running flow. Implements: blueprint flow-verification Change-Id: I499898f543505b7dd6f82716ae8f4011cb08f601
* Remove threaded and distributed flowsJoshua Harlow2013-09-132-726/+0
| | | | | | | | Both of these are currently being reworked and it doesn't make much sense to keep them around in the code tree at this point. Change-Id: I2fa257ae91d117effccbe6298d29e002a64e08b3
* Add provides and requires properties to FlowAnastasia Karpinska2013-09-112-0/+30
| | | | | | | | | | | Provides and requires properties are used to browse all required and provided values for the whole Flow. The same properties were added to the Task. Appropriate Task properties were renamed to rebind and save_as. Change-Id: I02eb02303a9701a13f1a54f06f20bbf9aebd1d04
* Engine, task, linear_flow unificationJoshua Harlow2013-09-054-260/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In order to move away from the existing flows having their own implementation of running, start moving the existing flows to be patterns that only structure tasks (and impose constraints about how the group of tasks can run) in useful ways. Let the concept of running those patterns be handled by an engine instead of being handled by the flow itself. This will allow for varying engines to be able to run flows in whichever way the engine chooses (as long as the constraints set up by the flow are observed). Currently threaded flow and graph flow are broken by this commit, since they have not been converted to being a structure of tasks + constraints. The existing engine has not yet been modified to run those structures either, work is underway to remediate this. Part of: blueprint patterns-and-engines Followup bugs that must be addressed: Bug: 1221448 Bug: 1221505 Change-Id: I3a8b96179f336d1defe269728ebae0caa3d832d7
* Merge "Combine multiple exceptions into a linked one"Jenkins2013-09-041-5/+6
|\
| * Combine multiple exceptions into a linked oneJoshua Harlow2013-09-041-5/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | When multiple tasks running at the same time throw exceptions previously we would not be able to rethrow all the combined ones which was suboptimal. Instead of doing that create a combined linked exception that can retain all of the exceptions that were thrown and rethrow that in the situation where more than one task fails. Change-Id: I8f882e0d58caa189d6bff2e33b0bc30c4cee553d
* | Initial implementation of action-based engineIvan A. Melnikov2013-09-031-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is first in series of changes that provides separation of flows into engines and pattens. Patterns define structure, while engine define how the task should be run according to said structure. This change adds the basic patterns and a very simple engine that is able to run flow defined with patterns by converting it to recursive structure of so-called 'actions'. For simplicity and ease of review certain major features like resumption, saving and passing on task results and notifications are left out for farther changes. Partially implements blueprint patterns-and-engines Co-authored-by: Anastasia Karpinska <akarpinska at griddynamics.com> Change-Id: I68515d8a5b30d5d047bf9beb67cc3e2111175190
* | Merge "Further updates to update.py"Jenkins2013-09-031-3/+0
|\ \
| * | Further updates to update.pyJoshua Harlow2013-09-031-3/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Adjust how import reforming is done to use a function to reform the import line (after modification). 2. Clean the old copy of taskflow (if it exists) before placing a new copy in its place, this is very useful to avoid stale files. 3. Move some constants to be offical global constants. 4. Show exactly what modules are being used from the containing projects openstack.common. 5. Fix how the distributed flow was breaking imports due to the location/usage of logbook in that module no longer being functionally correct. 6. Add a header to each file copied/created that tells people not to modify them directly. Change-Id: I6387722e092b93418bd63b722a95a8efcaef788d
* | | Split utils moduleIvan A. Melnikov2013-09-034-25/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In this commit we split utils module into several parts: - flow_utils, with code used in running flows; - threading_utils, with code that helps in working with threads; - reflection, with code that inspects python objects metadata; - misc, with all the other code that used to live in utils.py. We also move graph_utils into taskflow.utils package. This commit just moves code around. It should not change any logic (with exception of complex_graph example). Change-Id: Iebfe45395f0ff502bc00fc7ae14829130b2c6abe
* | | Rename Task.__call__ to Task.executeIvan A. Melnikov2013-09-032-4/+0
|/ / | | | | | | | | | | | | | | | | We'd like library interface to be nice and simple, which implies that it does not involve too many underscores. Also 'execute' matches with 'revert' nicely. Suggested-by: Anastasia Karpinska <akarpinska@griddynamics.com> Change-Id: Iea72e6230a32665b4236a1e7d1dcc41bd56d0418
* | Notify on task reversionJoshua Harlow2013-08-292-9/+4
| | | | | | | | | | | | | | | | | | When a runner and its associated task are reverted we should fire off notifications that this is occuring so that others can perform various activities when this occurs. Change-Id: I3fc797255e5cacd0d5cff9cb6ec444ccb9e7ac2e
* | Have runner keep the exceptionJoshua Harlow2013-08-292-15/+13
|/ | | | | | | | | If the runner when executing caused an exception to be thrown then it is nice to keep the exc_info() associated with that exception for later usage in stack-traces, rethrowing or similar... Change-Id: I32080eae06977acdfb367756693069245a8db5d7
* Add a locally running threaded flowJoshua Harlow2013-08-264-10/+661
| | | | | | | | | | | | | | | | | | | Propose a new graph flow that will run every task in the graph at the same time but will use a count down latch concept to ensure that a tasks dependents are provided before the task itself can run. This allows for tasks to run in parallel (if they have no dependents or are placed in disjoint parts of graph). Note: although this flow uses threads it is typically expected that the underlying threads would be greenthreads since python native threading is still suboptimal (for various reasons). Implements: blueprint locally-run-many-at-once Change-Id: If434abd77758aa12fc99da395a2559995305a853
* File movementsJoshua Harlow2013-08-171-1/+1
| | | | | | | | | | | | In order to rework some of the persistence layer I would like to move around some of the files first, keeping job specifics in a jobs folder. Having some of the items which are root level taskflow items (flow, task) be at the root of the hiearchy. Also for now until the celery work is commited move that, since it doesn't make sense in backends anyway. Change-Id: If6c149710b40f70d4ec69ee8e523defe8f5e766d
* Added Memory Persistence API and Generic Datatypeskchenweijie2013-08-092-216/+2
| | | | | | | | | | Added a generic persistence API for JobBoards, Jobs, LogBooks, Workflows, FlowDetails, Tasks, and TaskDetails. The currently implemented backends are in-memory. Also implemented generic return types (listed above) for the generic API. Change-Id: I6d09860ee08a900faf2c213a230429bf9e0dec01
* Merge "Quiet the provider linking."Jenkins2013-08-021-2/+0
|\
| * Quiet the provider linking.Joshua Harlow2013-08-021-2/+0
| | | | | | | | | | | | | | | | | | The log output actually seems to be less useful when this is integrated into cinder than I thought it would be so lets turn it off. Change-Id: I2832f6bc9153153fcf6e76e65d8430ba536539e8
* | Merge "Deep-copy not always possible."Jenkins2013-08-021-11/+1
|\ \
| * | Deep-copy not always possible.Joshua Harlow2013-08-021-11/+1
| |/ | | | | | | | | | | | | | | | | | | | | In working with cinder it appears that at last cinder is using objects which can not be deep copied (sqlalchemy objects) and which actually cause the deepcopy routine to fail. To get around this avoid the deepcopy until proven its really needed. Change-Id: Ia7b7c49e82c2151351cc0f78ecb1531c35540652
* | Merge "Adjust a bunch of hacking violations."Jenkins2013-08-025-17/+57
|\ \ | |/ |/|
| * Adjust a bunch of hacking violations.Joshua Harlow2013-08-015-17/+57
| | | | | | | | | | | | | | - Fix new hacking violations! - Add more comments! Change-Id: Ida0c1b6c98a8f45a15985b33529a1f479e595db5
* | Fix minor code styleChangbin Liu2013-07-301-1/+1
|/ | | | Change-Id: Ib051581cd820cbb478b95b1b4421adb2b577567d
* Merge "Provide the length of the flows."Jenkins2013-07-242-0/+6
|\
| * Provide the length of the flows.Joshua Harlow2013-07-102-0/+6
| | | | | | | | | | | | | | | | | | Its useful to be able to check if a flow contains anything by just inspecting the len() of the flow. This can be used to avoid running a flow in the first place if nothing is in said flow. Change-Id: Ibfea7f10fc67d5dd3ddfa6f6fe6c08805562998a
* | Parents should be frozen after creation.Joshua Harlow2013-07-091-2/+2
|/ | | | | | | Use a tuple to ensure that the parents list can not be mutated after flow creation. Change-Id: Ic2a0ed73a2a7211184deb92d57720ff417429dca
* Allow graph dependencies to be manually provided.Joshua Harlow2013-07-093-48/+43
| | | | | | | | | Instead of always automatically finding dependencies of tasks it would be nice to allow developers to manually provide them if they want to. Adjust how dependencies are located due to this alteration + a few test cases for it. Change-Id: I61027be7c458cd26b163ca58e703dc413735396c
* Merge "Use the runner interface as the best task lookup."Jenkins2013-07-103-8/+7
|\
| * Use the runner interface as the best task lookup.Joshua Harlow2013-07-063-8/+7
| | | | | | | | | | | | | | | | | | | | Instead of having to examine tasks for requires or provides or optional properties/attributes, have the runner class do this instead and then have the flows which use the runner class just access the known to exist attributes directly. Change-Id: I2db4ce5296bdbd50e882f5940572a4c513e73d20
* | Merge "Unify creation/usage of uuids."Jenkins2013-07-104-20/+38
|\ \
| * | Unify creation/usage of uuids.Joshua Harlow2013-07-064-20/+38
| |/ | | | | | | | | | | | | | | | | | | | | Instead of using the name it is likely better to give the flows a uuid and use that for tracking purposes rather than using the flow name (which maybe duplicated). Also unify the usage of the job tracking id -> uuid as well as adjust the visibility of said uuids (to be read-only). Change-Id: I592800bd9e08e3a7bde33ff250a454588324f052
* | Add helper reset internals function.Joshua Harlow2013-07-082-13/+15
|/ | | | | | | | Use a little helper function to reset the internal state of the flow when needed instead of duplicating code that does this. Change-Id: I51d83538a2920c7d387ffd1756e8d99413f4077e
* Merge "Clear out before connecting."Jenkins2013-07-061-16/+23
|\
| * Clear out before connecting.Joshua Harlow2013-07-061-16/+23
| | | | | | | | | | | | | | | | Since the node set may have been altered since the last connect we want to make sure clear out the edges and the providers of items before connecting. Change-Id: I73e6b3bc5d83a82d2b13dbb6b06c66e794d66e9c
* | Merge "Ensure we document and complete correct removal."Jenkins2013-07-062-19/+29
|\ \
| * | Ensure we document and complete correct removal.Joshua Harlow2013-07-062-19/+29
| | | | | | | | | | | | Change-Id: I156a7d53c3640f6be64532a5c6b74f71ed651d14
* | | Pass runners instead of task objects/uuids.Joshua Harlow2013-07-062-18/+13
|/ / | | | | | | | | | | | | | | Runners contain tasks and provide the identifying characteristics for the tasks, pass these along to the underlying components instead of tasks. Change-Id: I795a52d3a218e508c38ea209d757750ae6a936ea
* | Move how resuming is done to be disconnected from jobs/flows.Joshua Harlow2013-07-065-60/+208
|/ | | | | | | | | | | Instead of having resuming tied to a job allow a workflow to have a resumption strategy object that will split its initial work order into 2 segments. One that has finished previously and one that has not finished previously. Refactor the code that previously tied a single resumption strategy to the job class and move it to a more generic resumption module folder. Change-Id: I8709cd6cb7a9deecefe8d2927be517a00acb422d
* Make connection/validation of tasks be after they are added.Joshua Harlow2013-07-062-2/+39
| | | | | | | | | Instead of having validation be immediate when tasks are added to the linear flow, follow the same paradigm as the graph_flow where only upon connection (or determination of the order) will tasks be connected/validated. Change-Id: Ia8275ec88b0229f0793819249ae59fad0a2e9935
* Add helper to do notificationJoshua Harlow2013-07-062-54/+45
| | | | | | | | | Instead of embedding the notification calls into the ordered flow object provide a better util helper class that can be used to accomplish the notification registration and activation. Change-Id: Ia4d21cb126e8ac12c227a5c0aca467d96daa3987
* Store results by add() uuid instead of in array format.Joshua Harlow2013-06-281-4/+5
| | | | | | | | | | | Instead of storing the results in the previous linear format, just store the results indexed by the uuid returned from the add() method instead. This allows for easier access to individual task result as well as makes it easier to support non-linear result insertation. Change-Id: Id7461cb74eb1a380b05cf23643e7120df5682224
* Integrate better locking and a runner helper class.Joshua Harlow2013-06-283-169/+151
| | | | | | | | | | | | | | | | | | | | | | Ensure that when a linear flow or derivatives is running that it can not be modified by another thread at the same time it is running by putting a lock around sensitive functions. Also instead of using the raw task objects themselves integrate a helper 'runner' class that provides useful functionality that occurs before its tasks runs as well as member variables that are associated with the contained task. This helper class currently provides the following: - A uuid that can be returned to callers of the add method to identify there task (and later its results), allowing for multiple of the same tasks to be added. - Automatic extraction of the needed required and optional inputs for the contained task. Change-Id: Ib01939a4726155a629e4b4703656b9067868d8f3
* Cleaning up various components.Joshua Harlow2013-06-284-334/+391
| | | | | | | | | | | 1. In order to aid future features its useful to split split apart the linear_flow and have a good base class that is not connected to a ordered or linear flow. 2. Have the graph flow inherit from the linear flow since it is performing a linear run using a topological ordering of the tasks. Change-Id: I287cd4a666caf09ca3a4cdf1d53814e9c6feb765
* Move some of the ordered flow helper classes to utils.Joshua Harlow2013-06-271-30/+2
| | | | | | | Allow other flow types easier access to these helper classes by moving them to the shared utils file. Change-Id: I001754efc25b025e8070493d5861ffaedacafc7b
* Allow instance methods to be wrapped and unwrapped correctly.Joshua Harlow2013-06-272-10/+17
| | | | Change-Id: Ia29e0e8a54fa815dec88016a9444fe7cfad7b8ac
* Fix most of the hacking rulesAngus Salkeld2013-06-261-6/+6
| | | | | | Hacking rules: H302, H303, H304, H401, H404 Change-Id: I38e62696724a99c5ebe74d95d477999bd91a2c9a
* Fix all flake8 E* and F* errorsAngus Salkeld2013-06-261-1/+1
| | | | | | This was mostly unused modules or local variables. Change-Id: I39f9598ba8b64101a10f80bae448d27272abeddd
* Fix the current flake8 errorsAngus Salkeld2013-06-263-2/+4
| | | | | | | | The test pip ordering was resulting in the pep8 rules not really running so this fixes that up and the resulting errors so we can get a clean run. Change-Id: Icc27be05cd513c4f04135c7e768007dbf9d07b10
* Merge "Add the task to the accumulator before running."Jenkins2013-06-241-12/+21
|\
View Lorry Controller Status