summaryrefslogtreecommitdiff
path: root/taskflow/tests/utils.py
Commit message (Collapse)AuthorAgeFilesLines
* Merge "Change StrictRedis usage to Redis"Zuul2023-02-171-1/+1
|\
| * Change StrictRedis usage to RedisTobias Urdin2022-09-291-1/+1
| | | | | | | | | | | | | | The StrictRedis class is only an alias for Redis in >= 3.0.0 Change-Id: Ief27531f120a50805053c214cb61bb4151678d70
* | Remove sixTakashi Kajinami2022-05-181-10/+5
|/ | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Fix process based executor task proxying-back eventsJoshua Harlow2017-07-111-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Let's dive into what the problem is here. First a description of what happens to a task that is to be executed in a external (but local) process via the process executor mechanism. When a task is about to be sent to execute in the external (but local) process its first cloned, this is mainly done so that its notification callbacks can be altered in a safe manner (ie not altering the original task object to do this) and that clone has its notifier emptied out. What replaces the clone's notifier callbacks though is a new object (that has a __call__ method so it looks like just another callback) that will send messages to the parent process (the one that has the engine in it) over a secure(ish) channel whenever the local task triggers its notifier notify() method. This allows for callbacks in the parent process to get triggered because once the messages recieved the original tasks notifier object has its notify() method called (therefore those callbacks do not really know the task they are getting messages from is executing out of process). The issue though is that if the ANY(*) event type is registered due to how it works in the notifier is that if the child/cloned notifier has the ANY event type registered and the cloned task calls notify() with a specific event this will cause the ANY callback (in the clone) to transmit a message *and* it will cause the *specific* event callback to also transmit a message back to the parent process. On the engine process side it will get 2 messages and trigger the callbacks 3 times (twice for the specific event callback because how the local notifier has the ANY callback registered and one more time when the local process also sends the same event based on its registration of the ANY event in the child process). This is not what is expected (the message rcved on the engine process should only trigger one callback to get triggered if the engine process task has no ANY callback registered or two engine process callbacks to get triggered if the engine process task has the ANY callback registered). Closes-Bug: #1537948 Change-Id: I271bf1f23ad73df6c177cf00fd902c4881ba44ae
* Some classes not define __ne__() built-in functionJi-Wei2016-08-311-0/+3
| | | | | | | | | | Some classes defines __eq__() built-in function, but does not define __ne__() built-in function, so self.assertEqual works but self.assertNotEqual does not work at all in this test case in python2. This patch fixes it. Change-Id: I3e4f213081268bad44583a63a84795d39094117f Closes-Bug: #1586268
* Instead of a multiprocessing queue use sockets via asyncoreJoshua Harlow2016-05-241-8/+7
| | | | | | | | | | | | | | | | | | | | | | For a local process based executor usage currently to ensure that task emitted notifications are proxied we use the multi processing library and use its queue concept. This sadly creates a proxy process that gets associated, and this proxy process handles the queue and messages sent to and from it. Instead of doing this we can instead just create a temporary local socket using a random socket and have tasks (which are running in different processes) use that to communicate back any emitted notifications instead (and we can use the asyncore module to handle the emitted notifications since it handles the lower level socket reading, polling and dispatching). To ensure that the socket created is somewhat secure we use a similar process as the multi-processing library uses where we sign all messages with a hmac that uses a one time key that only the main process and the child process know about (and reject any messages that do not validate using this key). Change-Id: Iff9180054bf14495e5667af00ae2fafbdbc23791
* Make conductor.stop stop the running engine gracefullyGreg Hill2016-05-031-0/+6
| | | | | | | | | | | | Previously stopping the conductor would only prevent it from accepting new jobs. This change makes it abort the current job by telling the engine to stop processing tasks after the current ones finish. This allows for conductors to gracefully exit when receiving a kill signal (although the signal handling is not done automatically yet). Change-Id: Ie6ddcbb2df4508ad1e3f6698c6f4cb2fc26a278f
* Allow for revert to have a different argument list from executeGreg Hill2016-02-261-0/+8
| | | | | | | | | Also allows for people to create Atom's with a different rebind or requires structure for the revert method, if desired. Implements blueprint: seperate-revert-args Change-Id: Ie7d13c8000ef08ff303481d486d1ba1cfbdeea44
* Avoid adding 1 to a failure (if it gets triggered)Joshua Harlow2015-07-271-1/+5
| | | | | | | | | | Sometimes the CI gate times out and instead of returning normal results a failure object is returned, so be more careful on adding integers to those objects. Closes-Bug: 1478744 Change-Id: Ibdb9d30266d2a7f3bfeacc39e74cf61b44025a56
* Retain atom 'revert' result (or failure)Joshua Harlow2015-07-101-0/+9
| | | | | | | | | When a atom is reverted it can be useful to retain the result of that 'revert' method being called, so that it can be later analyzed (or used for various purposes) so adjust the storage, and actions to enable it to be stored. Change-Id: I38a9a5f3bf7550e924468bb4a86652cb8beb306c
* Build-out + test a redis backed jobboardJoshua Harlow2015-06-291-0/+14
| | | | | | Part of blueprint taskflow-redis-jobs Change-Id: I7c94e2201c5d933c8a1ec73fc0cf705962e5eef6
* Remove 2.6 classifier + 2.6 compatibility codeJoshua Harlow2015-06-211-2/+2
| | | | | | | | Fixes bug 1445827 Depends-On: I02e3c9aacef0b295a2f823a5cbaf11768a90cb82 Change-Id: I1db681803598ac1bc917fd74a99458bc61edf3f1
* Add a test that checks for task result visibilityJoshua Harlow2015-05-291-0/+7
| | | | | | | | | | Now that a task can provide the same thing that it requires, having a nice little test that ensures what is gotten is expected in a nice manner should be done to make sure what a task will be getting for its arguments respect the scoping that is defined by the flow ordering. Change-Id: I1cdc5c3bd3f56b39f20f28f51c36017405150cea
* Fix lookup scoping multi-match orderingJoshua Harlow2015-02-241-0/+7
| | | | | | | | | | | | | | | | | When look-up is occurring the possible provider ordering is used instead of the scope returned provider ordering. This causes incorrect matches when look-up argument names that are produced by multiple providers (at the same scope level) all providing the same requirement name. The scope order should be enforced as the de-facto order and not the order that the storage unit finds (which is hash based and varies depending on hash ordering). Closes-Bug: #1425326 Change-Id: I15f1ee5515758bdc470c0f7dd7a2f616923e5628
* adding check for str/unicode type in requiresMin Pae2015-02-151-0/+6
| | | | | | | | | | | | When the requires argument for an Atom is passed in as a string, each character of the string is iterated over to build up a requirement list. This works for simple one letter argument names but not for long argument names. Added check for str and unicode types to prevent iterating over a string. Change-Id: Ida584221b48966d26935fb2ede0075aabb7ce972
* Improve upon/adjust/move around new optional exampleJoshua Harlow2015-02-111-0/+6
| | | | | | | | | | Instead of having the optional requirements example be a example that is itself a unittest just move the example to be an actual unit test that gets tested using the various engine types and change the example to be something slightly different (but shows the same kind of usage information). Change-Id: Ia03a81a6be636c501a35e7e290f587f7d05f8b30
* Add a capturing listener (for test or other usage)Joshua Harlow2015-02-031-47/+16
| | | | | | | | | | | To enable those that are using taskflow to easily attach to an engine (say in a unit test) and capture all the transitions, details and state changes in a accumulating 'values' list provide a capturing listener that does just this (and change the one used for internal testing to use it) so that others may also benefit from the same functionality. Change-Id: I7a35e282dd4a6b3e14277bea2af1f275615bd212
* Remove 'SaveOrderTask' and test state in class variablesJoshua Harlow2015-01-151-28/+54
| | | | | | | | | | | | | | | | | | | Instead of saving task state in a class variable that is later introspected by further test code just remove that concept (which doesn't work in multiprocessing or worker engines which can not have access those types of shared/globally available concepts due to how they run) and use a specialized listener that can gather the same information in a more decoupled manner (and it will work in multiprocessing and worker engines correctly). This allows our engine test cases to work in those engine types which increases those engines test coverage (and future coverage and engine tests that are added). Fixes a bunch of occurrences of bug 1357117 as well that were removed during this cleanup and adjustment process... Change-Id: Ic9901de2902ac28ec255bef146be5846d18f9bfb
* Remove direct usage of the deprecated failure locationJoshua Harlow2014-10-211-2/+2
| | | | | | | | Internally we should be using the new location and not the deprecated location wherever possible. This avoids emitting warnings messages on our own code, which is a dirty habit. Change-Id: Idac5a772eca7529d92542ada3be1cea092880e25
* Use and verify event and latch wait() return using timeoutsJoshua Harlow2014-10-181-9/+9
| | | | | | | | | | | | | Instead of blocking up the whole test suite when a latch or event was not decremented to its desired value (or not set for an event) we should use a reasonably high value that we use when waiting for those actions to occur and verify that when those wait() functions return that we have reached the desired state and if not either raise an exception or stop further testing. Fixes bug 1363739 Change-Id: I8b40282ac2db9cabd48b0b65c8a2a49610d77c4f
* Merge "Expose only `ensure_atom` from storage"Jenkins2014-10-181-0/+10
|\
| * Expose only `ensure_atom` from storageJoshua Harlow2014-09-271-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Move the storage ensuring logic from being split across the engine and the storage layer and expose only a single `ensure_atom` function that does the work instead. This also removes the access to the `ensure_task` and `ensure_retry` methods as the internals of the `ensure_atom` function is now the only location that needs to use these two functions. This reduces the need to do type specific atom checks in the non-storage components (which we want to reduce overall). Breaking change: removes the public methods named `ensure_task` and `ensure_retry` (which should not be used externally anyway) from the storage object and makes those internal/private methods instead. Change-Id: I3a0f1f0dd777a1633b4937e16b50030275c84d1d
* | Switch to a custom NotImplementedError derivativeJoshua Harlow2014-10-081-1/+3
|/ | | | | | | | | | When a feature or method is not implemented it's useful to throw our own derivative of a NotImplementedError error so that we can distingush these types of errors vs actual usage of NotImplementedError which could be thrown from driver or user code. Change-Id: I8d5dfb56a254f315c5509dc600a078cfef2dde0b
* Tweak engine iteration 'close-up shop' runtime pathJoshua Harlow2014-09-041-0/+14
| | | | | | | | | | | | | | | | | | | | | | | 1. Have the runner yield the final set of failures instead of raising them, this allows the same yield syntax to be used for all exit points that the runner run_iter() produces and now raise failures from the main engine run loop to match this change. 2. Use a context manager instead of try/finally to start and stop the action engines task executor (teenie niceness...) 3. When the engine run_iter() is used and the generator that is returned is closed, instead of breaking from the run loop, which can leave running tasks incomplete instead continue running and signal to the runner that the engine has suspended itself. This ensures that the running atoms are not lost when the generator from run_iter() is closed (for whatever reason) before finishing. Also adds a bunch of useful tests that directly test the runner instead of the indirect testing that we were doing before. Fixes bug 1361013 Change-Id: I1b598e26f0b3877c8f7004f87bacdb7f5e9c9897
* Cleanup zookeeper integration testingJoshua Harlow2014-05-091-0/+25
| | | | | | | | | | | | In the persistence tests only use zake when zookeeper is not available (and of the right version). When zookeeper is available skip running zake. In the jobboard tests split out the tests which are not specific for zake into a base class (allowing for a future commit to add a zookeeper integration test). Change-Id: I50d51639a7f6c03c29d559c485676fddb9a7cf20
* Implement run iterationsJoshua Harlow2014-05-071-1/+1
| | | | | | | | | | | | | Instead of blocking the caller when they call run() allow there to be a new api run_iter() that will yield back the engine state transitions while running. This allows for a engine user to do alternate work while an engine is running (and come back to yield on there own time). Implements blueprint iterable-execution Change-Id: Ibb48c6c5618c97c59a6ab170dab5233ed47e5554
* Revert "Move taskflow.utils.misc.Failure to its own module"Jeremy Stanley2014-03-311-0/+16
| | | | | | | | | | | | This reverts commit 42ca240e8157b840c298d14fbf478ae570376633 which was a breaking change in a library consumed by other OpenStack projects with no deprecation or backwards compatibility considerations. It was able to merge because openstack/taskflow is apparently not yet part of the integrated gate via the proposed I202f4809afd689155e2cc4a00fc704fd772a0e92 change. Change-Id: I96cf36dc317499df91e43502efc85221f8177395 Closes-Bug: #1300161
* Move taskflow.utils.misc.Failure to its own moduleIvan A. Melnikov2014-03-311-16/+0
| | | | | | | | | | Failure class is important part of TaskFlow API, so it should be more visible and accessible. Breaking change: any client that used taskflow.utils.misc.Failure should be updated. Change-Id: Ib30000c9246bbcb227b34dfb0aba4d0b950bf926
* Add timeout to WaitForOneFromTaskIvan A. Melnikov2014-03-261-1/+7
| | | | | | We should never wait forever for anything. Change-Id: Icaa61febbf4745206abbe5bf9255c9401f341e71
* Merge "Drop obsolete test_unordered_flow"Jenkins2014-03-241-20/+0
|\
| * Drop obsolete test_unordered_flowIvan A. Melnikov2014-03-211-20/+0
| | | | | | | | | | | | Action engine tests were modified to keep few cases covered. Change-Id: Ibfcdd5378605532a4eb3fb1a43a70532a771ea16
* | tests: Fix WaitForOneFromTask constructor parameter introspectionIvan A. Melnikov2014-03-221-4/+3
|/ | | | | | | | Strings are iterable, so the check in WaitForOneFromTask was effectively useless. This lead to TypeErrors (caught by transition notifier) when it tried to find None in 'task1'. Change-Id: I3e91029f7b7cb960d6ca84490de76b5e044bc238
* Flow smart revert with retry controllerAnastasia Karpinska2014-03-181-1/+58
| | | | | | | | | | | | | | | | | | | - Remove flow REVERTING state. Now flow can be running and reverting simultaneously. Until flow isn't finished it is in RUNNING state. - Add RETRYING state for the retry controller. - Implement smart revertion and flow retries and retries resumption. - Default retry controllers: Times, ForEach and ParameterizedForEach. - Example and unit tests. Implements: blueprint subgraph-execution Implements: blueprint reversion-strategies Implements: blueprint smart-revert Change-Id: Ifa600bcad1edf2910f02ac36783cd458afbd880c
* Add retry action to execute retriesAnastasia Karpinska2014-03-141-0/+10
| | | | Change-Id: I093a49f88d9eecea5eb88fcecad9f7825cd7cb33
* Merge "Run action-engine tests with worker-based engine"Jenkins2014-02-201-0/+16
|\
| * Run action-engine tests with worker-based engineStanislav Kudriashev2014-02-191-0/+16
| | | | | | | | Change-Id: I00398fd2387b261c577f5d3eba9e2aebae3ba165
* | Merge "Message-oriented worker-based flow with kombu"Jenkins2014-02-201-0/+35
|\ \ | |/
| * Message-oriented worker-based flow with kombuStanislav Kudriashev2014-02-191-0/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Implemented Worker to be started on remote host for handling tasks request. * Implemented WorkerTaskExecutor that proxies tasks requests to remote workers. * Implemented Proxy that is used for consuming and publishing messages by Worker and Executor. * Added worker-based engine and worker task executor. * Added kombu dependency to requirements. * Added worker-based flow example. * Added unit-tests for worker-based flow components. Implements: blueprint worker-based-engine Change-Id: I8c6859ba4a1a56c2592e3d67cdfb8968b13ee99c
* | Remove extraneous vim configuration commentsyangxurong2014-02-141-2/+0
|/ | | | | | | | | Remove line containing comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4 Change-Id: I7581cc88b8de433d5609ed06c6570b0b45c13573 Closes-Bug:#1229324
* Engine tests refactoringIvan A. Melnikov2014-01-301-10/+6
| | | | | | | | | | | | This change makes tests from test_action_engine.py more focused and deterministic: - replace assertIsSubset with assertIsSuperAndSubsequence, which checks order, too; - remove all sleeping from test tasks, it does not help anything anyway; - refactor tests that verify behaviour in case of task failing in nested subflow. Change-Id: I87ce1e5eed4a3a93c3c6593b618e82cfdd68204f
* Tests: don't pass 'values' to task constructorIvan A. Melnikov2014-01-301-8/+7
| | | | | | This might make it easier for worker-based engine to run it. Change-Id: I94d7b8e0a447708e448c8c5d3dd7232f67c7adda
* Move six to the right locationJoshua Harlow2014-01-091-1/+2
| | | | | | | | | Third party imports are supposed to be after standard library imports so make six be after. Some newlines are also added to separate standard library imports from all the others. Co-authored-by: Ivan A. Melnikov <imelnikov@griddynamics.com> Change-Id: Ied067e9367612758666da726df195ed390215e1b
* Enable H202 warning for flake8Ivan A. Melnikov2014-01-021-1/+1
| | | | Change-Id: Iaeb6d4ab1af5063558ef1a7b9a85e46c6cf0ccb7
* Unit tests refactoringAnastasia Karpinska2013-10-171-16/+163
| | | | | | | * duplicated tests were removed * common tasks moved to utils Change-Id: I69c91a264ec668b1333db8fd907298262af098cb
* Removed unused utilitiesIvan A. Melnikov2013-10-141-16/+0
| | | | Change-Id: Ifa3e7b820d594303c6c02eae12f55bce2bd1dacc
* No module name for builtin type and exception namesIvan A. Melnikov2013-10-101-8/+12
| | | | | | | | Make utils.reflection.get_class_name() return just type name (without module) for builtin exceptions and types. This makes output less clumsy and makes this function return same strings for python 2 and 3. Change-Id: Ibc599ed34c804cf2cf2ab67e7466a48ebe7f6df3
* Task decorator was removed and examples updatedAnastasia Karpinska2013-09-231-5/+3
| | | | Change-Id: Ie49fe6c2f48a18130d1fd2a3aa5485cd8cee4ed4
* Add basic sanity tests for unordered flowJoshua Harlow2013-09-221-0/+22
| | | | | | Fixes: bug 1224194 Change-Id: Id5af06a2f10e357e8d042df6f41795cc9328469b
* Add provides and requires properties to FlowAnastasia Karpinska2013-09-111-1/+1
| | | | | | | | | | | Provides and requires properties are used to browse all required and provided values for the whole Flow. The same properties were added to the Task. Appropriate Task properties were renamed to rebind and save_as. Change-Id: I02eb02303a9701a13f1a54f06f20bbf9aebd1d04
* Allow provides to be a set and results to be a dictionaryJoshua Harlow2013-09-071-6/+17
| | | | | | | | | | | | Instead of forcing task provides to be a tuple or list and using the corresponding numerical index to determine how to interpret the tasks results we should also allow for the task provides to be a set and then allow for the tasks result to be interpreted as a dictionary. Fixes bug 1221998 Change-Id: Ibae689e7975d6782aa248d2b6e3691c44a89ef9f
View Lorry Controller Status