summaryrefslogtreecommitdiff
path: root/taskflow/tests/unit/worker_based
Commit message (Collapse)AuthorAgeFilesLines
* Remove sixTakashi Kajinami2022-05-182-5/+3
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Use LOG.warning instead of deprecated LOG.warnTakashi Kajinami2021-11-291-1/+1
| | | | | | | | | The LOG.warn method is deprecated[1] and the LOG.warning method should be used instead. [1] https://docs.python.org/3/library/logging.html#logging.warning Change-Id: I4321a489c56eb1aa650e776ee35d8f88d4d8910c
* Use unittest.mock instead of mockHervé Beraud2021-04-271-1/+1
| | | | | | | | The mock third party library was needed for mock support in py2 runtimes. Since we now only support py36 and later, we can use the standard lib unittest.mock module instead. Change-Id: Ib169e3deb7ddb2bc93a206ebec4043552281aa7f
* Fix invalid json unit testBen Nemec2018-02-081-1/+5
| | | | | | | | | | Recent versions of oslo.serialization have made it possible to dump exceptions to JSON, which broke a unit test in taskflow that assumed exceptions were unserializable. This change switches to an explicitly unserializable class for that test. Change-Id: If6d19bc9fcf1f1813cb087d42dc7ba6a61c71b3d Closes-Bug: 1748241
* Replace assertEqual(None, *) with assertIsNone in testsweiweigu2016-07-121-1/+1
| | | | | | | | Replace assertEqual(None, *) with assertIsNone in tests to have more clear messages in case of failure. Change-Id: I74452af6d840bcf612fd3bb2521db9134460dd63 Closes-Bug: #1280522
* Make tests less dependent on transient stateGreg Hill2016-06-021-3/+2
| | | | | | | | | Having to update the endpoint count every time you add a test class is really obnoxious and leads to a ton of pointless rebasing. Now we just check that it finds at least the task it knows about and call that good. Change-Id: I96b8c6cd6cbc1fdc58dee4b18cab5699e3daa844
* Instead of a multiprocessing queue use sockets via asyncoreJoshua Harlow2016-05-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | For a local process based executor usage currently to ensure that task emitted notifications are proxied we use the multi processing library and use its queue concept. This sadly creates a proxy process that gets associated, and this proxy process handles the queue and messages sent to and from it. Instead of doing this we can instead just create a temporary local socket using a random socket and have tasks (which are running in different processes) use that to communicate back any emitted notifications instead (and we can use the asyncore module to handle the emitted notifications since it handles the lower level socket reading, polling and dispatching). To ensure that the socket created is somewhat secure we use a similar process as the multi-processing library uses where we sign all messages with a hmac that uses a one time key that only the main process and the child process know about (and reject any messages that do not validate using this key). Change-Id: Iff9180054bf14495e5667af00ae2fafbdbc23791
* Make conductor.stop stop the running engine gracefullyGreg Hill2016-05-031-1/+1
| | | | | | | | | | | | Previously stopping the conductor would only prevent it from accepting new jobs. This change makes it abort the current job by telling the engine to stop processing tasks after the current ones finish. This allows for conductors to gracefully exit when receiving a kill signal (although the signal handling is not done automatically yet). Change-Id: Ie6ddcbb2df4508ad1e3f6698c6f4cb2fc26a278f
* Merge "Use a automaton machine for WBE request state machine"Jenkins2016-04-281-2/+2
|\
| * Use a automaton machine for WBE request state machineJoshua Harlow2016-02-211-2/+2
| | | | | | | | Change-Id: Ib26ff4418ab1128a578519be964c4d39cbd1d2f4
* | Allow for revert to have a different argument list from executeGreg Hill2016-02-261-1/+1
|/ | | | | | | | | Also allows for people to create Atom's with a different rebind or requires structure for the revert method, if desired. Implements blueprint: seperate-revert-args Change-Id: Ie7d13c8000ef08ff303481d486d1ba1cfbdeea44
* Add WBE worker expiryJoshua Harlow2016-02-142-5/+20
| | | | | | | | | When a worker hasn't responded to a notification request for a given amount of time remove it from being a useable worker that we can match task submissions to. Change-Id: I596bccc1c42f83ee79136dd27bc87039154ff7b1
* Some WBE protocol/executor cleanupsJoshua Harlow2016-02-144-32/+38
| | | | | | | | | | | | | | | | | | | Remove some of the usage of @property as none of these objects are publicly exposed (or have docstrings on them) to save some space/lines of code that aren't really adding any benefit. Use less **kwargs when we know exactly what the keyword arguments will or will not be. Being explicit makes it easier to understand these functions (vs not knowing what the arguments can or can't be). Removes base worker finder because right now we only have one implementation (at some point we will have two) but we can just wait to add a base class until then. Change-Id: I7107ff6b77a355b4c5d301948355fb6386605388
* Remove need for separate notify threadJoshua Harlow2016-02-141-3/+2
| | | | | | | | | | | | | | Instead of having a periodic notification thread that will drop messages to try to find workers we can just have this same work be done in the periodically called on_wait callback that is already used for expiring and matching workers to new/updated workers. This avoids having one more thread that doesn't do all that much (and activating it during waiting calls will be often enough to achieve its goal in life). Change-Id: If80233d13d914f2ed3665001a27627b78e6ee780
* Fix for WBE sporadic timeout of tasksJoshua Harlow2016-02-052-88/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes the sporadic of tasks that would happen under certain circumstances. What happened was that a new worker notification would be sent to a callback while at the same time a task submission would come in and there would be a small race period where the task would insert itself into the requests cache while the callback was processing. So to work around this the whole concept of a requests cache was revamped and now the WBE executor just maintains its own local dictionary of ongoing requests and accesses it safely. During the on_wait function that is periodically called by kombu the previous expiry of work happens but now any requests that are pending are matched to any new workers that may have appeared. This avoids the race (and ensures that even if a new worker is found but a submission is in progress that the duration until that submission happens will only be until the next on_wait call happens). Related-Bug: #1431097 Change-Id: I98b0caeedc77ab2f7214847763ae1eb0433d4a78
* Handle cases where exc_args can't be serialized as JSON in the WBEGreg Hill2016-01-281-0/+8
| | | | | | | | | First try to get them, but if they aren't able to be serialized, just return the failure without them. This only affects the WBE that serializes failures as part of the JSON response that is sent over the wire from the worker process to the WBE engine. Change-Id: I86e3255b612bc15097aabe63a684cf8b8808e61b
* Fix order of assertEqual for unit.worker_basedlin-hua-cheng2015-10-177-60/+60
| | | | | | | First parameter should be the expected value. Change-Id: I209cc4be621c62f60fca3584a21457988129c014 Partial-Bug: #1357117
* Bump futurist and remove waiting code in taskflowJoshua Harlow2015-07-251-3/+3
| | | | Change-Id: Ifc9780aa129a4a2804cead301a519895c2bfc0b5
* Create and use a serial retry executorJoshua Harlow2015-07-212-26/+3
| | | | | | | | | | | | To make it easily possible to change the retry atom execution from being in the engine thread this creates a retry executor (which is similar to the task executor) and provide that a serial executor (which it will use to execute with). This makes the retry and task actions closer to being the same and makes the surrounding code that much similar (which makes understanding it easier). Change-Id: I993e938280df3bd97f8076293183ef21989e2dba
* Merge "Remove direct usage of timeutils overrides and use fixture"Jenkins2015-07-151-3/+4
|\
| * Remove direct usage of timeutils overrides and use fixtureJoshua Harlow2015-07-131-3/+4
| | | | | | | | Change-Id: Ifa99497672dbc8fa60672ce4bfbfed1832b128af
* | Merge "Retain atom 'revert' result (or failure)"Jenkins2015-07-151-1/+1
|\ \ | |/ |/|
| * Retain atom 'revert' result (or failure)Joshua Harlow2015-07-101-1/+1
| | | | | | | | | | | | | | | | | | When a atom is reverted it can be useful to retain the result of that 'revert' method being called, so that it can be later analyzed (or used for various purposes) so adjust the storage, and actions to enable it to be stored. Change-Id: I38a9a5f3bf7550e924468bb4a86652cb8beb306c
* | Integrate futurist (and **remove** taskflow originating code)Joshua Harlow2015-07-094-8/+8
|/ | | | Change-Id: If89baa042695f19e42b6368034f3ccf22c2cf0aa
* Remove 2.6 classifier + 2.6 compatibility codeJoshua Harlow2015-06-212-4/+6
| | | | | | | | Fixes bug 1445827 Depends-On: I02e3c9aacef0b295a2f823a5cbaf11768a90cb82 Change-Id: I1db681803598ac1bc917fd74a99458bc61edf3f1
* Add a test that checks for task result visibilityJoshua Harlow2015-05-291-1/+1
| | | | | | | | | | Now that a task can provide the same thing that it requires, having a nice little test that ensures what is gotten is expected in a nice manner should be done to make sure what a task will be getting for its arguments respect the scoping that is defined by the flow ordering. Change-Id: I1cdc5c3bd3f56b39f20f28f51c36017405150cea
* Merge "Just let the future executors handle the max workers"Jenkins2015-05-121-10/+4
|\
| * Just let the future executors handle the max workersJoshua Harlow2015-03-181-10/+4
| | | | | | | | | | | | | | | | | | | | | | | | Instead of providing and retaining a thread count in the worker and action engine executors and checking it and handling the none case, we can just let the future types handle this already (which they already do). And when displaying this information in the worker banner use a new future executor attribute that is the maximum number of workers that will be ever created. Change-Id: I765c22936b53cdbb8a90195a764d4c67bcc3f34b
* | Move to using the oslo.utils stop watchJoshua Harlow2015-03-202-14/+17
|/ | | | | | | | | The code was moved to the oslo.utils package and we can now use it to avoid having our own type that does the same thing (less duplicated code and more sharing is good). Change-Id: I0545a978083ced75b2ba99280569ca2370756d33
* Add + use failure json schema validationJoshua Harlow2015-03-111-4/+4
| | | | Change-Id: Ie3aa386c831459a028ba494570bafd53b998126e
* Fix lookup scoping multi-match orderingJoshua Harlow2015-02-241-1/+1
| | | | | | | | | | | | | | | | | When look-up is occurring the possible provider ordering is used instead of the scope returned provider ordering. This causes incorrect matches when look-up argument names that are produced by multiple providers (at the same scope level) all providing the same requirement name. The scope order should be enforced as the de-facto order and not the order that the storage unit finds (which is hash based and varies depending on hash ordering). Closes-Bug: #1425326 Change-Id: I15f1ee5515758bdc470c0f7dd7a2f616923e5628
* Merge "Make the dispatcher handler be an actual type"Jenkins2015-02-202-8/+9
|\
| * Make the dispatcher handler be an actual typeJoshua Harlow2015-02-152-8/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having the dispatcher target be a tuple or a single callback, have it be an actual type with comments as to what the types fields are and how they are used. This makes it more obvious as to what those fields are and how they are used so that it is easier to understand how the WBE engine and components work. Change-Id: I5541ccd5a7aa6ae73ed9adceeac2c0524e2a1dc9
* | adding check for str/unicode type in requiresMin Pae2015-02-151-1/+1
|/ | | | | | | | | | | | When the requires argument for an Atom is passed in as a string, each character of the string is iterated over to build up a requirement list. This works for simple one letter argument names but not for long argument names. Added check for str and unicode types to prevent iterating over a string. Change-Id: Ida584221b48966d26935fb2ede0075aabb7ce972
* Improve upon/adjust/move around new optional exampleJoshua Harlow2015-02-111-1/+1
| | | | | | | | | | Instead of having the optional requirements example be a example that is itself a unittest just move the example to be an actual unit test that gets tested using the various engine types and change the example to be something slightly different (but shows the same kind of usage information). Change-Id: Ia03a81a6be636c501a35e7e290f587f7d05f8b30
* Abstract out the worker finding from the WBE engine0.7.0Joshua Harlow2015-01-314-64/+43
| | | | | | | | | | | | | | To be able to easily plug-in future types of ways to get which topics (and tasks) workers exist on (and can perform) and to identify and keep this information up-to date refactor the functionality that currently does this using periodic messages into a finder type and a periodic function that exists on it (that will be periodically activated by an updated and improved periodic worker). Part of blueprint wbe-worker-info Change-Id: Ib3ae29758af3d244b4ac4624ac380caf88b159fd
* Add and use a nicer kombu message formatterJoshua Harlow2015-01-291-12/+10
| | | | | | | | | | | Since the kombu message object that is recieved has no useful __str__ or __repr__ and on reception and processing we should the message (and the delivery tag) it is nice to have a more useful formatting of that message for debugging and such (where more detail about the messages are quite useful to see). Change-Id: I6730b10122a5de1a0a074525931f312fbd97b8c0
* Use monotonic time when/if availableJoshua Harlow2015-01-272-15/+11
| | | | | | | | | | Instead of using a time that can change depending on ntpd or other time adjustments use a monotonically increasing time if it's available from the underlying python library to avoid these types of time-shifts problems in the first place. Change-Id: Ib775a44026b9828536c905a1ed41f527358c0d39
* Add a thread bundle helper utility + testsJoshua Harlow2015-01-241-3/+0
| | | | | | | | | | | | To make it easier to create a bunch of threads in a single call (and stop them in a single call) create a concept of a thread bundle (similar to a thread group) that will call into a provided set of factories to get a thread, activate callbacks to notify others that a thread is about to start or stop and then perform the start or stop of the bound threads in a orderly manner. Change-Id: I7d233cccb230b716af41243ad27220b988eec14c
* Merge "Use explicit WBE worker object arguments (instead of kwargs)"Jenkins2015-01-241-6/+13
|\
| * Use explicit WBE worker object arguments (instead of kwargs)Joshua Harlow2015-01-231-6/+13
| | | | | | | | | | | | | | | | Removes the kwargs usage that is now uniform across the other WBE components from the workers module so that the usage of kwargs for setting up these objects no longer is valid. Change-Id: I4e25b88c5d2f7e2d7933ff270e2782cebe227025
* | Tidy up the WBE cache (now WBE types) moduleJoshua Harlow2015-01-222-1/+140
|/ | | | | | | | | | | | | | | Instead of using the expiring cache type as a means to store worker information just avoid using that type since we don't support expiry in the first place on worker information and use a worker container and a worker object that we can later extend as needed. Also add on clear methods to the cache type that will be used when the WBE executor stop occurs. This ensures we clear out the worker information and any unfinished requests. Change-Id: I6a520376eff1e8a6edcef0a59f2d8b9c0eb15752
* Use explicit WBE object arguments (instead of kwargs)Joshua Harlow2015-01-224-12/+19
| | | | | | | | | | | | | | Instead of passing around kwargs to root WBE classes and to contained classes prefer to use explicitnamed arguments that are passed around. This makes the code more obvious as to what the intended arguments are and makes it easier for error validation when other unknown arguments are passed (as well as for docs). Also moves the docs for the worker engine to be a sub-TOC under the main engine document so that it can be more easily explored and managed/found... Change-Id: I9413fad187c330fee494f0d4536cc27d9a90f0fb
* Increase robustness of WBE producer/consumersJoshua Harlow2015-01-202-26/+51
| | | | | | | | | Use the kombu provided ensure() decorator/wrapper along with sensible default settings to ensure that retries are attempted when kombu detects recoverable connection or recoverable channel errors have occurred. Change-Id: If47f72d02561d0b5d556ac386869a6122c8b871d
* Merge "Switch to using 'oslo_utils' vs 'oslo.utils'"Jenkins2015-01-166-7/+7
|\
| * Switch to using 'oslo_utils' vs 'oslo.utils'Joshua Harlow2015-01-146-7/+7
| | | | | | | | | | | | | | | | Prefer the non-deprecated 'oslo_utils' instead of the namespaced 'oslo.utils' wherever it was previously used. Change-Id: I9a78150ef5266e1ff22147278162fe3cfe1b2e3f
* | Remove 'SaveOrderTask' and test state in class variablesJoshua Harlow2015-01-151-1/+1
|/ | | | | | | | | | | | | | | | | | | Instead of saving task state in a class variable that is later introspected by further test code just remove that concept (which doesn't work in multiprocessing or worker engines which can not have access those types of shared/globally available concepts due to how they run) and use a specialized listener that can gather the same information in a more decoupled manner (and it will work in multiprocessing and worker engines correctly). This allows our engine test cases to work in those engine types which increases those engines test coverage (and future coverage and engine tests that are added). Fixes a bunch of occurrences of bug 1357117 as well that were removed during this cleanup and adjustment process... Change-Id: Ic9901de2902ac28ec255bef146be5846d18f9bfb
* Allow specifying the engine 'executor' as a stringJoshua Harlow2015-01-011-25/+41
| | | | | | | | | | | | | | | | To enable a parallel process executor to be used without having to pass in a futures executor allow for the executor option to be a string 'processes' (or similar) that will cause the default parallel process executor to be used automatically. Also allow for a 'threads' string that ensure a parallel thread executor is used to match the ability to uses processes. This also adjusts the WBE engine to have a similar executor fetching function (which in the WBE case now validates a provided executor to be of the desired type). Change-Id: I54a82584c32c697922507b4f6e01ea7b8acc73c6
* Move over to using oslo.utils [reflection, uuidutils]Joshua Harlow2014-12-185-5/+7
| | | | | | | | | | | | | | The reflection module is now part of oslo.utils so we should remove our local version and use that version instead; this also goes for the uuidutils module which is now part of oslo.utils as well so we no longer need our local version copied from the incubator... Note that one reflection method `find_subclasses` which was to specific to taskflow is now moved to the misc utility module instead of its prior home in the reflection module. Change-Id: I069881c80b0b2916cc0c414992b80171f7eeb79f
* Base task executor should provide 'wait_for_any'Joshua Harlow2014-12-151-3/+3
| | | | | | | | | | | Instead of having each task executor reproduce the same code for 'wait_for_any' we can just have the base task implementation provide the function that everyone is replicating instead; making common code common and save the headache caused by the same code being in multiple places (which is bad for multiple reasons). Change-Id: Icea4b7e3df605ab11b17c248d05acb3f9c02a1ca
View Lorry Controller Status