summaryrefslogtreecommitdiff
path: root/taskflow/engines/worker_based
Commit message (Collapse)AuthorAgeFilesLines
* Remove sixTakashi Kajinami2022-05-184-19/+14
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Use LOG.warning instead of deprecated LOG.warnTakashi Kajinami2021-11-291-2/+2
| | | | | | | | | The LOG.warn method is deprecated[1] and the LOG.warning method should be used instead. [1] https://docs.python.org/3/library/logging.html#logging.warning Change-Id: I4321a489c56eb1aa650e776ee35d8f88d4d8910c
* set default python to python3Doug Hellmann2018-04-171-1/+1
| | | | | | | | | | | | | | | | | | Set the default python to python3 except for the py27 environment. We have to set that explicitly to override the new default. Have py27 and py3* jobs run same commands by removing the commands entry for py27. The pep8 rules under python 3 are different, so we need to fix a couple of issues with blank space. The E721 rule is ignored entirely because we are not yet using a version of flake8 that supports ignoring individual rules on specific lines, although the instructions for that are added so that when we upgrade flake8 to a newer version we can remove the global rule. Change-Id: Id236b0632a83a188958130d04f1e9c476c5ace65 Signed-off-by: Doug Hellmann <doug@doughellmann.com>
* Merge "python3.0 has deprecated LOG.warn"Jenkins2017-05-033-25/+27
|\
| * python3.0 has deprecated LOG.warnxhzhf2017-03-073-25/+27
| | | | | | | | | | | | | | | | python3.0 has deprecated LOG.warn https://docs.python.org/3/library/logging.html#logging.warning Closes-Bug: #1650843 Change-Id: Ib8473cbdb84547f385e619fe08ed723b1ee7bcdb
* | Fix py35 test failureChangBo Guo(gcb)2017-05-031-1/+1
|/ | | | | | | | dict.keys() returns object dict_keys on Python 3, can't work well with kombu recently, convert it to list before futhur processing. Closes-Bug: #1687855 Change-Id: I9dcfadd8f16eccce06f4cde3b2a6669f28a3f8bc
* Some classes not define __ne__() built-in functionJi-Wei2016-08-311-0/+3
| | | | | | | | | | Some classes defines __eq__() built-in function, but does not define __ne__() built-in function, so self.assertEqual works but self.assertNotEqual does not work at all in this test case in python2. This patch fixes it. Change-Id: I3e4f213081268bad44583a63a84795d39094117f Closes-Bug: #1586268
* Merge "Spice up WBE banner and add simple worker __main__ entrypoint"Jenkins2016-04-281-71/+49
|\
| * Spice up WBE banner and add simple worker __main__ entrypointJoshua Harlow2016-02-081-71/+49
| | | | | | | | Change-Id: Ifdf275a623352aa3e42fbf0aa9a4394b64b54337
* | Merge "Use a automaton machine for WBE request state machine"Jenkins2016-04-282-38/+90
|\ \
| * | Use a automaton machine for WBE request state machineJoshua Harlow2016-02-212-38/+90
| | | | | | | | | | | | Change-Id: Ib26ff4418ab1128a578519be964c4d39cbd1d2f4
* | | Refactor Atom/BaseTask/Task/Retry class hierarchyGreg Hill2016-02-181-1/+1
|/ / | | | | | | | | | | | | | | * Moved common argument mapping logic to Atom * Removed BaseTask * Removed duplicated logic from subclasses Change-Id: I275745adb80cecb0c35c1230eac76436bf3b0157
* | Add WBE worker expiryJoshua Harlow2016-02-144-7/+58
| | | | | | | | | | | | | | | | | | When a worker hasn't responded to a notification request for a given amount of time remove it from being a useable worker that we can match task submissions to. Change-Id: I596bccc1c42f83ee79136dd27bc87039154ff7b1
* | Some WBE protocol/executor cleanupsJoshua Harlow2016-02-143-107/+69
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove some of the usage of @property as none of these objects are publicly exposed (or have docstrings on them) to save some space/lines of code that aren't really adding any benefit. Use less **kwargs when we know exactly what the keyword arguments will or will not be. Being explicit makes it easier to understand these functions (vs not knowing what the arguments can or can't be). Removes base worker finder because right now we only have one implementation (at some point we will have two) but we can just wait to add a base class until then. Change-Id: I7107ff6b77a355b4c5d301948355fb6386605388
* | Remove need for separate notify threadJoshua Harlow2016-02-142-46/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having a periodic notification thread that will drop messages to try to find workers we can just have this same work be done in the periodically called on_wait callback that is already used for expiring and matching workers to new/updated workers. This avoids having one more thread that doesn't do all that much (and activating it during waiting calls will be often enough to achieve its goal in life). Change-Id: If80233d13d914f2ed3665001a27627b78e6ee780
* | Don't bother scanning for workers if no new messages arrivedJoshua Harlow2016-02-112-7/+26
| | | | | | | | | | | | | | | | | | | | If the worker finder has not gotten any new notification messages letting it know about new (or updated) workers we can just skip trying to match existing waiting work to workers as without messages being processed the match will still not work (as the worker data doesn't change without those messages being processed). Change-Id: I41d50c676f04f85c49a03d9d503da1955af45f7d
* | Fix for WBE sporadic timeout of tasksJoshua Harlow2016-02-053-69/+64
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes the sporadic of tasks that would happen under certain circumstances. What happened was that a new worker notification would be sent to a callback while at the same time a task submission would come in and there would be a small race period where the task would insert itself into the requests cache while the callback was processing. So to work around this the whole concept of a requests cache was revamped and now the WBE executor just maintains its own local dictionary of ongoing requests and accesses it safely. During the on_wait function that is periodically called by kombu the previous expiry of work happens but now any requests that are pending are matched to any new workers that may have appeared. This avoids the race (and ensures that even if a new worker is found but a submission is in progress that the duration until that submission happens will only be until the next on_wait call happens). Related-Bug: #1431097 Change-Id: I98b0caeedc77ab2f7214847763ae1eb0433d4a78
* Handle cases where exc_args can't be serialized as JSON in the WBEGreg Hill2016-01-282-7/+20
| | | | | | | | | First try to get them, but if they aren't able to be serialized, just return the failure without them. This only affects the WBE that serializes failures as part of the JSON response that is sent over the wire from the worker process to the WBE engine. Change-Id: I86e3255b612bc15097aabe63a684cf8b8808e61b
* Move all internal blather usage/calls to trace usage/callsJoshua Harlow2015-12-171-4/+4
| | | | Change-Id: I415a81d3b6b15b17a9a91cc2a0681c159172a4e1
* Make more of the WBE logging and '__repr__' message more usefulJoshua Harlow2015-10-093-14/+23
| | | | | | | | | When running the examples, especially when running them in TRACE/BLATHER/DEBUG logging level these updates make it more clear = what is being processed, the messages being sent/acked/received and what there contents are. Change-Id: I94a497c9064df30197454ae480fe3d471ba1dc7d
* Provide a deprecated alias for the now removed stop watch classJoshua Harlow2015-07-123-6/+5
| | | | | | | | | | | | | This allows those who were using it to still continue using it until 2.0 where it will be removed; this makes it possible for those users to get off that code in a way that will be easily do-able (without totally breaking there code-bases, until we do that in the 2.0 release). This also removes all internal usage of that stop watch class so that the library itself will not reference it anymore. Change-Id: If313d8e7b9bdc8741db2e2e1dfb381aa3260b971
* Integrate futurist (and **remove** taskflow originating code)Joshua Harlow2015-07-094-8/+8
| | | | Change-Id: If89baa042695f19e42b6368034f3ccf22c2cf0aa
* Remove 2.6 classifier + 2.6 compatibility codeJoshua Harlow2015-06-211-2/+2
| | | | | | | | Fixes bug 1445827 Depends-On: I02e3c9aacef0b295a2f823a5cbaf11768a90cb82 Change-Id: I1db681803598ac1bc917fd74a99458bc61edf3f1
* Remove all 'lock_utils' now that fasteners provides equivalentsJoshua Harlow2015-06-041-2/+2
| | | | | | Depends-On: I442249783da4a6ae10c78b95e0b279409c95d2e6 Change-Id: I877928c858e8d6176d3f01ad9de2765104acf5c3
* Merge "Just let the future executors handle the max workers"Jenkins2015-05-121-8/+5
|\
| * Just let the future executors handle the max workersJoshua Harlow2015-03-181-8/+5
| | | | | | | | | | | | | | | | | | | | | | | | Instead of providing and retaining a thread count in the worker and action engine executors and checking it and handling the none case, we can just let the future types handle this already (which they already do). And when displaying this information in the worker banner use a new future executor attribute that is the maximum number of workers that will be ever created. Change-Id: I765c22936b53cdbb8a90195a764d4c67bcc3f34b
* | Use the ability to chain exceptions correctlyJoshua Harlow2015-04-031-12/+20
| | | | | | | | | | | | | | | | | | | | | | In the zookeeper jobboard (and persistence backends) we are associating the cause of a new exception when raising a exception. Using the new exception helper we can make this work better on py2.x and py3.x so that the py3.x version has the chain setup correctly (while the py2.x version just uses the class 'cause' attribute instead). Change-Id: Ieeac2f70e1834d4612556565762ffd3be3e5b5a1
* | Use the class name instead of the TYPE property in __str__Joshua Harlow2015-03-171-1/+2
| | | | | | | | Change-Id: Ie6fc31b0c89b9eaf9fabc1feb17d697edc5ff522
* | Just use the class name instead of TYPE constantJoshua Harlow2015-03-151-4/+7
| | | | | | | | | | | | | | | | | | | | The TYPE constant is more of an internal protocol detail and is not the best constant to use in exceptions that tell the received data is invalid, instead just prefer and use the class name instead which is more relevant than the TYPE constant. Change-Id: I60652a149eb1a3c3d8523544c355044bbd6fec94
* | Use a collections.namedtuple for the request work unitJoshua Harlow2015-03-132-17/+23
|/ | | | | | | | | | Instead of returning a tuple with hard to understand and read contents a namedtuple can help understand what the tuple is composed of and how it can be used/what it is, so we should prefer to use it when we can for the wbe requested work to perform. Change-Id: I8147814449d04ba9a03730547ac113e4ef7b272a
* Add + use failure json schema validationJoshua Harlow2015-03-112-43/+58
| | | | Change-Id: Ie3aa386c831459a028ba494570bafd53b998126e
* Make the worker banner template part of the worker classJoshua Harlow2015-03-031-33/+38
| | | | | | | | | Instead of having the banner template be part of the worker module, have it be part of the worker class so that anyone that desires to subclass and create new workers can alter the banner template more easily (if they so choose). Change-Id: Id56815ffd726f8952a74f9ab866b1bd006dcfcad
* Make the dispatcher handler be an actual typeJoshua Harlow2015-02-155-19/+49
| | | | | | | | | | | | | Instead of having the dispatcher target be a tuple or a single callback, have it be an actual type with comments as to what the types fields are and how they are used. This makes it more obvious as to what those fields are and how they are used so that it is easier to understand how the WBE engine and components work. Change-Id: I5541ccd5a7aa6ae73ed9adceeac2c0524e2a1dc9
* Remove delayed decorator and replace with nicer methodJoshua Harlow2015-02-131-20/+38
| | | | | | | | Instead of having a delayed decorator that is in the module have one that is attached onto the instance and that can log how long it took for the target method to finally be ran. Change-Id: Ie6e51b0a2a60a063554dfba7fcc321ecd6661949
* Merge "Default to using a thread-safe storage unit"Jenkins2015-02-071-3/+0
|\
| * Default to using a thread-safe storage unitJoshua Harlow2015-02-051-3/+0
| | | | | | | | | | | | | | | | | | | | | | Instead of having a nominally useful single-threaded storage unit that uses a dummy r/w lock and a multi-threaded storage unit just have the storage unit by default protect itself from multi-threading calls being used on it (with the appropriate reader/writer locks being activated to make this work correctly). Change-Id: Ib6879edb465156a8e54fd5b4002550d1cec49137
* | Use a notifier instead of a direct property assignmentJoshua Harlow2015-02-042-5/+11
|/ | | | | | | | | | | | | | To allow for finders to be shared (across executors) in the near future we need to avoid assigning our callback as the sole/single callback that the finder will use and instead use a notification mechanism (using our notifier type) that can be used to register many callbacks and decouple the emitting of new workers events from the reception of that event. Part of blueprint wbe-worker-info Change-Id: I0ab2ec649f759ec67e15960bdeb83b108174734b
* Abstract out the worker finding from the WBE engine0.7.0Joshua Harlow2015-01-315-136/+168
| | | | | | | | | | | | | | To be able to easily plug-in future types of ways to get which topics (and tasks) workers exist on (and can perform) and to identify and keep this information up-to date refactor the functionality that currently does this using periodic messages into a finder type and a periodic function that exists on it (that will be periodically activated by an updated and improved periodic worker). Part of blueprint wbe-worker-info Change-Id: Ib3ae29758af3d244b4ac4624ac380caf88b159fd
* Add and use a nicer kombu message formatterJoshua Harlow2015-01-294-30/+38
| | | | | | | | | | | Since the kombu message object that is recieved has no useful __str__ or __repr__ and on reception and processing we should the message (and the delivery tag) it is nice to have a more useful formatting of that message for debugging and such (where more detail about the messages are quite useful to see). Change-Id: I6730b10122a5de1a0a074525931f312fbd97b8c0
* Use kombu socket.timeout alias instead of socket.timeoutJoshua Harlow2015-01-291-2/+2
| | | | | | | | Use the kombu timeout error alias to socket.timeout to be more resilient against kombu ever changing this exception type to something different. Change-Id: I53c01db67742c65ee49264fd0d8d36417cec3ec1
* Merge "Link WBE docs together better (especially around arguments)"Jenkins2015-01-293-13/+18
|\
| * Link WBE docs together better (especially around arguments)Joshua Harlow2015-01-263-13/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Link the engine/worker 'retry_options' to the proxy which has linkage to what the defaults are and to where more information can be gathered about the option (which really gets sent in to kombu) so that users can more easily understand it. Also removes showing the docs about the following as its more of an internal API/class not meant for public consumption: * Module: taskflow.engines.worker_based.executor Change-Id: Ib16ee10765ec6b9a0af320bd6818d9649c2485b1
* | Merge "Stopwatch usage cleanup/tweak"Jenkins2015-01-291-8/+4
|\ \
| * | Stopwatch usage cleanup/tweakJoshua Harlow2015-01-281-8/+4
| |/ | | | | | | | | | | | | | | | | | | | | | | | | Instead of optionally creating a stopwatch when a provided timeout is not none (to avoid the stopwatch leftover() method raising a error) just allow the stopwatch leftover() method to not raise when no duration is provided to avoid these repeated styles of usage/checks in the first place. By default the leftover() method still raises an error (a new keyword argument is now accepted to turn off this behavior). Change-Id: If934ee6e6855adbb6975cd6ea41e273d40e73dac
* | Merge "Emit a warning when no routing keys provided on publish()"Jenkins2015-01-291-0/+8
|\ \
| * | Emit a warning when no routing keys provided on publish()Joshua Harlow2015-01-261-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | When no routing keys are provided the message will no be published anywhere; so when this happens emit a warning to tell the library user that something is likely misconfigured and/or wrong. Change-Id: Id27e8ab73c1ae8a02cb4252958d9dc896a5df1f0
* | | Add docstring to wbe proxy to denote not for public useJoshua Harlow2015-01-281-1/+4
| |/ |/| | | | | Change-Id: I9bd59ae5d58ef3ace960509903403252fbed6277
* | Add a thread bundle helper utility + testsJoshua Harlow2015-01-241-18/+10
| | | | | | | | | | | | | | | | | | | | | | | | To make it easier to create a bunch of threads in a single call (and stop them in a single call) create a concept of a thread bundle (similar to a thread group) that will call into a provided set of factories to get a thread, activate callbacks to notify others that a thread is about to start or stop and then perform the start or stop of the bound threads in a orderly manner. Change-Id: I7d233cccb230b716af41243ad27220b988eec14c
* | Merge "Use explicit WBE worker object arguments (instead of kwargs)"Jenkins2015-01-241-8/+14
|\ \ | |/ |/|
| * Use explicit WBE worker object arguments (instead of kwargs)Joshua Harlow2015-01-231-8/+14
| | | | | | | | | | | | | | | | Removes the kwargs usage that is now uniform across the other WBE components from the workers module so that the usage of kwargs for setting up these objects no longer is valid. Change-Id: I4e25b88c5d2f7e2d7933ff270e2782cebe227025
View Lorry Controller Status