summaryrefslogtreecommitdiff
path: root/taskflow/conductors
Commit message (Collapse)AuthorAgeFilesLines
* Remove sixTakashi Kajinami2022-05-183-8/+3
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Import modules, not classesSean McGinnis2020-05-011-6/+2
| | | | | | | | This addresses comments from I24b13c8654a33d99bf687a44a36cfdace39e3866 to follow guidelines to only import modules, not classes. Change-Id: I0e66c08bfcdff306bac0121602d45b56cfb2c712 Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
* set default python to python3Doug Hellmann2018-04-171-1/+1
| | | | | | | | | | | | | | | | | | Set the default python to python3 except for the py27 environment. We have to set that explicitly to override the new default. Have py27 and py3* jobs run same commands by removing the commands entry for py27. The pep8 rules under python 3 are different, so we need to fix a couple of issues with blank space. The E721 rule is ignored entirely because we are not yet using a version of flake8 that supports ignoring individual rules on specific lines, although the instructions for that are added so that when we upgrade flake8 to a newer version we can remove the global rule. Change-Id: Id236b0632a83a188958130d04f1e9c476c5ace65 Signed-off-by: Doug Hellmann <doug@doughellmann.com>
* Remove kwarg timeout in executor conductorchenghuiyu2017-09-011-3/+1
| | | | | | | The method stop returns immediately regardless of whether the conductor has been stopped, so kwarg timeout is no longer necessary, and it is deprecated for removal. Change-Id: I50e5cbb1a00eb9b6f358f150716d2a5d682a49c9
* Remove deprecated things for 2.0 releaseJoshua Harlow2016-05-102-51/+0
| | | | Change-Id: Id9adbc50bd51adc77ce88f698ad0ea2ee63fc5e2
* Make conductor.stop stop the running engine gracefullyGreg Hill2016-05-031-2/+18
| | | | | | | | | | | | Previously stopping the conductor would only prevent it from accepting new jobs. This change makes it abort the current job by telling the engine to stop processing tasks after the current ones finish. This allows for conductors to gracefully exit when receiving a kill signal (although the signal handling is not done automatically yet). Change-Id: Ie6ddcbb2df4508ad1e3f6698c6f4cb2fc26a278f
* Add periodic jobboard refreshing (incase of sync issues)Joshua Harlow2016-03-151-2/+20
| | | | | | Related-Bug: #1557107 Change-Id: I42672ef63ef02ec5ec6a842d263d0db83d91fe45
* Retrieve the store from flowdetails as well, if it existsGreg Hill2016-01-251-3/+7
| | | | | | | | Gives users a more permanent way to provide an initial set of arguments to a flow. Change-Id: Ib9c3d60882548120d467a645bbac9be78408bac3 Implements: blueprint flow-details-keep-store
* Merge "Move 'convert_to_timeout' to timing type as a helper function"Jenkins2015-12-241-12/+1
|\
| * Move 'convert_to_timeout' to timing type as a helper functionJoshua Harlow2015-11-121-12/+1
| | | | | | | | | | | | | | | | Also adds some basic tests to this newly exposed helper function so it continues to operate as expected and better docstrings on the timeout types methods. Change-Id: I9fa4c7d313084800d49cfc77f6ca93afcff1169d
* | Merge "Avoid recreating notify details for each dispatch iteration"Jenkins2015-12-161-10/+7
|\ \
| * | Avoid recreating notify details for each dispatch iterationJoshua Harlow2015-12-141-10/+7
| | | | | | | | | | | | | | | | | | | | | | | | We can just create the notification details once and reuse it for each iteration, instead of recreating it for each iteration; saves some resources... Change-Id: Ie8ad61647afe2eef813af87a915710cf91de8e46
* | | Change name of misc.ensure_dict to misc.safe_copy_dictJoshua Harlow2015-12-141-1/+1
| | | | | | | | | | | | Change-Id: I2ff3b9bb4d0b8143163e4bf370f10c74e79dbd31
* | | Use the misc.ensure_dict helper in conductor engine options savingJoshua Harlow2015-11-181-4/+1
|/ / | | | | | | Change-Id: I7d796f4ca4b59a22fdae417f1483b3778d3070d0
* | Adding notification points for job completionMin Pae2015-11-161-0/+24
|/ | | | | | | Adding notifications for job completion, both consumed and abandoned, so that a listener can take some action based on job completion. Change-Id: I826285d4bfccd2406df7b59e53a9b724702ed094
* Add a executor backed conductor and have existing impl. use itJoshua Harlow2015-11-124-243/+448
| | | | | | | | | | | | | | | | | | | This adds a executor backed job dispatching base class and has the existing blocking executor use it by running jobs and dispatching jobs into a sync executor. It also allows for dispatching jobs into a thread executor, or other executor via a new '_executor_factory' method that can generate executors (it can be overriden in the non-blocking executor to provide your own executors instances). This does alter the behavior in that now that jobs are dispatched into an executor we no longer can immediatly know if a job was dispatched and raised an exception or whether it will raise an exception in the future, so we now alter the 'local_dispatched' to just be a boolean that is used to determine if any dispatches happened (failure or not). Change-Id: I485770e8f4c85d3833892a453c9fb5168d8f0407
* Register conductor information on jobboardtonytan4ever2015-10-191-0/+26
| | | | Change-Id: I3bf935280a6e8b265045b09fde43d0ec7dc56f07
* feat: add max_dispatches arg to conductor's runSriram Madapusi Vasudevan2015-10-151-7/+27
| | | | | | | | | | | | | - This will cause the conductor to only do 'n' number of dispatches, after which it stops dispatching jobs. - This will allow the code that call the conductor, to monitor conductor.dispatching, and make a decision on what is to be done with it. Eg: Decomission a conductor, restart the conductor etc - Backward Compatible. Change-Id: I3386c7050806806b5ee44a74ba93e50515a5ab7b
* Found another removal_version=? that should be removal_version=2.0Joshua Harlow2015-07-141-1/+1
| | | | Change-Id: I515f36dfea037929231ca8c591c870ebfe895344
* Update all removal_version from being ? to being 2.0Joshua Harlow2015-07-122-2/+2
| | | | | | | | Before the 2.0 release it would be great to get these removed so let's mark that version as the version that will no longer have these in it. Change-Id: I66a74d270bf95db005e9febfce1a5e211c7a49f6
* Merge "Notify on the individual engine steps"Jenkins2015-07-082-2/+39
|\
| * Notify on the individual engine stepsJoshua Harlow2015-07-082-2/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a conductor is running it is quite useful to be able to how long each engine step takes. To enable this information being output, add a notifier to the base conductor and use it in the blocking conductor to emit events around engine activities. This makes it possible to track the timing (or other information that can be gathered from these events) in a non-intrusive manner. In the `99_bottles.py` demo we also now use this to be able to easily see what the conductor is actively doing (without having to enable the more verbose DEBUG level logging). Change-Id: Ifd8ff38f82fc8135fe5fec4c8e41f0e06f4fdee3
* | Merge "Handle conductor ctrl-c more appropriately"Jenkins2015-07-081-0/+4
|\ \ | |/
| * Handle conductor ctrl-c more appropriatelyJoshua Harlow2015-06-181-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | When a conductor program is interrupted via ctrl-c or equivalent it is much nicer log that that has happened and to reraise that exception. This also slightly tweaks the 99 bottles song to make it even better, by having more pieces/tasks, which makes it possible to kill the program during each task and see how the resumption works when a flow is composed of segments. Change-Id: I5d242eba9a043ef96646ba74ea5928daa0691ed0
* | Merge "Remove 2.6 classifier + 2.6 compatibility code"Jenkins2015-06-291-2/+3
|\ \
| * | Remove 2.6 classifier + 2.6 compatibility codeJoshua Harlow2015-06-211-2/+3
| |/ | | | | | | | | | | | | | | Fixes bug 1445827 Depends-On: I02e3c9aacef0b295a2f823a5cbaf11768a90cb82 Change-Id: I1db681803598ac1bc917fd74a99458bc61edf3f1
* | Merge "Use sphinx deprecated docstring markup"Jenkins2015-06-251-4/+6
|\ \ | |/ |/|
| * Use sphinx deprecated docstring markupJoshua Harlow2015-06-101-4/+6
| | | | | | | | | | | | | | | | | | | | Along with using the `debtcollector` library we should also be marking the docstrings of deprecated functions/methods/properties with the appropriate deprecated information so that the generated docs also show what is deprecated. Change-Id: I8a1960731fd53434bf04d22cc6ff208998adb4f3
* | Merge "Remove all 'lock_utils' now that fasteners provides equivalents"Jenkins2015-06-151-3/+3
|\ \ | |/ |/|
| * Remove all 'lock_utils' now that fasteners provides equivalentsJoshua Harlow2015-06-041-3/+3
| | | | | | | | | | | | Depends-On: I442249783da4a6ae10c78b95e0b279409c95d2e6 Change-Id: I877928c858e8d6176d3f01ad9de2765104acf5c3
* | Remove listener stack and replace with exit stackJoshua Harlow2015-05-111-3/+8
|/ | | | Change-Id: Ic9150b8208420d29865af297ae8d0c656becb81e
* Move to the newer debtcollector provided functionsJoshua Harlow2015-03-202-7/+7
| | | | | | | | Use the debtcollector library provided variants of these functions and remove our versions to avoid more duplications. Change-Id: I526789f44899afd284803b127bec5bb349d1aa2b
* Ensure we register & deregister conductor listenersJoshua Harlow2015-03-132-5/+12
| | | | | | | | | | Instead of just registering engine listeners that were returned, make sure we also deregister them when the engine has either finished or failed. This ensures that if a listener has hold of any resources (or other) that it can clean those up and be sure that its deregister call will be made. Change-Id: Ia1420c435156362698702fed2bda11c2a0fef803
* add _listeners_from_job method to Conductor baseDan Krause2015-03-111-4/+22
| | | | Change-Id: I4a58c2fd5a0c648ffe3f81e2170a622b6b0b43f0
* Merge "Use debtcollector library to replace internal utility"Jenkins2015-03-031-1/+3
|\
| * Use debtcollector library to replace internal utilityJoshua Harlow2015-02-191-1/+3
| | | | | | | | | | | | | | | | | | | | | | The new debtcollector library was extracted from this code and has now been released (and made available in the requirements list) so we can go ahead and just use it for most of the cases (minus the proxy moved class) in our code-base to reduce the amount of custom code we have to maintain to perform deprecation activities. Change-Id: Ieeda6ffe282b67a1c1bb4f72e5858d22df0f30a0
* | Stick to one space after a periodJoshua Harlow2015-02-241-2/+2
|/ | | | Change-Id: I72666a24883e5b42e1ef76d0146d3da40d376ff8
* Allow loading conductors via entrypointsJoshua Harlow2015-02-164-161/+235
| | | | | | | | | | | | | Add an easy to use helper function that is exposed to fetch a conductor in a way that matches how engines and persistence backends and jobboards are loaded. This also adjusts the single threaded conductor to be now named blocking conductor and exposes an entrypoint backend fetch() function to load current and future conductors. Change-Id: Id146d847c329d3e8510a8f24c3ec8b918680ddb5
* Merge "catch NotFound errors when consuming or abandoning"Jenkins2015-02-151-1/+1
|\
| * catch NotFound errors when consuming or abandoningDan Krause2015-02-121-1/+1
| | | | | | | | | | | | | | | | | | If the zk node disappears underneath the conductor before it attempts to abandon or consume a job, consumption/abandonment would fail with an uncaught exception, killing the conductor. Catching excp.NotFound corrects this. Change-Id: Iaa7d0d8d0ebbc76973f16d717cc3137c75fbfec2
* | Mark conductor 'stop' method deprecation kwarg with versionsJoshua Harlow2015-02-121-5/+6
|/ | | | | | | | | To make it known (when the warning message is emitted) what version this was deprecated in and when it may be removed in add those parameters to the 'removed_kwarg' decorator so that users of conductors are aware of this information. Change-Id: Iacc97831ccd96d68cea4bd7e65a967b5ff9e7dc5
* Modify stop and add wait on conductor to prevent lockupsMin Pae2015-02-091-11/+24
| | | | | | | | | | | | | | Removed _dead.wait from the stop method and added a wait method to explicitly wait for graceful shutdown. Implicitly waiting for graceful shutdown in the stop method is leading to potential deadlock situations when conductor is running in the main thread and a signal handler is used to kick off stop/shutdown. Also in larger systems, it is desirable to be able to initiate shutdown on all running services then wait/join everything together so as not to force a serial shutdown pattern. Change-Id: Ic74dab22b09100c5b653d12c2518f83a1bf8e859
* Add and use a new simple helper logging moduleJoshua Harlow2014-12-081-2/+1
| | | | | | | | | Add a new logging BLATHER level to easily allow its usage for messages that are below the normal DEBUG level such as compilation information and scope lookup info which can be very verbose in logs if always enabled. Change-Id: I828211403bd02bfd6777b10cdcfe58fb0637a52c
* Merge "Use and verify event and latch wait() return using timeouts"Jenkins2014-10-191-4/+3
|\
| * Use and verify event and latch wait() return using timeoutsJoshua Harlow2014-10-181-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | Instead of blocking up the whole test suite when a latch or event was not decremented to its desired value (or not set for an event) we should use a reasonably high value that we use when waiting for those actions to occur and verify that when those wait() functions return that we have reached the desired state and if not either raise an exception or stop further testing. Fixes bug 1363739 Change-Id: I8b40282ac2db9cabd48b0b65c8a2a49610d77c4f
* | Deprecate `engine_conf` and prefer `engine` insteadJoshua Harlow2014-10-182-12/+17
|/ | | | | | | | | | | | | | | | | | | | | | To avoid having one set of options coming from `engine_conf` and another set of options coming from `kwargs` and another set coming from `engine_conf` if it is a URI just start to shift toward `engine_conf` being deprecated and `engine` being a string type only (or a URI with additional query parameters) and having any additional **kwargs that are provided just get merged into the final engine options. This adds a new helper function that handles all these various options and adds in a keyword argument `engine` that will be shifted to in a future version (in that future version we can also then remove the `engine_conf` and just stick to a smaller set of option mechanisms). It also adjusts all examples to use this new and more easier to understand format and adjusts tests, conductor interface to use this new more easily understandable style of getting an engine. Change-Id: Ic7617057338e0c63775cf38a24643cff6e454950
* Have the dispatch_job function return a futureJoshua Harlow2014-08-212-5/+8
| | | | | | | | | | | | | To make it easier to add in a multi-threaded conductor convert the base dispatch_job function to return a future object. This future object will contain a single result, whether the job should be consumed or abandoned. In the single threaded conductor its dispatch_job function will return a future, after completing the job (in a multi threaded conductor it would not return a future after doing the work). Change-Id: I077334820d36c64e272e93d158e3a0cd0d66a937
* Merge "Add a pformat() failure method and use it in the conductor"Jenkins2014-08-211-5/+12
|\
| * Add a pformat() failure method and use it in the conductorJoshua Harlow2014-08-111-5/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the conductors engine raises a wrapped failure (which may contain a single failure from a remote worker, or may contain many failures if there were more than one such failures...) it is very useful to show more than just the basic exception information (or at least provide methods that can print out the information in a nice manner). This commit adds a pformat() failure method that formats the failure in a nice and understandable manner and then uses it in the conductor dispatching routine so that the conductors LOGs are much more useful. Change-Id: I46afacf54c9b4cae885c76c09b51e61d71fe623a
* | Avoid naming time type module the same as a builtinJoshua Harlow2014-08-121-1/+1
|/ | | | | | | | | In order to avoid naming conflicts which can easily occur if we name our time module as a python builtin just pick a name that doesn't conflict (it still retains the same meaning). Change-Id: Ia19f3776689d2b1f31f7cbfaa15e53f99a3ee900
View Lorry Controller Status