summaryrefslogtreecommitdiff
path: root/taskflow/utils/misc.py
Commit message (Collapse)AuthorAgeFilesLines
* Remove sixTakashi Kajinami2022-05-181-13/+14
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Replace deprecated import of ABCs from collections4.6.2Takashi Kajinami2021-07-171-2/+2
| | | | | | | | ABCs in collections should be imported from collections.abc and direct import from collections is deprecated since Python 3.3. Closes-Bug: #1936667 Change-Id: Ia6316894a67cc0ea7ee0d2110565b45546c813e7
* Update TaskFlow for networkx 2.xMichael Johnson2019-10-181-5/+0
| | | | | | | | | | | The networkx 2.x series has been out for two years now and supports python 3.6 and greater[1]. This patch updates TaskFlow to require a minimum of networkx 2.1. It also updates the code to support recent deprecation expiration introduced in the 2.4 release. [1] https://networkx.github.io/documentation/stable/news.html Change-Id: Ife31d353ba80824ebc63c8b21ee90943badc8da3
* Fix code to support networkx > 1.0Michal Arbet2018-07-111-0/+5
| | | | | | | | | | | | | With the release of NetworkX 2.0 the reporting API was moved to view/iterator model. Many methods were moved from reporting lists or dicts to iterating over the information. Methods that used to return containers now return views and methods that returned iterators have been removed in networkx. Because of this change in NetworkX 2.0 , taskflow code have to be changed also to support networkx > 2.0 Change-Id: I23c226f37bd85c1e38039fbcb302a2d0de49f333 Closes-Bug: #1778115
* Use method ensure_tree from oslo.utilsChangBo Guo(gcb)2016-09-251-19/+0
| | | | | | Oslo.utils provides same function and just use it. Change-Id: Iac245d4d98c41edea5294a4d8842db69a42b3794
* Instead of a multiprocessing queue use sockets via asyncoreJoshua Harlow2016-05-241-0/+8
| | | | | | | | | | | | | | | | | | | | | | For a local process based executor usage currently to ensure that task emitted notifications are proxied we use the multi processing library and use its queue concept. This sadly creates a proxy process that gets associated, and this proxy process handles the queue and messages sent to and from it. Instead of doing this we can instead just create a temporary local socket using a random socket and have tasks (which are running in different processes) use that to communicate back any emitted notifications instead (and we can use the asyncore module to handle the emitted notifications since it handles the lower level socket reading, polling and dispatching). To ensure that the socket created is somewhat secure we use a similar process as the multi-processing library uses where we sign all messages with a hmac that uses a one time key that only the main process and the child process know about (and reject any messages that do not validate using this key). Change-Id: Iff9180054bf14495e5667af00ae2fafbdbc23791
* Remove deprecated things for 2.0 releaseJoshua Harlow2016-05-101-12/+0
| | | | Change-Id: Id9adbc50bd51adc77ce88f698ad0ea2ee63fc5e2
* Allow cachedproperty to avoid lockingJoshua Harlow2016-02-081-12/+21
| | | | | | | | | Under some cirumstances it is ok to recompute a value when raced to by multiple threads, for these cases allow the cachedproperty to avoid needing and using a lock. Change-Id: Icb71cb5ea280185cfb120671be900435dabcd50b
* Allow for alterations in decider 'area of influence'Joshua Harlow2016-01-091-31/+0
| | | | | | | | Christmas came early. Closes-Bug: #1479466 Change-Id: I931d826690c925f022dbfffe9afb7bf41345b1d0
* Merge "Use the retrying lib. to do basic sqlalchemy engine validation"Jenkins2016-01-091-24/+0
|\
| * Use the retrying lib. to do basic sqlalchemy engine validationJoshua Harlow2016-01-051-24/+0
| | | | | | | | | | | | | | | | Instead of a custom loop and backoff just use the retrying library to do this same code and delegate the loop complexity to it instead. Change-Id: Iaf02cc728d2a2cfc7077300e03d7ef25522717b7
* | Use shared util helper for driver name + config extractionJoshua Harlow2015-12-221-0/+13
|/ | | | Change-Id: I43465b8f5868e64bdf38d2873417a8a4a403a23b
* Merge "Change name of misc.ensure_dict to misc.safe_copy_dict"Jenkins2015-12-151-2/+7
|\
| * Change name of misc.ensure_dict to misc.safe_copy_dictJoshua Harlow2015-12-141-2/+7
| | | | | | | | Change-Id: I2ff3b9bb4d0b8143163e4bf370f10c74e79dbd31
* | Move validation of compiled unit out of compiler1.25.0Joshua Harlow2015-11-201-13/+0
|/ | | | | | | | | | | | | Instead of having the compiler do any validation on the graph it has created instead have the compiler just compile and have the engine that uses that compiled result do any post compilation validation instead. This makes it more clear that the compiler just compiles a flow (and tasks and nested flows) into a graph, and that is all that it does. Change-Id: I96a35d732dc2be9fc8bc8dc6466256a19ac2df6d
* Add optional 'defer_reverts' behaviorGreg Hill2015-11-171-0/+8
| | | | | | | | | | | This makes it possible to REVERT a subflow and have it also revert the parent flow if the parent flow doesn't have its own retry strategy. We will probably want to make this new behavior the default or only behavior in a future release. Change-Id: Iea5ac366380ba7396a87d0185703549fb0c2f825
* Add a executor backed conductor and have existing impl. use itJoshua Harlow2015-11-121-0/+14
| | | | | | | | | | | | | | | | | | | This adds a executor backed job dispatching base class and has the existing blocking executor use it by running jobs and dispatching jobs into a sync executor. It also allows for dispatching jobs into a thread executor, or other executor via a new '_executor_factory' method that can generate executors (it can be overriden in the non-blocking executor to provide your own executors instances). This does alter the behavior in that now that jobs are dispatched into an executor we no longer can immediatly know if a job was dispatched and raised an exception or whether it will raise an exception in the future, so we now alter the 'local_dispatched' to just be a boolean that is used to determine if any dispatches happened (failure or not). Change-Id: I485770e8f4c85d3833892a453c9fb5168d8f0407
* Extend and improve failure loggingJoshua Harlow2015-08-071-0/+8
| | | | | | | | | Add the ability to show the intention, result, and state of predecessors of the atom that has failed so that more contextual information can be shown in the logs that are written. Change-Id: Ic77c0d4e94a147e54da74976c1d148aef82eccb3
* Update all removal_version from being ? to being 2.0Joshua Harlow2015-07-121-2/+2
| | | | | | | | Before the 2.0 release it would be great to get these removed so let's mark that version as the version that will no longer have these in it. Change-Id: I66a74d270bf95db005e9febfce1a5e211c7a49f6
* Build-out + test a redis backed jobboardJoshua Harlow2015-06-291-13/+36
| | | | | | Part of blueprint taskflow-redis-jobs Change-Id: I7c94e2201c5d933c8a1ec73fc0cf705962e5eef6
* Merge "Use monotonic lib. to avoid finding monotonic time function"Jenkins2015-06-241-29/+0
|\
| * Use monotonic lib. to avoid finding monotonic time functionJoshua Harlow2015-06-161-29/+0
| | | | | | | | | | | | | | | | That library already does this same/equivalent search and ensures that a monotonically increasing time is made available so we can just avoid looking around for it ourselves. Change-Id: I2b5c69430d75a095f6743c10e2d9281f7d9120e0
* | Merge "Remove listener stack and replace with exit stack"Jenkins2015-06-061-32/+0
|\ \ | |/ |/|
| * Remove listener stack and replace with exit stackJoshua Harlow2015-05-111-32/+0
| | | | | | | | Change-Id: Ic9150b8208420d29865af297ae8d0c656becb81e
* | Merge "Refactor/reduce shared 'ensure(task/retry)' code"Jenkins2015-05-301-6/+10
|\ \
| * | Refactor/reduce shared 'ensure(task/retry)' codeJoshua Harlow2015-05-011-6/+10
| |/ | | | | | | | | | | | | | | These methods are nearly identical so we should refactor them to use the same code for sanity and understanding purposes. Change-Id: Ibaf270bd451b6d02d7782901bc2327afe04d3847
* | Small refactoring of 'merge_uri' utility functionJoshua Harlow2015-05-021-12/+20
|/ | | | | | | | Perform some small adjustments/cleanups and add some unit tests to ensure this function keeps operating as expected. Change-Id: I496bd6844072f57624de31fc7ddb0362f163cc53
* Merge "Ensure we register & deregister conductor listeners"Jenkins2015-03-171-0/+32
|\
| * Ensure we register & deregister conductor listenersJoshua Harlow2015-03-131-0/+32
| | | | | | | | | | | | | | | | | | | | Instead of just registering engine listeners that were returned, make sure we also deregister them when the engine has either finished or failed. This ensures that if a listener has hold of any resources (or other) that it can clean those up and be sure that its deregister call will be made. Change-Id: Ia1420c435156362698702fed2bda11c2a0fef803
* | Merge "Added a map and a reduce task"Jenkins2015-03-141-0/+15
|\ \ | |/ |/|
| * Added a map and a reduce taskMin Pae2015-03-121-0/+15
| | | | | | | | | | | | | | - Added a MapFunctorTask and ReduceFunctorTask to mimic the python builtin functions map and reduce, respectively Change-Id: I29cd1249e1671aedfd2d25460d746b8d2154df2b
* | Merge "Add a frozen checking decorator"Jenkins2015-03-041-0/+17
|\ \
| * | Add a frozen checking decoratorJoshua Harlow2015-03-021-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since quite a few of the types check for being frozen and disallow mutations on there instances we can take advantage of a common decorator that checks the frozen attribute and raises instead of duplicating the same logic at the start of the mutating methods. Change-Id: I8c81a26d2d39bb9da4f68d64e07f67ac26ee0b08
* | | Merge "Use debtcollector library to replace internal utility"Jenkins2015-03-031-4/+6
|\ \ \
| * | | Use debtcollector library to replace internal utilityJoshua Harlow2015-02-191-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The new debtcollector library was extracted from this code and has now been released (and made available in the requirements list) so we can go ahead and just use it for most of the cases (minus the proxy moved class) in our code-base to reduce the amount of custom code we have to maintain to perform deprecation activities. Change-Id: Ieeda6ffe282b67a1c1bb4f72e5858d22df0f30a0
* | | | Merge "Tweak functor used to find flatteners/storage routines"Jenkins2015-03-031-0/+12
|\ \ \ \ | |_|/ / |/| | |
| * | | Tweak functor used to find flatteners/storage routinesJoshua Harlow2015-03-021-0/+12
| | |/ | |/| | | | | | | | | | | | | | | | | | | Make both of these finding functions use similar routines that the utility module now provides since the logic that both use can be shared. Change-Id: Ib941b99945d42f5c0d791e9b2a0696d0e62a2388
* | | Fix lookup scoping multi-match orderingJoshua Harlow2015-02-241-0/+31
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When look-up is occurring the possible provider ordering is used instead of the scope returned provider ordering. This causes incorrect matches when look-up argument names that are produced by multiple providers (at the same scope level) all providing the same requirement name. The scope order should be enforced as the de-facto order and not the order that the storage unit finds (which is hash based and varies depending on hash ordering). Closes-Bug: #1425326 Change-Id: I15f1ee5515758bdc470c0f7dd7a2f616923e5628
* | Merge "Use oslo.utils encodeutils for encode/decode functions"Jenkins2015-02-241-13/+12
|\ \
| * | Use oslo.utils encodeutils for encode/decode functionsJoshua Harlow2015-02-221-13/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having similar logic that is nearly the same as what oslo.utils provides just call into the oslo.utils functionality and use it instead. This also allows us to pass a 'errors' keyword parameter and have it be used in oslo.utils to be used when some kind of encoding or decoding issue occurs (by default this is 'strict', as that is the default used by oslo.utils and python itself). Change-Id: Ia73340da7b47e1d450aa4ba41e3905d447691743
* | | Refactor parts of the periodic workerJoshua Harlow2015-02-221-0/+6
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | Perform some adjustments to the periodic worker type to make it easier to later change and make it better in the future. These changes hopefully also reduce complexity and increase understandability. In part this deprecates providing a tombstone via __init__() since that will restrict future enhancements we can make (in retrospect we need more internal control over that argument & type to build better workers). Change-Id: I1965e157c303c2a45b9950e9f4a921c638f57fd1
* | Use the enum library for the retry strategy enumerationsJoshua Harlow2015-02-191-0/+12
|/ | | | | | | | | | | | | | | Instead of having a set of string constants that are the retry strategy/decision result have it instead be an enumerator that can be more clearly associated in docs and in code that these values are explicitly the strategies that the retry code can use (making it more obvious to users of these constants what they are for). This also updates the retry docs to use this new linkable class when describing the various strategies that can be used/returned. Change-Id: I944e521562be26f5315d7553da8d5820fc284c49
* Improve multilock class and its associated unit testJoshua Harlow2015-02-131-0/+15
| | | | | | | | | | | | | | | | | | In the multilock unit test check that all values were acquired (as expected). Also use a class constant for the number of threads to spin up when testing timing overlaps and use the module constant for the duration to wait (to try to trigger overlaps) and use a new helper utility function to make it more obvious what is happening in the multilock release method. Also swaps out the usage of time.time in the lock utils unit test to attempt to use monotonic time (when it can be used) so that false positives are unable to happen (using time.time it is possible for time to go backwards and cause a non-existent overlap to appear). Change-Id: Ifb3967e1c1da41e1b7ac0793fc6a99f84de2a907
* Use monotonic time when/if availableJoshua Harlow2015-01-271-0/+29
| | | | | | | | | | Instead of using a time that can change depending on ntpd or other time adjustments use a monotonically increasing time if it's available from the underlying python library to avoid these types of time-shifts problems in the first place. Change-Id: Ib775a44026b9828536c905a1ed41f527358c0d39
* Make all/most usage of type errors follow a similar patternJoshua Harlow2015-01-231-13/+20
| | | | | | | | | | | | | | | To make it easy to add new type errors and to make the existing ones have a common string pattern adjust the current type errors to contain at least the following string format: '%s' (%s) where these two places will be filled in with the object of the wrong type and the type of that object. This information is useful when analyzing the exception (by the user) to know exactly what they passed in and what type it was. This convention is not maintained where it would interpolate large text blobs (such as in binary encoding/decoding and json decoding). Change-Id: Id84b0e7ce684a543cc407b15016e77804e6f03ed
* Merge "Switch to using 'oslo_serialization' vs 'oslo.serialization'"Jenkins2015-01-231-1/+1
|\
| * Switch to using 'oslo_serialization' vs 'oslo.serialization'Joshua Harlow2015-01-151-1/+1
| | | | | | | | | | | | | | | | Prefer the non-deprecated 'oslo_serialization' instead of the namespaced 'oslo.serialization' wherever it was previously used. Change-Id: I652cf0b56e28d727c59fe0c060949bb2bd920d11
* | Use a class provided logger before falling back to moduleJoshua Harlow2015-01-221-0/+8
|/ | | | | | | | | | | | | If a subclass overrides the logging listeners and provides a class specific logger (under the attribute named _LOGGER) try to use that before using the taskflow specific logging module (and only use this one as a last resort). This allows subclasses to easily override the default logger without having to continually pass in a constructor argument to do the same. Change-Id: I91fea3db6cdd1dfb39963ff9589fd530fe087278
* Switch to using 'oslo_utils' vs 'oslo.utils'Joshua Harlow2015-01-141-3/+3
| | | | | | | | Prefer the non-deprecated 'oslo_utils' instead of the namespaced 'oslo.utils' wherever it was previously used. Change-Id: I9a78150ef5266e1ff22147278162fe3cfe1b2e3f
* Use platform neutral line separator(s)Joshua Harlow2015-01-121-0/+5
| | | | | | | | To at least try to support things like windows it's better if we can make an attempt to use the platform neutral characters for line separator(s) where appropriate. Change-Id: Icc533ed4d4c94f461b7f19600b74146221f17b18
View Lorry Controller Status