summaryrefslogtreecommitdiff
path: root/taskflow/persistence/backends/impl_dir.py
Commit message (Collapse)AuthorAgeFilesLines
* Use method ensure_tree from oslo.utilsChangBo Guo(gcb)2016-09-251-1/+2
| | | | | | Oslo.utils provides same function and just use it. Change-Id: Iac245d4d98c41edea5294a4d8842db69a42b3794
* Fix how the dir persistence backend was not listing logbooksJoshua Harlow2015-09-041-2/+6
| | | | | | | | | | Due to the usage of the os.path.islink check this means that no logbooks would be returned when get_logbooks was called, which is not the behavior we want. Closes-Bug: #1492403 Change-Id: Ife6a5bec777c9e2d820391914ce2c6fbbadf4f79
* Use io.open vs raw openJoshua Harlow2015-07-101-4/+4
| | | | | | | | The io.open call can take in a encoding so we don't need to read in binary mode, then convert it since it can just do that on our behalf. Change-Id: I0cce2841b40f1566ba07ff95a553cb18ea9059ee
* Make the default file encoding a class constant with a docstringJoshua Harlow2015-06-101-1/+7
| | | | | | | | Having this exposed makes it easier to know what it is being used for and what the default is (and also makes it show up in generated docs, so people can read all about it). Change-Id: I2e0f85d9c087d220671e2dbf82d497677c462a9e
* Use a lru cache to limit the size of the internal file cacheJoshua Harlow2015-06-101-2/+13
| | | | | | | | | | | | Instead of having an unbounded internal file cache which will eventually absorb all memory of the running/containing python process have the size of that cache be limited by an optionally provided size (and have eviction be based on how recent a cached entry was used). Fixes bug 1458248 Change-Id: I5e59efc4edd51b05cfb1e67d3e7014e378e352aa
* Replace lock_utils lock(s) with fasteners packageJoshua Harlow2015-06-021-3/+3
| | | | | | | | | The usage of this package allows us to get rid of the reader/writer lock (since that package provides that) as well as the interprocess lock as all of these are now provided by that package instead. Change-Id: I87990b46d397f6df779de7028bcc40e28621e1ba
* Use the ability to chain exceptions correctlyJoshua Harlow2015-04-031-2/+6
| | | | | | | | | | | In the zookeeper jobboard (and persistence backends) we are associating the cause of a new exception when raising a exception. Using the new exception helper we can make this work better on py2.x and py3.x so that the py3.x version has the chain setup correctly (while the py2.x version just uses the class 'cause' attribute instead). Change-Id: Ieeac2f70e1834d4612556565762ffd3be3e5b5a1
* Merge "Ensure thread-safety of persistence dir backend"Jenkins2015-03-211-2/+5
|\
| * Ensure thread-safety of persistence dir backendJoshua Harlow2015-03-201-2/+5
| | | | | | | | | | | | | | | | | | | | Even though this backend is protected by process locks against other processes we still need to protect against threads in the same process; so to do this grab a write lock and use it in the same manner that the memory backend does; as a per-backend write-lock. Change-Id: Ifb86631033aad1ca622be9cf3c3677293d8c56bd
* | Use binary/encode decode helper routines in dir backendJoshua Harlow2015-03-181-4/+5
|/ | | | | | | | | | | | | | Instead of using smaller versions of those routines just call into the same helper routines that the rest of the code uses. This also allows that backend to take a configuration option that specifies the encoding that will be used for when translating unicode strings into binary (and vice versa); this defaults to utf-8 to retain the existing behavior. Change-Id: Icfcbb22310e163753d8b34af212efc166c82634b
* Persistence backend refactorDan Krause2015-03-091-338/+68
| | | | | | | Factors lots of duplicate code out of persistence backends Adds get_flows_for_book to all backends Change-Id: I0434bd4931cd9274876f9e9c92909531f244bcac
* add get_flow_details and get_atom_details to all backendsDan Krause2015-02-191-0/+6
| | | | Change-Id: Ic21a883b4eee827d90065f8ecfc5af2599ebece5
* Merge "Switch to using 'oslo_serialization' vs 'oslo.serialization'"Jenkins2015-01-231-1/+1
|\
| * Switch to using 'oslo_serialization' vs 'oslo.serialization'Joshua Harlow2015-01-151-1/+1
| | | | | | | | | | | | | | | | Prefer the non-deprecated 'oslo_serialization' instead of the namespaced 'oslo.serialization' wherever it was previously used. Change-Id: I652cf0b56e28d727c59fe0c060949bb2bd920d11
* | Merge "Move the persistence base to the parent directory"Jenkins2015-01-211-1/+1
|\ \
| * | Move the persistence base to the parent directoryJoshua Harlow2014-12-191-1/+1
| |/ | | | | | | | | | | | | | | | | | | | | | | In order to match the directory/module layout of the other pluggable backends better move the persistence base module that defines the base abstract classes up into the parent directory. This makes it easier to look at the taskflow code-base and understand the common layout. Change-Id: I7887cb0241b8fe65cbdfee32c101c3df5f05d27c
* | Move implementation(s) to there own sectionsJoshua Harlow2015-01-161-4/+4
|/ | | | | | | | | | Instead of putting implementation(s) under the interfaces section put the implementation(s) under there own section. This also includes some other tweaks to refer to those implementation(s) where appropriate. Change-Id: Iffdc0439c843e7f70cf873e5a75501feb51f96c7
* Add and use a new simple helper logging moduleJoshua Harlow2014-12-081-1/+1
| | | | | | | | | Add a new logging BLATHER level to easily allow its usage for messages that are below the normal DEBUG level such as compilation information and scope lookup info which can be very verbose in logs if always enabled. Change-Id: I828211403bd02bfd6777b10cdcfe58fb0637a52c
* Switch to using oslo.utils and oslo.serializationJoshua Harlow2014-09-201-1/+1
| | | | | | | | | | | | Instead of copying modules from the incubator into taskflow we can now directly use these same modules from supported libraries instead so this moves the usage of everything except uuidutils which wasn't moved over to using those newly published libraries. Part of blueprint integrate-and-use-oslo-utils-serialization Change-Id: I1183bda96e1ddb062d9cab91990186f0f56f0a0e
* Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-4/+18
| | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Persistence cleanup part oneJoshua Harlow2014-03-261-69/+66
| | | | | | | | | | | | | | | | | | | | | | | | | | | - Convert the various functions that take a task detail into ones that take atom details (since this is now the generic type they should take). - Don't expose the detail type strings as part of the atom detail api, leave those as private hidden strings and provide conversion functions from string<->class instead. - Have the logbook objects contain the following new methods to reduce the dependence on persistence_utils to do the same. - to_dict() which converts the current object into a dict - from_dict() which converts the provided dict into a object - merge() which merges a incoming objects data with the current objects - Have the persistence backends + storage + action engine use these new methods instead of there current usage. - Don't compare to logbook.RETRY_DETAIL or logbook.TASK_DETAIL since python has the isinstance function just use it (ideally we should fix the code so that this isn't even needed, usage of isinstance means something is not designed/structured right). - In storage tests we can't assume that failures will be non-lossy since under certain backends when a failure is stored information about the internally held exc_info is lost, so take this into account when testing by using matches() where applicable. Change-Id: Ie8a274cfd4cb4e64e87c355dc99d466d74a4e82c
* Adjust the exception hierachyJoshua Harlow2014-03-241-12/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Group the exceptions into the following groups * Storage * Jobs * Execution * Other (wrapped failure here) This grouping makes it easier to understand where one type of exception should be used vs using another type of exception. Backwards incompatible changes: * StorageError -> StorageFailure * AlreadyExists -> Duplicate * WrappedFailure now inherits from Exception and not directly from TaskFlowException since it wraps arbitrary other exceptions and is not specific to taskflow. Cleanups: * JobNotFound -> NotFound * EmptyFlow -> Empty * JobAlreadyExists -> AlreadyExists * InvariantViolation (X) * ConnectionFailure (X) Change-Id: I0e1e81b513fbbc7adb8bfaa1244993e345ab70d3
* Unify usage of storage error exception typeJoshua Harlow2014-02-241-16/+33
| | | | | | | | | | | | - Instead of formatting the exception by creating a storage error instance we can just let the already included cause make this formatting occur. - Handle cases in impl_dir which were not catching and raising storage errors. - Unify usage of 'backend internal error' messaging. - Pass the cause to storage errors where applicable. Change-Id: Ieb14256b202ccbfa3a96f68fa35db7e40f92114d
* Switch to using the type checking decode_jsonJoshua Harlow2014-02-201-3/+3
| | | | | | | | | To avoid the problem of user file modification into json of a non-dictionary type we should use the decode_json helper that checks the root elements type to help protect against these types of misuse. Change-Id: I85889eaeee90650bc9b33f7e9ef7424eab8750b2
* Remove extraneous vim configuration commentsyangxurong2014-02-141-2/+0
| | | | | | | | | Remove line containing comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4 Change-Id: I7581cc88b8de433d5609ed06c6570b0b45c13573 Closes-Bug:#1229324
* Storage protects lower level backend against thread safetyJoshua Harlow2014-02-101-38/+2
| | | | | | | | | | | | | | | | | | | | | | | The storage changes to use a read/write lock provide locking guarantees that we do not need to duplicate in the underlying storage. This is predicated on the fact that we do the following: 1. A user before running upgrades the schema of there backend before giving the logbook/backend to the engine. This means that the backend upgrade() method does not need to be thread safe. 2. One engine working on one logbook at a given time. 3. An engine accesses and uses the logbook and its flow/task details by accessing data via the storage accessor interface. 4. Tasks do not currently get direct access to storage or the underlying logbook while running (in the future we could provide access to a storage object, but we would need to have a contract that ensures a task can not hijack an engines usage of storage by deadlocking the engine). Change-Id: I0efd5333928698802d25e1ff124e781f9a4907d9
* Add a validate method to dir and memory backendsJoshua Harlow2014-02-021-0/+13
| | | | | | | | | | In order to allow all connections to be validated post-creation add a new method onto the memory backend (a no-op) and add a new method onto the dir backend (which checks directory existence) and raise appropriate errors when validation fails. Change-Id: Ie476d0c33a970e543ffba21ee8b669ac02258923
* Implement ZooKeeper as persistence storage backendChangbin Liu2014-01-281-86/+9
| | | | | | | | | | | | | | | | | Test cases are in taskflow/tests/unit/persistence/test_zk_persistence and taskflow/tests/unit/persistence/test_zake_persistence. The former requires a working ZooKeeper cluster, and the latter uses a fake in-memory backend. Right now, due to the lack of ZooKeeper in Jenkins, the former test is disabled. To enable it, simply uncomment the code and change "hosts" in conf to your ZooKeeper cluster's addresses. Also moved some shared helper functions from taskflow/persistence/backends/impl_dir to taskflow/utils/persistence_utils. Implements: blueprint zk-logbook Change-Id: Ia1b7ef8312a1761515515fd206b36be135119737
* Doc strings and comments clean-upskudriashev2014-01-261-2/+2
| | | | | | | | * Added missing period for doc strings * Correct syntax errors * Remove H402 from flake8 ignore list Change-Id: Ia8592bf99378e3658d6cca2ceb148bf9eb0b5de8
* Move six to the right locationJoshua Harlow2014-01-091-1/+2
| | | | | | | | | Third party imports are supposed to be after standard library imports so make six be after. Some newlines are also added to separate standard library imports from all the others. Co-authored-by: Ivan A. Melnikov <imelnikov@griddynamics.com> Change-Id: Ied067e9367612758666da726df195ed390215e1b
* Fix misspellingsStanislav Kudriashev2013-12-101-2/+2
| | | | Change-Id: I6f332f01d197c6ba1b02de1145714718f8aea6fb
* Fix several python3 incompatibilitiesIvan A. Melnikov2013-10-111-2/+5
| | | | Change-Id: Ibf5dd6c0b6bcd161364daf35d618641f6079acf5
* Remove decorators and move to utilsJoshua Harlow2013-10-071-8/+7
| | | | | | | | | | | | | In order to avoid the circular import in threading utils move the decorators functionality to utils/misc and move the locking functionality to utils/lock_utils and then use these functions from the threading util (and elsewhere). Fixes bug: 1236080 Change-Id: I9e71c2ba15782cbb6dd5ab7e1264b77ed47bc29e
* Wrapping and serializing failuresIvan A. Melnikov2013-10-041-4/+3
| | | | | | | | | | | | * save task failures to sqlalchemy backend and restore them from there; * for any wrapped exception use common WrappedFailre exception type, which makes it easier to handle particular exception types; * several minor improvements. Affects-db: this change adds db migration that looses exception information for saved task details. Change-Id: I575282002e6999646bbf51f492b82a7e3525787a
* Add a directory/filesystem based persistence layerJoshua Harlow2013-09-211-0/+474
Add a nice non-memory-based but also non-db based persistence layer which is another good example of how a persistence layer can be created (and used). Example directory structure: /books /books/247c5311-d4ec-461b-9e76-51830d6a75b2 /books/247c5311-d4ec-461b-9e76-51830d6a75b2/metadata /books/247c5311-d4ec-461b-9e76-51830d6a75b2/flows /books/247c5311-d4ec-461b-9e76-51830d6a75b2/flows/25f18828-a067-411e-9035-8217536f925d /flows /flows/25f18828-a067-411e-9035-8217536f925d /flows/25f18828-a067-411e-9035-8217536f925d/metadata /flows/25f18828-a067-411e-9035-8217536f925d/tasks /flows/25f18828-a067-411e-9035-8217536f925d/tasks/a352fa2e-82cf-4c37-89ae-3aa10dbf1437 /tasks /tasks/a352fa2e-82cf-4c37-89ae-3aa10dbf1437 Change-Id: I63aaf56497187e21469bc500a49dd02de0c67f29
View Lorry Controller Status