summaryrefslogtreecommitdiff
path: root/taskflow/persistence/backends/impl_memory.py
Commit message (Collapse)AuthorAgeFilesLines
* Remove sixTakashi Kajinami2022-05-181-2/+1
| | | | | | | | This library no longer supports Python 2, thus usage of six can be removed. This also removes workaround about pickle library used in Python 2 only. Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
* Fix some misspellings in the function name and descriptionsPablo Iranzo Gómez2016-06-301-1/+1
| | | | Change-Id: I7e3451feb94b1f25b00c5e7b197bb6b527548306
* Remove deprecated things for 2.0 releaseJoshua Harlow2016-05-101-26/+4
| | | | Change-Id: Id9adbc50bd51adc77ce88f698ad0ea2ee63fc5e2
* Add in-memory backend delete() in recursive/non-recursive modesJoshua Harlow2015-10-141-8/+27
| | | | Change-Id: I957c875bf493b2e38bf82af6faccbab8a4861ade
* Update all removal_version from being ? to being 2.0Joshua Harlow2015-07-121-1/+1
| | | | | | | | Before the 2.0 release it would be great to get these removed so let's mark that version as the version that will no longer have these in it. Change-Id: I66a74d270bf95db005e9febfce1a5e211c7a49f6
* Merge "Use a class constant for the default path based backend path"Jenkins2015-06-211-2/+4
|\
| * Use a class constant for the default path based backend pathJoshua Harlow2015-06-101-2/+4
| | | | | | | | | | | | | | | | When no path is provided to a path based backend via configuration use a class constant to provide the default, and override this in backends that support providing defaults. Change-Id: I0a6c88398403a162b113e34abe7e56821d1f02bc
* | Merge "Use hash path lookup vs path finding"0.11.0Jenkins2015-06-151-12/+11
|\ \ | |/ |/|
| * Use hash path lookup vs path findingJoshua Harlow2015-06-061-12/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When creating a fake symlink to a path just use hash lookup via `fetch_node` and handle that failing if the destination nodes does not exist vs. fetching the parent and linear searching for the existing target node (which gets slower as the parent node gets more children). This makes the preparing code using the little speed test helper on my not-very-fast box change to be the following: Old (preparing) - Took 29.724 seconds to run New (preparing) - Took 21.343 seconds to run Part of ongoing blueprint make-things-speedy Change-Id: I608b90ae58b4e4b6724b7f1bb8faebd118a1ec79
* | Add a new `ls_r` methodJoshua Harlow2015-06-041-5/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of having a `ls` method that when used recursively *always* returns the absolute path of items in the fake in memory storage tree and *relative* paths (when used in non-recursive mode) add a new `ls_r` method that can return absolute *or* relative paths. In the future it is highly likely that the the `ls` recursive keyword argument will be removed (so preferring and moving to the `ls_r` should occur earlier rather than later), so this also adds a debtcollector removed keyword argument decorator over the existing `ls` to ensure that users are aware of this change (as well as a adjusted docstring). Fixes bug 1458114 Change-Id: Id2a5869e94ac44679020a14297d1073d1dc2718f
* | Merge "Replace lock_utils lock(s) with fasteners package"Jenkins2015-06-041-2/+2
|\ \
| * | Replace lock_utils lock(s) with fasteners packageJoshua Harlow2015-06-021-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | The usage of this package allows us to get rid of the reader/writer lock (since that package provides that) as well as the interprocess lock as all of these are now provided by that package instead. Change-Id: I87990b46d397f6df779de7028bcc40e28621e1ba
* | | Merge "Ensure empty paths raise a value error"Jenkins2015-05-301-0/+3
|\ \ \ | |/ / |/| |
| * | Ensure empty paths raise a value errorJoshua Harlow2015-05-111-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Empty paths should not be allowed to be set or fetched or normalized so check for those and raise an error when one is encountered. Also add some basic test conditions that ensure the normpath method raises when it should. Change-Id: I7f7e6600f03c67376ba310ab231b2e33cd7528db
* | | Merge "Avoid creating temporary removal lists"Jenkins2015-05-141-3/+3
|\ \ \ | |_|/ |/| |
| * | Avoid creating temporary removal listsJoshua Harlow2015-05-081-3/+3
| |/ | | | | | | | | | | | | | | | | | | | | | | Instead of creating a temporary list, only to then iterate over it (and then never again use that temporary list) just use itertools and create an iterator that we use for iterating (therefore avoiding any need to create a temporary list). Part of ongoing blueprint make-things-speedy Change-Id: I5322e5bdf613d485fbc8851c1319a907b425e2dd
* | Merge "Expose in memory backend split staticmethod"Jenkins2015-05-121-2/+5
|\ \
| * | Expose in memory backend split staticmethodJoshua Harlow2015-05-111-2/+5
| |/ | | | | | | | | | | | | | | | | | | | | | | Just like we expose join we should also expose split to ensure that both of these operations can easily performed by users of this backend. This also refactors the internal usage to call into that static method vs calling into the posix path module specific one. Change-Id: I496c730b86f0af6d4b637862a92482c0df6a63b3
* | Avoid re-normalizing paths when following linksJoshua Harlow2015-05-071-3/+3
|/ | | | | | | | | | | | | | | There is not any need to re-normalize internal paths once the initial get() or __get_item__() have normalized there path arguments so in _get_item() just tell the fetching function to avoid re-normalizing the paths its provided. This also works for link following since the target placed in the fake inodes metadata is already normalized when it is inserted. Part of blueprint make-things-speedy Change-Id: If31d61b744f8df618c7db3afffc4a118cb17003a
* Speed up memory backend via a path -> node reverse mappingJoshua Harlow2015-05-051-34/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A simple reverse mapping/dict is added and used to avoid repeated linear finding of paths, and metadata (and a little tiny helper inode class) is added so that each node knows its full path (avoiding repeated traversal to determine this same info). For linear flow with 500 'dummy' nodes on an old machine running python 2.6 (your numbers will vary): Old --- Took 0.255 seconds to run building Took 0.456 seconds to run compiling Took 50.039 seconds to run preparing Took 0.796 seconds to run validating Took 2.853 seconds to run running New --- Took 0.254 seconds to run building Took 0.457 seconds to run compiling Took 30.780 seconds to run preparing Took 0.776 seconds to run validating Took 2.222 seconds to run running Change-Id: Idc9ce88865b58a2dffd2e8955f0fab7f056d92b3
* Expose fake filesystem 'join' and 'normpath'Joshua Harlow2015-04-231-9/+16
| | | | Change-Id: I9572b2b739c2a1743df1bd6222a9d8dcff77cb4d
* Merge "Use the ability to chain exceptions correctly"Jenkins2015-04-181-3/+4
|\
| * Use the ability to chain exceptions correctlyJoshua Harlow2015-04-031-3/+4
| | | | | | | | | | | | | | | | | | | | | | In the zookeeper jobboard (and persistence backends) we are associating the cause of a new exception when raising a exception. Using the new exception helper we can make this work better on py2.x and py3.x so that the py3.x version has the chain setup correctly (while the py2.x version just uses the class 'cause' attribute instead). Change-Id: Ieeac2f70e1834d4612556565762ffd3be3e5b5a1
* | Add note about thread safety of fake filesystemJoshua Harlow2015-04-081-0/+10
| | | | | | | | Change-Id: Ic9851455b559a94bb011c27ba55ad050e6bd6d55
* | Merge "Add in memory filesystem clearing"Jenkins2015-04-081-0/+4
|\ \
| * | Add in memory filesystem clearingJoshua Harlow2015-03-291-0/+4
| |/ | | | | | | Change-Id: Ibd22eddeec87f660e729099f702f52cdd6b828ea
* | Merge "Allow providing a node stringify function to tree pformat"Jenkins2015-04-081-1/+10
|\ \
| * | Allow providing a node stringify function to tree pformatJoshua Harlow2015-03-291-1/+10
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | Allow providing a node to string function that will be activated when the root node pformat routine needs to format a node into a string (for viewing purposes); this allows for subclasses or other users of nodes to provide there own function that they can specialize as they see fit (without duplicating the pformat routine itself). To start use it to print out the node link target in the fake in-memory filesystem when a link is encountered. Change-Id: I33b8f1ec8d30d72ae35971e35f5edd84a9145db7
* | Merge "Add memory backend get() support"Jenkins2015-04-051-0/+9
|\ \
| * | Add memory backend get() supportJoshua Harlow2015-03-251-0/+9
| |/ | | | | | | | | | | | | | | | | | | Instead of requiring users to use the __getitem__ fetch entrypoint and deal with the value not being found provide a dictionary equivalent get() support that handles the not found case and returns a default when this is triggered. Change-Id: Ie3795745515a9d39ef4321fecb78e9b0e7539d72
* | Allow ls() to list recursively (using breadth-first)Joshua Harlow2015-03-311-2/+18
|/ | | | | | | | | | | | Make it easier to list the contents of the full in-memory filesystem by allowing the ls() function to take a recursive keyword argument (which by default is false). This makes it easier to analyze the contents of the full in memory filesystem; saving people from creating similar code themselves. Change-Id: I573797945255cb81728e7a86b58768b848110ee7
* Add more comments to fake in-memory filesystemJoshua Harlow2015-03-251-1/+25
| | | | | | | | | | | Ensure users of this code/objects are aware that only posixpath style paths will work (and not the variant that is used in os.path which can change depending on operating system) as well as adding docstrings on other methods so that they show up in the generated docs. Change-Id: I5a21cb75b2452b9c9d65860b63658a9d0980025f
* Prefer posixpath to os.pathJoshua Harlow2015-03-241-9/+9
| | | | | | | | | | | | | | | | | | To ensure that the memory fake filesystem works uniformly across distributions prefer to use the posixpath module which always works with '/' and friends instead of the os.path and os.path.sep and friends which may vary depending on operating system used. Since we have tested with the usage of '/' we might as well just restrict to that; and avoid the cross operating system issues that may pop up when using this fake filesystem. In general isolating one self from the operating system specifics is IMHO preferable for this; as it avoids edge cases that we don't care about. Change-Id: I3f61f380e1bcb131bc42b627adf9dfe8a7f2d992
* Allow using shallow copy instead of deep copyJoshua Harlow2015-03-221-4/+9
| | | | | | | | | | | Certain projects (cinder at the least) seems to be storing items that can not be deep copied without issues; so to make it so that we can easily support having this backend be the default backend always used we need to be able to support a shallow copy for when it is needed. Change-Id: I16215b44870cef9ea60d144ba710c5d730e8b284
* Merge "Rename memory backend filesystem -> fake filesystem"Jenkins2015-03-231-2/+2
|\
| * Rename memory backend filesystem -> fake filesystemJoshua Harlow2015-03-181-2/+2
| | | | | | | | | | | | | | | | To ensure that no-one gets confused and believes this is a real filesystem class, rename it to be a name that clearly tells it is not a real filesystem. Change-Id: I47b4018ff50019662492e1865d9cac5e8d43d3bf
* | Ensure thread-safety of persistence dir backendJoshua Harlow2015-03-201-1/+1
|/ | | | | | | | | | Even though this backend is protected by process locks against other processes we still need to protect against threads in the same process; so to do this grab a write lock and use it in the same manner that the memory backend does; as a per-backend write-lock. Change-Id: Ifb86631033aad1ca622be9cf3c3677293d8c56bd
* Add a in-memory backend dumping exampleJoshua Harlow2015-03-121-8/+11
| | | | Change-Id: Ie29735a16129cedd574e09ed06b44a98560f0d90
* Restructure the in-memory node usageJoshua Harlow2015-03-121-34/+123
| | | | | | | | | | | | | Instead of retaining full paths only retain partial paths; this then allows for finding a parent node and then easily finding a child by using the find() routines. This more naturally emulates a real filesystem using our tree structure and simplifies certain operations to make them more efficient/simple. Change-Id: I207b02c979704a15eac8be46d11a0af61b05d9b9
* Persistence backend refactorDan Krause2015-03-091-208/+85
| | | | | | | Factors lots of duplicate code out of persistence backends Adds get_flows_for_book to all backends Change-Id: I0434bd4931cd9274876f9e9c92909531f244bcac
* add get_flow_details and get_atom_details to all backendsDan Krause2015-02-191-0/+14
| | | | Change-Id: Ic21a883b4eee827d90065f8ecfc5af2599ebece5
* Make all/most usage of type errors follow a similar patternJoshua Harlow2015-01-231-2/+2
| | | | | | | | | | | | | | | To make it easy to add new type errors and to make the existing ones have a common string pattern adjust the current type errors to contain at least the following string format: '%s' (%s) where these two places will be filled in with the object of the wrong type and the type of that object. This information is useful when analyzing the exception (by the user) to know exactly what they passed in and what type it was. This convention is not maintained where it would interpolate large text blobs (such as in binary encoding/decoding and json decoding). Change-Id: Id84b0e7ce684a543cc407b15016e77804e6f03ed
* Merge "Move the persistence base to the parent directory"Jenkins2015-01-211-1/+1
|\
| * Move the persistence base to the parent directoryJoshua Harlow2014-12-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | In order to match the directory/module layout of the other pluggable backends better move the persistence base module that defines the base abstract classes up into the parent directory. This makes it easier to look at the taskflow code-base and understand the common layout. Change-Id: I7887cb0241b8fe65cbdfee32c101c3df5f05d27c
* | Rework the in-memory backendJoshua Harlow2014-12-181-68/+153
|/ | | | | | | | | | | | | | | | | | | | This avoids storing direct copies of incoming objects and makes sure that we always merge incoming objects (if a saved object already exists) or create a copy of the incoming object if it does not exist when storing. On retrieval we also always return copies instead of returning the data that is stored internally to avoid the problems that can be hard to detect when users (engine or other) modify those source objects. Fixes bug 1365830 Also fixes a retry test case issue that was discovered due to this more easily useable/understandable memory backend changes... Change-Id: I2afdda7beb71e35f7e12d9fd7ccf90b6c5447274
* Add and use a new simple helper logging moduleJoshua Harlow2014-12-081-2/+1
| | | | | | | | | Add a new logging BLATHER level to easily allow its usage for messages that are below the normal DEBUG level such as compilation information and scope lookup info which can be very verbose in logs if always enabled. Change-Id: I828211403bd02bfd6777b10cdcfe58fb0637a52c
* Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-4/+4
| | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Default the impl_memory conf to noneJoshua Harlow2014-05-101-1/+1
| | | | | | | | This configuration isn't used for anything in the memory backend so default it to none so that it doesn't need to be provided (since it's not used). Change-Id: I9e333aec52b697ebd3d5b1f4ec14d810e99bab19
* Persistence cleanup part oneJoshua Harlow2014-03-261-49/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | - Convert the various functions that take a task detail into ones that take atom details (since this is now the generic type they should take). - Don't expose the detail type strings as part of the atom detail api, leave those as private hidden strings and provide conversion functions from string<->class instead. - Have the logbook objects contain the following new methods to reduce the dependence on persistence_utils to do the same. - to_dict() which converts the current object into a dict - from_dict() which converts the provided dict into a object - merge() which merges a incoming objects data with the current objects - Have the persistence backends + storage + action engine use these new methods instead of there current usage. - Don't compare to logbook.RETRY_DETAIL or logbook.TASK_DETAIL since python has the isinstance function just use it (ideally we should fix the code so that this isn't even needed, usage of isinstance means something is not designed/structured right). - In storage tests we can't assume that failures will be non-lossy since under certain backends when a failure is stored information about the internally held exc_info is lost, so take this into account when testing by using matches() where applicable. Change-Id: Ie8a274cfd4cb4e64e87c355dc99d466d74a4e82c
* Extend logbook and storage to work with retryAnastasia Karpinska2014-03-141-2/+6
| | | | | | | | - Store AtomType in TaskDetail (TASK_DETAIL or RETRY_DETAIL) - Extend storage to work with retries - Add alembic migration for database Change-Id: I07896d4b79ebe5de33f65e00a3bd32af543a09b4
View Lorry Controller Status