summaryrefslogtreecommitdiff
path: root/taskflow/utils/persistence_utils.py
Commit message (Collapse)AuthorAgeFilesLines
* python3.0 has deprecated LOG.warnxhzhf2017-03-071-2/+2
| | | | | | | | python3.0 has deprecated LOG.warn https://docs.python.org/3/library/logging.html#logging.warning Closes-Bug: #1650843 Change-Id: Ib8473cbdb84547f385e619fe08ed723b1ee7bcdb
* Retrieve the store from flowdetails as well, if it existsGreg Hill2016-01-251-2/+9
| | | | | | | | Gives users a more permanent way to provide an initial set of arguments to a flow. Change-Id: Ib9c3d60882548120d467a645bbac9be78408bac3 Implements: blueprint flow-details-keep-store
* Remove **most** usage of taskflow.utils in examplesJoshua Harlow2015-07-171-75/+0
| | | | | | | | | | It appears folks are using the taskflow.utils code in there own code-bases (likely taking it from the examples) which we do not want to encourage, so remove the usage of **most** of taskflow.utils code from the examples so that people are less likely to copy/paste/reference it. Change-Id: I0ce3c520de347e3e746e7912aa1366a515458424
* Rename logbook module -> models moduleJoshua Harlow2015-07-081-5/+5
| | | | | | | | | | | Since this module contains more than the logbook class and really is a our generic models that are used to hold the runtime structure it is more appropriate to place it under a models module and deprecate the usage of the old module by placing a warning there (so that when it is imported that warning is triggered). Change-Id: I79def5ee08f560d38f2c9dcefd0b33becc2a4d36
* Switch to using 'oslo_utils' vs 'oslo.utils'Joshua Harlow2015-01-141-2/+2
| | | | | | | | Prefer the non-deprecated 'oslo_utils' instead of the namespaced 'oslo.utils' wherever it was previously used. Change-Id: I9a78150ef5266e1ff22147278162fe3cfe1b2e3f
* Use platform neutral line separator(s)Joshua Harlow2015-01-121-3/+4
| | | | | | | | To at least try to support things like windows it's better if we can make an attempt to use the platform neutral characters for line separator(s) where appropriate. Change-Id: Icc533ed4d4c94f461b7f19600b74146221f17b18
* Move over to using oslo.utils [reflection, uuidutils]Joshua Harlow2014-12-181-1/+1
| | | | | | | | | | | | | | The reflection module is now part of oslo.utils so we should remove our local version and use that version instead; this also goes for the uuidutils module which is now part of oslo.utils as well so we no longer need our local version copied from the incubator... Note that one reflection method `find_subclasses` which was to specific to taskflow is now moved to the misc utility module instead of its prior home in the reflection module. Change-Id: I069881c80b0b2916cc0c414992b80171f7eeb79f
* Add and use a new simple helper logging moduleJoshua Harlow2014-12-081-1/+1
| | | | | | | | | Add a new logging BLATHER level to easily allow its usage for messages that are below the normal DEBUG level such as compilation information and scope lookup info which can be very verbose in logs if always enabled. Change-Id: I828211403bd02bfd6777b10cdcfe58fb0637a52c
* Switch to using oslo.utils and oslo.serializationJoshua Harlow2014-09-201-1/+2
| | | | | | | | | | | | Instead of copying modules from the incubator into taskflow we can now directly use these same modules from supported libraries instead so this moves the usage of everything except uuidutils which wasn't moved over to using those newly published libraries. Part of blueprint integrate-and-use-oslo-utils-serialization Change-Id: I1183bda96e1ddb062d9cab91990186f0f56f0a0e
* Upgrade hacking version and fix some of the issuesJoshua Harlow2014-06-131-6/+14
| | | | | | | | | | | | | | | | | Update hacking to the new requirements version and fix about half of the new reported issues. The other hacking issues are for now ignored until fixed by adjusting our tox.ini file. This commit fixes the following new hacking errors: H405 - multi line docstring summary not separated with an empty line E265 - block comment should start with '# ' F402 - import 'endpoint' from line 21 shadowed by loop variable Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
* Persistence cleanup part oneJoshua Harlow2014-03-261-220/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | - Convert the various functions that take a task detail into ones that take atom details (since this is now the generic type they should take). - Don't expose the detail type strings as part of the atom detail api, leave those as private hidden strings and provide conversion functions from string<->class instead. - Have the logbook objects contain the following new methods to reduce the dependence on persistence_utils to do the same. - to_dict() which converts the current object into a dict - from_dict() which converts the provided dict into a object - merge() which merges a incoming objects data with the current objects - Have the persistence backends + storage + action engine use these new methods instead of there current usage. - Don't compare to logbook.RETRY_DETAIL or logbook.TASK_DETAIL since python has the isinstance function just use it (ideally we should fix the code so that this isn't even needed, usage of isinstance means something is not designed/structured right). - In storage tests we can't assume that failures will be non-lossy since under certain backends when a failure is stored information about the internally held exc_info is lost, so take this into account when testing by using matches() where applicable. Change-Id: Ie8a274cfd4cb4e64e87c355dc99d466d74a4e82c
* Add atom intentions for tasks and retriesAnastasia Karpinska2014-03-181-4/+6
| | | | | | | | | | | | Add atom intentions: REVERT, EXECUTE, RETRY and IGNORE. Intentions will be used by action engine to schedule tasks correctly. Add intention to task detail and extend storage to work with atom intentions. Add alembic migration to add intentions column to database. Change-Id: I79c9bb5f11861658dbfedfd64ef93eb92b29cb2d
* Extend logbook and storage to work with retryAnastasia Karpinska2014-03-141-4/+35
| | | | | | | | - Store AtomType in TaskDetail (TASK_DETAIL or RETRY_DETAIL) - Extend storage to work with retries - Add alembic migration for database Change-Id: I07896d4b79ebe5de33f65e00a3bd32af543a09b4
* Correct LOG.warning in persistence utilsStanislav Kudriashev2014-03-071-1/+1
| | | | Change-Id: Ibfd0bd8ce452810161d25d3ac5441eb9ed1e0e13
* Remove extraneous vim configuration commentsyangxurong2014-02-141-2/+0
| | | | | | | | | Remove line containing comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4 Change-Id: I7581cc88b8de433d5609ed06c6570b0b45c13573 Closes-Bug:#1229324
* Implement ZooKeeper as persistence storage backendChangbin Liu2014-01-281-0/+73
| | | | | | | | | | | | | | | | | Test cases are in taskflow/tests/unit/persistence/test_zk_persistence and taskflow/tests/unit/persistence/test_zake_persistence. The former requires a working ZooKeeper cluster, and the latter uses a fake in-memory backend. Right now, due to the lack of ZooKeeper in Jenkins, the former test is disabled. To enable it, simply uncomment the code and change "hosts" in conf to your ZooKeeper cluster's addresses. Also moved some shared helper functions from taskflow/persistence/backends/impl_dir to taskflow/utils/persistence_utils. Implements: blueprint zk-logbook Change-Id: Ia1b7ef8312a1761515515fd206b36be135119737
* Doc strings and comments clean-upskudriashev2014-01-261-9/+8
| | | | | | | | * Added missing period for doc strings * Correct syntax errors * Remove H402 from flake8 ignore list Change-Id: Ia8592bf99378e3658d6cca2ceb148bf9eb0b5de8
* Fix function name typo in persistence utilsskudriashev2014-01-191-4/+4
| | | | Change-Id: Id34669d0fdaeba99e81b2bae55d529f81214e4fc
* Fix up python 3.3 incompatabilitiesJoshua Harlow2013-11-211-1/+1
| | | | | | | | | | | | Make the python 3.3 testing work by selectively disabling & including eventlet, switch to testtools and testrepository which has 2.6, 2.7, 3.2+ unified testing support so that we can correctly run our tests in all supported python versions. Closes-Bug: #1251660 Co-authored-by: Alexander Gorodnev <agorodnev@griddynamics.com> Change-Id: I23b6f04387cfd3bf6b5a044edffa446ca897ce3a
* Remove uuid from taskflow.flow.FlowIvan A. Melnikov2013-10-151-10/+5
| | | | | | | In a way our resumption works it may not correspond uuid from flow details, and so it is hardly useful. Change-Id: I090d017e2f0f8475594af94a2430a34e6ed1ce70
* Merge "Helpers to save flow factory in metadata"Jenkins2013-10-141-1/+6
|\
| * Helpers to save flow factory in metadataIvan A. Melnikov2013-10-141-1/+6
| | | | | | | | | | | | | | | | | | | | | | This change adds new helpers that, taken together, allow to resume the flows having nothing but flow detail at hands. First one, load_from_factory, gets flow factory function as a parameter and saves its fully qualified name and arguments to flow metadata. Others can be used to re-create the flow using that metadata, and load it into engine. Change-Id: Ia3cd989b3b0388ec0a9f09fe527f768eec5cc904
* | Merge "Create logbook if not provided for create_flow_details"Jenkins2013-10-141-2/+5
|\ \ | |/
| * Create logbook if not provided for create_flow_detailsIvan A. Melnikov2013-10-141-2/+5
| | | | | | | | | | | | | | This simplifies code and gives flow details correct name even if log book was not provided. Change-Id: Icef03ff843002f15c3ed90fd8c44d69c0bd4d9f0
* | Add a persistence util logbook formatting functionJoshua Harlow2013-10-121-0/+72
|/ | | | | | | | It is quite useful to be able to dump what is in a logbook in a human readable format so this adds a helper utility function to do just this. Change-Id: I84d8a9caf531de018fb361211624fc5b8c030260
* Correctly save task versionsIvan A. Melnikov2013-10-051-0/+2
| | | | | | | When task is added to storage or task detail is saved to persistence backend, task version should be saved with it. Change-Id: I43e97cb621c3ffbbc27cafdc206ac39389336a24
* Wrapping and serializing failuresIvan A. Melnikov2013-10-041-4/+38
| | | | | | | | | | | | * save task failures to sqlalchemy backend and restore them from there; * for any wrapped exception use common WrappedFailre exception type, which makes it easier to handle particular exception types; * several minor improvements. Affects-db: this change adds db migration that looses exception information for saved task details. Change-Id: I575282002e6999646bbf51f492b82a7e3525787a
* Simpler API to load flows into enginesIvan A. Melnikov2013-10-031-11/+13
| | | | | | | | | | | | | | | | | | | | | Previously to run a flow client code had to put together the flow, an engine, logbook, flowdetail, and storage backend. This commit adds two helper functions, run() and load(), so that simplest usecase now looks like taskflow.engines.run(flow) Client code may also provide configuration for storage and engine if needed, but if not needed it just works with defaults. Engines are loaded via stevedore, as drivers in 'taskflow.engines' backend. Now three entry points are defined in that namespace: - 'default', for SingleThreadedActionEngine, used by default; - 'serial', as another synonym for SingleThreadedActionEngine; - 'parallel', for MultiThreadedActionEngine. Closes-bug: #1224726 Change-Id: I7f4cb5c8ff7f5f12831ddd0952c202d2fd8cd6ef
* Memory persistence backend improvementsIvan A. Melnikov2013-09-301-18/+43
| | | | | | | * got rid of globals -- all data now belongs to backend instance; * be more accurate and consistent with copying saved data. Change-Id: I3e54415e23b7df03e48de3d6c0800f0149051560
* Add a directory/filesystem based persistence layerJoshua Harlow2013-09-211-0/+130
Add a nice non-memory-based but also non-db based persistence layer which is another good example of how a persistence layer can be created (and used). Example directory structure: /books /books/247c5311-d4ec-461b-9e76-51830d6a75b2 /books/247c5311-d4ec-461b-9e76-51830d6a75b2/metadata /books/247c5311-d4ec-461b-9e76-51830d6a75b2/flows /books/247c5311-d4ec-461b-9e76-51830d6a75b2/flows/25f18828-a067-411e-9035-8217536f925d /flows /flows/25f18828-a067-411e-9035-8217536f925d /flows/25f18828-a067-411e-9035-8217536f925d/metadata /flows/25f18828-a067-411e-9035-8217536f925d/tasks /flows/25f18828-a067-411e-9035-8217536f925d/tasks/a352fa2e-82cf-4c37-89ae-3aa10dbf1437 /tasks /tasks/a352fa2e-82cf-4c37-89ae-3aa10dbf1437 Change-Id: I63aaf56497187e21469bc500a49dd02de0c67f29
View Lorry Controller Status