summaryrefslogtreecommitdiff
path: root/swiftclient/utils.py
Commit message (Collapse)AuthorAgeFilesLines
* shell: Allow timeouts to have unitsTim Burke2023-03-201-0/+14
| | | | | Related-Change: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab Change-Id: Ifeaaea790d1dadc84b157a7cf2be7590949c70f0
* Allow tempurl to be used to sign /info requestsTim Burke2022-08-291-3/+11
| | | | Change-Id: I1f9d26541e9c8f5aec7a6790c87df397d178efe6
* Allow tempurl times to have unitsTim Burke2022-08-191-34/+55
| | | | | | | Specifically, let users add a suffix for seconds, minutes, hours, or days. Change-Id: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab
* tempurl: Support sha256 and sha512 signatures4.0.1Tim Burke2022-06-081-3/+19
| | | | | | | | | | | | | | | Up the default to sha256 since * the proxy has supported (and defaulted to allowing) it for four years now, and * Rackspace has supported it for even longer. Include a note in the --help about older clusters likely requiring sha1. Change-Id: Ibac2bb7e2e4c9946c7384f0aab8e43d0d79ba645 Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046 Related-Bug: #1733634 Closes-Bug: #1977867
* Remove unnecessary unicode prefixesStephen Finucane2022-03-211-4/+4
| | | | | | | | All strings are unicode by default in Python 3. No need to mark them as such. Signed-off-by: Stephen Finucane <stephenfin@redhat.com> Change-Id: I68fb60ef271abfddebcc9d2137424f5db2a17e92
* Remove unnecessary object subclassingStephen Finucane2022-03-211-3/+3
| | | | | | | All classes subclass from object by default in Python 3. Signed-off-by: Stephen Finucane <stephenfin@redhat.com> Change-Id: I5a1ad57bcc092861ce969759b06a07c880ad3d35
* Remove sixStephen Finucane2022-03-211-14/+10
| | | | | | | This mostly affects tests. Nothing too complicated Signed-off-by: Stephen Finucane <stephenfin@redhat.com> Change-Id: Iabc78f651e1d48db35638280722f8019798eccd6
* Clean up some warningsTim Burke2020-06-171-2/+5
| | | | Change-Id: Iae149533d04c7b173c4ef88fb775f5fe13c16466
* Optionally display listings in raw jsonClay Gerrard2019-07-091-0/+22
| | | | | | | | | | | | Symlinks have recently added some new keys to container listings. It's very convenient to be able to see and reason about the extra information in container listings. Allowing raw json output is similar with what the client already does for the info command, and it's forward compatible with any listing enhancements added by future middleware development. Change-Id: I88fb38529342ac4e4198aeccd2f10c69c7396704
* Fix SLO re-uploadTim Burke2019-06-271-0/+8
| | | | | | | | | | | | | | Previously, if you uploaded a file as an SLO then re-uploaded it with the same segment size and mtime, the second upload would go delete the segments it just (re)uploaded. This was due to us tracking old_slo_manifest_paths and new_slo_manifest_paths in different formats; one would have a leading slash while the other would not. Now, normalize to the stripped-slash version so we stop deleting segments we just uploaded. Change-Id: Ibcbed3df4febe81cdf13855656e2daaca8d521b4
* Isolate docs requirementsTim Burke2019-06-271-1/+1
| | | | | | | | | | | ...since modern sphinx won't install on py27. While we're at it, clean up some warnings and treat warnings as errors. Also, fix up how we parse test configs so we can run func tests. Related-Change: Id3c2ed87230c5918c18e2c01d086df8157f036b1 Change-Id: I3718f69610545b0dbcb0a2ab45b400da3a45682c
* Add ability to generate a temporary URL with anmmcardle2018-07-101-3/+22
| | | | | | IP range restriction Change-Id: I4734599886e4f4a563162390d0ff3bb1ef639db4
* ISO 8601 timestamps for tempurlChristopher Bartz2017-03-291-19/+54
| | | | | | | | | Client-side implementation for ISO 8601 timestamp support of tempurl middleware. Please see https://review.openstack.org/#/c/422679/ Change-Id: I76da28b48948475ec1bae5258e0b39a316553fb7
* Fix some reST field lists in docstringsliuyamin2017-03-291-1/+1
| | | | | | | | | | Probably the most common format for documenting arguments is reST field lists [1]. This change updates some docstrings to comply with the field lists syntax. [1] http://sphinx-doc.org/domains.html#info-field-lists Change-Id: Ic011fd3e3a8c5bafa24a3438a6ed5bb126b50e95
* Fix logging of the gzipped bodyVitaly Gridnev2017-03-081-2/+7
| | | | | Change-Id: I6d7ccbf4ef9b46e890ecec58842c5cdd2804c7a9 Closes-bug: 1670620
* Merge "Accept more types of input for headers/meta"Jenkins2017-01-241-6/+16
|\
| * Accept more types of input for headers/metaTim Burke2016-11-181-6/+16
| | | | | | | | | | | | | | | | | | | | | | | | Previously, we only accepted iterables of strings like 'Header: Value'. Now, we'll also accept lists of tuples like ('Header', 'Value') as well as dictionaries like {'Header': 'Value'}. This should be more intuitive for application developers, who are already used to being able to pass dicts or lists of tuples to libraries like requests. Change-Id: I93ed2f1e8305f0168b7a4bd90c205b04730da836
* | prefix-based tempurls supportChristopher Bartz2017-01-191-6/+16
|/ | | | | | | | | Implements client-side functionality for prefix-based tempurls. Please see: https://review.openstack.org/#/c/274048/ Change-Id: I8d7701daee888ed1120271a96c0660b01543ca2d
* Add additional headers for HEAD/GET/DELETE requests.Charles Hsu2016-11-071-0/+14
| | | | | Change-Id: I69276ba711057c122f97deac412e492e313c34dd Closes-Bug: 1615830
* Make tempurl subcommand insist on whole number secondsAlistair Coles2016-09-141-5/+7
| | | | | | | | | | | | | | | Previously the tempurl subcommand would dump a traceback due to a TypeError if the seconds option was not an int value. With this patch it will now return the same error message as if the seconds option were negative or not a number. Also changes the error message to state that the seconds option should be a "whole number" rather than a "positive integer", since 0 is a valid value. Change-Id: Ie940d470f2be8006aa8eb7fe242f092457aeae21 Closes-Bug: #1621817
* Make tempurl command check for valid object pathAlistair Coles2016-09-141-3/+13
| | | | | | | | | | | | | | If the supplied path is not of the form /v1/a/c/o then swift tempurl subcommand will now return an error message. Also removes redundant check for seconds parameter being an int from shell.py because the same check is made when calling utils.generate_temp_url. Drive-by fix for missing param definition for generate_temp_url. Change-Id: I41f4389948b01fadaa5fc4939ea12e0ed2167345 Related-Change: I0fb2ce125fe12d660e4deb778265016bdd5ff31b
* Accept gzip-encoded API responsesTim Burke2016-08-301-0/+5
| | | | | | | | | | | Previously, we would accept gzip-encoded responses, but only because we were letting requests decode *all* responses (even object data). This restores the previous capability, but with tighter controls about which requests will accept gzipped responses and where the decoding happens. Change-Id: I4fd8b97207b9ab01b1bcf825cc16efd8ad46344a Related-Bug: 1282861 Related-Bug: 1338464
* Fix unicode issues in tempurl commandTim Burke2016-07-191-17/+28
| | | | | | | Previously, we weren't encoding paths and keys as UTF-8, which would trigger a UnicodeEncodeError on py27. Change-Id: I2fad428369406c2ae32343a5e943ffb2cd1ca6ef
* Use bulk-delete middleware when availableTim Burke2016-01-121-0/+10
| | | | | | | | | | | | | | | | | | | | | | When issuing `delete` commands that would require three or more individual deletes, check whether the cluster supports bulk deletes and use that if it's available. Additionally, a new option is added to the `delete` command: * --prefix <prefix> Delete all objects that start with <prefix>. This is similar to the --prefix option for the `list` command. Example: $ swift delete c --prefix obj_prefix/ ...will delete from container "c" all objects whose name begins with "obj_prefix/", such as "obj_prefix/foo" and "obj_prefix/bar". Change-Id: I6b9504848d6ef562cf4f570bbcd17db4e3da8264
* Retry file uploads via SwiftServiceTim Burke2016-01-111-4/+28
| | | | | | | | | | | | When we introduced LengthWrapper, we neglected to make it resettable. As a result, upload failures result in errors like: put_object(...) failure and no ability to reset contents for reupload. Now, LengthWrappers will be resettable if their _readable has seek/tell. Related-Change: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce Change-Id: I21f43f06e8c78b24d1fc081efedf2687942e042f
* Wrap raw iterators to ensure we send entire contents to serverTim Burke2015-12-301-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | Currently, if you attempt to stream an upload from an iterator, as in def data(): yield 'foo' yield '' yield 'bar' conn.put_object('c', 'o', data()) ... requests will faithfully emit a zero-length chunk, ending the transfer. Swift will then close the connection, possibly (if Connection: keep-alive was set) after attempting to parse the next chunk as a new request. Now, Swift will receive all of the bytes from the iterable, and any zero-byte chunks will be ignored. This will be fixed in requests [1], but not until an eventual 3.0.0 release. [1] https://github.com/kennethreitz/requests/pull/2631 Change-Id: I19579ed7a0181ac3f488433e7c1839f7f7a040b8
* Merge "enable autodocumentation for utils module; docstring fixes"Jenkins2015-10-291-9/+9
|\
| * enable autodocumentation for utils module; docstring fixesZack M. Davis2015-09-101-9/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit adds the utils module to those for which Sphinx automatically generates documentation from docstrings. (Many of the functions here may be of little interest to users, but `generate_temp_url`, at least, definitely deserves to be in the documentation; in this way, this commit can be seen as a spiritual companion to ca70dd9e.) Also, a few markup errors and perceived infelicities in existing docstrings are amended. Change-Id: I8c66a23cb359d7dd9302a16459fad9825fedb690
* | Miscellaneous (mostly test) cleanupTim Burke2015-10-071-21/+17
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | * Always use testtools.TestCase, since we're relying on testtools * Always use mock (as opposed to unittest.mock) since we're relying on mock * Add note about when a missing logging handler was added * Stop %-formatting the giant usage string that doesn't actually need any formatting * Prefer assertIs, assertIn, assertIsInstance over assertTrue * Use else-self.fail instead of sentinel values * Check resp.get('error') is None before checking resp['success'] is True, so test failures actually tell you something useful * Tighten some isinstance assertions * Import MockHttpTest from correct location * Only populate clean_os_environ once * Use setUp for setup, not __init__ * Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with assertEqual(foo, dict.get(key)) when key is a literal and foo is not None * Use mock.patch.object instead of manually patching for tests * Use six.binary_type instead of type(''.encode('utf-8')) * Stop shadowing builtin bytes * Reclaim some margin * Stop checking the return-type of encode_utf8; we already know it's bytes Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
* absolute expiry option for tempURL generationZack M. Davis2015-09-041-3/+6
| | | | | | | | | | | | | | | | | | | | The `tempurl` subcommand's second positional argument is called `seconds` and has heretofore interpreted as the number of seconds for which the tempURL should be valid, counting from the moment of running the command. This is indeed a common, if not the most common, use-case. But some users, occasionally, might want to generate a tempURL that expires at some particular ("absolute") time, rather than a particular amount of time relative to the moment of happening to run the command. (One might make an analogy to the way in which Swift's expiring object support supports an `X-Delete-At` header in addition to `X-Delete-After`—and it's the former that must be regarded as ontologically prior.) Thus, this commit adds an `--absolute` optional argument to the `tempurl` subcommand; if present, the `seconds` argument will be interpreted as a Unix timestamp of when the tempURL should be expire, rather than a duration for which the tempURL should be valid starting from "now". Change-Id: If9ded96f2799800958d5063127f3de812f50ef06
* Log and report trace on service operation failsJoel Wright2015-08-311-2/+16
| | | | | | | | | This patch adds exception logging to the swift service API. Each operation that results in failure of any operation will now log the exception as well as report a timestamp and full stack trace in the results returned by the service API calls. Change-Id: I7336b28354e7740ea7d048bdf355e3c1a1b4436c
* Remove simplejson dependencyTim Burke2015-05-201-1/+12
| | | | | | | In addition to removing an unnecessary dependency, this closes another hole that was allowing raw bytes to appear in user-facing messages. Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
* Add improvements to MD5 validation.Daniel Wakefield2015-03-041-0/+1
| | | | | | | | | | | | | | | With MD5Sum checking being added, a concern was brought up that It was a change with no possibility of reverting to the old behaviour. This change adds the flag '--ignore-checksum' to the upload subcommand allowing the checks to be turned off. Changed occurrences of the magic string for a null md5 to use a descriptive constant instead. Updated Error messages generated when validation fails. They should now be more descriptive and not output a literal newline sequence. Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
* Verify MD5 of uploaded objects.Daniel Wakefield2015-02-191-7/+89
| | | | | | | | | | | Changed existing code to calculate the MD5 of the object during the upload stream. Checks this MD5 against the etag returned in the response. An exception is raised if they do not match. Closes-Bug: 1379263 Change-Id: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce
* Adding Swift Temporary URL supportJosh Gachnang2014-07-111-0/+49
| | | | | | | | | | Temporary URLs allow a user to sign an object URL with a shared secret to so that the object can be downloaded without auth for a specified amount of time. http://docs.openstack.org/trunk/config-reference/content/object-storage-tempurl.html Change-Id: Ife0b6c98c975e074d4dad0a31145573b784747c5
* Python 3: Get compatible types from sixVictor Stinner2014-03-281-1/+3
| | | | | | | | | | | * Replace unicode with six.text_type * Replace basestring with six.string_types * The long type doesn't exist in Python 3 anymore: replace 1L with long(1) and only test this type with Python 2 * Fix quote(): quote the URL if the string is a byte string. Use "bytes" type instead of "str" to be Python 3 compatible. Change-Id: I1df5aa85e4e7d07191fb5c654d52fc4bd8b9f440
* Add LengthWrapper in put_object to honor content_length paramClay Gerrard2014-02-251-0/+18
| | | | | | Closes-Bug:#1284360 Change-Id: Iec63a3fde77bb8195bfe46c764403b367999ff43
* Port to python-requestsTristan Cacqueray2014-02-121-50/+0
| | | | | | | | | | | | | | | | | | | | | Currently, httplib implementation does not support SSL certificate verification. This patch fixes this. Note that ssl compression parameter and 100-continue thing is still missing from requests, though those are lower priority. Requests now takes care of: * proxy configuration (get_environ_proxies), * chunked encoding (with data generator), * bulk uploading (with files dictionary), * SSL certificate verification (with 'insecure' and 'cacert' parameter). This patch have been tested with requests 1.1.0 (CentOS 6) and requests 2.2.1 (current version). Change-Id: Ib5de962f4102d57c71ad85fd81a615362ef175dc Closes-Bug: #1199783 DocImpact SecurityImpact
* Enable usage of proxies defined in environment (http(s)_proxy).Davide Guerri2013-12-101-0/+50
| | | | | | | | | As far as proxies usage is concerned, keystone-client API and swift-client API behave differently because the former uses python Request library while the latter uses raw httplib. As a result, Keystone authentication honors environment variables http_proxy, https_proxy and no_proxy while Swift doesn't. This patch, which code is mainly borrowed from Python Requests, makes Swift data connections and Swift authentication connections behaving homogeneously. Change-Id: Ic8a0089c35c458d7ed96e572e22429014298fe4c
* Add verbose output to all stat commandsClay Gerrard2013-10-091-0/+30
| | | | | | | | | | | | | When you stat a container or object with the verbose flag the full path of the reousrce will be displayed with the token similarlly to how an account stat displays the auth url and token. * move some logic out of bin/swift.st_stat to test it * new module swiftclient.commnad_helpers for code you want to test * moved prt_bytes into swiftclient.utils to test it * fixed IndexError with prt_bytes on sizes >= 1024Y Change-Id: Iaaa96e0308b08c554205b0055b8a04de581fefa4
* Move multi-threading code to a library.Darrell Bishop2013-07-281-1/+0
| | | | | | | | | | | | | | | | | | | | | | This patch extracts the multi-threading code from bin/swift into swiftclient/multithreading and adds tests. In particular, this new way of doing it (with context managers) will prevent non-daemonic threads from wedging the process when unexpected exceptions happen. I enabled reporting of which lines, specifically, are not covered by unit tests (added -m option to "coverage report" in .unittests). This patch includes a drive-by fix for uploading a segmented file with --use-slo when that object already exists. A key of "name" was used instead of "path", raising KeyError. There's also another drive-by fix for uploading segmented objects with --use-slo. Commit 874e0e4427b80e1b15b74a1557b73ba9d61443ca regressed this by removing the capturing of thread-worker results in QueueFunctionThread.run(). This patch restores that functionality and the feature (uploading SLO objects). Change-Id: I0b4f677e4a734e83d1a25088d9a74f7d46384e53
* Add env[SWIFTCLIENT_INSECURE]You Yamagata2013-01-101-0/+28
Add env[SWIFTCLIENT_INSECURE] as default of --insecure option. If set to 'true', allow to access insecure keystone server. The name follows 'NOVACLIENT_INSECURE' in novaclient. Change-Id: I322674eba9c07e6def97bce339815fa15191a92d
View Lorry Controller Status