summaryrefslogtreecommitdiff
path: root/swiftclient/shell.py
Commit message (Collapse)AuthorAgeFilesLines
...
* | | Miscellaneous (mostly test) cleanupTim Burke2015-10-071-10/+10
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Always use testtools.TestCase, since we're relying on testtools * Always use mock (as opposed to unittest.mock) since we're relying on mock * Add note about when a missing logging handler was added * Stop %-formatting the giant usage string that doesn't actually need any formatting * Prefer assertIs, assertIn, assertIsInstance over assertTrue * Use else-self.fail instead of sentinel values * Check resp.get('error') is None before checking resp['success'] is True, so test failures actually tell you something useful * Tighten some isinstance assertions * Import MockHttpTest from correct location * Only populate clean_os_environ once * Use setUp for setup, not __init__ * Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with assertEqual(foo, dict.get(key)) when key is a literal and foo is not None * Use mock.patch.object instead of manually patching for tests * Use six.binary_type instead of type(''.encode('utf-8')) * Stop shadowing builtin bytes * Reclaim some margin * Stop checking the return-type of encode_utf8; we already know it's bytes Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
* | absolute expiry option for tempURL generationZack M. Davis2015-09-041-3/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The `tempurl` subcommand's second positional argument is called `seconds` and has heretofore interpreted as the number of seconds for which the tempURL should be valid, counting from the moment of running the command. This is indeed a common, if not the most common, use-case. But some users, occasionally, might want to generate a tempURL that expires at some particular ("absolute") time, rather than a particular amount of time relative to the moment of happening to run the command. (One might make an analogy to the way in which Swift's expiring object support supports an `X-Delete-At` header in addition to `X-Delete-After`—and it's the former that must be regarded as ontologically prior.) Thus, this commit adds an `--absolute` optional argument to the `tempurl` subcommand; if present, the `seconds` argument will be interpreted as a Unix timestamp of when the tempURL should be expire, rather than a duration for which the tempURL should be valid starting from "now". Change-Id: If9ded96f2799800958d5063127f3de812f50ef06
* | Merge "Reduce memory usage for download/delete and add --no-shuffle option ↵Jenkins2015-08-281-0/+18
|\ \ | | | | | | | | | to st_download"
| * | Reduce memory usage for download/delete and add --no-shuffle option to ↵Joel Wright2015-07-201-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | st_download The current code builds a full object listing before performing either a multiple download or delete operation (and also shuffles this complete list in the case of a download). This patch removes the creation of the full object list and adds the ability to turn off shuffle for files when downloading. Also added is a limit on the number of list results that can be queued by a single call to service.list without consuming results (reduces memory overhead for large listings). Some tests added for service.py download and list. Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
* | | flake8 ignores same hacks as swiftHiroshi Miura2015-08-181-4/+4
| |/ |/| | | | | | | | | | | | | | | | | | | - blacklisted flake8 hacking - fix against E122 continuation line missing indentation or outdented Closes-bug: #1475516 Change-Id: I708d0a3466a1f85c84e478873e142821ce0774cb Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
* | Merge "swiftclient: add short options to help message"Jenkins2015-07-311-34/+43
|\ \ | |/ |/|
| * swiftclient: add short options to help messageHiroshi Miura2015-07-171-34/+43
| | | | | | | | | | | | | | | | | | | | | | | | - add usage strings for short option such as '-a' for '--all' This add all short options to usage text. - add missing --header usage help for download command - some cometic changes Closes-bug: #1475511 Change-Id: Ibfecac8764669540fa025787548133a50fa50b10 Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
* | Merge "Always decode command-line arguments as UTF-8"2.5.0Jenkins2015-07-161-6/+9
|\ \ | |/ |/|
| * Always decode command-line arguments as UTF-8Tim Burke2015-07-071-6/+9
| | | | | | | | | | | | | | | | | | There was always an implicit assumption that they were UTF-8 before, and by converting them to unicode we close another hole allowing raw bytes to appear in user-facing messages. Closes-Bug: #1431866 Change-Id: If2e41d9a592c3ad02818e9c6f0959fd4b73cd0e0
* | Merge "Correct the help message of swift tempurl"Jenkins2015-07-111-4/+4
|\ \
| * | Correct the help message of swift tempurlYangLei2015-06-301-4/+4
| | | | | | | | | | | | | | | | | | | | | correct the help message of swift tempurl use <> instead of [] in Positional arguments. Change-Id: Ib60ce97cef03e0423082c497604525eba2300fa9
* | | Merge "Add some bash helpers for auth stuff"Jenkins2015-07-111-4/+50
|\ \ \
| * | | Add some bash helpers for auth stuffClay Gerrard2015-06-151-4/+50
| | | | | | | | | | | | | | | | Change-Id: If61ac9a050e7a115f37dbf4e74b904ac5dfd2052
* | | | Merge "Add ability to download objects to particular folder."Jenkins2015-07-111-2/+24
|\ \ \ \ | |_|_|/ |/| | |
| * | | Add ability to download objects to particular folder.Charles Hsu2015-07-081-2/+24
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds "--output-dir" and "--remove-prefix" options to the "download" command and unit tests for it. Example: $ swift list example --prefix swift2.2 swift2.2/bin/swift-object-auditor swift2.2/bin/swift-object-expirer swift2.2/bin/swift-object-info swift2.2/bin/swift-object-replicator swift2.2/bin/swift-object-server swift2.2/bin/swift-object-updater When given "--output-dir <directory>", client downloads objects to <directory>. $ swift download example --prefix swift2.2 \ --output-dir new/swift/dir The folder structure: . └── new └── swift └── dir └── swift2.2 └── bin ├── swift-object-auditor ├── swift-object-expirer ├── swift-object-info ├── swift-object-replicator ├── swift-object-server └── swift-object-updater When given "--remove-prefix", client downloads objects without <prefix>. $ swift download example --prefix swift2.2 \ --remove-prefix \ --output-dir swift The folder structure: . └── swift └── bin ├── swift-object-auditor ├── swift-object-expirer ├── swift-object-info ├── swift-object-replicator ├── swift-object-server └── swift-object-updater Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
* | | Added check for negative segment-sizePradeep Kumar Singh2015-06-231-0/+3
| |/ |/| | | | | | | | | Closes-Bug: #1453135 Change-Id: Ia9c2b27d998e6ac1889cc74c12e456c06ecd84d9
* | Fix inconsistent usage of "Positional argument"Andreas Jaeger2015-06-081-1/+1
|/ | | | | | | | | | | All help texts uses "Positional argument" with the exception of tempurl. Update tempurl to use this as well so that the formatting tools work fine and can show this nicely on http://docs.openstack.org/cli-reference/content/swiftclient_commands.html#swiftclient_subcommand_tempurl like it's done for other options. Change-Id: Ib5502c23b236986bea5a4d4a63a46fca411a8494 Closes-Bug: #1463081
* be more explicit in the --version CLI optionJohn Dickinson2015-05-131-1/+1
| | | | Change-Id: I14f33c7ea90ab1fe58aa67f1ff70e527c88cd141
* Mention --segment-size option after 413 response2.4.0Tim Burke2015-03-051-0/+6
| | | | | | | | | | | | Previously, it would only output something like ClientException: Object PUT failed <url> 413 Request Entity Too Large Your request is too large. ... which is misleading as it implies that Swift simply can't handle large objects. Change-Id: Icec9e76b2671490c3a2249ec08992950107c6828
* Merge "Unindent a chunk of st_list"Jenkins2015-03-051-57/+52
|\
| * Unindent a chunk of st_listPete Zaitcev2015-02-261-57/+52
| | | | | | | | | | | | | | | | The chunk looks like nicely containable, but it pushes against the right edge of the screen annoyingly. This patch _only_ prettifies, without any observable changes. Change-Id: I96fcb5848e3e5e0e8a8ebbe398b1d4a0bea5fa24
* | Add improvements to MD5 validation.Daniel Wakefield2015-03-041-2/+6
|/ | | | | | | | | | | | | | | With MD5Sum checking being added, a concern was brought up that It was a change with no possibility of reverting to the old behaviour. This change adds the flag '--ignore-checksum' to the upload subcommand allowing the checks to be turned off. Changed occurrences of the magic string for a null md5 to use a descriptive constant instead. Updated Error messages generated when validation fails. They should now be more descriptive and not output a literal newline sequence. Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
* Fix crash with -l, -d /, and pseudo foldersPete Zaitcev2015-02-181-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | Using -l, -d /, or a pseduo folder by itself works fine, but if you combine all of them, this happens: [zaitcev@guren s]$ swift list -d / testcont amazon_lili.jpg pseudodir/ rin_shibuya_pixiv.jpg swift-c134855-5a.diff [zaitcev@guren s]$ swift list -d / -l testcont 151954 2014-06-20 17:13:52 amazon_lili.jpg Traceback (most recent call last): File "/q/zaitcev/hail/python-swiftclient-tip/bin/swift", line 24, in <module> sys.exit(main()) File "/q/zaitcev/hail/python-swiftclient-tip/swiftclient/shell.py", line 1301, in main globals()['st_%s' % args[0]](parser, argv[1:], output) File "/q/zaitcev/hail/python-swiftclient-tip/swiftclient/shell.py", line 414, in st_list total_bytes += item_bytes TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType' The patch is a straightforward fix that assigns expected variables. Change-Id: If24dcc2491e8b8982455107d2aca30c0491a398a Closes-Bug: 1399678
* Merge "This patch fixes downloading files to stdout."Jenkins2015-02-031-3/+4
|\
| * This patch fixes downloading files to stdout.Joel Wright2015-01-231-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | This patch fixes downloading files to stdout and modifies _SwiftReader to operate as an iterator that performs file checks at the end of iteration as well as a context manager. File verification checks have been removed from __exit__ and added to __iter__. Change-Id: I3250bdeeef8484a9122c4b5b854756a7c8f8731e Closes-Bug: 1395922 Closes-Bug: 1387376
* | Fix cross account upload using --os-storage-urlAlistair Coles2015-01-061-22/+15
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removes an account stat from the object upload path. This stat fails when user is not account admin even though the user may have container ACL permission to write objects. Reduces the severity of the CLI output message when upload fails to create the given container (this is not an error since the container may exist - the user just does not have permission to PUT or POST the container). Changes the 'swift upload' exit return code from 1 to 0 if container PUT fails but object PUT succeeds. For segment uploads, makes the attempt to create the segment container conditional on it not being the same as the manifest container. This avoids an unnecessary container PUT. Fixes another bug that became apparent: with segmented upload a container HEAD may be attempted to determine the policy to be used for the segment container. When this failed the result dict has headers=None which was causing an exception in the shell result handler. Add unit tests for object upload/download and container list with --os-storage-url option. Closes-Bug: #1371650 Change-Id: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
* Print info message about incorrect --totals usageJiří Suchomel2015-01-051-0/+5
| | | | | | | | when neither -l nor --lh is provided. Added test coverage for --totals. Change-Id: I3245e715c26ec28457a21dec07311a58c475c066 Closes-Bug: 1258392
* Merge "Change tests to use new CaptureOutput class."Jenkins2014-12-301-1/+1
|\
| * Change tests to use new CaptureOutput class.Daniel Wakefield2014-12-051-1/+1
| | | | | | | | | | | | | | | | | | | | Added the ability to clear the buffers in the CaptureOutput class so it can be easily used multiple times in the same context manager. Also added a option to suppress the SystemExit associated with printing an error. Change-Id: Ib59bbbe88256f215eed0a8ebc8282e02181d4377
* | Fix misnamed dictionary key.Daniel Wakefield2014-12-081-21/+16
|/ | | | | | | | | | | | | The response dictionary in _delete_segment attach's any error it encounters to the dict key 'exception', all other response dict's use 'error' to store the exception. Changed to make it consistent and added tests Also added a third branch in st_delete as messages created in _delete_container where being silently dropped. Change-Id: Ifbc3b1fae78910fbc6acf4a86cfb0f60bb1aa336
* Merge "Make swift post output an error message when failing"Jenkins2014-11-131-7/+7
|\
| * Make swift post output an error message when failingAlistair Coles2014-10-241-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | swiftclient.shell.st_post was ignoring the result dict returned from SwiftService.post and therefore giving no indication when posts failed e.g. due to invalid auth credentials. This patch ensures that SwiftService always returns a result_dict from post() and then checks the result dict. On failure the shell now outputs the ClientException message and exits with error code 1. Also adds unit tests and cleans up some unnecessary mocked return values in existing tests. Closes-Bug: 1383243 Change-Id: I111150eb3d026c8d01c2cac13d3613ca7304e5b9
* | Replaces Stacktraces with useful error messages.Daniel Wakefield2014-10-211-11/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Changed the message shown when a user doesn't enter project/tenant id/name to be more informative. When attempting to stat a container without supplying project/tenant name or id, an empty response was being returned instead of an error being raised. Changed the error raised in swiftclient.client when no tenant or project is specified to be more specific. Add tests for basic regression checking. Closes-Bug: #1372589 Change-Id: I4eb6964d9f1702db119cc0294edc02841b1ecd5f
* | Fix race in shell when testing for errors to raise SysExitAlistair Coles2014-10-201-3/+1
|/ | | | | | | | | | | | | | | | | SysExit is raised if the OutputManager has non-zero error_count. However, error_count is incremented in a print thread, but tested on the main thread. Currently error_count is tested before the OutputManager print threads have exited, which results in a race. This race means that SysExit is not always raised even when an error has occured. This change moves the error_count test after the OutputManager context manager has exited, which ensures that the OutputManager threads have exited. Change-Id: I5ef7d9f10057fe2b41f48ab95066a7265b86a3ac Closes-Bug: #1379229
* Merge "Allow segment size to be specified in a human readable way."Jenkins2014-10-171-1/+17
|\
| * Allow segment size to be specified in a human readable way.Daniel Wakefield2014-10-101-1/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of always specifying segment size in bytes the user can now use B,K,M or G as suffixes for the corresponding size. Conversion is done with Binary units (1024) rather than SI units (1000). e.g swift upload test_container -S 1073741824 large_file can now be written swift upload test_container -S 1G large_file The change is backwards compatible as it ignores arguments to -S that don't have a valid suffix. Updated unit tests and help message. Change-Id: I6314b4e45cf2fbffde2fe57a02df77a25e911e84
* | Do not crash with "swift list --lh" for Ceph RadosGW.Jiří Suchomel2014-10-011-1/+1
|/ | | | | | | | | | If 'x-timestamp' header is not present in 'swift list' result, date string cannot be correctly generated. Check for correct exception to catch this case and return a special string value. This is just a workaround for faulty RadosGW behavior. Change-Id: I172843a2668e33efac00924906ff0dca3e790050 Closes-Bug: 1349528
* Add importable SwiftService incorporating shell.py logicJoel Wright2014-08-261-883/+454
| | | | | | | | | | | | | | | | This patch adds a SwiftService class that incorporates the high level logic from swiftclient/shell.py. It also ports shell.py to use the new class, and updates the code in swiftclient/multithreading.py to allow the SwiftService to be used for multiple operations whilst using only one thread pool. Currently, code that imports swiftclient has to have its own logic for things like creating large objects, parallel uploads, and parallel downloads. This patch adds a SwiftService class that makes that functionality available in Python code as well as through the shell. Change-Id: I08c5796b4c01001d79fd571651c3017c16462ffd Implements: blueprint bin-swift-logic-as-importable-library
* Merge "Fix crash when downloading a pseudo-directory"Jenkins2014-08-171-12/+22
|\
| * Fix crash when downloading a pseudo-directoryPete Zaitcev2014-08-141-12/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a user creates an object with name ending with a slash, then downloading such container ends in a traceback like this: .............. test5g.file [auth 1.516s, headers 1.560s, total 244.565s, 22.089 MB/s] Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/swiftclient/multithreading.py", lin result = self.func(item, *self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/swiftclient/shell.py", line 403, in fp = open(path, 'wb') IOError: [Errno 21] Is a directory: 'first-pseudo-folder/' The proposed fix is not to save this object. Note that the contents of the object are available with --output option, as before. Only the crash is fixed. Even though we do not use the contents, we download the object and check its Etag, in case. We also create a corresponding directory, in case the pseudo-directory contains no objects. The format of printout is changed, so user realizes easier when pseudo-directory convention is in effect. Note that this is not a compatibility issue because previously there was crash in such case. Change-Id: I3352f7a4eaf9970961af0cc84c4706fc1eab281d
* | Add keystone v3 auth supportanc2014-07-231-44/+150
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Enables swiftclient to authenticate using the keystone v3 API, allowing user id's, user domains and tenant/project domains to be specified. Since swiftclient imports keystoneclient, the main changes in swiftclient/client.py are to selectively import the correct keystoneclient library version and pass a number of new options to it via the get_auth() function. In addition the get_keystoneclient_2_0 method has been renamed get_auth_keystone to better reflect its purpose since it now deals with both v2 and v3 use cases. In swiftclient/shell.py the new options are added to the parser. To make the default help message shorter, help for all the --os-* options (including the existing v2 options) is only displayed when explicitly requested usng a new --os-help option. A new set of unit tests is added to test_shell.py to verify the parser. A comment in tests/sample.conf explains how to configure the existing functional tests to run using keystone v3 API. Note that to use keystone v3 with swift you will need to set auth_version = v3.0 in the auth_token middleware config section of proxy-server.conf. Change-Id: Ifda0b3263eb919a8c6a1b204ba0a1215ed6f642f
* Fix context sensitive help for info and tempurl2.2.0Alistair Coles2014-07-211-22/+9
| | | | | | | | | | Make it so that swift <cmd> --help will print the info subcommand help for info and tempurl just like all the other subcommands. Also add unit tests to verify subcommand help. Change-Id: Id3666dcf72a9727fbfda2f74c23293ada1c53aa0
* Merge "Allow to specify storage policy when uploading objects"Jenkins2014-07-211-6/+42
|\
| * Allow to specify storage policy when uploading objectsYuan Zhou2014-07-171-6/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Client already supports -H/--header option when creating container or uploading objects. This patch extends this option to support Storage Policy. e.g., swift post con -H 'X-Storage-Policy:p1' This creates one container 'con' with storage policy 'p1'. swift upload con obj -H 'X-Storage-Policy:p2' This creates container 'con' with storage policy 'p2' and uploads object 'obj' into it. Also fixes segmented uploading to non-default storage policy container When uploading large objects with segmentation to container with non-default storage policy, there will be another 'xxx_segments' container created, but with the default storage policy. This results all the segments to be stored with the wrong policy. This patch is for the Storage Policy feature, and also compatible with old versions w/o Storage Policy support. Change-Id: I5c19e90604a0bcf2c85e1732b8a0b97ae6801994
* | Merge "Adding Swift Temporary URL support"Jenkins2014-07-181-2/+46
|\ \ | |/ |/|
| * Adding Swift Temporary URL supportJosh Gachnang2014-07-111-2/+46
| | | | | | | | | | | | | | | | | | | | Temporary URLs allow a user to sign an object URL with a shared secret to so that the object can be downloaded without auth for a specified amount of time. http://docs.openstack.org/trunk/config-reference/content/object-storage-tempurl.html Change-Id: Ife0b6c98c975e074d4dad0a31145573b784747c5
* | Add context sensitive helpMark Seger2014-06-251-1/+22
|/ | | | Change-Id: I9aaadbaba2c43a12a83f8200eaaa96f48c52028e
* Merge "fixed typos found by RETF rules"Jenkins2014-05-161-2/+2
|\
| * fixed typos found by RETF rulesChristian Berendt2014-05-121-2/+2
| | | | | | | | | | | | rules are avaialble at https://en.wikipedia.org/wiki/Wikipedia:AutoWikiBrowser/Typos Change-Id: I98b6e46cb84962ffdac02e9007b5ad3b30bc2817
* | Add "." for help stringsAndreas Jaeger2014-05-081-8/+8
|/ | | | | | | | | The command descriptions did not include a "." at the end but the argument description does. Add "." everywhere. Also capitalize and add "." for parameter of capabilities. Change-Id: I2355e0d0c38170d669c8e568361faf4b68203d16
View Lorry Controller Status