diff options
author | David Cournapeau <cournape@gmail.com> | 2009-02-20 16:37:01 +0000 |
---|---|---|
committer | David Cournapeau <cournape@gmail.com> | 2009-02-20 16:37:01 +0000 |
commit | da364a18e447c334dfa2ca5083b08e1b6a7c0d10 (patch) | |
tree | 7783575e8fe37f8e13ddfc050e9c9d94529bca8e /numpy/lib/io.py | |
parent | 745cfe4c60a59c01a8c2fd6ef68a8ca87f2180f5 (diff) | |
parent | a271ac2e41aacab2903864cf371c4ba94cf4ec3f (diff) | |
download | numpy-da364a18e447c334dfa2ca5083b08e1b6a7c0d10.tar.gz |
Merged revisions 6185-6187,6191-6221,6235-6238,6240-6241,6244,6250-6251,6253,6256,6258,6260-6261,6263,6265-6266,6268,6271,6283-6286,6291-6316,6320-6352,6354,6356,6358-6368,6370-6373,6398-6400,6410,6421-6424 via svnmerge from
http://svn.scipy.org/svn/numpy/trunk
................
r6185 | cdavid | 2008-12-22 01:19:14 +0900 (Mon, 22 Dec 2008) | 1 line
Add doc sources so that sdist tarball contains them.
................
r6186 | pierregm | 2008-12-22 19:01:51 +0900 (Mon, 22 Dec 2008) | 4 lines
testutils:
assert_array_compare : make sure that the comparison is performed on ndarrays, and make sure we use the np version of the comparison function.
core:
* Try not to touch the data in unary/binary ufuncs, (including inplace)
................
r6187 | pearu | 2008-12-22 19:05:00 +0900 (Mon, 22 Dec 2008) | 1 line
Fix a bug.
................
r6191 | cdavid | 2008-12-23 13:10:59 +0900 (Tue, 23 Dec 2008) | 1 line
Fix typos in the comments for manifest.
................
r6192 | cdavid | 2008-12-23 13:11:12 +0900 (Tue, 23 Dec 2008) | 1 line
Use msvcrt values if available for manifest generation: only there starting from python 2.6.1.
................
r6193 | pearu | 2008-12-23 18:02:15 +0900 (Tue, 23 Dec 2008) | 1 line
Fix issue 964: f2py python 2.6, 2.6.1 support.
................
r6194 | pierregm | 2008-12-24 08:43:43 +0900 (Wed, 24 Dec 2008) | 12 lines
testutils:
* assert_equal : use assert_equal_array on records
* assert_array_compare : prevent the common mask to be back-propagated to the initial input arrays.
* assert_equal_array : use operator.__eq__ instead of ma.equal
* assert_equal_less: use operator.__less__ instead of ma.less
core:
* Fixed _check_fill_value for nested flexible types
* Add a ndtype option to _make_mask_descr
* Fixed mask_or for nested flexible types
* Fixed the printing of masked arrays w/ flexible types.
................
r6195 | cdavid | 2008-12-26 21:16:45 +0900 (Fri, 26 Dec 2008) | 1 line
Update to handle numscons 0.10.0 and above.
................
r6196 | cdavid | 2008-12-26 21:36:19 +0900 (Fri, 26 Dec 2008) | 1 line
Do not import msvcrt globally in mingw32compiler module, since the module is imported on all platforms.
................
r6197 | cdavid | 2008-12-26 23:39:55 +0900 (Fri, 26 Dec 2008) | 1 line
Do not test for functions already tested by python configure script.
................
r6198 | cdavid | 2008-12-27 14:56:58 +0900 (Sat, 27 Dec 2008) | 1 line
BUG: Add a runtime check about endianness, to detect bug 4728 in python on Mac OS X.
................
r6199 | cdavid | 2008-12-27 19:06:25 +0900 (Sat, 27 Dec 2008) | 1 line
Fix some typo/syntax errors when converting dict access to a function in manifest generation.
................
r6200 | cdavid | 2008-12-27 19:15:30 +0900 (Sat, 27 Dec 2008) | 1 line
BUG (#970): fix a python 2.6 bug in distutils which caused an unhelpful Error:None message when trying to build with no VS installed and without the -c mingw32 option.
................
r6201 | cdavid | 2008-12-27 19:30:49 +0900 (Sat, 27 Dec 2008) | 1 line
Improve the error message when initializing compiler failed.
................
r6202 | cdavid | 2008-12-27 19:32:05 +0900 (Sat, 27 Dec 2008) | 1 line
Try to initialize the msvc compiler before the general code to detect the error early.
................
r6203 | cdavid | 2008-12-27 19:43:41 +0900 (Sat, 27 Dec 2008) | 1 line
BUG (#970): this commit should fix the actual bug, which albeeit linked to commir r6200, was caused in anoter code path.
................
r6204 | cdavid | 2008-12-27 19:57:05 +0900 (Sat, 27 Dec 2008) | 1 line
Fix manifest generation.
................
r6205 | cdavid | 2008-12-27 20:46:08 +0900 (Sat, 27 Dec 2008) | 1 line
BUG (#827): close temp file before reopning them on windows, and make sure they are not automatically deleted on close either (2.6and higher specific).
................
r6206 | cdavid | 2008-12-27 21:18:47 +0900 (Sat, 27 Dec 2008) | 1 line
Do not define the union for runtime endianness detection if we don't check endianness.
................
r6207 | cdavid | 2008-12-27 22:48:52 +0900 (Sat, 27 Dec 2008) | 1 line
Start working on formatting failure on 2.6: copy how python does complex formatting.
................
r6208 | cdavid | 2008-12-27 23:44:11 +0900 (Sat, 27 Dec 2008) | 1 line
Fix formatting for purely imaginary complex numbers.
................
r6209 | cdavid | 2008-12-27 23:53:15 +0900 (Sat, 27 Dec 2008) | 1 line
More work on formatting float.
................
r6210 | cdavid | 2008-12-27 23:59:41 +0900 (Sat, 27 Dec 2008) | 1 line
Finish formatting fixes for float scalar arrays.
................
r6211 | cdavid | 2008-12-28 00:12:20 +0900 (Sun, 28 Dec 2008) | 1 line
Include umath_funcs_c99 in multiarray so that we can use isinf and co macros.
................
r6212 | cdavid | 2008-12-28 01:15:04 +0900 (Sun, 28 Dec 2008) | 1 line
Include config.h before our C99 math compat layer.
................
r6213 | cdavid | 2008-12-28 01:15:41 +0900 (Sun, 28 Dec 2008) | 1 line
Fix formatting.
................
r6214 | cdavid | 2008-12-28 01:16:18 +0900 (Sun, 28 Dec 2008) | 1 line
Do not define FMTR and FMTI macros, as those are already defined on some platforms.
................
r6215 | cdavid | 2008-12-28 01:16:52 +0900 (Sun, 28 Dec 2008) | 1 line
More formatting fixes.
................
r6216 | cdavid | 2008-12-28 01:17:27 +0900 (Sun, 28 Dec 2008) | 1 line
Remove undef of removed macro.
................
r6217 | cdavid | 2008-12-28 01:33:40 +0900 (Sun, 28 Dec 2008) | 1 line
Do not use PyOS_ascii_formatd, as it does not handle long double correctly.
................
r6218 | cdavid | 2008-12-28 02:19:40 +0900 (Sun, 28 Dec 2008) | 1 line
Try ugly hack to circumvent long double brokenness with mingw.
................
r6219 | cdavid | 2008-12-28 02:25:50 +0900 (Sun, 28 Dec 2008) | 1 line
Use ugly hack for mingw long double pb with complex format function as well.
................
r6220 | cdavid | 2008-12-28 12:18:20 +0900 (Sun, 28 Dec 2008) | 1 line
Revert formatting changes: ascii_formatd only works for double, so we can't use it as it is for our formatting needs.
................
r6221 | cdavid | 2008-12-28 15:44:06 +0900 (Sun, 28 Dec 2008) | 1 line
Do not add doc sources through add_data_dir: it will put the docs alongside numpy, as a separate package, which is not what we want. Use the manifest instead, since that's the only way I know of to include something in sdist-generated tarballs.
................
r6235 | cdavid | 2008-12-29 16:57:52 +0900 (Mon, 29 Dec 2008) | 13 lines
Merged revisions 6233-6234 via svnmerge from
http://svn.scipy.org/svn/numpy/branches/fix_float_format
........
r6233 | cdavid | 2008-12-29 12:49:09 +0900 (Mon, 29 Dec 2008) | 1 line
Use parametric tests for format tests so that it is clearer which type is failing.
........
r6234 | cdavid | 2008-12-29 12:49:27 +0900 (Mon, 29 Dec 2008) | 1 line
Fix formatting tests: cfloat and cdouble as well as np.float and np.double are the same; make sure we test 4 bytes float.
........
................
r6236 | cdavid | 2008-12-29 17:02:15 +0900 (Mon, 29 Dec 2008) | 1 line
Add nan/inf tests for formatting.
................
r6237 | cdavid | 2008-12-29 17:26:04 +0900 (Mon, 29 Dec 2008) | 1 line
Add test for real float types locale independance.
................
r6238 | cdavid | 2008-12-29 17:35:06 +0900 (Mon, 29 Dec 2008) | 1 line
Clearer error messages for formatting failures.
................
r6240 | cdavid | 2008-12-30 12:48:11 +0900 (Tue, 30 Dec 2008) | 1 line
Add tests for print of float types.
................
r6241 | cdavid | 2008-12-30 12:56:54 +0900 (Tue, 30 Dec 2008) | 1 line
Add print tests for complex types.
................
r6244 | cdavid | 2008-12-30 13:20:48 +0900 (Tue, 30 Dec 2008) | 1 line
Fix test for print: forgot to make sure the value is a float before comparing it.
................
r6250 | cdavid | 2008-12-30 14:02:28 +0900 (Tue, 30 Dec 2008) | 17 lines
Merged revisions 6247-6249 via svnmerge from
http://svn.scipy.org/svn/numpy/branches/fix_float_format
........
r6247 | cdavid | 2008-12-30 13:41:37 +0900 (Tue, 30 Dec 2008) | 1 line
Handle 1e10 specially, as it is the limit where exp notation is shorter than decimal for single precision, but not for double (python native one).
........
r6248 | cdavid | 2008-12-30 13:47:38 +0900 (Tue, 30 Dec 2008) | 1 line
Refactor a bit redirected output print test.
........
r6249 | cdavid | 2008-12-30 13:49:31 +0900 (Tue, 30 Dec 2008) | 1 line
Fix test for single precision print.
........
................
r6251 | cdavid | 2008-12-30 14:12:50 +0900 (Tue, 30 Dec 2008) | 1 line
Use np.inf instead of float('inf'), as the later does not work on windows for python < 2.6.
................
r6253 | cdavid | 2008-12-30 14:15:09 +0900 (Tue, 30 Dec 2008) | 1 line
Fix typo in test.
................
r6256 | cdavid | 2008-12-30 14:34:22 +0900 (Tue, 30 Dec 2008) | 1 line
Special case float tests on windows: python 2.5 and below have >=3 digits in the exp.
................
r6258 | cdavid | 2008-12-30 14:42:03 +0900 (Tue, 30 Dec 2008) | 1 line
Hardcode reference for inf/nan-involved values.
................
r6260 | cdavid | 2008-12-30 14:50:18 +0900 (Tue, 30 Dec 2008) | 1 line
Fix more formatting tests on win32.
................
r6261 | cdavid | 2008-12-30 14:52:16 +0900 (Tue, 30 Dec 2008) | 1 line
Fix some more redirected output print tests.
................
r6263 | cdavid | 2008-12-30 15:01:31 +0900 (Tue, 30 Dec 2008) | 1 line
More fixes for print tests.
................
r6265 | cdavid | 2008-12-30 15:03:56 +0900 (Tue, 30 Dec 2008) | 1 line
Fix typo.
................
r6266 | cdavid | 2008-12-30 15:08:06 +0900 (Tue, 30 Dec 2008) | 1 line
Fix typo.
................
r6268 | cdavid | 2008-12-30 15:12:26 +0900 (Tue, 30 Dec 2008) | 1 line
complex scalar arrays cannot be created from real/imag args: wrap init values in a complex.
................
r6271 | cdavid | 2008-12-30 15:32:03 +0900 (Tue, 30 Dec 2008) | 1 line
Do not use dict for reference: hashing on scalar arrays does not work as I expected.
................
r6283 | ptvirtan | 2008-12-31 10:14:47 +0900 (Wed, 31 Dec 2008) | 1 line
Fix #951: make tests to clean temp files properly
................
r6284 | jarrod.millman | 2009-01-01 08:25:03 +0900 (Thu, 01 Jan 2009) | 2 lines
ran reindent
................
r6285 | alan.mcintyre | 2009-01-01 08:46:34 +0900 (Thu, 01 Jan 2009) | 15 lines
Remove the following deprecated items from numpy.testing:
- ParametricTestCase
- The following arguments from numpy.testing.Tester.test(): level,
verbosity,
all, sys_argv, testcase_pattern
- Path manipulation functions: set_package_path, set_local_path,
restore_path
- NumpyTestCase, NumpyTest
Also separated testing parameter setup from NoseTester.test into
NoseTester.prepare_test_args for use in a utility script for valgrind
testing (see NumPy ticket #784).
................
r6286 | jarrod.millman | 2009-01-01 16:56:53 +0900 (Thu, 01 Jan 2009) | 2 lines
add default include dir for Fedora/Red Hat (see SciPy ticket 817)
................
r6291 | cdavid | 2009-01-04 19:57:39 +0900 (Sun, 04 Jan 2009) | 1 line
Do not import md5 on python >= 2.6; use hashlib instead.
................
r6292 | cdavid | 2009-01-04 20:08:16 +0900 (Sun, 04 Jan 2009) | 1 line
Do not use popen* but subprocess.Popen instead.
................
r6293 | cdavid | 2009-01-04 21:03:29 +0900 (Sun, 04 Jan 2009) | 1 line
Revert md5 change: hashlib.md5 is not a drop-in replacement for md5.
................
r6294 | pierregm | 2009-01-05 05:16:00 +0900 (Mon, 05 Jan 2009) | 2 lines
* adapted default_fill_value for flexible datatype
* fixed max/minimum_fill_value for flexible datatype
................
r6295 | stefan | 2009-01-06 06:51:18 +0900 (Tue, 06 Jan 2009) | 1 line
Credit more developers.
................
r6296 | pierregm | 2009-01-06 07:52:21 +0900 (Tue, 06 Jan 2009) | 1 line
*moved the printing templates out of MaskedArray.__repr__
................
r6297 | stefan | 2009-01-06 19:09:00 +0900 (Tue, 06 Jan 2009) | 1 line
Use new-style classes with multiple-inheritance to address bug in IronPython.
................
r6298 | pierregm | 2009-01-07 05:35:37 +0900 (Wed, 07 Jan 2009) | 1 line
* Bugfix #961
................
r6299 | pierregm | 2009-01-08 03:14:12 +0900 (Thu, 08 Jan 2009) | 1 line
* Fixed iadd/isub/imul when the base array has no mask but the other array does
................
r6300 | pierregm | 2009-01-08 07:34:51 +0900 (Thu, 08 Jan 2009) | 3 lines
* Renamed `torecords` to `toflex`, keeping `torecords` as an alias
* Introduced `fromflex`, to reconstruct a masked_array from the output of `toflex` (can?\226?\128?\153t `use fromrecords` as it would clash with `numpy.ma.mrecords.fromrecords`)
* Fixed a bug in MaskedBinaryOperation (#979) (wrong array broadcasting)
................
r6301 | cdavid | 2009-01-08 18:19:00 +0900 (Thu, 08 Jan 2009) | 1 line
Avoid putting things into stderr when errors occurs in f2py wrappers; put all the info in the python error string instead.
................
r6302 | cdavid | 2009-01-09 00:11:32 +0900 (Fri, 09 Jan 2009) | 1 line
Fix python 2.4 issue.
................
r6303 | chanley | 2009-01-09 01:30:01 +0900 (Fri, 09 Jan 2009) | 1 line
Fix test_print.py function _test_locale_independance() since str(1.2) does not use the LC_NUMERIC locale to convert numbers. Fix from Mark Sienkiewicz.
................
r6304 | cdavid | 2009-01-09 04:22:21 +0900 (Fri, 09 Jan 2009) | 1 line
Revert buggy test fix for locale independecce.
................
r6305 | pierregm | 2009-01-09 05:02:29 +0900 (Fri, 09 Jan 2009) | 2 lines
* Add __eq__ and __ne__ for support of flexible arrays.
* Fixed .filled for nested structures
................
r6306 | pierregm | 2009-01-09 06:51:04 +0900 (Fri, 09 Jan 2009) | 1 line
* Remove a debugging print statement.
................
r6307 | jarrod.millman | 2009-01-09 11:14:35 +0900 (Fri, 09 Jan 2009) | 2 lines
Updated license file
................
r6308 | cdavid | 2009-01-09 14:26:58 +0900 (Fri, 09 Jan 2009) | 1 line
Tag formatting unit tests as known failures.
................
r6309 | jarrod.millman | 2009-01-09 17:59:29 +0900 (Fri, 09 Jan 2009) | 2 lines
should be more reliable way to determine what bit platform
................
r6310 | jarrod.millman | 2009-01-09 18:14:17 +0900 (Fri, 09 Jan 2009) | 2 lines
better default library paths for 64bit arch
................
r6311 | jarrod.millman | 2009-01-09 18:57:15 +0900 (Fri, 09 Jan 2009) | 2 lines
simplification suggested by stefan
................
r6312 | jarrod.millman | 2009-01-09 19:02:09 +0900 (Fri, 09 Jan 2009) | 2 lines
switch the order [lib,lib64] --> [lib64,lib]
................
r6313 | jarrod.millman | 2009-01-09 19:18:29 +0900 (Fri, 09 Jan 2009) | 2 lines
removed unneeded import
................
r6314 | jarrod.millman | 2009-01-10 04:37:16 +0900 (Sat, 10 Jan 2009) | 2 lines
can't use append an int to a string
................
r6315 | pierregm | 2009-01-10 05:18:12 +0900 (Sat, 10 Jan 2009) | 2 lines
* Added flatten_structured_arrays
* Fixed _get_recordarray for nested structures
................
r6316 | pierregm | 2009-01-10 10:53:05 +0900 (Sat, 10 Jan 2009) | 1 line
* Add flatten_structured_array to the namespace
................
r6320 | pierregm | 2009-01-14 06:01:58 +0900 (Wed, 14 Jan 2009) | 9 lines
numpy.ma.core:
* introduced baseclass, sharedmask and hardmask as readonly properties of MaskedArray
* docstrings update
numpy.ma.extras:
* docstring updates
docs/reference
* introduced maskedarray, maskedarray.baseclass, maskedarray.generic
................
r6321 | stefan | 2009-01-14 16:14:27 +0900 (Wed, 14 Jan 2009) | 2 lines
Docstring: remove old floating point arithmetic, parallel
execution and postponed import references.
................
r6322 | stefan | 2009-01-14 16:55:16 +0900 (Wed, 14 Jan 2009) | 1 line
Fix printing of limits.
................
r6323 | stefan | 2009-01-14 16:56:10 +0900 (Wed, 14 Jan 2009) | 1 line
Fix finfo to work on all instances, not just NumPy scalars.
................
r6324 | pierregm | 2009-01-17 09:15:15 +0900 (Sat, 17 Jan 2009) | 1 line
* fixed _arraymethod.__call__ for structured arrays
................
r6325 | ptvirtan | 2009-01-18 06:24:13 +0900 (Sun, 18 Jan 2009) | 3 lines
Make `trapz` accept 1-D `x` parameter for n-d `y`, even if axis != -1.
Additional tests included.
................
r6326 | pierregm | 2009-01-19 17:53:53 +0900 (Mon, 19 Jan 2009) | 3 lines
* renamed FlatIter to MaskedIterator
* added __getitem__ to MaskedIterator
................
r6327 | pierregm | 2009-01-19 18:01:24 +0900 (Mon, 19 Jan 2009) | 2 lines
* replace np.asarray by np.asanyarray in unique1d
................
r6328 | pierregm | 2009-01-19 18:04:20 +0900 (Mon, 19 Jan 2009) | 2 lines
* add intersect1d, intersect1d_nu, setdiff1d, setmember1d, setxor1d, unique1d, union1d
* use np.atleast1d instead of ma.atleast1d
................
r6329 | pierregm | 2009-01-20 06:22:52 +0900 (Tue, 20 Jan 2009) | 3 lines
* lib : introduced _iotools
* lib.io : introduced genfromtxt, ndfromtxt, mafromtxt, recfromtxt, recfromcsv.
................
r6330 | pierregm | 2009-01-22 14:37:36 +0900 (Thu, 22 Jan 2009) | 1 line
* genfromtxt : if names is True, accept a line starting with a comment character as header.
................
r6331 | pierregm | 2009-01-22 14:40:25 +0900 (Thu, 22 Jan 2009) | 1 line
* added recfunctions, a collection of utilities to manipulate structured arrays.
................
r6332 | pierregm | 2009-01-23 03:21:32 +0900 (Fri, 23 Jan 2009) | 2 lines
* fixed a machine-dependent issue on default int ('<i4' on OS X, '<i8' on linux) ?
* fixed an machine-dependent issue on argsort ?
................
r6333 | cdavid | 2009-01-24 17:02:14 +0900 (Sat, 24 Jan 2009) | 1 line
Fix compilation error on 2.4.
................
r6334 | pierregm | 2009-01-27 06:04:26 +0900 (Tue, 27 Jan 2009) | 7 lines
* _iotools.StringConverter :
- add a _checked attribute to indicate whether the converter has been upgraded or not.
- switched the default value for bool to False
* io.genfromtxt:
- fixed for the case where a whole column is masked: switch to bool or the common dtype (if needed)
................
r6335 | pierregm | 2009-01-27 11:46:26 +0900 (Tue, 27 Jan 2009) | 1 line
* prevent MaskedBinaryOperation and DomainedBinaryOperation to shrink the mask of the output when at least one of the inputs has a mask full of False
................
r6336 | matthew.brett@gmail.com | 2009-01-30 09:26:44 +0900 (Fri, 30 Jan 2009) | 1 line
New docstrings for byteorder and newbyteorder()
................
r6337 | pierregm | 2009-02-02 14:20:17 +0900 (Mon, 02 Feb 2009) | 2 lines
* Added a 'autoconvert' option to stack_arrays.
* Fixed 'stack_arrays' to work with fields with titles.
................
r6338 | pierregm | 2009-02-04 02:11:44 +0900 (Wed, 04 Feb 2009) | 1 line
* Make sure that StringConverter.update sets the type to object if it can't define it.
................
r6339 | pierregm | 2009-02-05 05:52:36 +0900 (Thu, 05 Feb 2009) | 2 lines
* test__iotools : prevent test_upgrademapper if dateutil is not installed
* MaskedArray.__rmul__ : switch to multiply(self, other)
................
r6340 | pierregm | 2009-02-05 06:53:05 +0900 (Thu, 05 Feb 2009) | 1 line
test_upgrademapper : got rid of the dateutil import
................
r6341 | pierregm | 2009-02-05 13:31:51 +0900 (Thu, 05 Feb 2009) | 2 lines
* genfromtxt : Fixed when a dtype involving objects is explicitly given. Raise a NotImplementedError if the dtype is nested.
* _iotools : make sure StringConverter gets properly initiated when a function returning a np.object is used as input parameter.
................
r6342 | alan.mcintyre | 2009-02-06 05:11:40 +0900 (Fri, 06 Feb 2009) | 6 lines
Issue #957:
- Fix problems with test decorators when used on test generators.
- The skip/fail arguments for skipif and knownfailureif can now be
either a bool or a callable that returns a bool.
- Added tests for the test decorators.
................
r6343 | ptvirtan | 2009-02-06 09:27:08 +0900 (Fri, 06 Feb 2009) | 1 line
doc/numpydoc: work better together with Sphinx's config option
................
r6344 | ptvirtan | 2009-02-06 09:51:41 +0900 (Fri, 06 Feb 2009) | 1 line
doc: Move maskedarray docs upward in TOC
................
r6345 | oliphant | 2009-02-06 15:25:50 +0900 (Fri, 06 Feb 2009) | 1 line
Avoid re-creating the sequence when there is only one field in the regular expression.
................
r6346 | oliphant | 2009-02-06 15:31:11 +0900 (Fri, 06 Feb 2009) | 1 line
Removed an unneccessary return statement in a unit test.
................
r6347 | pearu | 2009-02-06 23:36:58 +0900 (Fri, 06 Feb 2009) | 1 line
Fix a bug: python system_info.py failed because _pkg_config_info defined section to be None.
................
r6348 | pearu | 2009-02-06 23:38:57 +0900 (Fri, 06 Feb 2009) | 1 line
Fix another bug, see last commit.
................
r6349 | pierregm | 2009-02-07 18:19:12 +0900 (Sat, 07 Feb 2009) | 2 lines
MaskedArray.resize : systematically raise a TypeError exception, as a masked array never owns its data
MaskedIterator : fixed to allow .flat on masked matrices
................
r6350 | pierregm | 2009-02-08 03:51:31 +0900 (Sun, 08 Feb 2009) | 1 line
................
r6351 | ptvirtan | 2009-02-10 05:18:08 +0900 (Tue, 10 Feb 2009) | 1 line
Fix #955: fix errobj leak in scalarmath floating point error handling
................
r6352 | pierregm | 2009-02-10 09:42:40 +0900 (Tue, 10 Feb 2009) | 1 line
* prevent modifications to the mask to be back-propagated w/ __array_wrap__
................
r6354 | cdavid | 2009-02-10 19:44:01 +0900 (Tue, 10 Feb 2009) | 1 line
Fix hyphen (patch from debian package).
................
r6356 | pierregm | 2009-02-11 10:51:28 +0900 (Wed, 11 Feb 2009) | 1 line
* MaskedArray.__array_wrap__ : forces the domain (if any) to a ndarray (fill with True)
................
r6358 | oliphant | 2009-02-12 13:22:03 +0900 (Thu, 12 Feb 2009) | 1 line
Add multiple-field access by making a copy of the array and filling with the selected fields.
................
r6359 | stefan | 2009-02-12 14:44:07 +0900 (Thu, 12 Feb 2009) | 2 lines
Trust user's specification of MACOSX_DEPLOYMENT_TARGET [patch by Brian
Granger].
................
r6360 | cdavid | 2009-02-14 23:54:26 +0900 (Sat, 14 Feb 2009) | 1 line
Merge fix_float_format branch into the trunk.
................
r6361 | cdavid | 2009-02-15 00:02:39 +0900 (Sun, 15 Feb 2009) | 1 line
Fix typo in multiarray tests.
................
r6362 | cdavid | 2009-02-15 00:03:22 +0900 (Sun, 15 Feb 2009) | 1 line
Remove leftover in TestIO.
................
r6363 | cdavid | 2009-02-15 02:03:51 +0900 (Sun, 15 Feb 2009) | 1 line
Include C99 math compatbility layer in multiarray - isnan and co needed by numpyos.c
................
r6364 | ptvirtan | 2009-02-15 07:09:26 +0900 (Sun, 15 Feb 2009) | 1 line
More add_newdocs entries, and make add_newdoc capable of adding docs also to normal Python objects.
................
r6365 | ptvirtan | 2009-02-15 07:10:24 +0900 (Sun, 15 Feb 2009) | 1 line
Move (un)packbits docstrings to add_newdocs.py. Fix typos.
................
r6366 | ptvirtan | 2009-02-15 07:11:19 +0900 (Sun, 15 Feb 2009) | 1 line
Document constants in numpy.doc.constants
................
r6367 | ptvirtan | 2009-02-15 07:38:32 +0900 (Sun, 15 Feb 2009) | 1 line
Move numpy.lib __doc__ back to info.py; was moved to __init__.py by mistake.
................
r6368 | pierregm | 2009-02-15 07:42:29 +0900 (Sun, 15 Feb 2009) | 1 line
* genfromtxt : fixed case when using explicit converters and explicit dtype.
................
r6370 | ptvirtan | 2009-02-16 00:44:47 +0900 (Mon, 16 Feb 2009) | 1 line
sphinxext: clean up plot directive, and merge some features from matplotlib
................
r6371 | charris | 2009-02-18 06:19:10 +0900 (Wed, 18 Feb 2009) | 2 lines
Coding style cleanups.
................
r6372 | cdavid | 2009-02-18 14:06:57 +0900 (Wed, 18 Feb 2009) | 1 line
Fix some const issues in NumPyOs_ascii_strtod.
................
r6373 | charris | 2009-02-18 14:36:35 +0900 (Wed, 18 Feb 2009) | 1 line
Coding style cleanups.
................
r6398 | charris | 2009-02-19 05:54:52 +0900 (Thu, 19 Feb 2009) | 2 lines
Coding style cleanups.
................
r6399 | charris | 2009-02-19 09:45:14 +0900 (Thu, 19 Feb 2009) | 2 lines
Coding style cleanups.
................
r6400 | charris | 2009-02-19 13:58:23 +0900 (Thu, 19 Feb 2009) | 1 line
Coding style cleanups.
................
r6410 | cdavid | 2009-02-19 19:05:28 +0900 (Thu, 19 Feb 2009) | 1 line
Tag known failure on win32.
................
r6421 | stefan | 2009-02-20 04:28:08 +0900 (Fri, 20 Feb 2009) | 1 line
Fix tests using strptime to be Python 2.4 compatible.
................
r6422 | charris | 2009-02-20 08:25:01 +0900 (Fri, 20 Feb 2009) | 2 lines
Coding style cleanups.
................
r6423 | charris | 2009-02-20 12:40:53 +0900 (Fri, 20 Feb 2009) | 1 line
Coding style cleanups.
................
r6424 | cdavid | 2009-02-20 22:30:20 +0900 (Fri, 20 Feb 2009) | 1 line
Unhelpful message for compaq fortran compiler.
................
Diffstat (limited to 'numpy/lib/io.py')
-rw-r--r-- | numpy/lib/io.py | 521 |
1 files changed, 511 insertions, 10 deletions
diff --git a/numpy/lib/io.py b/numpy/lib/io.py index e9a012db1..12765e17c 100644 --- a/numpy/lib/io.py +++ b/numpy/lib/io.py @@ -1,4 +1,5 @@ __all__ = ['savetxt', 'loadtxt', + 'genfromtxt', 'ndfromtxt', 'mafromtxt', 'recfromtxt', 'recfromcsv', 'load', 'loads', 'save', 'savez', 'packbits', 'unpackbits', @@ -15,7 +16,11 @@ from cPickle import load as _cload, loads from _datasource import DataSource from _compiled_base import packbits, unpackbits +from _iotools import LineSplitter, NameValidator, StringConverter, \ + _is_string_like, has_nested_fields, flatten_dtype + _file = file +_string_like = _is_string_like class BagObj(object): """A simple class that converts attribute lookups to @@ -264,10 +269,6 @@ def _getconv(dtype): return str -def _string_like(obj): - try: obj + '' - except (TypeError, ValueError): return 0 - return 1 def loadtxt(fname, dtype=float, comments='#', delimiter=None, converters=None, skiprows=0, usecols=None, unpack=False): @@ -342,7 +343,7 @@ def loadtxt(fname, dtype=float, comments='#', delimiter=None, converters=None, if usecols is not None: usecols = list(usecols) - if _string_like(fname): + if _is_string_like(fname): if fname.endswith('.gz'): import gzip fh = gzip.open(fname) @@ -520,7 +521,7 @@ def savetxt(fname, X, fmt='%.18e',delimiter=' '): """ - if _string_like(fname): + if _is_string_like(fname): if fname.endswith('.gz'): import gzip fh = gzip.open(fname,'wb') @@ -603,8 +604,508 @@ def fromregex(file, regexp, dtype): seq = regexp.findall(file.read()) if seq and not isinstance(seq[0], tuple): - # make sure np.array doesn't interpret strings as binary data - # by always producing a list of tuples - seq = [(x,) for x in seq] - output = np.array(seq, dtype=dtype) + # Only one group is in the regexp. + # Create the new array as a single data-type and then + # re-interpret as a single-field structured array. + newdtype = np.dtype(dtype[dtype.names[0]]) + output = np.array(seq, dtype=newdtype) + output.dtype = dtype + else: + output = np.array(seq, dtype=dtype) + return output + + + + +#####-------------------------------------------------------------------------- +#---- --- ASCII functions --- +#####-------------------------------------------------------------------------- + + + +def genfromtxt(fname, dtype=float, comments='#', delimiter=None, skiprows=0, + converters=None, missing='', missing_values=None, usecols=None, + names=None, excludelist=None, deletechars=None, + case_sensitive=True, unpack=None, usemask=False, loose=True): + """ + Load data from a text file. + + Each line past the first `skiprows` ones is split at the `delimiter` + character, and characters following the `comments` character are discarded. + + + + Parameters + ---------- + fname : file or string + File or filename to read. If the filename extension is `.gz` or `.bz2`, + the file is first decompressed. + dtype : data-type + Data type of the resulting array. If this is a flexible data-type, + the resulting array will be 1-dimensional, and each row will be + interpreted as an element of the array. In this case, the number + of columns used must match the number of fields in the data-type, + and the names of each field will be set by the corresponding name + of the dtype. + If None, the dtypes will be determined by the contents of each + column, individually. + comments : {string}, optional + The character used to indicate the start of a comment. + All the characters occurring on a line after a comment are discarded + delimiter : {string}, optional + The string used to separate values. By default, any consecutive + whitespace act as delimiter. + skiprows : {int}, optional + Numbers of lines to skip at the beginning of the file. + converters : {None, dictionary}, optional + A dictionary mapping column number to a function that will convert + values in the column to a number. Converters can also be used to + provide a default value for missing data: + ``converters = {3: lambda s: float(s or 0)}``. + missing : {string}, optional + A string representing a missing value, irrespective of the column where + it appears (e.g., `'missing'` or `'unused'`). + missing_values : {None, dictionary}, optional + A dictionary mapping a column number to a string indicating whether the + corresponding field should be masked. + usecols : {None, sequence}, optional + Which columns to read, with 0 being the first. For example, + ``usecols = (1,4,5)`` will extract the 2nd, 5th and 6th columns. + names : {None, True, string, sequence}, optional + If `names` is True, the field names are read from the first valid line + after the first `skiprows` lines. + If `names` is a sequence or a single-string of comma-separated names, + the names will be used to define the field names in a flexible dtype. + If `names` is None, the names of the dtype fields will be used, if any. + excludelist : {sequence}, optional + A list of names to exclude. This list is appended to the default list + ['return','file','print']. Excluded names are appended an underscore: + for example, `file` would become `file_`. + deletechars : {string}, optional + A string combining invalid characters that must be deleted from the names. + case_sensitive : {True, False, 'upper', 'lower'}, optional + If True, field names are case_sensitive. + If False or 'upper', field names are converted to upper case. + If 'lower', field names are converted to lower case. + unpack : {bool}, optional + If True, the returned array is transposed, so that arguments may be + unpacked using ``x, y, z = loadtxt(...)`` + usemask : {bool}, optional + If True, returns a masked array. + If False, return a regular standard array. + + Returns + ------- + out : MaskedArray + Data read from the text file. + + Notes + -------- + * When spaces are used as delimiters, or when no delimiter has been given + as input, there should not be any missing data between two fields. + * When the variable are named (either by a flexible dtype or with `names`, + there must not be any header in the file (else a :exc:ValueError exception + is raised). + + Warnings + -------- + * Individual values are not stripped of spaces by default. + When using a custom converter, make sure the function does remove spaces. + + See Also + -------- + numpy.loadtxt : equivalent function when no data is missing. + + """ + # + if usemask: + from numpy.ma import MaskedArray, make_mask_descr + # Check the input dictionary of converters + user_converters = converters or {} + if not isinstance(user_converters, dict): + errmsg = "The input argument 'converter' should be a valid dictionary "\ + "(got '%s' instead)" + raise TypeError(errmsg % type(user_converters)) + # Check the input dictionary of missing values + user_missing_values = missing_values or {} + if not isinstance(user_missing_values, dict): + errmsg = "The input argument 'missing_values' should be a valid "\ + "dictionary (got '%s' instead)" + raise TypeError(errmsg % type(missing_values)) + defmissing = [_.strip() for _ in missing.split(',')] + [''] + + # Initialize the filehandle, the LineSplitter and the NameValidator +# fhd = _to_filehandle(fname) + if isinstance(fname, basestring): + fhd = np.lib._datasource.open(fname) + elif not hasattr(fname, 'read'): + raise TypeError("The input should be a string or a filehandle. "\ + "(got %s instead)" % type(fname)) + else: + fhd = fname + split_line = LineSplitter(delimiter=delimiter, comments=comments, + autostrip=False)._handyman + validate_names = NameValidator(excludelist=excludelist, + deletechars=deletechars, + case_sensitive=case_sensitive) + + # Get the first valid lines after the first skiprows ones + for i in xrange(skiprows): + fhd.readline() + first_values = None + while not first_values: + first_line = fhd.readline() + if first_line == '': + raise IOError('End-of-file reached before encountering data.') + if names is True: + first_values = first_line.strip().split(delimiter) + else: + first_values = split_line(first_line) + if names is True: + fval = first_values[0].strip() + if fval in comments: + del first_values[0] + + # Check the columns to use + if usecols is not None: + usecols = list(usecols) + nbcols = len(usecols or first_values) + + # Check the names and overwrite the dtype.names if needed + if dtype is not None: + dtype = np.dtype(dtype) + dtypenames = getattr(dtype, 'names', None) + if names is True: + names = validate_names([_.strip() for _ in first_values]) + first_line ='' + elif _is_string_like(names): + names = validate_names([_.strip() for _ in names.split(',')]) + elif names: + names = validate_names(names) + elif dtypenames: + dtype.names = validate_names(dtypenames) + if names and dtypenames: + dtype.names = names + + # If usecols is a list of names, convert to a list of indices + if usecols: + for (i, current) in enumerate(usecols): + if _is_string_like(current): + usecols[i] = names.index(current) + + # If user_missing_values has names as keys, transform them to indices + missing_values = {} + for (key, val) in user_missing_values.iteritems(): + # If val is a list, flatten it. In any case, add missing &'' to the list + if isinstance(val, (list, tuple)): + val = [str(_) for _ in val] + else: + val = [str(val),] + val.extend(defmissing) + if _is_string_like(key): + try: + missing_values[names.index(key)] = val + except ValueError: + pass + else: + missing_values[key] = val + + + # Initialize the default converters + if dtype is None: + # Note: we can't use a [...]*nbcols, as we would have 3 times the same + # ... converter, instead of 3 different converters. + converters = [StringConverter(None, + missing_values=missing_values.get(_, defmissing)) + for _ in range(nbcols)] + else: + flatdtypes = flatten_dtype(dtype) + # Initialize the converters + if len(flatdtypes) > 1: + # Flexible type : get a converter from each dtype + converters = [StringConverter(dt, + missing_values=missing_values.get(i, defmissing), + locked=True) + for (i, dt) in enumerate(flatdtypes)] + else: + # Set to a default converter (but w/ different missing values) + converters = [StringConverter(dtype, + missing_values=missing_values.get(_, defmissing), + locked=True) + for _ in range(nbcols)] + missing_values = [_.missing_values for _ in converters] + + # Update the converters to use the user-defined ones + uc_update = [] + for (i, conv) in user_converters.iteritems(): + # If the converter is specified by column names, use the index instead + if _is_string_like(i): + i = names.index(i) + if usecols: + try: + i = usecols.index(i) + except ValueError: + # Unused converter specified + continue + converters[i].update(conv, default=None, + missing_values=missing_values[i], + locked=True) + uc_update.append((i, conv)) + # Make sure we have the corrected keys in user_converters... + user_converters.update(uc_update) + + # Reset the names to match the usecols + if (not first_line) and usecols: + names = [names[_] for _ in usecols] + + rows = [] + append_to_rows = rows.append + if usemask: + masks = [] + append_to_masks = masks.append + # Parse each line + for line in itertools.chain([first_line,], fhd): + values = split_line(line) + # Skip an empty line + if len(values) == 0: + continue + # Select only the columns we need + if usecols: + values = [values[_] for _ in usecols] + # Check whether we need to update the converter + if dtype is None: + for (converter, item) in zip(converters, values): + converter.upgrade(item) + # Store the values + append_to_rows(tuple(values)) + if usemask: + append_to_masks(tuple([val.strip() in mss + for (val, mss) in zip(values, + missing_values)])) + + # Convert each value according to the converter: + # We want to modify the list in place to avoid creating a new one... + if loose: + conversionfuncs = [conv._loose_call for conv in converters] + else: + conversionfuncs = [conv._strict_call for conv in converters] + for (i, vals) in enumerate(rows): + rows[i] = tuple([convert(val) + for (convert, val) in zip(conversionfuncs, vals)]) + + # Reset the dtype + data = rows + if dtype is None: + # Get the dtypes from the types of the converters + coldtypes = [conv.type for conv in converters] + # Find the columns with strings... + strcolidx = [i for (i, v) in enumerate(coldtypes) + if v in (type('S'), np.string_)] + # ... and take the largest number of chars. + for i in strcolidx: + coldtypes[i] = "|S%i" % max(len(row[i]) for row in data) + # + if names is None: + # If the dtype is uniform, don't define names, else use '' + base = set([c.type for c in converters if c._checked]) + + if len(base) == 1: + (ddtype, mdtype) = (list(base)[0], np.bool) + else: + ddtype = [('', dt) for dt in coldtypes] + mdtype = [('', np.bool) for dt in coldtypes] + else: + ddtype = zip(names, coldtypes) + mdtype = zip(names, [np.bool] * len(coldtypes)) + output = np.array(data, dtype=ddtype) + if usemask: + outputmask = np.array(masks, dtype=mdtype) + else: + # Overwrite the initial dtype names if needed + if names and dtype.names: + dtype.names = names + flatdtypes = flatten_dtype(dtype) + # Case 1. We have a structured type + if len(flatdtypes) > 1: + # Nested dtype, eg [('a', int), ('b', [('b0', int), ('b1', 'f4')])] + # First, create the array using a flattened dtype: + # [('a', int), ('b1', int), ('b2', float)] + # Then, view the array using the specified dtype. + if has_nested_fields(dtype): + if 'O' in (_.char for _ in flatdtypes): + errmsg = "Nested fields involving objects "\ + "are not supported..." + raise NotImplementedError(errmsg) + rows = np.array(data, dtype=[('', t) for t in flatdtypes]) + output = rows.view(dtype) + else: + output = np.array(data, dtype=dtype) + # Now, process the rowmasks the same way + if usemask: + rowmasks = np.array(masks, + dtype=np.dtype([('', np.bool) + for t in flatdtypes])) + # Construct the new dtype + mdtype = make_mask_descr(dtype) + outputmask = rowmasks.view(mdtype) + # Case #2. We have a basic dtype + else: + # We used some user-defined converters + if user_converters: + ishomogeneous = True + descr = [] + for (i, ttype) in enumerate([conv.type for conv in converters]): + # Keep the dtype of the current converter + if i in user_converters: + ishomogeneous &= (ttype == dtype.type) + if ttype == np.string_: + ttype = "|S%i" % max(len(row[i]) for row in data) + descr.append(('', ttype)) + else: + descr.append(('', dtype)) + # So we changed the dtype ? + if not ishomogeneous: + # We have more than one field + if len(descr) > 1: + dtype = np.dtype(descr) + # We have only one field: drop the name if not needed. + else: + dtype = np.dtype(ttype) + # + output = np.array(data, dtype) + if usemask: + if dtype.names: + mdtype = [(_, np.bool) for _ in dtype.names] + else: + mdtype = np.bool + outputmask = np.array(masks, dtype=mdtype) + # Try to take care of the missing data we missed + if usemask and output.dtype.names: + for (name, conv) in zip(names or (), converters): + missing_values = [conv(_) for _ in conv.missing_values if _ != ''] + for mval in missing_values: + outputmask[name] |= (output[name] == mval) + # Construct the final array + if usemask: + output = output.view(MaskedArray) + output._mask = outputmask + if unpack: + return output.squeeze().T + return output.squeeze() + + + +def ndfromtxt(fname, dtype=float, comments='#', delimiter=None, skiprows=0, + converters=None, missing='', missing_values=None, + usecols=None, unpack=None, names=None, + excludelist=None, deletechars=None, case_sensitive=True,): + """ + Load ASCII data stored in fname and returns a ndarray. + + Complete description of all the optional input parameters is available in + the docstring of the `genfromtxt` function. + + See Also + -------- + numpy.genfromtxt : generic function. + + """ + kwargs = dict(dtype=dtype, comments=comments, delimiter=delimiter, + skiprows=skiprows, converters=converters, + missing=missing, missing_values=missing_values, + usecols=usecols, unpack=unpack, names=names, + excludelist=excludelist, deletechars=deletechars, + case_sensitive=case_sensitive, usemask=False) + return genfromtxt(fname, **kwargs) + +def mafromtxt(fname, dtype=float, comments='#', delimiter=None, skiprows=0, + converters=None, missing='', missing_values=None, + usecols=None, unpack=None, names=None, + excludelist=None, deletechars=None, case_sensitive=True,): + """ + Load ASCII data stored in fname and returns a MaskedArray. + + Complete description of all the optional input parameters is available in + the docstring of the `genfromtxt` function. + + See Also + -------- + numpy.genfromtxt : generic function. + """ + kwargs = dict(dtype=dtype, comments=comments, delimiter=delimiter, + skiprows=skiprows, converters=converters, + missing=missing, missing_values=missing_values, + usecols=usecols, unpack=unpack, names=names, + excludelist=excludelist, deletechars=deletechars, + case_sensitive=case_sensitive, + usemask=True) + return genfromtxt(fname, **kwargs) + + +def recfromtxt(fname, dtype=None, comments='#', delimiter=None, skiprows=0, + converters=None, missing='', missing_values=None, + usecols=None, unpack=None, names=None, + excludelist=None, deletechars=None, case_sensitive=True, + usemask=False): + """ + Load ASCII data stored in fname and returns a standard recarray (if + `usemask=False`) or a MaskedRecords (if `usemask=True`). + + Complete description of all the optional input parameters is available in + the docstring of the `genfromtxt` function. + + See Also + -------- + numpy.genfromtxt : generic function + + Warnings + -------- + * by default, `dtype=None`, which means that the dtype of the output array + will be determined from the data. + """ + kwargs = dict(dtype=dtype, comments=comments, delimiter=delimiter, + skiprows=skiprows, converters=converters, + missing=missing, missing_values=missing_values, + usecols=usecols, unpack=unpack, names=names, + excludelist=excludelist, deletechars=deletechars, + case_sensitive=case_sensitive, usemask=usemask) + output = genfromtxt(fname, **kwargs) + if usemask: + from numpy.ma.mrecords import MaskedRecords + output = output.view(MaskedRecords) + else: + output = output.view(np.recarray) + return output + + +def recfromcsv(fname, dtype=None, comments='#', skiprows=0, + converters=None, missing='', missing_values=None, + usecols=None, unpack=None, names=True, + excludelist=None, deletechars=None, case_sensitive='lower', + usemask=False): + """ + Load ASCII data stored in comma-separated file and returns a recarray (if + `usemask=False`) or a MaskedRecords (if `usemask=True`). + + Complete description of all the optional input parameters is available in + the docstring of the `genfromtxt` function. + + See Also + -------- + numpy.genfromtxt : generic function + """ + kwargs = dict(dtype=dtype, comments=comments, delimiter=",", + skiprows=skiprows, converters=converters, + missing=missing, missing_values=missing_values, + usecols=usecols, unpack=unpack, names=names, + excludelist=excludelist, deletechars=deletechars, + case_sensitive=case_sensitive, usemask=usemask) + output = genfromtxt(fname, **kwargs) + if usemask: + from numpy.ma.mrecords import MaskedRecords + output = output.view(MaskedRecords) + else: + output = output.view(np.recarray) + return output + |