diff options
Diffstat (limited to 'doc/source/user')
| -rw-r--r-- | doc/source/user/basic.io.genfromtxt.rst | 449 | ||||
| -rw-r--r-- | doc/source/user/basic.io.rst | 7 | ||||
| -rw-r--r-- | doc/source/user/basics.rst | 1 |
3 files changed, 457 insertions, 0 deletions
diff --git a/doc/source/user/basic.io.genfromtxt.rst b/doc/source/user/basic.io.genfromtxt.rst new file mode 100644 index 000000000..28130b311 --- /dev/null +++ b/doc/source/user/basic.io.genfromtxt.rst @@ -0,0 +1,449 @@ +.. sectionauthor:: Pierre Gerard-Marchant <pierregmcode@gmail.com> + +********************************************* +Importing data with :func:`~numpy.genfromtxt` +********************************************* + +Numpy provides several functions to create arrays from tabular data. +We focus here on the :func:`~numpy.genfromtxt` function. + +In a nutshell, :func:`~numpy.genfromtxt` runs two main loops. +The first loop converts each line of the file in a sequence of strings. +The second loop converts each string to the appropriate data type. +This mechanism is slower than a single loop, but gives more flexibility. +In particular, :func:`~numpy.genfromtxt` is able to take missing data into account, when other faster and simpler functions like :func:`~numpy.loadtxt` cannot + + +.. note:: + When giving examples, we will use the following conventions + + >>> import numpy as np + >>> from StringIO import StringIO + + + +Defining the input +================== + +The only mandatory argument of :func:`~numpy.genfromtxt` is the source of the data. +It can be a string corresponding to the name of a local or remote file, or a file-like object with a :meth:`read` method (such as an actual file or a :class:`StringIO.StringIO` object). +If the argument is the URL of a remote file, this latter is automatically downloaded in the current directory. + +The input file can be a text file or an archive. +Currently, the function recognizes :class:`gzip` and :class:`bz2` (`bzip2`) archives. +The type of the archive is determined by examining the extension of the file: +if the filename ends with ``'.gz'``, a :class:`gzip` archive is expected; if it ends with ``'bz2'``, a :class:`bzip2` archive is assumed. + + + +Splitting the lines into columns +================================ + +The :keyword:`delimiter` argument +--------------------------------- + +Once the file is defined and open for reading, :func:`~numpy.genfromtxt` splits each non-empty line into a sequence of strings. +Empty or commented lines are just skipped. +The :keyword:`delimiter` keyword is used to define how the splitting should take place. + +Quite often, a single character marks the separation between columns. +For example, comma-separated files (CSV) use a comma (``,``) or a semicolon (``;``) as delimiter. + + >>> data = "1, 2, 3\n4, 5, 6" + >>> np.genfromtxt(StringIO(data), delimiter=",") + array([[ 1., 2., 3.], + [ 4., 5., 6.]]) + +Another common separator is ``"\t"``, the tabulation character. +However, we are not limited to a single character, any string will do. +By default, :func:`~numpy.genfromtxt` assumes ``delimiter=None``, meaning that the line is split along white spaces (including tabs) and that consecutive white spaces are considered as a single white space. + +Alternatively, we may be dealing with a fixed-width file, where columns are defined as a given number of characters. +In that case, we need to set :keyword:`delimiter` to a single integer (if all the columns have the same size) or to a sequence of integers (if columns can have different sizes). + + >>> data = " 1 2 3\n 4 5 67\n890123 4" + >>> np.genfromtxt(StringIO(data), delimiter=3) + array([[ 1., 2., 3.], + [ 4., 5., 67.], + [ 890., 123., 4.]]) + >>> data = "123456789\n 4 7 9\n 4567 9" + >>> np.genfromtxt(StringIO(data), delimiter=(4, 3, 2)) + array([[ 1234., 567., 89.], + [ 4., 7., 9.], + [ 4., 567., 9.]]) + + +The :keyword:`autostrip` argument +--------------------------------- + +By default, when a line is decomposed into a series of strings, the individual entries are not stripped of leading nor trailing white spaces. +This behavior can be overwritten by setting the optional argument :keyword:`autostrip` to a value of ``True``. + + >>> data = "1, abc , 2\n 3, xxx, 4" + >>> # Without autostrip + >>> np.genfromtxt(StringIO(data), dtype="|S5") + array([['1', ' abc ', ' 2'], + ['3', ' xxx', ' 4']], + dtype='|S5') + >>> # With autostrip + >>> np.genfromtxt(StringIO(data), dtype="|S5", autostrip=True) + array([['1', 'abc', '2'], + ['3', 'xxx', '4']], + dtype='|S5') + + +The :keyword:`comments` argument +-------------------------------- + +The optional argument :keyword:`comments` is used to define a character string that marks the beginning of a comment. +By default, :func:`~numpy.genfromtxt` assumes ``comments='#'``. +The comment marker may occur anywhere on the line. +Any character present after the comment marker(s) is simply ignored. + + >>> data = """# + ... # Skip me ! + ... # Skip me too ! + ... 1, 2 + ... 3, 4 + ... 5, 6 #This is the third line of the data + ... 7, 8 + ... # And here comes the last line + ... 9, 0 + ... """ + >>> np.genfromtxt(StringIO(data), comments="#", delimiter=",") + [[ 1. 2.] + [ 3. 4.] + [ 5. 6.] + [ 7. 8.] + [ 9. 0.]] + +.. note:: + There is one notable exception to this behavior: if the optional argument ``names=True``, the first commented line will be examined for names. + + + +Skipping lines and choosing columns +=================================== + +The :keyword:`skip_header` and :keyword:`skip_footer` arguments +--------------------------------------------------------------- + +The presence of a header in the file can hinder data processing. +In that case, we need to use the :keyword:`skip_header` optional argument. +The values of this argument must be an integer which corresponds to the number of lines to skip at the beginning of the file, before any other action is performed. +Similarly, we can skip the last ``n`` lines of the file by using the :keyword:`skip_footer` attribute and giving it a value of ``n``. + + >>> data = "\n".join(str(i) for i in range(10)) + >>> np.genfromtxt(StringIO(data),) + array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) + >>> np.genfromtxt(StringIO(data), + ... skip_header=3, skip_footer=5) + array([ 3., 4.]) + +By default, ``skip_header=0`` and ``skip_footer=0``, meaning that no lines are skipped. + + +The :keyword:`usecols` argument +------------------------------- + +In some cases, we are not interested in all the columns of the data but only a few of them. +We can select which columns to import with the :keyword:`usecols` argument. +This argument accepts a single integer or a sequence of integers corresponding to the indices of the columns to import. +Remember that by convention, the first column has an index of 0. +Negative integers correspond to + +For example, if we want to import only the first and the last columns, we can use ``usecols=(0, -1)``: + >>> data = "1 2 3\n4 5 6" + >>> np.genfromtxt(StringIO(data), usecols=(0, -1)) + array([[ 1., 3.], + [ 4., 6.]]) + +If the columns have names, we can also select which columns to import by giving their name to the :keyword:`usecols` argument, either as a sequence of strings or a comma-separated string. + >>> data = "1 2 3\n4 5 6" + >>> np.genfromtxt(StringIO(data), + ... names="a, b, c", usecols=("a", "c")) + array([(1.0, 3.0), (4.0, 6.0)], + dtype=[('a', '<f8'), ('c', '<f8')]) + >>> np.genfromtxt(StringIO(data), + ... names="a, b, c", usecols=("a, c")) + array([(1.0, 3.0), (4.0, 6.0)], + dtype=[('a', '<f8'), ('c', '<f8')]) + + + + +Choosing the data type +====================== + +The main way to control how the sequences of strings we have read from the file are converted to other types is to set the :keyword:`dtype` argument. +Acceptable values for this argument are: + +* a single type, such as ``dtype=float``. + The output will be 2D with the given dtype, unless a name has been associated with each column with the use of the :keyword:`names` argument (see below). + Note that ``dtype=float`` is the default for :func:`~numpy.genfromtxt`. +* a sequence of types, such as ``dtype=(int, float, float)``. +* a comma-separated string, such as ``dtype="i4,f8,|S3"``. +* a dictionary with two keys ``'names'`` and ``'formats'``. +* a sequence of tuples ``(name, type)``, such as ``dtype=[('A', int), ('B', float)]``. +* an existing :class:`numpy.dtype` object. +* the special value ``None``. + In that case, the type of the columns will be determined from the data itself (see below). + +In all the cases but the first one, the output will be a 1D array with a structured dtype. +This dtype has as many fields as items in the sequence. +The field names are defined with the :keyword:`names` keyword. + + +When ``dtype=None``, the type of each column is determined iteratively from its data. +We start by checking whether a string can be converted to a boolean (that is, if the string matches ``true`` or ``false`` in lower cases); +then whether it can be converted to an integer, then to a float, then to a complex and eventually to a string. +This behavior may be changed by modifying the default mapper of the :class:`~numpy.lib._iotools.StringConverter` class. + +The option ``dtype=None`` is provided for convenience. +However, it is significantly slower than setting the dtype explicitly. + + + +Setting the names +================= + +The :keyword:`names` argument +----------------------------- + +A natural approach when dealing with tabular data is to allocate a name to each column. +A first possibility is to use an explicit structured dtype, as mentioned previously. + + >>> data = StringIO("1 2 3\n 4 5 6") + >>> np.genfromtxt(data, dtype=[(_, int) for _ in "abc"]) + array([(1, 2, 3), (4, 5, 6)], + dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8')]) + +Another simpler possibility is to use the :keyword:`names` keyword with a sequence of strings or a comma-separated string. + >>> data = StringIO("1 2 3\n 4 5 6") + >>> np.genfromtxt(data, names="A, B, C") + array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], + dtype=[('A', '<f8'), ('B', '<f8'), ('C', '<f8')]) + +In the example above, we used the fact that by default, ``dtype=float``. +By giving a sequence of names, we are forcing the output to a structured dtype. + +We may sometimes need to define the column names from the data itself. +In that case, we must use the :keyword:`names` keyword with a value of ``True``. +The names will then be read from the first line (after the ``skip_header`` ones), even if the line is commented out. + + >>> data = StringIO("So it goes\n#a b c\n1 2 3\n 4 5 6") + >>> np.genfromtxt(data, skip_header=1, names=True) + array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], + dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')]) + +The default value of :keyword:`names` is ``None``. +If we give any other value to the keyword, the new names will overwrite the field names we may have defined with the dtype. + + >>> data = StringIO("1 2 3\n 4 5 6") + >>> ndtype=[('a',int), ('b', float), ('c', int)] + >>> names = ["A", "B", "C"] + >>> np.genfromtxt(data, names=names, dtype=ndtype) + array([(1, 2.0, 3), (4, 5.0, 6)], + dtype=[('A', '<i8'), ('B', '<f8'), ('C', '<i8')]) + + +The :keyword:`defaultfmt` argument +---------------------------------- + +If ``names=None`` but a structured dtype is expected, names are defined with the standard NumPy default of ``"f%i"``, yielding names like ``f0``, ``f1`` and so forth. + >>> data = StringIO("1 2 3\n 4 5 6") + >>> np.genfromtxt(data, dtype=(int, float, int)) + array([(1, 2.0, 3), (4, 5.0, 6)], + dtype=[('f0', '<i8'), ('f1', '<f8'), ('f2', '<i8')]) + +In the same way, if we don't give enough names to match the length of the dtype, the missing names will be defined with this default template. + >>> data = StringIO("1 2 3\n 4 5 6") + >>> np.genfromtxt(data, dtype=(int, float, int), names="a") + array([(1, 2.0, 3), (4, 5.0, 6)], + dtype=[('a', '<i8'), ('f0', '<f8'), ('f1', '<i8')]) + +We can overwrite this default with the :keyword:`defaultfmt` argument, that takes any format string: + >>> data = StringIO("1 2 3\n 4 5 6") + >>> np.genfromtxt(data, dtype=(int, float, int), defaultfmt="var_%02i") + array([(1, 2.0, 3), (4, 5.0, 6)], + dtype=[('var_00', '<i8'), ('var_01', '<f8'), ('var_02', '<i8')]) + +.. note:: + We need to keep in mind that ``defaultfmt`` is used only if some names are expected but not defined. + + +Validating names +---------------- + +Numpy arrays with a structured dtype can also be viewed as :class:`~numpy.recarray`, where a field can be accessed as if it were an attribute. +For that reason, we may need to make sure that the field name doesn't contain any space or invalid character, or that it does not correspond to the name of a standard attribute (like ``size`` or ``shape``), which would confuse the interpreter. +:func:`~numpy.genfromtxt` accepts three optional arguments that provide a finer control on the names: + + :keyword:`deletechars` + Gives a string combining all the characters that must be deleted from the name. By default, invalid characters are ``~!@#$%^&*()-=+~\|]}[{';: /?.>,<``. + :keyword:`excludelist` + Gives a list of the names to exclude, such as ``return``, ``file``, ``print``... + If one of the input name is part of this list, an underscore character (``'_'``) will be appended to it. + :keyword:`case_sensitive` + Whether the names should be case-sensitive (``case_sensitive=True``), + converted to upper case (``case_sensitive=False`` or ``case_sensitive='upper'``) or to lower case (``case_sensitive='lower'``). + + + +Tweaking the conversion +======================= + +The :keyword:`converters` argument +---------------------------------- + +Usually, defining a dtype is sufficient to define how the sequence of strings must be converted. +However, some additional control may sometimes be required. +For example, we may want to make sure that a date in a format ``YYYY/MM/DD`` is converted to a :class:`datetime` object, or that a string like ``xx%`` is properly converted to a float between 0 and 1. +In such cases, we should define conversion functions with the :keyword:`converters` arguments. + +The value of this argument is typically a dictionary with column indices or column names as keys and a conversion function as values. +These conversion functions can either be actual functions or lambda functions. In any case, they should accept only a string as input and output only a single element of the wanted type. + +In the following example, the second column is converted from as string representing a percentage to a float between 0 and 1 + >>> convertfunc = lambda x: float(x.strip("%"))/100. + >>> data = "1, 2.3%, 45.\n6, 78.9%, 0" + >>> names = ("i", "p", "n") + >>> # General case ..... + >>> np.genfromtxt(StringIO(data), delimiter=",", names=names) + array([(1.0, nan, 45.0), (6.0, nan, 0.0)], + dtype=[('i', '<f8'), ('p', '<f8'), ('n', '<f8')]) + +We need to keep in mind that by default, ``dtype=float``. +A float is therefore expected for the second column. +However, the strings ``' 2.3%'`` and ``' 78.9%'`` cannot be converted to float and we end up having ``np.nan`` instead. +Let's now use a converter. + + >>> # Converted case ... + >>> np.genfromtxt(StringIO(data), delimiter=",", names=names, + ... converters={1: convertfunc}) + array([(1.0, 0.023, 45.0), (6.0, 0.78900000000000003, 0.0)], + dtype=[('i', '<f8'), ('p', '<f8'), ('n', '<f8')]) + +The same results can be obtained by using the name of the second column (``"p"``) as key instead of its index (1). + + >>> # Using a name for the converter ... + >>> np.genfromtxt(StringIO(data), delimiter=",", names=names, + ... converters={"p": convertfunc}) + array([(1.0, 0.023, 45.0), (6.0, 0.78900000000000003, 0.0)], + dtype=[('i', '<f8'), ('p', '<f8'), ('n', '<f8')]) + + +Converters can also be used to provide a default for missing entries. +In the following example, the converter ``convert`` transforms a stripped string into the corresponding float or into -999 if the string is empty. +We need to explicitly strip the string from white spaces as it is not done by default. + + >>> data = "1, , 3\n 4, 5, 6" + >>> convert = lambda x: float(x.strip() or -999) + >>> np.genfromtxt(StringIO(data), delimiter=",", + ... converter={1: convert}) + array([[ 1., -999., 3.], + [ 4., 5., 6.]]) + + + + +Using missing and filling values +-------------------------------- + +Some entries may be missing in the dataset we are trying to import. +In a previous example, we used a converter to transform an empty string into a float. +However, user-defined converters may rapidly become cumbersome to manage. + +The :func:`~nummpy.genfromtxt` function provides two other complementary mechanisms: the :keyword:`missing_values` argument is used to recognize missing data and a second argument, :keyword:`filling_values`, is used to process these missing data. + +:keyword:`missing_values` +------------------------- + +By default, any empty string is marked as missing. +We can also consider more complex strings, such as ``"N/A"`` or ``"???"`` to represent missing or invalid data. +The :keyword:`missing_values` argument accepts three kind of values: + + a string or a comma-separated string + This string will be used as the marker for missing data for all the columns + a sequence of strings + In that case, each item is associated to a column, in order. + a dictionary + Values of the dictionary are strings or sequence of strings. + The corresponding keys can be column indices (integers) or column names (strings). In addition, the special key ``None`` can be used to define a default applicable to all columns. + + +:keyword:`filling_values` +------------------------- + +We know how to recognize missing data, but we still need to provide a value for these missing entries. +By default, this value is determined from the expected dtype according to this table: + ++---------------+-------------+ ++ Expected type + Default + ++---------------+-------------+ ++ ``bool`` + ``False``+ ++---------------+-------------+ ++ ``int`` + ``-1`` + ++---------------+-------------+ ++ ``float`` + ``np.nan`` + ++---------------+-------------+ ++ ``complex`` +``np.nan+0j``+ ++---------------+-------------+ ++ ``string`` + ``'???'`` + ++---------------+-------------+ + +We can get a finer control on the conversion of missing values with the :keyword:`filling_values` optional argument. +Like :keyword:`missing_values`, this argument accepts different kind of values: + + a single value + This will be the default for all columns + a sequence of values + Each entry will be the default for the corresponding column + a dictionary + Each key can be a column index or a column name, and the corresponding value should be a single object. + We can use the special key ``None`` to define a default for all columns. + +In the following example, we suppose that the missing values are flagged with ``"N/A"`` in the first column and by ``"???"`` in the third column. +We wish to transform these missing values to 0 if they occur in the first and second column, and to -999 if they occur in the last column. + +>>> data = "N/A, 2, 3\n4, ,???" +>>> kwargs = dict(delimiter=",", +... dtype=int, +... names="a,b,c", +... missing_values={0:"N/A", 'b':" ", 2:"???"}, +... filling_values={0:0, 'b':0, 2:-999}) +>>> np.genfromtxt(StringIO.StringIO(data), **kwargs) +array([(0, 2, 3), (4, 0, -999)], + dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8')]) + + +:keyword:`usemask` +------------------ + +We may also want to keep track of the occurrence of missing data by constructing a boolean mask, with ``True`` entries where data was missing and ``False`` otherwise. +To do that, we just have to set the optional argument :keyword:`usemask` to ``True`` (the default is ``False``). +The output array will then be a :class:`~numpy.ma.MaskedArray`. + + + +.. unpack=None, loose=True, invalid_raise=True) + + +Shortcut functions +================== + +In addition to :func:`~numpy.genfromtxt`, the :mod:`numpy.lib.io` module provides several convenience functions derived from :func:`~numpy.genfromtxt`. +These functions work the same way as the original, but they have different default values. + +:func:`~numpy.ndfromtxt` + Always set ``usemask=False``. + The output is always a standard :class:`numpy.ndarray`. +:func:`~numpy.mafromtxt` + Always set ``usemask=True``. + The output is always a :class:`~numpy.ma.MaskedArray` +:func:`~numpy.recfromtxt` + Returns a standard :class:`numpy.recarray` (if ``usemask=False``) or a :class:`~numpy.ma.MaskedRecords` array (if ``usemaske=True``). + The default dtype is ``dtype=None``, meaning that the types of each column will be automatically determined. +:func:`~numpy.recfromcsv` + Like :func:`~numpy.recfromtxt`, but with a default ``delimiter=","``. + diff --git a/doc/source/user/basic.io.rst b/doc/source/user/basic.io.rst new file mode 100644 index 000000000..54ef758d5 --- /dev/null +++ b/doc/source/user/basic.io.rst @@ -0,0 +1,7 @@ +************** +I/O with Numpy +************** + +.. toctree:: + + basic.io.genfromtxt
\ No newline at end of file diff --git a/doc/source/user/basics.rst b/doc/source/user/basics.rst index 74c92127b..03f0a1d85 100644 --- a/doc/source/user/basics.rst +++ b/doc/source/user/basics.rst @@ -11,6 +11,7 @@ Numpy basics basics.types basics.creation + basics.io basics.indexing basics.broadcasting basics.byteswapping |