path: root/numpy/f2py/lib/extgen/doc.txt

.. -*- rest -*-

============================================
ExtGen --- Python extension module generator
============================================

:Author:
  Pearu Peterson <pearu.peterson@gmail.com>
:Created: August 2007

.. contents:: Table of Contents

Introduction
============

ExtGen is a pure Python package that provides a high-level
tool for constructing and building Python extension modules.
Even an inexperienced user with no background on writing extension
modules can build extension modules on fly when using ExtGen tool!

Hello example follows

  >>> from numpy.f2py.lib.extgen import *
  >>> m = PyCModule('foo')         # define extension module component
  >>> f = PyCFunction('hello')     # define function component
  >>> f += 'printf("Hello!\\n");'  # put a C statement into function body
  >>> m += f                       # add function to module
  >>> print m.generate()           # shows a string containing C source to extension module
                                   # useful for debugging
  >>> foo = m.build()              # compile, build, and return extension module object
  >>> foo.hello()                  # call function
  Hello!


Users reference manual
======================

Writing a python extension module requires a knowledge of Pyhton C/API
details and may take lots of effort to get your first extension module
compile and working, even for a simple problem. See the `Simple Example`__
in Python reference manual. ExtGen provides a high level tool for
constructing extension modules by automatically taking care of the
Pyhton C/API details while providing full control how an extension
module is created. 

__ http://docs.python.org/ext/

Getting started
---------------

Creating the `Simple Example`__ with the help of ExtGen tool is really simple

  >>> system = PyCFunction('system',
                           PyCArgument('command', 'c_const_char_ptr'),
                           PyCReturn('sts','c_int'))
  >>> system += 'sts = system(command);'
  >>> module = PyCModule('spam', system)
  >>> spam = module.build()
  >>> spam.system('pwd')
  /home/pearu/svn/numpy/numpy/f2py/lib
  0

__ http://docs.python.org/ext/

ExtGen generated modules have automatically generated documentation
strings that accept also user input

  >>> a = PyCArgument('command', 'c_const_char_ptr',
                      input_description='a shell command string')
  >>> r = PyCReturn('sts', 'c_int',
                      output_description='status value returned by shell command')
  >>> system = PyCFunction('system', title='Execute a shell command.')
  >>> system += a                              # add argument component to function
  >>> system += r                              # add return value component to function
  >>> system += 'sts = system(command);'       # add C code to functon body
  >>> module = PyCModule('spam', system)       # create module instance with function component
  >>> spam = module.build()
  >>> print spam.__doc__
  This module 'spam' is generated with ExtGen from NumPy version 1.0.4.dev3744.
  :Functions:
    system(command) -> sts
  >>> print spam.system.__doc__
    system(command) -> sts  
  Execute a shell command.
  :Parameters:
    command : a to C const char ptr convertable object
      a shell command string
  :Returns:
    sts : a to C int convertable object
      status value returned by shell command
  >>>

To see the source code that ExtGen generates, use `.generate()` method for any component instance

  >>> print system.generate()
  static
  char pyc_function_system_doc[] = 
  "  system(command) -> sts"
  "\n\nExecute a shell command."
  "\n\n:Parameters:\n"
  "  command : a to C const char ptr convertable object\n"
  "    a shell command string"
  "\n\n:Returns:\n"
  "  sts : a to C int convertable object\n"
  "    status value returned by shell command"
  ;
  static
  PyObject*
  pyc_function_system(PyObject *pyc_self, PyObject *pyc_args, PyObject *pyc_keywds) {
    PyObject * volatile pyc_buildvalue = NULL;
    volatile int capi_success = 1;
    const char * command = NULL;
    int sts = 0;
    static char *capi_kwlist[] = {"command", NULL};
    if (PyArg_ParseTupleAndKeywords(pyc_args, pyc_keywds,"z",
                                    capi_kwlist, &command)) {
      sts = system(command);
      capi_success = !PyErr_Occurred();
      if (capi_success) {
        pyc_buildvalue = Py_BuildValue("i", sts);
      }
    }
    return pyc_buildvalue;
  }
  >>> print module.generate()   # prints full extension module source
  ...

Components
----------

All components are subclassed of `Component` base class that provides
the following methods:

- `.generate(self)` --- return a generated component string
- `.add(self, component, container_label=None)` --- add subcomponent
  to component instance. The `container_label` argument can be used to tell
  `ExtGen` where this component should be used, see `Developers reference
  manual` for more details, otherwise `EgtGen` figures that out by
  looking at subcomponent class properties.
- `.__add__(self, other)`, `.__iadd__(self, other)` --- shortcuts
  for `self.add(other)` call.

ExtGen provides the following Python C/API related components:

- `SetupPy(<build directory>, *components)` --- generates a `setup.py` file
  that is used to build extension modules to the given build directory.
  It provides the following methods:

  - `.execute(self, *args)` --- runs `python setup.py` command with given
    arguments.

  One can add `PyCModule` and `PySource` components to `SetupPy`.

- `PyCModule(<modulename>, *components, title=..., description=...)` ---
  represents python extension module source. It provides the following
  methods:

  - `.build(self, build_dir=None, clean_at_exit=None)` --- compilers,
    builds, and returns extension module object.

  One can add `PyCFunction` components to `PyCModule`.

- `PyCFunction(<funcname>, *components, title=..., description=...)` ---
  represents python extension module function source.

  One can add `PyCArgument`, `PyCReturn`, `CCode` components to `PyCfunction`.
  String components are converted `CCode` components by default.

- `PyCArgument(<argname>, ctype=<expected C type>, **components, 
  input_intent='required', input_title=..., input_description=...,
  output_intent='hide', output_title=..., output_description=...,
  title=..., description=...,
  depends=<list of argument dependencies>)` --- represents argument
  to python extension module function.

  `ctype` is `PyCTypeSpec` component instance or string. In the latter case
  it is converted to `PyCTypeSpec` component.

- `PyCReturn(<retname>, ctype=<expected C type>, **components)` ---
  same as `PyCArgument` but with `input_intent='hide'` and `output_intent='return'`
  options set.

- `PyCTypeSpec(<typeobj>)` --- represents variable type in a
  python extension module. Over 70 types are supported:

  >>> typenames = PyCTypeSpec.typeinfo_map.keys()
  >>> typenames.sort()
  >>> print ', '.join(typenames)
  buffer, c_Py_UNICODE, c_Py_complex, c_Py_ssize_t, c_char, c_char1,
  c_const_char_ptr, c_double, c_float, c_int, c_long, c_long_long,
  c_short, c_unsigned_char, c_unsigned_int, c_unsigned_long,
  c_unsigned_long_long, c_unsigned_short, cell, cobject, complex, dict,
  file, float, frozenset, function, generator, instance, int, iter,
  list, long, method, module, numeric_array, numpy_complex128,
  numpy_complex160, numpy_complex192, numpy_complex256, numpy_complex32,
  numpy_complex64, numpy_descr, numpy_float128, numpy_float16,
  numpy_float32, numpy_float64, numpy_float80, numpy_float96,
  numpy_int128, numpy_int16, numpy_int32, numpy_int64, numpy_int8,
  numpy_iter, numpy_multiiter, numpy_ndarray, numpy_ufunc,
  numpy_uint128, numpy_uint16, numpy_uint32, numpy_uint64, numpy_uint8,
  object, property, set, slice, str, tuple, type, unicode

  `typeobj` can be python type instance (e.g. `int`) or one of the string values
  from the list of typenames above.

- `PySource(<filename>, *components)`  --- represents pure python module file.

  One can add `Code` components to `PySource`.

ExtGen provides the following C related components:

- `CSource` --- represents C file. Derived from `FileSource`.

  One can add `CCode` components to `CSource`.
  String input is converted to `CCode` component.  

- `CHeader(<header filename>)`, `CStdHeader(<std header filename>)`. Derived from `Line`.

- `CCode(*components)` --- represents any C code block. Derived from `Code`.

- `CFunction(<funcname>, rctype=CTypeSpec('int'), *components)` --- represents
  a C function.

  One can add `CArgument`, `CDeclaration`, `CCode` components to `CFunction`.

- `CDeclaration(<ctype>, *declarators)` --- represenets `<ctype> <declarators list>`
  code idiom. `<ctype>` is `CTypeSpec` instance.

  One can add `CDeclarator` components to `CDeclaration`.

- `CArgument(<argument name>, <ctype>)` --- C function argument. Derived from `CDeclaration`.

- `CTypeSpec(<typename>)` --- C type specifier. Derived from `Line`.

- `CDeclarator(<name>, *initvalues, is_string=..., is_scalar=...)` --- represents
  `<name> [= <initialzer>]` code idiom.

  String input is converted to `CInitExpr` component.

ExtGen provides the following general purpose components:

- `Word(<word>)`

  Nothing can be added to `Word`.

- `Line(*strings)`

  One can add `Line` component to `Line`.
  String input is converted to `Line` component.

- `Code(*lines)`

  One can add `Line` and `Code` components to `Code`.
  String input is converted to `Line` component.

- `FileSource(<filename>, *components)`.

  One can add `Line` and `Code` components to `FileSource`.
  String input is converted to `Code` component.

Developers reference manual
===========================

To extend ExtGen, one needs to understand the infrastructure of
generating extension modules.

There are two important concepts in ExtGen model: components and
containers. Components (ref. class `Component`) define code blocks or
code idioms used in building up a code sources. Containers (ref. class
`Container`) are named string lists that are joined together with
specified rules resulting actual code sources. ExtGen uses two steps
for constructing code sources:

- creating code components and adding them together to a parent
  component. For example, the `PyCModule` instance in the
  hello example becomes a parent component to a `PyCFunction` instance
  after executing `m += f`.

- generating code source by calling `.generate()` method of the
  parent component.

One can iterate the above process as one wishes.

The method `PyCModule.build()` is defined for convenience.
It compiles the generated sources, builds an extension module,
imports the resulting module to Python, and returns the module object.

All component classes must be derived from the base class `Component`
defined in `extgen/base.py` file. `Component` class defines the
following methods and attributes:

- `.initialize(self, *args, **kws)` is used to initialize the attributes
  and subcomponents of the `Component` instance. Derived classes
  usually redefine it to define the signature of the component
  constructor.

- `.add(self, component, container_label=None)` is used to add
  subcomponents to the `Component`. Derived classes can affect
  the behavior of the `.add()` method by redefining the following
  class attributes:

  - `.default_component_class_name` is used when the `component`
    argument is not a `Component` instance.

  - `.default_container_label` is used when component
    `container_label` is undefined. 

  - `.component_containe_map` is used to find `container_label`
    corresponding to `component` argument class.

- `.update_parent(self, parent)` is called after `parent.add(self,..)`.

- `.finalize(self)` is called after finishing adding new components
  and before `.generate()` method call.

- `.generate(self)` returns a source code string. It recursively
  processes all subcomponents, creates code containers, and
  evaluates code templates.

- `.provides(self)` property method returns an unique string
  labeling the current component. The label is used to name
  the result of `.generate()` method when storing it to a container.
  The result is saved to container only if container does not
  contain the given provides label. With this feature one avoids
  redefining the same functions, variables, types etc that are needed
  by different components.

- `.init_containers(self)` is called before processing subcomponents.
  Derived classes may redefine it.

- `.update_containers(self)` is called after processing subcomponents.
  Derived classes usually define it to fill up any containers.

- `.get_templates(self)` is used by `.generate()` method to evaluate
  the templates and return results. By default, `.get_templates()`
  returns `.template` attribute. Derived classes may redefine it
  to return a tuple of templates, then also `.generate()` will
  return a tuple of source code strings.

- `.get_container(self, name)` or `.container_<name>` can be used
  to retrive a container with a given name. If the current component
  does not have requested container then the method tries to find
  the container from parent classes. If it still does not find it,
  then a new container with the given name will be created for
  the current component. One should acctually avoid the last
  solution and always define the containers in `.container_options`
  class attribute. This attribute is a mapping between container
  names and keyword options to the `Container` constructor.
  See `Container` options below for more detail.

- `.evaluate(self, template)` will evaluate `template` using
  the attributes (with string values) and the code from containers.

- `.info(message)`, `.warning(message)` are utility methods and
  will write messages to `sys.stderr`.

- `.register(*components)` will register predefined components
  that can be retrived via `.get(provides)` method.

Deriving a new `Component` class involves the following
tasks:

- A component class must have a base class `Component`.

- A component class may redefine `.initialize()`,
  `.init_containers()`, `.add()`, `update_parent()`, 
  `.update_containers()`, `.get_templates()`
  methods, `.provides()` property method and `.container_options`,
  `.component_container_map`, `.default_container_label`,
  `.default_component_class_name`, `.template` attributes.

- In `.initialize()` method one can process constructor options,
  set new attributes and add predefined components. It must
  return a `Component` instance.

- In `.init_containers()` and `.update_containers()` methods
  one may retrive containers from parents via `.get_container(<name>)`
  method or `.container_<name>` attribute and fill them using
  `.add()` method of the container.

- The attribute `.template` is a string containing formatting mapping keys
  that correspond to containers names or instance attribute names.

- The attribute `.container_options` is a mapping of container
  names and keyword argument dictionaries used as options
  to a `Container` constructor.

- The attribute `.component_container_map` is a mapping between
  subcomponent class names and the names of containers that should
  be used to save the code generation results.

- All classes derived from `Component` are available as
  `Component.<subclass name>`.

See `extgen/*.py` files for more examples how to redefine `Component`
class methods and attributes.


Using `Container` class
-----------------------
    
`Container` class has the following optional arguments:

  - `separator='\n'`
  - `prefix=''`
  - `suffix=''`
  - `skip_prefix_when_empty=False`
  - `skip_suffix_when_empty=False`
  - `default=''`
  - `reverse=False`
  - `use_indent=False`
  - `use_firstline_indent=False`
  - `indent_offset=0`
  - `user_defined_str=None`
  - `replace_map={}`
  - `ignore_empty_content=False`
  - `skip_prefix_suffix_when_single=False`

that are used to enhance the behaviour of `Container.__str__()`
method.  By default, `Container.__str__()` returns
`prefix+separator.join(<Container instance>.list)+suffix`.

One can add items to `Container` instance using `.add(<string>,
label=None)` method. The items are saved in `.list` and `.label_map`
attributes.

`Container` instances can be combined using `+` operator and
copied with `.copy()` method. The `.copy()` method has the
same arguments as `Container` constructor and can be used
to change certain container properties.

The `label` argument should contain an unique value that represents
the content of `<string>`.  If `label` is `None` then `label =
time.time()` will be set.

If one tries to add items with the same label to the container then
the equality of the corresponding string values will be checked. If
they are not equal then `ValueError` is raised, otherwise adding an
item is ignored.

If `reverse` is `True` then the `.list` is reversed before joining
its items. If `use_indent` is `True` then each item in `.list` will
be prefixed with `indent_offset` spaces. If `use_firstline_indent` is
`True` then additional indention of the number of starting spaces
in `.line[0]` is used. The `replace_map` is used to apply
`.replace(key, value)` method to the result of `__str__()`.
Full control over the `__str__()` method is obtained via
defining `user_defined_str` that should be a callable object taking
list as input and return a string.