.. -*- rest -*- ============================================ ExtGen --- Python extension module generator ============================================ :Author: Pearu Peterson :Created: August 2007 .. contents:: Table of Contents Introduction ============ ExtGen is a pure Python package that provides a high-level tool for constructing and building Python extension modules. Even an inexperienced user with no background on writing extension modules can build extension modules on fly when using ExtGen tool! Hello example follows >>> from numpy.f2py.lib.extgen import * >>> m = PyCModule('foo') # define extension module component >>> f = PyCFunction('hello') # define function component >>> f += 'printf("Hello!\\n");' # put a C statement into function body >>> m += f # add function to module >>> print m.generate() # shows a string containing C source to extension module # useful for debugging >>> foo = m.build() # compile, build, and return extension module object >>> foo.hello() # call function Hello! Users reference manual ====================== Writing a python extension module requires a knowledge of Pyhton C/API details and may take lots of effort to get your first extension module compile and working, even for a simple problem. See the `Simple Example`__ in Python reference manual. ExtGen provides a high level tool for constructing extension modules by automatically taking care of the Pyhton C/API details while providing full control how an extension module is created. __ http://docs.python.org/ext/ Getting started --------------- Creating the `Simple Example`__ with the help of ExtGen tool is really simple >>> system = PyCFunction('system', PyCArgument('command', 'c_const_char_ptr'), PyCReturn('sts','c_int')) >>> system += 'sts = system(command);' >>> module = PyCModule('spam', system) >>> spam = module.build() >>> spam.system('pwd') /home/pearu/svn/numpy/numpy/f2py/lib 0 __ http://docs.python.org/ext/ ExtGen generated modules have automatically generated documentation strings that accept also user input >>> a = PyCArgument('command', 'c_const_char_ptr', input_description='a shell command string') >>> r = PyCReturn('sts', 'c_int', output_description='status value returned by shell command') >>> system = PyCFunction('system', title='Execute a shell command.') >>> system += a # add argument component to function >>> system += r # add return value component to function >>> system += 'sts = system(command);' # add C code to functon body >>> module = PyCModule('spam', system) # create module instance with function component >>> spam = module.build() >>> print spam.__doc__ This module 'spam' is generated with ExtGen from NumPy version 1.0.4.dev3744. :Functions: system(command) -> sts >>> print spam.system.__doc__ system(command) -> sts Execute a shell command. :Parameters: command : a to C const char ptr convertable object a shell command string :Returns: sts : a to C int convertable object status value returned by shell command >>> To see the source code that ExtGen generates, use `.generate()` method for any component instance >>> print system.generate() static char pyc_function_system_doc[] = " system(command) -> sts" "\n\nExecute a shell command." "\n\n:Parameters:\n" " command : a to C const char ptr convertable object\n" " a shell command string" "\n\n:Returns:\n" " sts : a to C int convertable object\n" " status value returned by shell command" ; static PyObject* pyc_function_system(PyObject *pyc_self, PyObject *pyc_args, PyObject *pyc_keywds) { PyObject * volatile pyc_buildvalue = NULL; volatile int capi_success = 1; const char * command = NULL; int sts = 0; static char *capi_kwlist[] = {"command", NULL}; if (PyArg_ParseTupleAndKeywords(pyc_args, pyc_keywds,"z", capi_kwlist, &command)) { sts = system(command); capi_success = !PyErr_Occurred(); if (capi_success) { pyc_buildvalue = Py_BuildValue("i", sts); } } return pyc_buildvalue; } >>> print module.generate() # prints full extension module source ... Components ---------- All components are subclassed of `Component` base class that provides the following methods: - `.generate(self)` --- return a generated component string - `.add(self, component, container_label=None)` --- add subcomponent to component instance. The `container_label` argument can be used to tell `ExtGen` where this component should be used, see `Developers reference manual` for more details, otherwise `EgtGen` figures that out by looking at subcomponent class properties. - `.__add__(self, other)`, `.__iadd__(self, other)` --- shortcuts for `self.add(other)` call. ExtGen provides the following Python C/API related components: - `SetupPy(, *components)` --- generates a `setup.py` file that is used to build extension modules to the given build directory. It provides the following methods: - `.execute(self, *args)` --- runs `python setup.py` command with given arguments. One can add `PyCModule` and `PySource` components to `SetupPy`. - `PyCModule(, *components, title=..., description=...)` --- represents python extension module source. It provides the following methods: - `.build(self, build_dir=None, clean_at_exit=None)` --- compilers, builds, and returns extension module object. One can add `PyCFunction` components to `PyCModule`. - `PyCFunction(, *components, title=..., description=...)` --- represents python extension module function source. One can add `PyCArgument`, `PyCReturn`, `CCode` components to `PyCfunction`. String components are converted `CCode` components by default. - `PyCArgument(, ctype=, **components, input_intent='required', input_title=..., input_description=..., output_intent='hide', output_title=..., output_description=..., title=..., description=..., depends=)` --- represents argument to python extension module function. `ctype` is `PyCTypeSpec` component instance or string. In the latter case it is converted to `PyCTypeSpec` component. - `PyCReturn(, ctype=, **components)` --- same as `PyCArgument` but with `input_intent='hide'` and `output_intent='return'` options set. - `PyCTypeSpec()` --- represents variable type in a python extension module. Over 70 types are supported: >>> typenames = PyCTypeSpec.typeinfo_map.keys() >>> typenames.sort() >>> print ', '.join(typenames) buffer, c_Py_UNICODE, c_Py_complex, c_Py_ssize_t, c_char, c_char1, c_const_char_ptr, c_double, c_float, c_int, c_long, c_long_long, c_short, c_unsigned_char, c_unsigned_int, c_unsigned_long, c_unsigned_long_long, c_unsigned_short, cell, cobject, complex, dict, file, float, frozenset, function, generator, instance, int, iter, list, long, method, module, numeric_array, numpy_complex128, numpy_complex160, numpy_complex192, numpy_complex256, numpy_complex32, numpy_complex64, numpy_descr, numpy_float128, numpy_float16, numpy_float32, numpy_float64, numpy_float80, numpy_float96, numpy_int128, numpy_int16, numpy_int32, numpy_int64, numpy_int8, numpy_iter, numpy_multiiter, numpy_ndarray, numpy_ufunc, numpy_uint128, numpy_uint16, numpy_uint32, numpy_uint64, numpy_uint8, object, property, set, slice, str, tuple, type, unicode `typeobj` can be python type instance (e.g. `int`) or one of the string values from the list of typenames above. - `PySource(, *components)` --- represents pure python module file. One can add `Code` components to `PySource`. ExtGen provides the following C related components: - `CSource` --- represents C file. Derived from `FileSource`. One can add `CCode` components to `CSource`. String input is converted to `CCode` component. - `CHeader(
)`, `CStdHeader()`. Derived from `Line`. - `CCode(*components)` --- represents any C code block. Derived from `Code`. - `CFunction(, rctype=CTypeSpec('int'), *components)` --- represents a C function. One can add `CArgument`, `CDeclaration`, `CCode` components to `CFunction`. - `CDeclaration(, *declarators)` --- represenets ` ` code idiom. `` is `CTypeSpec` instance. One can add `CDeclarator` components to `CDeclaration`. - `CArgument(, )` --- C function argument. Derived from `CDeclaration`. - `CTypeSpec()` --- C type specifier. Derived from `Line`. - `CDeclarator(, *initvalues, is_string=..., is_scalar=...)` --- represents ` [= ]` code idiom. String input is converted to `CInitExpr` component. ExtGen provides the following general purpose components: - `Word()` Nothing can be added to `Word`. - `Line(*strings)` One can add `Line` component to `Line`. String input is converted to `Line` component. - `Code(*lines)` One can add `Line` and `Code` components to `Code`. String input is converted to `Line` component. - `FileSource(, *components)`. One can add `Line` and `Code` components to `FileSource`. String input is converted to `Code` component. Developers reference manual =========================== To extend ExtGen, one needs to understand the infrastructure of generating extension modules. There are two important concepts in ExtGen model: components and containers. Components (ref. class `Component`) define code blocks or code idioms used in building up a code sources. Containers (ref. class `Container`) are named string lists that are joined together with specified rules resulting actual code sources. ExtGen uses two steps for constructing code sources: - creating code components and adding them together to a parent component. For example, the `PyCModule` instance in the hello example becomes a parent component to a `PyCFunction` instance after executing `m += f`. - generating code source by calling `.generate()` method of the parent component. One can iterate the above process as one wishes. The method `PyCModule.build()` is defined for convenience. It compiles the generated sources, builds an extension module, imports the resulting module to Python, and returns the module object. All component classes must be derived from the base class `Component` defined in `extgen/base.py` file. `Component` class defines the following methods and attributes: - `.initialize(self, *args, **kws)` is used to initialize the attributes and subcomponents of the `Component` instance. Derived classes usually redefine it to define the signature of the component constructor. - `.add(self, component, container_label=None)` is used to add subcomponents to the `Component`. Derived classes can affect the behavior of the `.add()` method by redefining the following class attributes: - `.default_component_class_name` is used when the `component` argument is not a `Component` instance. - `.default_container_label` is used when component `container_label` is undefined. - `.component_containe_map` is used to find `container_label` corresponding to `component` argument class. - `.update_parent(self, parent)` is called after `parent.add(self,..)`. - `.finalize(self)` is called after finishing adding new components and before `.generate()` method call. - `.generate(self)` returns a source code string. It recursively processes all subcomponents, creates code containers, and evaluates code templates. - `.provides(self)` property method returns an unique string labeling the current component. The label is used to name the result of `.generate()` method when storing it to a container. The result is saved to container only if container does not contain the given provides label. With this feature one avoids redefining the same functions, variables, types etc that are needed by different components. - `.init_containers(self)` is called before processing subcomponents. Derived classes may redefine it. - `.update_containers(self)` is called after processing subcomponents. Derived classes usually define it to fill up any containers. - `.get_templates(self)` is used by `.generate()` method to evaluate the templates and return results. By default, `.get_templates()` returns `.template` attribute. Derived classes may redefine it to return a tuple of templates, then also `.generate()` will return a tuple of source code strings. - `.get_container(self, name)` or `.container_` can be used to retrive a container with a given name. If the current component does not have requested container then the method tries to find the container from parent classes. If it still does not find it, then a new container with the given name will be created for the current component. One should acctually avoid the last solution and always define the containers in `.container_options` class attribute. This attribute is a mapping between container names and keyword options to the `Container` constructor. See `Container` options below for more detail. - `.evaluate(self, template)` will evaluate `template` using the attributes (with string values) and the code from containers. - `.info(message)`, `.warning(message)` are utility methods and will write messages to `sys.stderr`. - `.register(*components)` will register predefined components that can be retrived via `.get(provides)` method. Deriving a new `Component` class involves the following tasks: - A component class must have a base class `Component`. - A component class may redefine `.initialize()`, `.init_containers()`, `.add()`, `update_parent()`, `.update_containers()`, `.get_templates()` methods, `.provides()` property method and `.container_options`, `.component_container_map`, `.default_container_label`, `.default_component_class_name`, `.template` attributes. - In `.initialize()` method one can process constructor options, set new attributes and add predefined components. It must return a `Component` instance. - In `.init_containers()` and `.update_containers()` methods one may retrive containers from parents via `.get_container()` method or `.container_` attribute and fill them using `.add()` method of the container. - The attribute `.template` is a string containing formatting mapping keys that correspond to containers names or instance attribute names. - The attribute `.container_options` is a mapping of container names and keyword argument dictionaries used as options to a `Container` constructor. - The attribute `.component_container_map` is a mapping between subcomponent class names and the names of containers that should be used to save the code generation results. - All classes derived from `Component` are available as `Component.`. See `extgen/*.py` files for more examples how to redefine `Component` class methods and attributes. Using `Container` class ----------------------- `Container` class has the following optional arguments: - `separator='\n'` - `prefix=''` - `suffix=''` - `skip_prefix_when_empty=False` - `skip_suffix_when_empty=False` - `default=''` - `reverse=False` - `use_indent=False` - `use_firstline_indent=False` - `indent_offset=0` - `user_defined_str=None` - `replace_map={}` - `ignore_empty_content=False` - `skip_prefix_suffix_when_single=False` that are used to enhance the behaviour of `Container.__str__()` method. By default, `Container.__str__()` returns `prefix+separator.join(.list)+suffix`. One can add items to `Container` instance using `.add(, label=None)` method. The items are saved in `.list` and `.label_map` attributes. `Container` instances can be combined using `+` operator and copied with `.copy()` method. The `.copy()` method has the same arguments as `Container` constructor and can be used to change certain container properties. The `label` argument should contain an unique value that represents the content of ``. If `label` is `None` then `label = time.time()` will be set. If one tries to add items with the same label to the container then the equality of the corresponding string values will be checked. If they are not equal then `ValueError` is raised, otherwise adding an item is ignored. If `reverse` is `True` then the `.list` is reversed before joining its items. If `use_indent` is `True` then each item in `.list` will be prefixed with `indent_offset` spaces. If `use_firstline_indent` is `True` then additional indention of the number of starting spaces in `.line[0]` is used. The `replace_map` is used to apply `.replace(key, value)` method to the result of `__str__()`. Full control over the `__str__()` method is obtained via defining `user_defined_str` that should be a callable object taking list as input and return a string.