diff options
Diffstat (limited to 'Doc/faq/extending.rst')
| -rw-r--r-- | Doc/faq/extending.rst | 481 |
1 files changed, 481 insertions, 0 deletions
diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst new file mode 100644 index 0000000000..3389877d45 --- /dev/null +++ b/Doc/faq/extending.rst @@ -0,0 +1,481 @@ +======================= +Extending/Embedding FAQ +======================= + +.. contents:: + +.. highlight:: c + + +Can I create my own functions in C? +----------------------------------- + +Yes, you can create built-in modules containing functions, variables, exceptions +and even new types in C. This is explained in the document +:ref:`extending-index`. + +Most intermediate or advanced Python books will also cover this topic. + + +Can I create my own functions in C++? +------------------------------------- + +Yes, using the C compatibility features found in C++. Place ``extern "C" { +... }`` around the Python include files and put ``extern "C"`` before each +function that is going to be called by the Python interpreter. Global or static +C++ objects with constructors are probably not a good idea. + + +Writing C is hard; are there any alternatives? +---------------------------------------------- + +There are a number of alternatives to writing your own C extensions, depending +on what you're trying to do. + +.. XXX make sure these all work; mention Cython + +If you need more speed, `Psyco <http://psyco.sourceforge.net/>`_ generates x86 +assembly code from Python bytecode. You can use Psyco to compile the most +time-critical functions in your code, and gain a significant improvement with +very little effort, as long as you're running on a machine with an +x86-compatible processor. + +`Pyrex <http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_ is a compiler +that accepts a slightly modified form of Python and generates the corresponding +C code. Pyrex makes it possible to write an extension without having to learn +Python's C API. + +If you need to interface to some C or C++ library for which no Python extension +currently exists, you can try wrapping the library's data types and functions +with a tool such as `SWIG <http://www.swig.org>`_. `SIP +<http://www.riverbankcomputing.co.uk/sip/>`_, `CXX +<http://cxx.sourceforge.net/>`_ `Boost +<http://www.boost.org/libs/python/doc/index.html>`_, or `Weave +<http://www.scipy.org/site_content/weave>`_ are also alternatives for wrapping +C++ libraries. + + +How can I execute arbitrary Python statements from C? +----------------------------------------------------- + +The highest-level function to do this is :cfunc:`PyRun_SimpleString` which takes +a single string argument to be executed in the context of the module +``__main__`` and returns 0 for success and -1 when an exception occurred +(including ``SyntaxError``). If you want more control, use +:cfunc:`PyRun_String`; see the source for :cfunc:`PyRun_SimpleString` in +``Python/pythonrun.c``. + + +How can I evaluate an arbitrary Python expression from C? +--------------------------------------------------------- + +Call the function :cfunc:`PyRun_String` from the previous question with the +start symbol :cdata:`Py_eval_input`; it parses an expression, evaluates it and +returns its value. + + +How do I extract C values from a Python object? +----------------------------------------------- + +That depends on the object's type. If it's a tuple, :cfunc:`PyTuple_Size` +returns its length and :cfunc:`PyTuple_GetItem` returns the item at a specified +index. Lists have similar functions, :cfunc:`PyListSize` and +:cfunc:`PyList_GetItem`. + +For strings, :cfunc:`PyString_Size` returns its length and +:cfunc:`PyString_AsString` a pointer to its value. Note that Python strings may +contain null bytes so C's :cfunc:`strlen` should not be used. + +To test the type of an object, first make sure it isn't *NULL*, and then use +:cfunc:`PyString_Check`, :cfunc:`PyTuple_Check`, :cfunc:`PyList_Check`, etc. + +There is also a high-level API to Python objects which is provided by the +so-called 'abstract' interface -- read ``Include/abstract.h`` for further +details. It allows interfacing with any kind of Python sequence using calls +like :cfunc:`PySequence_Length`, :cfunc:`PySequence_GetItem`, etc.) as well as +many other useful protocols. + + +How do I use Py_BuildValue() to create a tuple of arbitrary length? +------------------------------------------------------------------- + +You can't. Use ``t = PyTuple_New(n)`` instead, and fill it with objects using +``PyTuple_SetItem(t, i, o)`` -- note that this "eats" a reference count of +``o``, so you have to :cfunc:`Py_INCREF` it. Lists have similar functions +``PyList_New(n)`` and ``PyList_SetItem(l, i, o)``. Note that you *must* set all +the tuple items to some value before you pass the tuple to Python code -- +``PyTuple_New(n)`` initializes them to NULL, which isn't a valid Python value. + + +How do I call an object's method from C? +---------------------------------------- + +The :cfunc:`PyObject_CallMethod` function can be used to call an arbitrary +method of an object. The parameters are the object, the name of the method to +call, a format string like that used with :cfunc:`Py_BuildValue`, and the +argument values:: + + PyObject * + PyObject_CallMethod(PyObject *object, char *method_name, + char *arg_format, ...); + +This works for any object that has methods -- whether built-in or user-defined. +You are responsible for eventually :cfunc:`Py_DECREF`\ 'ing the return value. + +To call, e.g., a file object's "seek" method with arguments 10, 0 (assuming the +file object pointer is "f"):: + + res = PyObject_CallMethod(f, "seek", "(ii)", 10, 0); + if (res == NULL) { + ... an exception occurred ... + } + else { + Py_DECREF(res); + } + +Note that since :cfunc:`PyObject_CallObject` *always* wants a tuple for the +argument list, to call a function without arguments, pass "()" for the format, +and to call a function with one argument, surround the argument in parentheses, +e.g. "(i)". + + +How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)? +---------------------------------------------------------------------------------------- + +In Python code, define an object that supports the ``write()`` method. Assign +this object to :data:`sys.stdout` and :data:`sys.stderr`. Call print_error, or +just allow the standard traceback mechanism to work. Then, the output will go +wherever your ``write()`` method sends it. + +The easiest way to do this is to use the StringIO class in the standard library. + +Sample code and use for catching stdout: + + >>> class StdoutCatcher: + ... def __init__(self): + ... self.data = '' + ... def write(self, stuff): + ... self.data = self.data + stuff + ... + >>> import sys + >>> sys.stdout = StdoutCatcher() + >>> print 'foo' + >>> print 'hello world!' + >>> sys.stderr.write(sys.stdout.data) + foo + hello world! + + +How do I access a module written in Python from C? +-------------------------------------------------- + +You can get a pointer to the module object as follows:: + + module = PyImport_ImportModule("<modulename>"); + +If the module hasn't been imported yet (i.e. it is not yet present in +:data:`sys.modules`), this initializes the module; otherwise it simply returns +the value of ``sys.modules["<modulename>"]``. Note that it doesn't enter the +module into any namespace -- it only ensures it has been initialized and is +stored in :data:`sys.modules`. + +You can then access the module's attributes (i.e. any name defined in the +module) as follows:: + + attr = PyObject_GetAttrString(module, "<attrname>"); + +Calling :cfunc:`PyObject_SetAttrString` to assign to variables in the module +also works. + + +How do I interface to C++ objects from Python? +---------------------------------------------- + +Depending on your requirements, there are many approaches. To do this manually, +begin by reading :ref:`the "Extending and Embedding" document +<extending-index>`. Realize that for the Python run-time system, there isn't a +whole lot of difference between C and C++ -- so the strategy of building a new +Python type around a C structure (pointer) type will also work for C++ objects. + +For C++ libraries, you can look at `SIP +<http://www.riverbankcomputing.co.uk/sip/>`_, `CXX +<http://cxx.sourceforge.net/>`_, `Boost +<http://www.boost.org/libs/python/doc/index.html>`_, `Weave +<http://www.scipy.org/site_content/weave>`_ or `SWIG <http://www.swig.org>`_ + + +I added a module using the Setup file and the make fails; why? +-------------------------------------------------------------- + +Setup must end in a newline, if there is no newline there, the build process +fails. (Fixing this requires some ugly shell script hackery, and this bug is so +minor that it doesn't seem worth the effort.) + + +How do I debug an extension? +---------------------------- + +When using GDB with dynamically loaded extensions, you can't set a breakpoint in +your extension until your extension is loaded. + +In your ``.gdbinit`` file (or interactively), add the command:: + + br _PyImport_LoadDynamicModule + +Then, when you run GDB:: + + $ gdb /local/bin/python + gdb) run myscript.py + gdb) continue # repeat until your extension is loaded + gdb) finish # so that your extension is loaded + gdb) br myfunction.c:50 + gdb) continue + +I want to compile a Python module on my Linux system, but some files are missing. Why? +-------------------------------------------------------------------------------------- + +Most packaged versions of Python don't include the +:file:`/usr/lib/python2.{x}/config/` directory, which contains various files +required for compiling Python extensions. + +For Red Hat, install the python-devel RPM to get the necessary files. + +For Debian, run ``apt-get install python-dev``. + + +What does "SystemError: _PyImport_FixupExtension: module yourmodule not loaded" mean? +------------------------------------------------------------------------------------- + +This means that you have created an extension module named "yourmodule", but +your module init function does not initialize with that name. + +Every module init function will have a line similar to:: + + module = Py_InitModule("yourmodule", yourmodule_functions); + +If the string passed to this function is not the same name as your extension +module, the :exc:`SystemError` exception will be raised. + + +How do I tell "incomplete input" from "invalid input"? +------------------------------------------------------ + +Sometimes you want to emulate the Python interactive interpreter's behavior, +where it gives you a continuation prompt when the input is incomplete (e.g. you +typed the start of an "if" statement or you didn't close your parentheses or +triple string quotes), but it gives you a syntax error message immediately when +the input is invalid. + +In Python you can use the :mod:`codeop` module, which approximates the parser's +behavior sufficiently. IDLE uses this, for example. + +The easiest way to do it in C is to call :cfunc:`PyRun_InteractiveLoop` (perhaps +in a separate thread) and let the Python interpreter handle the input for +you. You can also set the :cfunc:`PyOS_ReadlineFunctionPointer` to point at your +custom input function. See ``Modules/readline.c`` and ``Parser/myreadline.c`` +for more hints. + +However sometimes you have to run the embedded Python interpreter in the same +thread as your rest application and you can't allow the +:cfunc:`PyRun_InteractiveLoop` to stop while waiting for user input. The one +solution then is to call :cfunc:`PyParser_ParseString` and test for ``e.error`` +equal to ``E_EOF``, which means the input is incomplete). Here's a sample code +fragment, untested, inspired by code from Alex Farber:: + + #include <Python.h> + #include <node.h> + #include <errcode.h> + #include <grammar.h> + #include <parsetok.h> + #include <compile.h> + + int testcomplete(char *code) + /* code should end in \n */ + /* return -1 for error, 0 for incomplete, 1 for complete */ + { + node *n; + perrdetail e; + + n = PyParser_ParseString(code, &_PyParser_Grammar, + Py_file_input, &e); + if (n == NULL) { + if (e.error == E_EOF) + return 0; + return -1; + } + + PyNode_Free(n); + return 1; + } + +Another solution is trying to compile the received string with +:cfunc:`Py_CompileString`. If it compiles without errors, try to execute the +returned code object by calling :cfunc:`PyEval_EvalCode`. Otherwise save the +input for later. If the compilation fails, find out if it's an error or just +more input is required - by extracting the message string from the exception +tuple and comparing it to the string "unexpected EOF while parsing". Here is a +complete example using the GNU readline library (you may want to ignore +**SIGINT** while calling readline()):: + + #include <stdio.h> + #include <readline.h> + + #include <Python.h> + #include <object.h> + #include <compile.h> + #include <eval.h> + + int main (int argc, char* argv[]) + { + int i, j, done = 0; /* lengths of line, code */ + char ps1[] = ">>> "; + char ps2[] = "... "; + char *prompt = ps1; + char *msg, *line, *code = NULL; + PyObject *src, *glb, *loc; + PyObject *exc, *val, *trb, *obj, *dum; + + Py_Initialize (); + loc = PyDict_New (); + glb = PyDict_New (); + PyDict_SetItemString (glb, "__builtins__", PyEval_GetBuiltins ()); + + while (!done) + { + line = readline (prompt); + + if (NULL == line) /* CTRL-D pressed */ + { + done = 1; + } + else + { + i = strlen (line); + + if (i > 0) + add_history (line); /* save non-empty lines */ + + if (NULL == code) /* nothing in code yet */ + j = 0; + else + j = strlen (code); + + code = realloc (code, i + j + 2); + if (NULL == code) /* out of memory */ + exit (1); + + if (0 == j) /* code was empty, so */ + code[0] = '\0'; /* keep strncat happy */ + + strncat (code, line, i); /* append line to code */ + code[i + j] = '\n'; /* append '\n' to code */ + code[i + j + 1] = '\0'; + + src = Py_CompileString (code, "<stdin>", Py_single_input); + + if (NULL != src) /* compiled just fine - */ + { + if (ps1 == prompt || /* ">>> " or */ + '\n' == code[i + j - 1]) /* "... " and double '\n' */ + { /* so execute it */ + dum = PyEval_EvalCode ((PyCodeObject *)src, glb, loc); + Py_XDECREF (dum); + Py_XDECREF (src); + free (code); + code = NULL; + if (PyErr_Occurred ()) + PyErr_Print (); + prompt = ps1; + } + } /* syntax error or E_EOF? */ + else if (PyErr_ExceptionMatches (PyExc_SyntaxError)) + { + PyErr_Fetch (&exc, &val, &trb); /* clears exception! */ + + if (PyArg_ParseTuple (val, "sO", &msg, &obj) && + !strcmp (msg, "unexpected EOF while parsing")) /* E_EOF */ + { + Py_XDECREF (exc); + Py_XDECREF (val); + Py_XDECREF (trb); + prompt = ps2; + } + else /* some other syntax error */ + { + PyErr_Restore (exc, val, trb); + PyErr_Print (); + free (code); + code = NULL; + prompt = ps1; + } + } + else /* some non-syntax error */ + { + PyErr_Print (); + free (code); + code = NULL; + prompt = ps1; + } + + free (line); + } + } + + Py_XDECREF(glb); + Py_XDECREF(loc); + Py_Finalize(); + exit(0); + } + + +How do I find undefined g++ symbols __builtin_new or __pure_virtual? +-------------------------------------------------------------------- + +To dynamically load g++ extension modules, you must recompile Python, relink it +using g++ (change LINKCC in the python Modules Makefile), and link your +extension module using g++ (e.g., ``g++ -shared -o mymodule.so mymodule.o``). + + +Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)? +---------------------------------------------------------------------------------------------------------------- + +In Python 2.2, you can inherit from builtin classes such as :class:`int`, +:class:`list`, :class:`dict`, etc. + +The Boost Python Library (BPL, http://www.boost.org/libs/python/doc/index.html) +provides a way of doing this from C++ (i.e. you can inherit from an extension +class written in C++ using the BPL). + + +When importing module X, why do I get "undefined symbol: PyUnicodeUCS2*"? +------------------------------------------------------------------------- + +You are using a version of Python that uses a 4-byte representation for Unicode +characters, but some C extension module you are importing was compiled using a +Python that uses a 2-byte representation for Unicode characters (the default). + +If instead the name of the undefined symbol starts with ``PyUnicodeUCS4``, the +problem is the reverse: Python was built using 2-byte Unicode characters, and +the extension module was compiled using a Python with 4-byte Unicode characters. + +This can easily occur when using pre-built extension packages. RedHat Linux +7.x, in particular, provided a "python2" binary that is compiled with 4-byte +Unicode. This only causes the link failure if the extension uses any of the +``PyUnicode_*()`` functions. It is also a problem if an extension uses any of +the Unicode-related format specifiers for :cfunc:`Py_BuildValue` (or similar) or +parameter specifications for :cfunc:`PyArg_ParseTuple`. + +You can check the size of the Unicode character a Python interpreter is using by +checking the value of sys.maxunicode: + + >>> import sys + >>> if sys.maxunicode > 65535: + ... print 'UCS4 build' + ... else: + ... print 'UCS2 build' + +The only way to solve this problem is to use extension modules compiled with a +Python binary built using the same size for Unicode characters. + + + |