NBEP 4: Defining C callbacks
Interfacing with some native libraries (for example written in C
or C++) can necessitate writing native callbacks to provide business logic
to the library. Some Python-facing libraries may also provide the
alternative of passing a ctypes-wrapped native callback instead of a
Python callback for better performance. A simple example is the
scipy.integrate package where the user passes the function to be
integrated as a callback.
Users of those libraries may want to benefit from the performance advantage of running purely native code, while writing their code in Python. This proposal outlines a scheme to provide such a functionality in Numba.
We propose adding a new decorator,
@cfunc, importable from the main
package. This decorator allows defining a callback as in the following
from numba import cfunc from numba.types import float64 # A callback with the C signature `double(double)` @cfunc(float64(float64), nopython=True) def integrand(x): return 1 / x
@cfunc decorator returns a “C function” object holding the
resources necessary to run the given compiled function (for example its
LLVM module). This object has several attributes and methods:
ctypesattribute is a ctypes function object representing the native function.
addressattribute is the address of the native function code, as an integer (note this can also be computed from the
native_nameattribute is the symbol under which the function can be looked up inside the current process.
inspect_llvm()method returns the IR for the LLVM module in which the function is compiled. It is expected that the
native_nameattribute corresponds to the function’s name in the LLVM IR.
The general signature of the decorator is
signature must specify the argument types and return type of the
function using Numba types. In contrary to
@jit, the return type cannot
options are keyword-only parameters specifying compilation options.
We are expecting that the standard
@jit options (
cache) can be made to work with
Calling from Numba-compiled functions
While the intended use is to pass a callback’s address to foreign C code expecting a function pointer, it should be made possible to call the C callback from a Numba-compiled function.
Passing array data
Native platform ABIs as used by C or C++ don’t have the notion of a shaped array as in Numpy. One common solution is to pass a raw data pointer and one or several size arguments (depending on dimensionality). Numba must provide a way to rebuild an array view of this data inside the callback.
from numba import cfunc, carray from numba.types import float64, CPointer, void, intp # A callback with the C signature `void(double *, double *, size_t)` @cfunc(void(CPointer(float64), CPointer(float64), intp)) def invert(in_ptr, out_ptr, n): in_ = carray(in_ptr, (n,)) out = carray(out_ptr, (n,)) for i in range(n): out[i] = 1 / in_[i]
carray function takes
(pointer, shape, dtype) arguments
dtype being optional) and returns a C-layout array view over the
data pointer, with the given shape and dtype. pointer must
be a ctypes pointer object (not a Python integer). The array’s
dimensionality corresponds to the shape tuple’s length. If dtype
is not given, the array’s dtype corresponds to the pointer’s pointee
farray function is similar except that it returns a F-layout
There is no standard mechanism in C for error reporting. Unfortunately,
Numba currently doesn’t handle
try..except blocks, which makes it more
difficult for the user to implement the required error reporting scheme.
The current stance of this proposal is to let users guard against invalid
arguments where necessary, and do whatever is required to inform the caller
of the error.
Based on user feedback, we can later add support for some error reporting
schemes, such as returning an integer error code depending on whether an
exception was raised, or setting
This proposal doesn’t make any provision for AOT compilation of C callbacks.
It would probably necessitate a separate API (a new method on the
numba.pycc.CC object), and the implementation would require exposing
a subset of the C function object’s functionality from the compiled C
Opaque data pointers
Some libraries allow passing an opaque data pointer (
void *) to a
user-provided callback, to provide any required context for execution
of the callback. Taking advantage of this functionality would require
adding specific support in Numba, for example the ability to do generic
types.voidptr and to take the address of a