A Map of the Numba Repository
The Numba repository is quite large, and due to age has functionality spread around many locations. To help orient developers, this document will try to summarize where different categories of functionality can be found.
Support Files
Build and Packaging
setup.py - Standard Python distutils/setuptools script
MANIFEST.in - Distutils packaging instructions
requirements.txt - Pip package requirements, not used by conda
versioneer.py - Handles automatic setting of version in installed package from git tags
.flake8 - Preferences for code formatting. Files should be fixed and removed from the exception list as time allows.
.pre-commit-config.yaml - Configuration file for pre-commit hooks.
.readthedocs.yml - Configuration file for Read the Docs.
buildscripts/condarecipe.local - Conda build recipe
Continuous Integration
azure-pipelines.yml - Azure Pipelines CI config (active: Win/Mac/Linux)
buildscripts/azure/ - Azure Pipeline configuration for specific platforms
buildscripts/incremental/ - Generic scripts for building Numba on various CI systems
codecov.yml - Codecov.io coverage reporting
Documentation
LICENSE - License for Numba
LICENSES.third-party - License for third party code vendored into Numba
README.rst - README for repo, also uploaded to PyPI
CONTRIBUTING.md - Documentation on how to contribute to project (out of date, should be updated to point to Sphinx docs)
CHANGE_LOG - History of Numba releases, also directly embedded into Sphinx documentation
docs/ - Documentation source
docs/_templates/ - Directory for templates (to override defaults with Sphinx theme)
docs/Makefile - Used to build Sphinx docs with
make
docs/source - ReST source for Numba documentation
docs/_static/ - Static CSS and image assets for Numba docs
docs/make.bat - Not used (remove?)
docs/requirements.txt - Pip package requirements for building docs with Read the Docs.
numba/scripts/generate_lower_listing.py - Dump all registered implementations decorated with
@lower*
for reference documentation. Currently misses implementations from the higher level extension API.
Numba Source Code
Numba ships with both the source code and tests in one package.
numba/ - all of the source code and tests
Public API
These define aspects of the public Numba interface.
numba/core/decorators.py - User-facing decorators for compiling regular functions on the CPU
numba/core/extending.py - Public decorators for extending Numba (
overload
,intrinsic
, etc) - numba/experimental/structref.py - Public API for defining a mutable structnumba/core/ccallback.py -
@cfunc
decorator for compiling functions to a fixed C signature. Used to make callbacks.numba/np/ufunc/decorators.py - ufunc/gufunc compilation decorators
numba/core/config.py - Numba global config options and environment variable handling
numba/core/annotations - Gathering and printing type annotations of Numba IR
numba/core/annotations/pretty_annotate.py - Code highlighting of Numba functions and types (both ANSI terminal and HTML)
numba/core/event.py - A simple event system for applications to listen to specific compiler events.
Dispatching
numba/core/dispatcher.py - Dispatcher objects are compiled functions produced by
@jit
. A dispatcher has different implementations for different type signatures.numba/_dispatcher.cpp - C++ dispatcher implementation (for speed on common data types)
Compiler Pipeline
numba/core/compiler.py - Compiler pipelines and flags
numba/core/errors.py - Numba exception and warning classes
numba/core/ir.py - Numba IR data structure objects
numba/core/bytecode.py - Bytecode parsing and function identity (??)
numba/core/interpreter.py - Translate Python interpreter bytecode to Numba IR
numba/core/analysis.py - Utility functions to analyze Numba IR (variable lifetime, prune branches, etc)
numba/core/controlflow.py - Control flow analysis of Numba IR and Python bytecode
numba/core/typeinfer.py - Type inference algorithm
numba/core/transforms.py - Numba IR transformations
numba/core/rewrites - Rewrite passes used by compiler
numba/core/rewrites/__init__.py - Loads all rewrite passes so they are put into the registry
numba/core/rewrites/registry.py - Registry object for collecting rewrite passes
numba/core/rewrites/ir_print.py - Write print() calls into special print nodes in the IR
numba/core/rewrites/static_raise.py - Converts exceptions with static arguments into a special form that can be lowered
numba/core/rewrites/static_getitem.py - Rewrites getitem and setitem with constant arguments to allow type inference
numba/core/rewrites/static_binop.py - Rewrites binary operations (specifically
**
) with constant arguments so faster code can be generatednumba/core/inline_closurecall.py - Inlines body of closure functions to call site. Support for array comprehensions, reduction inlining, and stencil inlining.
numba/core/postproc.py - Postprocessor for Numba IR that computes variable lifetime, inserts del operations, and handles generators
numba/core/lowering.py - General implementation of lowering Numba IR to LLVM numba/core/environment.py - Runtime environment object
numba/core/withcontexts.py - General scaffolding for implementing context managers in nopython mode, and the objectmode context manager
numba/core/pylowering.py - Lowering of Numba IR in object mode
numba/core/pythonapi.py - LLVM IR code generation to interface with CPython API
numba/core/targetconfig.py - Utils for target configurations such as compiler flags.
Type Management
numba/core/typeconv/ - Implementation of type casting and type signature matching in both C++ and Python
numba/capsulethunk.h - Used by typeconv
numba/core/types/ - definition of the Numba type hierarchy, used everywhere in compiler to select implementations
numba/core/consts.py - Constant inference (used to make constant values available during codegen when possible)
numba/core/datamodel - LLVM IR representations of data types in different contexts
numba/core/datamodel/models.py - Models for most standard types
numba/core/datamodel/registry.py - Decorator to register new data models
numba/core/datamodel/packer.py - Pack typed values into a data structure
numba/core/datamodel/testing.py - Data model tests (this should move??)
numba/core/datamodel/manager.py - Map types to data models
Compiled Extensions
Numba uses a small amount of compiled C/C++ code for core functionality, like dispatching and type matching where performance matters, and it is more convenient to encapsulate direct interaction with CPython APIs.
numba/_arraystruct.h - Struct for holding NumPy array attributes. Used in helperlib and the Numba Runtime.
numba/_helperlib.c - C functions required by Numba compiled code at runtime. Linked into ahead-of-time compiled modules
numba/_helpermod.c - Python extension module with pointers to functions from
_helperlib.c
numba/_dynfuncmod.c - Python extension module exporting _dynfunc.c functionality
numba/_dynfunc.c - C level Environment and Closure objects (keep in sync with numba/target/base.py)
numba/mathnames.h - Macros for defining names of math functions
numba/_pymodule.h - C macros for Python 2/3 portable naming of C API functions
numba/mviewbuf.c - Handles Python memoryviews
numba/_typeof.h | numba/_typeof.cpp - C++ implementation of type fingerprinting, used by dispatcher
numba/_numba_common.h - Portable C macro for marking symbols that can be shared between object files, but not outside the library.
Misc Support
numba/_version.py - Updated by versioneer
numba/core/runtime - Language runtime. Currently manages reference-counted memory allocated on the heap by Numba-compiled functions
numba/core/ir_utils.py - Utility functions for working with Numba IR data structures
numba/core/cgutils.py - Utility functions for generating common code patterns in LLVM IR
numba/core/utils.py - Python 2 backports of Python 3 functionality (also imports local copy of
six
)numba/misc/appdirs.py - Vendored package for determining application config directories on every platform
numba/misc/POST.py - A power-on-self-test script Numba uses in CI to make sure the test runner and compilation is working.
numba/core/compiler_lock.py - Global compiler lock because Numba’s usage of LLVM is not thread-safe
numba/misc/special.py - Python stub implementations of special Numba functions (prange, gdb*)
numba/core/itanium_mangler.py - Python implementation of Itanium C++ name mangling
numba/misc/findlib.py - Helper function for locating shared libraries on all platforms
numba/core/debuginfo.py - Helper functions to construct LLVM IR debug info
numba/core/unsafe/refcount.py - Read reference count of object
numba/core/unsafe/eh.py - Exception handling helpers
numba/core/unsafe/nrt.py - Numba runtime (NRT) helpers
numba/cpython/unsafe/tuple.py - Replace a value in a tuple slot
numba/np/unsafe/ndarray.py - NumPy array helpers
numba/core/unsafe/bytes.py - Copying and dereferencing data from void pointers
numba/core/callwrapper.py - Handles argument unboxing and releasing the GIL when moving from Python to nopython mode
numba/np/numpy_support.py - Helper functions for working with NumPy and translating Numba types to and from NumPy dtypes.
numba/core/tracing.py - Decorator for tracing Python calls and emitting log messages
numba/core/funcdesc.py - Classes for describing function metadata (used in the compiler)
numba/core/sigutils.py - Helper functions for parsing and normalizing Numba type signatures
numba/core/serialize.py - Support for pickling compiled functions
numba/core/caching.py - Disk cache for compiled functions
numba/np/npdatetime.py - Helper functions for implementing NumPy datetime64 support
numba/misc/llvm_pass_timings.py - Helper to record timings of LLVM passes.
numba/cloudpickle - Vendored cloudpickle subpackage
Core Python Data Types
numba/_hashtable.h | numba/_hashtable.cpp - Adaptation of the Python 3.7 hash table implementation
numba/cext/dictobject.h | numba/cext/dictobject.c - C level implementation of typed dictionary
numba/typed/dictobject.py - Nopython mode wrapper for typed dictionary
numba/cext/listobject.h | numba/cext/listobject.c - C level implementation of typed list
numba/typed/listobject.py - Nopython mode wrapper for typed list
numba/typed/typedobjectutils.py - Common utilities for typed dictionary and list
numba/cpython/unicode.py - Unicode strings (Python 3.5 and later)
numba/typed - Python interfaces to statically typed containers
numba/typed/typeddict.py - Python interface to typed dictionary
numba/typed/typedlist.py - Python interface to typed list
numba/experimental/jitclass - Implementation of experimental JIT compilation of Python classes
numba/core/generators.py - Support for lowering Python generators
Math
numba/_random.c - Reimplementation of NumPy / CPython random number generator
numba/_lapack.c - Wrappers for calling BLAS and LAPACK functions (requires SciPy)
ParallelAccelerator
Code transformation passes that extract parallelizable code from a function and convert it into multithreaded gufunc calls.
numba/parfors/parfor.py - General ParallelAccelerator
numba/parfors/parfor_lowering.py - gufunc lowering for ParallelAccelerator
numba/parfors/array_analysis.py - Array analysis passes used in ParallelAccelerator
Stencil
Implementation of @stencil
:
numba/stencils/stencil.py - Stencil function decorator (implemented without ParallelAccelerator)
numba/stencils/stencilparfor.py - ParallelAccelerator implementation of stencil
Debugging Support
numba/misc/gdb_hook.py - Hooks to jump into GDB from nopython mode
numba/misc/cmdlang.gdb - Commands to setup GDB for setting explicit breakpoints from Python
Type Signatures (CPU)
Some (usually older) Numba supported functionality separates the declaration of allowed type signatures from the definition of implementations. This package contains registries of type signatures that must be matched during type inference.
numba/core/typing - Type signature module
numba/core/typing/templates.py - Base classes for type signature templates
numba/core/typing/cmathdecl.py - Python complex math (
cmath
) modulenumba/core/typing/bufproto.py - Interpreting objects supporting the buffer protocol
numba/core/typing/mathdecl.py - Python
math
modulenumba/core/typing/listdecl.py - Python lists
numba/core/typing/builtins.py - Python builtin global functions and operators
numba/core/typing/setdecl.py - Python sets
numba/core/typing/npydecl.py - NumPy ndarray (and operators), NumPy functions
numba/core/typing/arraydecl.py - Python
array
modulenumba/core/typing/context.py - Implementation of typing context (class that collects methods used in type inference)
numba/core/typing/collections.py - Generic container operations and namedtuples
numba/core/typing/ctypes_utils.py - Typing ctypes-wrapped function pointers
numba/core/typing/enumdecl.py - Enum types
numba/core/typing/cffi_utils.py - Typing of CFFI objects
numba/core/typing/typeof.py - Implementation of typeof operations (maps Python object to Numba type)
numba/core/typing/asnumbatype.py - Implementation of
as_numba_type
operations (maps Python types to Numba type)numba/core/typing/npdatetime.py - Datetime dtype support for NumPy arrays
Target Implementations (CPU)
Implementations of Python / NumPy functions and some data models. These modules are responsible for generating LLVM IR during lowering. Note that some of these modules do not have counterparts in the typing package because newer Numba extension APIs (like overload) allow typing and implementation to be specified together.
numba/core/cpu.py - Context for code gen on CPU
numba/core/base.py - Base class for all target contexts
numba/core/codegen.py - Driver for code generation
numba/core/boxing.py - Boxing and unboxing for most data types
numba/core/intrinsics.py - Utilities for converting LLVM intrinsics to other math calls
numba/core/callconv.py - Implements different calling conventions for Numba-compiled functions
numba/core/options.py - Container for options that control lowering
numba/core/optional.py - Special type representing value or
None
numba/core/registry.py - Registry object for collecting implementations for a specific target
numba/core/imputils.py - Helper functions for lowering
numba/core/externals.py - Registers external C functions needed to link generated code
numba/core/fastmathpass.py - Rewrite pass to add fastmath attributes to function call sites and binary operations
numba/core/removerefctpass.py - Rewrite pass to remove unnecessary incref/decref pairs
numba/core/descriptors.py - empty base class for all target descriptors (is this needed?)
numba/cpython/builtins.py - Python builtin functions and operators
numba/cpython/cmathimpl.py - Python complex math module
numba/cpython/enumimpl.py - Enum objects
numba/cpython/hashing.py - Hashing algorithms
numba/cpython/heapq.py - Python
heapq
modulenumba/cpython/iterators.py - Iterable data types and iterators
numba/cpython/listobj.py - Python lists
numba/cpython/mathimpl.py - Python
math
modulenumba/cpython/numbers.py - Numeric values (int, float, etc)
numba/cpython/printimpl.py - Print function
numba/cpython/randomimpl.py - Python and NumPy
random
modulesnumba/cpython/rangeobj.py - Python range objects
numba/cpython/slicing.py - Slice objects, and index calculations used in slicing
numba/cpython/setobj.py - Python set type
numba/cpython/tupleobj.py - Tuples (statically typed as immutable struct)
numba/misc/cffiimpl.py - CFFI functions
numba/misc/quicksort.py - Quicksort implementation used with list and array objects
numba/misc/mergesort.py - Mergesort implementation used with array objects
numba/np/arraymath.py - Math operations on arrays (both Python and NumPy)
numba/np/arrayobj.py - Array operations (both NumPy and buffer protocol)
numba/np/linalg.py - NumPy linear algebra operations
numba/np/npdatetime.py - NumPy datetime operations
numba/np/npyfuncs.py - Kernels used in generating some NumPy ufuncs
numba/np/npyimpl.py - Implementations of most NumPy ufuncs
numba/np/polynomial/polynomial_functions.py - Implementations of NumPy
Polynomial
functionsnumba/np/polynomial/polynomial_core.py - Implementations of NumPy
Polynomial
classnumba/np/ufunc_db.py - Big table mapping types to ufunc implementations
Ufunc Compiler and Runtime
numba/np/ufunc - ufunc compiler implementation
numba/np/ufunc/_internal.h | numba/np/ufunc/_internal.c - Python extension module with helper functions that use CPython & NumPy C API
numba/np/ufunc/_ufunc.c - Used by _internal.c
numba/np/ufunc/gufunc_scheduler.h | numba/np/ufunc/gufunc_scheduler.cpp - Schedule work chunks to threads
numba/np/ufunc/dufunc.py - Special ufunc that can compile new implementations at call time
numba/np/ufunc/ufuncbuilder.py - Top-level orchestration of ufunc/gufunc compiler pipeline
numba/np/ufunc/sigparse.py - Parser for generalized ufunc indexing signatures
numba/np/ufunc/parallel.py - Codegen for
parallel
targetnumba/np/ufunc/array_exprs.py - Rewrite pass for turning array expressions in regular functions into ufuncs
numba/np/ufunc/wrappers.py - Wrap scalar function kernel with loops
numba/np/ufunc/workqueue.h | numba/np/ufunc/workqueue.c - Threading backend based on pthreads/Windows threads and queues
numba/np/ufunc/omppool.cpp - Threading backend based on OpenMP
numba/np/ufunc/tbbpool.cpp - Threading backend based on TBB
Unit Tests (CPU)
CPU unit tests (GPU target unit tests listed in later sections
runtests.py - Convenience script that launches test runner and turns on full compiler tracebacks
.coveragerc - Coverage.py configuration
numba/runtests.py - Entry point to unittest runner
numba/testing/_runtests.py - Implementation of custom test runner command line interface
numba/tests/test_* - Test cases
numba/tests/*_usecases.py - Python functions compiled by some unit tests
numba/tests/support.py - Helper functions for testing and special TestCase implementation
numba/tests/dummy_module.py - Module used in
test_dispatcher.py
numba/tests/npyufunc - ufunc / gufunc compiler tests
numba/testing - Support code for testing
numba/testing/loader.py - Find tests on disk
numba/testing/notebook.py - Support for testing notebooks
numba/testing/main.py - Numba test runner
Command Line Utilities
bin/numba - Command line stub, delegates to main in
numba_entry.py
numba/misc/numba_entry.py - Main function for
numba
command line toolnumba/pycc - Ahead of time compilation of functions to shared library extension
numba/pycc/__init__.py - Main function for
pycc
command line toolnumba/pycc/cc.py - User-facing API for tagging functions to compile ahead of time
numba/pycc/compiler.py - Compiler pipeline for creating standalone Python extension modules
numba/pycc/llvm_types.py - Aliases to LLVM data types used by
compiler.py
numba/pycc/modulemixin.c - C file compiled into every compiled extension. Pulls in C source from Numba core that is needed to make extension standalone
numba/pycc/platform.py - Portable interface to platform-specific compiler toolchains
numba/pycc/decorators.py - Deprecated decorators for tagging functions to compile. Use
cc.py
instead.
CUDA GPU Target
Note that the CUDA target does reuse some parts of the CPU target.
numba/cuda/ - The implementation of the CUDA (NVIDIA GPU) target and associated unit tests
numba/cuda/decorators.py - Compiler decorators for CUDA kernels and device functions
numba/cuda/deviceufunc.py - Custom ufunc dispatch for CUDA
numba/cuda/dispatcher.py - Dispatcher for CUDA JIT functions
numba/cuda/printimpl.py - Special implementation of device printing
numba/cuda/libdevice.py - Registers libdevice functions
numba/cuda/kernels/ - Custom kernels for reduction and transpose
numba/cuda/device_init.py - Initializes the CUDA target when imported
numba/cuda/compiler.py - Compiler pipeline for CUDA target
numba/cuda/intrinsic_wrapper.py - CUDA device intrinsics (shuffle, ballot, etc)
numba/cuda/initialize.py - Deferred initialization of the CUDA device and subsystem. Called only when user imports
numba.cuda
numba/cuda/simulator_init.py - Initializes the CUDA simulator subsystem (only when user requests it with env var)
numba/cuda/random.py - Implementation of random number generator
numba/cuda/api.py - User facing APIs imported into
numba.cuda.*
numba/cuda/stubs.py - Python placeholders for functions that only can be used in GPU device code
numba/cuda/simulator/ - Simulate execution of CUDA kernels in Python interpreter
numba/cuda/vectorizers.py - Subclasses of ufunc/gufunc compilers for CUDA
numba/cuda/args.py - Management of kernel arguments, including host<->device transfers
numba/cuda/target.py - Typing and target contexts for GPU
numba/cuda/cudamath.py - Type signatures for math functions in CUDA Python
numba/cuda/errors.py - Validation of kernel launch configuration
numba/cuda/nvvmutils.py - Helper functions for generating NVVM-specific IR
numba/cuda/testing.py - Support code for creating CUDA unit tests and capturing standard out
numba/cuda/cudadecl.py - Type signatures of CUDA API (threadIdx, blockIdx, atomics) in Python on GPU
numba/cuda/cudaimpl.py - Implementations of CUDA API functions on GPU
numba/cuda/codegen.py - Code generator object for CUDA target
numba/cuda/cudadrv/ - Wrapper around CUDA driver API
numba/cuda/cudadrv/dummyarray.py - Used to hold array information on the host, but not the data.
numba/cuda/tests/ - CUDA unit tests, skipped when CUDA is not detected
numba/cuda/tests/cudasim/ - Tests of CUDA simulator
numba/cuda/tests/nocuda/ - Tests for NVVM functionality when CUDA not present
numba/cuda/tests/cudapy/ - Tests of compiling Python functions for GPU
numba/cuda/tests/cudadrv/ - Tests of Python wrapper around CUDA API