Python And Native Modules

When we think about running Python code, particularly in the CPython interpreter, we normally think about interpreted code (a bytecode interpreter), but we also know that we can write native modules (extension modules) normally in C or C++. but also using cython, nuitka or even rust. These extension modules are not used just by some third party libraries, but the python runtime/environment/standard library also makes good use of them. In this sense, as explained in this article we have 2 types of these modules:

• A built-in extension module is a module built and shipped with the Python interpreter. A built-in module is statically linked into the interpreter, thereby lacking a __file__ attribute.
• A shared (or dynamic) extension module is built as a shared library (.so or .dll file) and is dynamically linked into the interpreter. In particular, the module’s __file__ attribute contains the path to the .so or .dll file.

Normally the Python interpreter is contained in the python3.XX binary (.exe in Windows), that is like 30 MBs in size, but in some builds (those done with the --enable-shared flag), most of the code is put in a libpython3.14.so (.dll in Windows) dynamic libray, and the python binary just bootstraps it.

built-in extension modules (like sys, builtins, _thread, gc...) are obviously part of the Python distribution (they are inside the interpreter as we've seen), while for shared extension modules, we have those that are part of the python distribution (like math, array, _ssl, _socket) and those that are developed by third parties.

You can check all the built-in extension modules like this:

>>> sys.builtin_module_names
('_abc', '_ast', '_codecs', '_collections', '_contextvars', '_datetime', '_functools', '_imp', '_io', '_locale', '_opcode', '_operator', '_signal', '_sre', '_stat', '_string', '_suggestions', '_symtable', '_sysconfig', '_thread', '_tokenize', '_tracemalloc', '_types', '_typing', '_warnings', '_weakref', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time')

While for the shared extension modules that are part of your distribution, just check the lib-dynload folder in your installation, e.g.:

ls -la /usr/local/lib/python3.14/lib-dynload/
total 27032
drwxr-xr-x  2 root root    4096 mar 14 13:26 .
drwxr-xr-x 43 root root    4096 mar 14 13:26 ..
-rwxr-xr-x  1 root root  298704 mar 14 13:26 array.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root  399192 mar 14 13:26 _asyncio.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root  201048 mar 14 13:26 binascii.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root   97240 mar 14 13:26 _bisect.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root 1705384 mar 14 13:26 _blake2.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root  105048 mar 14 13:26 _bz2.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root  166800 mar 14 13:26 cmath.cpython-314-x86_64-linux-gnu.so
...

So when running a Python application you have normal Python functions (that as first step of the execution have been compiled to Python bytecodes) that have to be interpreted, and other functions, living in those extension modules that are already native code and have to be executed as such, without interpretation. How does Python manage that?

The essential part in most interpreters is the interpreter loop. This is the code that loops through the bytecode instructions to be interpreted (or traverses the tree of nodes in Tree-parsing interpreters). I say "most" rather than "all" because in tree parsing interpreters we can have things like Truffle, where each node execute() method takes care of moving to the next node (and also manages its own specialization). In CPython, this interpreter loop lives in the _PyEval_EvalFrameDefault function. It's nicely explained here

When a Python function is invoked we have a Function object and a CALL bytecode instruction. If the function being called is not a native one (so a normal function that was written in pure Python and compiled to bytecodes to be interpreted) the interpreter handles this CALL by creating a frame object (the optimized _PyInterpreterFrame that I discussed here), that contains all the information needed for the function execution: local variables (including arguments and closure cells), the code object... and invokes _PyEval_EvalFrameDefault() with that frame. Indeed, there's a very interesting performance optimization added in Python3.11, _PyEval_EvalFrameDefault no longer recursively calls itself when it finds a new CALL, so we keep a flat C stack. From a GPT:

Ordinary Python→Python calls are frameless on the C stack. A chain of plain function calls runs inside a single _PyEval_EvalFrameDefault C invocation. The CALL opcode pushes a lightweight _PyInterpreterFrame onto a per thread data stack (chunked heap) and dispatches within the same C frame. Plain recursion is bounded by sys.getrecursionlimit() (a logical / data-stack counter), not the C stack.

And from this very in depth article

In CPython 3.10 and earlier, the CALL instruction used to create a new interpreter stackframe for the function being called and then it used to recursively reenter the interpreter by calling its entry point _PyEval_EvalFrameDefault.

This was bad for performance from many angles at the hardware level. The recursive call into the interpreter required saving the registers for the current function, and pushing a new C stackframe. It would lead to increased memory usage because each recursive interpreter call would allocate its own local variables on the stack, and other heap allocations. Apart from that it would also lead to poor instruction cache locality due to the constant jumps in and out of the bytecode evaluation loop.

In the 3.11 release this was fixed by eliminating the recursive call to the interpreter. Now the CALL instruction simply creates the stackframe for the called function, after that it immediately starts evaluating the new function’s bytecode without ever leaving the loop.

I've explained so far how the interpreter manages "normal" functions, but what about native functions? The Function object for a native function does not have a code object (as obviously they don't have associated bytecodes), but a function pointer to the native code. From a GPT:

Native functions: Stored as PyCFunctionObject with a function pointer
No __code__ attribute
Instead, has a function pointer (ml_meth) stored in PyMethodDef
CPython calls the C function pointer directly, bypassing the interpreter loop
The C function executes natively and returns a PyObject*

How does the interpreter check if this 'CALL function_object' should be treated one way or another (_PyInterpreterFrame + _PyEval_EvalFrameDefault() vs native call)? Basically at a low level different objects are used for a "normal" function and a native function. But the whole story goes like this (GPT explanation)

At the C level, all Python objects are represented by the PyObject structure, which points to a PyTypeObject (its type). The type object defines how instances of that type behave.

To make calls fast and avoid creating temporary tuple/dict arguments, CPython uses the vectorcall protocol (Py_TPFLAGS_HAVE_VECTORCALL).
When the interpreter encounters a CALL opcode, it ultimately looks at the target object's type to find its vectorcall entry point:

- Pure Python Functions (PyFunction_Type): The vectorcall pointer points to _PyFunction_Vectorcall. This function extracts the function's PyCodeObject (func->func_code), allocates a new _PyInterpreterFrame on the evaluation stack, and pushes it to _PyEval_EvalFrameDefault.

- Built-in/Native Functions (PyCFunction_Type): Native functions (like math.sqrt or print) are wrapped in a PyCFunctionObject. Their vectorcall pointer points to _PyCFunction_Vectorcall. This function extracts the underlying C function pointer (func->m_ml->ml_meth) and invokes the compiled C code directly.

But things are even more interesting. I already talked about how in Python3.11 the Python interpreter turned into a Python Adaptive Specializing Interpreter. Thanks to that, Function Calls are optimized like this (as explained by a GPT)

If the interpreter had to look up the type and vectorcall pointer from scratch on every single loop iteration, it would be slow. To solve this, CPython uses Opcodes Specialization (PEP 659 Adaptive Interpreter). When a generic CALL opcode executes, it starts in an "adaptive" state. It looks at the object being called and patches itself in memory to a specialized version based on what it sees:

- Python-to-Python Calls
If the target is a pure Python function, the CALL opcode specializes itself into CALL_PY_EXACT_ARGS (or CALL_PY_BOUND_METHOD).

The Check: It performs a strict pointer comparison on the target's type to ensure it is exactly PyFunction_Type.
The Action: It skips the generic lookup, directly grabs the code object, creates the _PyInterpreterFrame, and increments the frame depth.

- Python-to-Native Calls

If the target is a native C function, the CALL opcode specializes into CALL_BUILTIN_FAST or CALL_BUILTIN_CLASS.

The Check: It verifies that the object's type is PyCFunction_Type and often checks if the specific function handler matches what was cached.
The Action: It bypasses the interpreter frame creation entirely, sets up the C arguments, and jumps straight into the native C function.

That's fascinating!

If out of curiosity we want to know if a function is pure Python or native code we can check its type. For normal functions its type is FunctionType, for native functions it's BuiltinFunctionType

# Normal functions:
import types
import math

def f1(): pass
l1 = lambda x: x

isinstance(f1, types.FunctionType)
True
isinstance(l1, types.FunctionType)
True

isinstance(len, types.FunctionType)
False
isinstance(len, types.BuiltinFunctionType)
True
isinstance(math.sqrt, types.BuiltinFunctionType)

Additionally, as I've mentioned, native functions lack a __code__ attribute.

f1.__code__
code object f1 at 0x76be17bfcac0, file "", line 1

len.__code__
Traceback (most recent call last):
  File "", line 1, in 
    len.__code__
AttributeError: 'builtin_function_or_method' object has no attribute '__code__'. 

Python unpacking and multiple assignment (again)

When I talked about Python destructuring assignment I also mentioned the * and ** syntax, and referred to it as "operators" which indeed is not right (though it's common to name them like that). They should be better referred as unpacking/packing syntax or star expressions. For the different use cases of this syntax:

• function invokation: f1(*args, **kwargs) or function definition: def f2(*args, **kwargs) we should talk about arguments unpacking or packing
• When used in collection literals we should talk about iterator unpacking: [*items1, *items2] and dictionary unpacking: {**dict1, **dict2}, with the expressions being called "starred expressions"
• When used on the left side of an assignment: first, *reminder = my_list, we talk of a "starred target".

As we know the unpacking (destructuring) happens automatically when performing a multiple assignment, with no need of using * at all

x, y = [1, 2]
print(f"x: {x}, y: {y}")
# x: 1, y: 2

There are some advanced uses that I tend to forget. We can use multiple assignment with object attributes or dictionary keys:

@dataclass
class Person:
    name: str
    age: int
    country: str

# multiple assignment to attributes of an object
p1 = Person("Antoine", 47, "France")
p1.name, p1.age = ["Francois", 48]
print(f"p1.name: {p1.name}, p1.age: {p1.age}")

# multiple assignment to dictionary keys
d1 = {}
d1["name"], d1["age"] = ["Francois", 48]
print(f"d1['name']: {d1['name']}, d1['age']: {d1['age']}")

As I explained in this post we can unpack nested structures:

x, [a, b], y = [1, [2, 3], 4]
print(f"x: {x}, a: {a}, b: {b}, y: {y}")
# x: 1, a: 2, b: 3, y: 4

But notice that nested unpacking works for assignment, but not for function parameters. Surprisingly this is something that worked in Python2 but was lost in Python3.

>>> def f(a, (b, c)):
...     return c

SyntaxError: Function parameters cannot be parenthesized


>>> def f(a, [b, c]):
...     return c
...     
            
SyntaxError: invalid syntax

Python is missing the object destructuring assignment feature present JavaScript, I mean:

const user = {
  id: 42,
  isVerified: true,
};

const { id, isVerified } = user;

So we have to use this more verbose approach (notice that itemgetter and attrgetter are a nice option for dynamic scenarios)

p1 = Person("Antoine", 47, "France")

name, country = p1.name, p1.country
name, country = attrgetter("name", "country")(p1)

name, age = d1["name"], d1["age"]
name, age = itemgetter("name", "age")(d1)

Python Class Body to the Limit

In my 2 previous posts [1] and [2] we've seen how the class body of a Python class statement is just executable code that is put in a code object around which a synthetic function is created. This code is executed during the class creation (receiving a namespace object, a dictionary, created by the __prepare__ method of the metaclass). This is pretty powerful, as you can put complex initialization code there, not just a normal assignment). We can apply a decorator conditionally, create multiple function alias, define functions conditionally... Let's see some examples.

def log_deco(fn):
    def log_wrapper(*args, **kwargs):
        print(f"invoking {fn.__name__}")
        return fn(*args, **kwargs)
    return log_wrapper

def do_nothing_deco(fn):
    return fn

log_mode = True
profile_mode = Trumult

class Person:
    def __init__(self, name: str): 
        self.name = name

    # conditional decorator
    @(log_deco if log_mode else do_nothing_deco)
    def say_hi(self, to_x: str):
        print(f"hi {to_x} I'm {self.name}")

    # declare a function conditionally
    if profile_mode:
        def get_memory_consumption(self):
            print("my memory consumtpion is ...")

    def do_something(self):
        print(f"{self.name} is doing_something")

    # function aliases
    work = do_something
    cook = do_something
    sleep = do_something

p1 = Person("Francois")
p1.say_hi("Iyan")
p1.get_memory_consumption()
p1.sleep()


# invoking say_hi
# hi Iyan I'm Francois
# my memory consumtpion is ...
# Francois is doing_something

That's pretty nice, right? As the class body ends up being executed as a function you can put any code in it, and the compiler will compile any declarations that you put in it as attributes in the namespace object (that then is passed to the __new__ and __init__ of the metaclass to create the class). But there's one limitation. What if I want to create several alias for a same function using a loop? in principle we can't.

class MyGeoManager:
    _not_implemented = lambda self, item:  print(f"Method is not implemented yet")
    for name in ["get_city", "get_country"]:
        # obviously this does not do what we would like
        name = _not_implemented 

Obviously the above is not doing what we want. It's just creating an attribute named "name" and assigning it in a loop. How could we add an attribute get_city and an attribute get_countr?

Well, we can leverage something we've seen in previous posts, our friend frame.f_locals. I mentioned it in my last post and talked about it in depthhere. f_locals gives us a write-through proxy to the locals of a function. When using an optimized scope, adding variables to that proxy has no particular useful effect, as the function has been compiled to acces by index to the variables that were found at compile time, but in the kind of scope used in a class body, that is a real dictionary, not a fast-array, adding new variables via f_locals adds them to the namespace dictionary that then is passed to __new__ and __init__. So we can do this:

class MyGeoManager: 
    local_namespace = inspect.currentframe().f_locals
    not_implemented = lambda self, item:  print(f"Method is not implemented yet")
    for name in ["get_city", "get_country"]:
        local_namespace[name] = not_implemented

print(f"get_country: {MyGeoManager.get_country}")
print(f"get_city: {MyGeoManager.get_city}")

It works like a charm!

There's another way to do this, derived from how class scope differs from optimized scopes. We are used to the builtin functions exec, compile and eval working with a snapshot of the locals namespace (because we are used to work in optimized scopes), but in a class scope, these functions receive the real namespace object. So we can leverage exec() like this:

class MyGeoManager:
    not_implemented = lambda self, item: print("Method is not implemented yet")
    for name in ["get_city", "get_country"]:
        exec(f"{name} = not_implemented")