Linux Dynamic Linking

A few months ago I wrote a post about glibc, it's a good time now to follow up with a related element of linux systems, ld-linux.so, the dynamic linker or dynamic loader.

If you need a refresh on what linking is, this wikipedia article should be enough. In this post I'll be talking about dynamic linking on Linux (performed by ld-linux.so), do not confuse it with the static linking performed by the ld tool (part of the binutils package, in turn part of the build-essential metapackage) to create an executable copying inside it the library code it references.

So dynamic linking means loading into a process memory at runtime the libraries (shared objects, .so's) that it needs. This loading is mainly done when the process starts (based on the imports defined in the ELF binary file), but can also be performed dynamically at any point during the process execution, with the dlopen function. In both cases, ld-linux.so (at least in Ubuntu its exact name is: /lib64/ld-linux-x86-64.so.2) is used to perform this loading and linking.

If ld-linux.so is used to load so's, and ld-linux is a so itself, this looks like a chicken-egg problem. Well, it's the kernel itself who takes care of loading it when a process is launched:

A binary names its loader in the ELF .interp section (readelf -p .interp /bin/ls → /lib64/ld-linux-x86-64.so.2). On execve the kernel reads .interp, hands control to that interpreter, which maps the libraries and jumps into the program.

Notice that the dynamic linker/loader is also known as the ELF interpreter.

When you run a dynamically linked program, the Linux kernel reads the executable's Executable and Linkable Format (ELF) header. It looks for a specific section called .interp (or the PT_INTERP program header), which contains the hardcoded string path to this exact interpreter. How the ELF Interpreter Works: Rather than running your program directly, the operating system kernel actually loads and hands control over to ld-linux first. The interpreter then performs several crucial tasks:
- Finds Dependencies: It scans your binary to see which shared libraries (such as libc.so) it needs.
- Loads Libraries: It locates those .so files on the disk and maps them into the program's memory space.
- Resolves Symbols: It performs "relocations," fixing memory references so your program knows exactly where library functions exist in memory.
- Launches Program: Once the environment is ready, it hands control back to your program's main entry point.

There's a strong relation between glibc and ld-linux. They are built together carrying the same version:

The loader (ld.so) and libc.so.6 are a version-locked matched pair. They are built from the same source tree in the same build and talk to each other over a private, unstable ABI: the GLIBC_PRIVATE symbols.
nm -D /lib64/ld-linux-x86-64.so.2 | grep GLIBC_PRIVATE # loader DEFINES them (T/D)nm -D /lib/x86_64-linux-gnu/libc.so.6 | grep GLIBC_PRIVATE # libc has them UNDEFINED (U)

libc.so.6 lists symbols like _dl_allocate_tls@GLIBC_PRIVATE, __tunable_get_val@GLIBC_PRIVATE, _dl_find_dso_for_object@GLIBC_PRIVATE as undefined — “the loader must hand these to me at startup.” The names, semantics, and the layouts behind them (TCB/TLS, _rtld_global) change freely between glibc versions.

ld-linux.so can be executed on its own, as a normal executable. If you run it with the --help version (/lib64/ld-linux-x86-64.so.2 --help) you get this interesting information.

You have invoked 'ld.so', the program interpreter for dynamically-linked ELF programs. Usually, the program interpreter is invoked automatically when a dynamically-linked executable is started.

You may invoke the program interpreter program directly from the command line to load and run an ELF executable file; this is like executing that file itself, but always uses the program interpreter you invoked, instead of the program interpreter specified in the executable file you run. Invoking the program interpreter directly provides access to additional diagnostics, and changing the dynamic linker behavior without setting environment variables (which would be inherited by subprocesses).

libc.so also happens to have an entry point, but it seems to have no other function that showing some information about itself: (/lib/x86_64-linux-gnu/libc.so.6 --version)

GNU C Library (Ubuntu GLIBC 2.39-0ubuntu8.8) stable release version 2.39.
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 13.3.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
Minimum supported kernel: 3.2.0

Invoking ld-linux.so with --version in the same system you'll see that it's the same version as glibc, /lib64/ld-linux-x86-64.so.2 --version

ld.so (Ubuntu GLIBC 2.39-0ubuntu8.8) stable release version 2.39.

Notice that in the glibc information you can see "Minimum supported kernel: 3.2.0". Bearing in mind that my current kernel is 6.8.0, so we can say that glibc is quite little demanding with regards to kernel evolution.

Probably the ldd tool resonates with you. It's just a bash script wrapper (/usr/bin/ldd) around ld-linux.so

Python Function Call

Over the years I've come to appreciate a real lot how coherent the Python object model is. For example, the callable concept applies to every object that can be invoked, starting by functions. When you define a function, you have an object that is an instance of the types.FunctionType class, a class that has a __call__ method. In contrast, Kotlin has the invokable concept, but this is an extra (the invoke operator) that you can add to your classes, but the basic invokable element, a method/function, is not an instance of a class with an invoke operator. Functions/methods in Kotlin (at the Kotlin language level, then at a JVM or JavaScript runtime things differ) are not objects, we have to get a function reference or a method reference (with the :: syntax) to treat them as objects.

So having in mind that the callable concept applies to normal functions and methods has made me to dive deeper into what I explained in a recent post:

At the C level, all Python objects are represented by the PyObject structure, which points to a PyTypeObject (its type). The type object defines how instances of that type behave.

To make calls fast and avoid creating temporary tuple/dict arguments, CPython uses the vectorcall protocol (Py_TPFLAGS_HAVE_VECTORCALL).
When the interpreter encounters a CALL opcode, it ultimately looks at the target object's type to find its vectorcall entry point:

- Pure Python Functions (PyFunction_Type): The vectorcall pointer points to _PyFunction_Vectorcall. This function extracts the function's PyCodeObject (func->func_code), allocates a new _PyInterpreterFrame on the evaluation stack, and pushes it to _PyEval_EvalFrameDefault.

- Built-in/Native Functions (PyCFunction_Type): Native functions (like math.sqrt or print) are wrapped in a PyCFunctionObject. Their vectorcall pointer points to _PyCFunction_Vectorcall. This function extracts the underlying C function pointer (func->m_ml->ml_meth) and invokes the compiled C code directly.

But, based on the "callable protocol", when the interpreter encounters a CALL opcode, it should check if the type of the object has a __call__ method and invoke it, right? How does this fit with the above? Well, we have to separate language semantics from implementation:

It's a classic example of language semantics vs. implementation details:

At the language level (Python semantics): Everything is an object, and every object that can be invoked implements the callable protocol via __call__. Defining __call__ on function and builtin_function_or_method keeps the object model clean, predictable, and fully inspectable.

At the runtime level (C implementation): The interpreter is allowed to use any shortcut it wants as long as it doesn't break the language semantics. Since invoking fn() is the most critical operation in the language, CPython uses type-checking fast paths (ob_type) to skip the overhead of attribute lookups.

Going deeper:

Looking up __call__ via the attribute resolution mechanism on every function call would carry massive performance overhead. Here is how CPython handles this under the hood and how it fits with the previous explanation. 1. Fast Paths Bypass __call__ Entirely

For pure Python functions (types.FunctionType) and standard C builtins (types.BuiltinFunctionType), CPython completely bypasses the __call__ method lookup.

Even though type(func).__call__ exists at the Python language level (defined on function / PyFunction_Type), the bytecode interpreter does not execute a dictionary lookup for __call__ when you write func().

Instead:

The interpreter checks Py_TYPE(callable).

If it matches &PyFunction_Type or &PyCFunction_Type, it executes the direct fast paths explained previously (allocating a _PyInterpreterFrame or invoking the vectorcall C function pointer directly).

As for custom callables (instances of classes with a __call__ method), the mechanism is also optimized for performance (it's not doing an expensive MRO __call__ search for any object). The structure representing and object's type, PyTypeObject, has 2 fields, tp_call and tp_vectorcall_offset serving different generations of CPython's calling API. tp_vectorcall_offset is used among others for bound methods and functools.partial. tp_call is used for classes that define a __call__ method (and their derived classes). In that case tp_call points to the generic C wrapper that performs the lookup for "__call__" in the object's class dictionary/MRO and invokes it (and so, if the object does not have a tp_call no search is performed at all).

If we have a function: def fn(), we can invoke it normally, via fn(), but also like this: fn.__call__(). The latter invokation form ends up creating the corresponding _PyInterpreterFrame and running fn's code_object in _PyEval_EvalFrameDefault(), but does it through quite a few extra steps. First we have the look up of the __call__ attribute in the fn object, which will find it in the Function class (types.FunctionType). Retrieving it will give us a bound method (types.MethodType), where __self__ is the fn function, and __func__ is the __call__ function. Then, we have the normal invokation of that bound method, that will end up executing the code object for the fn function.

AsyncLoop 2026

With "async loop" in this post I'm not talking about something like JavaScript's for-await or Python's async-for, where the iterator is asynchronous. What I'm talking about here is about performing an asynchronous operation in a loop (like doing a loop of http requests). Well, in the async/away world that's not something to be scared of, but in the past, when asynchronous code was based on callbacks (not even Promises) this was a bit more complicated. 15 years ago I published a couple of posts about that [1] and [2]. For whatever the reason recently my mind came to think a bit about that old stuff, and I've decided to do a sort of generic AsyncLoop class that could be used with callback based functions. It's intended for use as a sort of for-of loop (so for iterator based loops) and for normal for loops (with a specific logic for getting the next item and checking for stop).

The AsyncLoop class receives a function to obtain the next item in the iteration, a function to check if the iteration must finish and a function (actionFn) for the body of the loop. That function is the one that behaves asynchronously and expects a callback. The class provides to the action/body a callback that will take care of continuing (or stopping) with the next iteration.

class AsyncLoop {
    constructor(nextItemFn, conditionFn, actionFn, onEndFn) {
        this.nextItemFn = nextItemFn; // function that returns the next item
        this.conditionFn = conditionFn; // function that receives the current item 
        this.actionFn = actionFn; // function that receives the current item, a callback to invoke when done
        this.onEndFn = onEndFn; // function that is invoked when the loop ends
    }

    run() {
        let item = this.nextItemFn();
        if (this.conditionFn(item)) {
            this.actionFn(item, () => {
                    this.run();
                }
            );
        } 
        else {
            this.onEndFn();
        }
    }
}

We can use it like this:

// callback based asynchronous function
function getContent(item, callback) {
    setTimeout(() => {
        let content = "Content for: " + item;
        callback(content);
    }, 500);
}

function testAsyncLoop() {
    let countriesIter = ["France", "Germany", "Italy"].values();
    new AsyncLoop(
        () => countriesIter.next().value, // nextItemFn
        (item) => item !== undefined, // conditionFn
        // actionFn
        (item, nextFn) => {
            getContent(item, result => {
                console.log(result);
                nextFn();
            }); 
        },
        () => console.log("All items processed") // onEndFn
    ).run();
}

We can use it also in nested loops:

class Continent {
    constructor(name, countries) {
        this.name = name;
        this.countries = countries;
    }   
}

function testNestedAsyncLoop() {
    let continentsIter = [
        new Continent("Europe", ["France", "Germany", "Italy"]),
        new Continent("Asia", ["China", "Japan", "India"]),
        new Continent("Africa", ["Nigeria", "Egypt", "South Africa"])
    ].values();

    //nested loop
    new AsyncLoop(
        () => continentsIter.next().value, // nextItemFn
        (continent) => continent !== undefined, // conditionFn
        (continent, nextContinentFn) => {
            console.log("Processing continent: " + continent.name);
            let countriesIter = continent.countries.values();
            new AsyncLoop(
                () => countriesIter.next().value,
                (country) => country !== undefined,
                (country, nextCountryFn) => {
                    getContent(country, result => {
                        console.log(result);
                        nextCountryFn();
                    });
                },
                () => nextContinentFn() // onEndFn
            ).run();
        },
        () => console.log("All continents processed")
    ).run();
}

Async/await has always felt a bit like magic (all what the compiler does under the covers), but looking to the above code that we would have written in 2011 and comparing it to how we would write it now with async/await is like a quantum leap!

function getContentPromisified(item) {
    return new Promise((res, rej) => getContent(item, res));
}

async function testUsingAsyncAwait(){
    let continents = [
        new Continent("Europe", ["France", "Germany", "Italy"]),
        new Continent("Asia", ["China", "Japan", "India"]),
        new Continent("Africa", ["Nigeria", "Egypt", "South Africa"])
    ];
    for (let continent of continents) {
        console.log("Processing continent: " + continent.name);
        for (let country of continent.countries) {  
            console.log(await getContentPromisified(country));  
        }
    }
    console.log("All continents processed");
}

Python And Native Modules

When we think about running Python code, particularly in the CPython interpreter, we normally think about interpreted code (a bytecode interpreter), but we also know that we can write native modules (extension modules) normally in C or C++. but also using cython, nuitka or even rust. These extension modules are not used just by some third party libraries, but the python runtime/environment/standard library also makes good use of them. In this sense, as explained in this article we have 2 types of these modules:

• A built-in extension module is a module built and shipped with the Python interpreter. A built-in module is statically linked into the interpreter, thereby lacking a __file__ attribute.
• A shared (or dynamic) extension module is built as a shared library (.so or .dll file) and is dynamically linked into the interpreter. In particular, the module’s __file__ attribute contains the path to the .so or .dll file.

Normally the Python interpreter is contained in the python3.XX binary (.exe in Windows), that is like 30 MBs in size, but in some builds (those done with the --enable-shared flag), most of the code is put in a libpython3.14.so (.dll in Windows) dynamic libray, and the python binary just bootstraps it.

built-in extension modules (like sys, builtins, _thread, gc...) are obviously part of the Python distribution (they are inside the interpreter as we've seen), while for shared extension modules, we have those that are part of the python distribution (like math, array, _ssl, _socket) and those that are developed by third parties.

You can check all the built-in extension modules like this:

>>> sys.builtin_module_names
('_abc', '_ast', '_codecs', '_collections', '_contextvars', '_datetime', '_functools', '_imp', '_io', '_locale', '_opcode', '_operator', '_signal', '_sre', '_stat', '_string', '_suggestions', '_symtable', '_sysconfig', '_thread', '_tokenize', '_tracemalloc', '_types', '_typing', '_warnings', '_weakref', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time')

While for the shared extension modules that are part of your distribution, just check the lib-dynload folder in your installation, e.g.:

ls -la /usr/local/lib/python3.14/lib-dynload/
total 27032
drwxr-xr-x  2 root root    4096 mar 14 13:26 .
drwxr-xr-x 43 root root    4096 mar 14 13:26 ..
-rwxr-xr-x  1 root root  298704 mar 14 13:26 array.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root  399192 mar 14 13:26 _asyncio.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root  201048 mar 14 13:26 binascii.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root   97240 mar 14 13:26 _bisect.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root 1705384 mar 14 13:26 _blake2.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root  105048 mar 14 13:26 _bz2.cpython-314-x86_64-linux-gnu.so
-rwxr-xr-x  1 root root  166800 mar 14 13:26 cmath.cpython-314-x86_64-linux-gnu.so
...

So when running a Python application you have normal Python functions (that as first step of the execution have been compiled to Python bytecodes) that have to be interpreted, and other functions, living in those extension modules that are already native code and have to be executed as such, without interpretation. How does Python manage that?

The essential part in most interpreters is the interpreter loop. This is the code that loops through the bytecode instructions to be interpreted (or traverses the tree of nodes in Tree-parsing interpreters). I say "most" rather than "all" because in tree parsing interpreters we can have things like Truffle, where each node execute() method takes care of moving to the next node (and also manages its own specialization). In CPython, this interpreter loop lives in the _PyEval_EvalFrameDefault function. It's nicely explained here

When a Python function is invoked we have a Function object and a CALL bytecode instruction. If the function being called is not a native one (so a normal function that was written in pure Python and compiled to bytecodes to be interpreted) the interpreter handles this CALL by creating a frame object (the optimized _PyInterpreterFrame that I discussed here), that contains all the information needed for the function execution: local variables (including arguments and closure cells), the code object... and invokes _PyEval_EvalFrameDefault() with that frame. Indeed, there's a very interesting performance optimization added in Python3.11, _PyEval_EvalFrameDefault no longer recursively calls itself when it finds a new CALL, so we keep a flat C stack. From a GPT:

Ordinary Python→Python calls are frameless on the C stack. A chain of plain function calls runs inside a single _PyEval_EvalFrameDefault C invocation. The CALL opcode pushes a lightweight _PyInterpreterFrame onto a per thread data stack (chunked heap) and dispatches within the same C frame. Plain recursion is bounded by sys.getrecursionlimit() (a logical / data-stack counter), not the C stack.

And from this very in depth article

In CPython 3.10 and earlier, the CALL instruction used to create a new interpreter stackframe for the function being called and then it used to recursively reenter the interpreter by calling its entry point _PyEval_EvalFrameDefault.

This was bad for performance from many angles at the hardware level. The recursive call into the interpreter required saving the registers for the current function, and pushing a new C stackframe. It would lead to increased memory usage because each recursive interpreter call would allocate its own local variables on the stack, and other heap allocations. Apart from that it would also lead to poor instruction cache locality due to the constant jumps in and out of the bytecode evaluation loop.

In the 3.11 release this was fixed by eliminating the recursive call to the interpreter. Now the CALL instruction simply creates the stackframe for the called function, after that it immediately starts evaluating the new function’s bytecode without ever leaving the loop.

I've explained so far how the interpreter manages "normal" functions, but what about native functions? The Function object for a native function does not have a code object (as obviously they don't have associated bytecodes), but a function pointer to the native code. From a GPT:

Native functions: Stored as PyCFunctionObject with a function pointer
No __code__ attribute
Instead, has a function pointer (ml_meth) stored in PyMethodDef
CPython calls the C function pointer directly, bypassing the interpreter loop
The C function executes natively and returns a PyObject*

How does the interpreter check if this 'CALL function_object' should be treated one way or another (_PyInterpreterFrame + _PyEval_EvalFrameDefault() vs native call)? Basically at a low level different objects are used for a "normal" function and a native function. But the whole story goes like this (GPT explanation)

At the C level, all Python objects are represented by the PyObject structure, which points to a PyTypeObject (its type). The type object defines how instances of that type behave.

To make calls fast and avoid creating temporary tuple/dict arguments, CPython uses the vectorcall protocol (Py_TPFLAGS_HAVE_VECTORCALL).
When the interpreter encounters a CALL opcode, it ultimately looks at the target object's type to find its vectorcall entry point:

- Pure Python Functions (PyFunction_Type): The vectorcall pointer points to _PyFunction_Vectorcall. This function extracts the function's PyCodeObject (func->func_code), allocates a new _PyInterpreterFrame on the evaluation stack, and pushes it to _PyEval_EvalFrameDefault.

- Built-in/Native Functions (PyCFunction_Type): Native functions (like math.sqrt or print) are wrapped in a PyCFunctionObject. Their vectorcall pointer points to _PyCFunction_Vectorcall. This function extracts the underlying C function pointer (func->m_ml->ml_meth) and invokes the compiled C code directly.

But things are even more interesting. I already talked about how in Python3.11 the Python interpreter turned into a Python Adaptive Specializing Interpreter. Thanks to that, Function Calls are optimized like this (as explained by a GPT)

If the interpreter had to look up the type and vectorcall pointer from scratch on every single loop iteration, it would be slow. To solve this, CPython uses Opcodes Specialization (PEP 659 Adaptive Interpreter). When a generic CALL opcode executes, it starts in an "adaptive" state. It looks at the object being called and patches itself in memory to a specialized version based on what it sees:

- Python-to-Python Calls
If the target is a pure Python function, the CALL opcode specializes itself into CALL_PY_EXACT_ARGS (or CALL_PY_BOUND_METHOD).

The Check: It performs a strict pointer comparison on the target's type to ensure it is exactly PyFunction_Type.
The Action: It skips the generic lookup, directly grabs the code object, creates the _PyInterpreterFrame, and increments the frame depth.

- Python-to-Native Calls

If the target is a native C function, the CALL opcode specializes into CALL_BUILTIN_FAST or CALL_BUILTIN_CLASS.

The Check: It verifies that the object's type is PyCFunction_Type and often checks if the specific function handler matches what was cached.
The Action: It bypasses the interpreter frame creation entirely, sets up the C arguments, and jumps straight into the native C function.

That's fascinating!

If out of curiosity we want to know if a function is pure Python or native code we can check its type. For normal functions its type is FunctionType, for native functions it's BuiltinFunctionType

# Normal functions:
import types
import math

def f1(): pass
l1 = lambda x: x

isinstance(f1, types.FunctionType)
True
isinstance(l1, types.FunctionType)
True

isinstance(len, types.FunctionType)
False
isinstance(len, types.BuiltinFunctionType)
True
isinstance(math.sqrt, types.BuiltinFunctionType)

Additionally, as I've mentioned, native functions lack a __code__ attribute.

f1.__code__
code object f1 at 0x76be17bfcac0, file "", line 1

len.__code__
Traceback (most recent call last):
  File "", line 1, in 
    len.__code__
AttributeError: 'builtin_function_or_method' object has no attribute '__code__'. 

Python unpacking and multiple assignment (again)

When I talked about Python destructuring assignment I also mentioned the * and ** syntax, and referred to it as "operators" which indeed is not right (though it's common to name them like that). They should be better referred as unpacking/packing syntax or star expressions. For the different use cases of this syntax:

• function invokation: f1(*args, **kwargs) or function definition: def f2(*args, **kwargs) we should talk about arguments unpacking or packing
• When used in collection literals we should talk about iterator unpacking: [*items1, *items2] and dictionary unpacking: {**dict1, **dict2}, with the expressions being called "starred expressions"
• When used on the left side of an assignment: first, *reminder = my_list, we talk of a "starred target".

As we know the unpacking (destructuring) happens automatically when performing a multiple assignment, with no need of using * at all

x, y = [1, 2]
print(f"x: {x}, y: {y}")
# x: 1, y: 2

There are some advanced uses that I tend to forget. We can use multiple assignment with object attributes or dictionary keys:

@dataclass
class Person:
    name: str
    age: int
    country: str

# multiple assignment to attributes of an object
p1 = Person("Antoine", 47, "France")
p1.name, p1.age = ["Francois", 48]
print(f"p1.name: {p1.name}, p1.age: {p1.age}")

# multiple assignment to dictionary keys
d1 = {}
d1["name"], d1["age"] = ["Francois", 48]
print(f"d1['name']: {d1['name']}, d1['age']: {d1['age']}")

As I explained in this post we can unpack nested structures:

x, [a, b], y = [1, [2, 3], 4]
print(f"x: {x}, a: {a}, b: {b}, y: {y}")
# x: 1, a: 2, b: 3, y: 4

But notice that nested unpacking works for assignment, but not for function parameters. Surprisingly this is something that worked in Python2 but was lost in Python3.

>>> def f(a, (b, c)):
...     return c

SyntaxError: Function parameters cannot be parenthesized


>>> def f(a, [b, c]):
...     return c
...     
            
SyntaxError: invalid syntax

Python is missing the object destructuring assignment feature present JavaScript, I mean:

const user = {
  id: 42,
  isVerified: true,
};

const { id, isVerified } = user;

So we have to use this more verbose approach (notice that itemgetter and attrgetter are a nice option for dynamic scenarios)

p1 = Person("Antoine", 47, "France")

name, country = p1.name, p1.country
name, country = attrgetter("name", "country")(p1)

name, age = d1["name"], d1["age"]
name, age = itemgetter("name", "age")(d1)

Python Class Body to the Limit

In my 2 previous posts [1] and [2] we've seen how the class body of a Python class statement is just executable code that is put in a code object around which a synthetic function is created. This code is executed during the class creation (receiving a namespace object, a dictionary, created by the __prepare__ method of the metaclass). This is pretty powerful, as you can put complex initialization code there, not just a normal assignment). We can apply a decorator conditionally, create multiple function alias, define functions conditionally... Let's see some examples.

def log_deco(fn):
    def log_wrapper(*args, **kwargs):
        print(f"invoking {fn.__name__}")
        return fn(*args, **kwargs)
    return log_wrapper

def do_nothing_deco(fn):
    return fn

log_mode = True
profile_mode = Trumult

class Person:
    def __init__(self, name: str): 
        self.name = name

    # conditional decorator
    @(log_deco if log_mode else do_nothing_deco)
    def say_hi(self, to_x: str):
        print(f"hi {to_x} I'm {self.name}")

    # declare a function conditionally
    if profile_mode:
        def get_memory_consumption(self):
            print("my memory consumtpion is ...")

    def do_something(self):
        print(f"{self.name} is doing_something")

    # function aliases
    work = do_something
    cook = do_something
    sleep = do_something

p1 = Person("Francois")
p1.say_hi("Iyan")
p1.get_memory_consumption()
p1.sleep()


# invoking say_hi
# hi Iyan I'm Francois
# my memory consumtpion is ...
# Francois is doing_something

That's pretty nice, right? As the class body ends up being executed as a function you can put any code in it, and the compiler will compile any declarations that you put in it as attributes in the namespace object (that then is passed to the __new__ and __init__ of the metaclass to create the class). But there's one limitation. What if I want to create several alias for a same function using a loop? in principle we can't.

class MyGeoManager:
    _not_implemented = lambda self, item:  print(f"Method is not implemented yet")
    for name in ["get_city", "get_country"]:
        # obviously this does not do what we would like
        name = _not_implemented 

Obviously the above is not doing what we want. It's just creating an attribute named "name" and assigning it in a loop. How could we add an attribute get_city and an attribute get_countr?

Well, we can leverage something we've seen in previous posts, our friend frame.f_locals. I mentioned it in my last post and talked about it in depthhere. f_locals gives us a write-through proxy to the locals of a function. When using an optimized scope, adding variables to that proxy has no particular useful effect, as the function has been compiled to acces by index to the variables that were found at compile time, but in the kind of scope used in a class body, that is a real dictionary, not a fast-array, adding new variables via f_locals adds them to the namespace dictionary that then is passed to __new__ and __init__. So we can do this:

class MyGeoManager: 
    local_namespace = inspect.currentframe().f_locals
    not_implemented = lambda self, item:  print(f"Method is not implemented yet")
    for name in ["get_city", "get_country"]:
        local_namespace[name] = not_implemented

print(f"get_country: {MyGeoManager.get_country}")
print(f"get_city: {MyGeoManager.get_city}")

It works like a charm!

There's another way to do this, derived from how class scope differs from optimized scopes. We are used to the builtin functions exec, compile and eval working with a snapshot of the locals namespace (because we are used to work in optimized scopes), but in a class scope, these functions receive the real namespace object. So we can leverage exec() like this:

class MyGeoManager:
    not_implemented = lambda self, item: print("Method is not implemented yet")
    for name in ["get_city", "get_country"]:
        exec(f"{name} = not_implemented")

Python Class Body and Lexical Scope

In my previous post about class creation in Python I mentioned how a code object is created for the code in the class body, and then that code object is executed as a function receiving a namespace object (created by __prepare__) as its locals. The class body will add attributes to that namespace, and can use whatever is already present in that namespace (if __prepare__ has put something there). This has made me wonder if apart from that, the class body can have access to its enclosing scope. Just remember that in this post we saw that methods in a class have access to its enclosing scope (they close over variables defined outside the class).

So the answer is YES, and it's just the closures mechanism in action. Let's see an example:

def create_class(id: str):
    class MyClass:
        # the class initialiation (the class body) has access to "external" variables in the enclosing scope, such as "id"
        # the code in the class body is placed in a codeobject that will run in a function having trapped (closed over) the "id" free variable
        class_id = id
        
        print(f"Free variables: {inspect.currentframe().f_code.co_freevars}")
        # Free variables: ('id',)

        def __init__(self, value):
            self.value = value

        def display(self):
            print(f"MyClass value: {self.value}, Class ID: {self.class_id}")

    return MyClass

cl = create_class("123")
instance = cl("aa")
instance.display()

# Free variables: ('id',)
# locals: {'__module__': '__main__', '__qualname__': 'create_class..MyClass', '__firstlineno__': 7, 'class_id': '123'}
# MyClass value: aa, Class ID: 123

As you can see, that class body (my understanding is that Python will execute the code object corresponding to that class body by putting it in a "synthetic function") has access to the id variable in the outer scope and can assign it to one of its attributes. We can see that 'id' in the list of freevars for the code object of the class body (that we get accessing the current frame from the class body itself). However, I came across something that confused me. If I print locals() from the class body, I can't see 'id' there. That's very strange, if I do the same from a normal function, locals shows both "normal" variables and those that the function has trapped in its closure, but, as I've said, for the class body, 'id' is missing in locals:

def create_class(id: str):
    class MyClass:
        # the class initialiation (the class body) has access to "external" variables in the enclosing scope, such as "id"
        # the code in the class body is placed in a codeobject that will run in a function having trapped (closed over) the "id" free variable
        class_id = id
        
        print(f"Free variables: {inspect.currentframe().f_code.co_freevars}")
        # Free variables: ('id',)

        # notice that locals() does not show the id free var
        # that's because we are running in "class scope" and locals() just shows its namespace, not the closure cells. The closure is a separate object that holds references to the free variables, and it is not part of the local namespace of the class body. However, the class body can still access the free variable "id" through the closure.
        print(f"locals: {locals()}")
        # {'__module__': '__main__', '__qualname__': 'create_class..MyClass', '__firstlineno__': 4, 'class_id': '123'}

While for a normal function, it's well there:

def outer(id):
    def inner(value):
        # locals() shows "id" free variable because we are in a function scope (optimized scope)
        print(f"locals: {locals()}")
        # {'value': 'bb', 'id': '456'}
        print(f"Inner function value: {value}, Outer ID: {id}")
    return inner

inner_func = outer("456")
inner_func("bb")

So, why is that? At the time of the Python3.13 release I wrote a post about locals(), f_locals and the "local namespace" (this is related to PEP-667). Well, indeed what I mention on that post (based on other articles) about the "local namespace" as a sort of dictionary is not correct for normal functions in recent Python versions. In "normal" functions we are working in an Optimized Scope. In this optimized scope local variables are not placed in a dictionary and accessed by key, but in a fastarray, and accessed by index (you can see this using dis to check the bytecodes of a function). This locals fastarray, part of the _PyInterpreterFrame for each running function, contains both local variables (including function arguments and those local variables that are cells, cellvars, because they are trapped by inner functions in its closure) and variables trapped by the function itself in its closure:

In CPython's frame object, the `fastlocals` array is laid out as:

[regular locals] [cellvars] [freevars]

At the beginning of a function that has variables in its __closure__ the `COPY_FREE_VARS` bytecode instruction copies cell references from `__closure__` into the frame's fastlocals array for quick access!*
After `COPY_FREE_VARS` executes, all variables (normal locals, cellvars and freevars) are accessed from the fastlocals array during function execution.

By the way, regarding the aforementioned _PyInterpreterFrame, I'll leverage to copy here some GPT wisdom about frames in recent (Python 3.11 and above) Python versions

The _PyInterpreterFrame is an internal C struct introduced in Python 3.11 that represents a stack frame for execution, aiming to improve performance by reducing the overhead of allocating full Python PyFrameObject objects.

Purpose: It holds the execution state for code objects, including local variables, globals, builtins, and the instruction pointer (f_lasti).
Performance: Unlike older Python versions where every frame was a full heap-allocated PyFrameObject, _PyInterpreterFrame is designed to be lightweight and often lives on the C stack, reducing garbage collection pressure.

The traditional PyFrameObject still exists, but it has been relegated to a "shadow" role. It is now treated purely as a compatibility API wrapper.

Python only creates a PyFrameObject on demand when a tool or a piece of code explicitly asks to inspect the call stack. This process is often referred to as materializing a frame.

Class Statement vs Dynamic Class Creation

We know that along with the standard class statement, Python also allows us to create classes dynamically by calling type() (or another metaclass if our class has a metaclass other than type)

I already dedicated a rather thick post to type. Basically we use it for creating a new class like this: type(classname, superclasses, namespace). (the namespace is just a dictionary with the attributes).

So I was wondering if the compiler translates a class statement into a call to type(), and yes, more or less we can say so, but there are some extras. I've had a really insightful conversation with a GPT about this, and additionally I've found an excellent article that explains it in full detail

The steps that Python follows when it comes across with a class statement (class Foo(Base, metaclass=Meta): x = 1) are these (I'm taking it from a GPT discussion, it's basically the same that is explained in the linked article)

• Step 1 — Determine the metaclass. Python calls __build_class__ (a builtin), which inspects the bases and the explicit metaclass= kwarg to resolve which metaclass to use (with MRO-based metaclass conflict resolution).
• Step 2 — Prepare the namespace. The metaclass's __prepare__ classmethod is called: namespace = Meta.__prepare__('Foo', (Base,), **kwargs). This returns the dict (or dict-like object) that will serve as the class namespace. For type, this is just a plain dict. For enum.EnumMeta, for example, it returns a special _EnumDict.
• Step 3 — Execute the class body. The compiled code object for the class body is executed as a function, with the namespace from step 2 as its locals(). This is the key insight: it's essentially exec(body_code, globals(), namespace). After this, namespace contains {'x': 1, '__module__': ..., '__qualname__': ...}.
• Step 4 — Call the metaclass. Meta('Foo', (Base,), namespace) is called — which, for the default type, invokes type.__call__ → type.__new__ → type.__init__. This is where the actual class object is constructed.

How do the above steps look at the bytecode level? When the Python compiler (yes, in Python, where compilation is like a hidden step that happens the first time our code is run (or has changed), it's sometimes confusing to establish the difference between compilation time and execution time), comes across a class statement, it creates a code object for the code that we've placed inside that statement (the body of the class statement), along with code objects for each function (method) defined in that code, and creates a sequence of bytecode instructions that at runtime will make use of that code object (and many more things) to create a class object (yes, remember that classes are objects).

That sequence of bytecode instructions can vary slightly with Python versions (what I'll show below, that corresponds to python 3.14 is slightly different from what is shown in the aforementioned article), but the intent is the same.

class Person:
	pass

# translates into:

 0           RESUME                   0

  1           LOAD_BUILD_CLASS
              PUSH_NULL
              LOAD_CONST               0 (code object Person at 0x78d07576e730, file "class_creation.py", line 1)
              MAKE_FUNCTION
              LOAD_CONST               1 ('Person')
              CALL                     2
              STORE_NAME               0 (Person)
              LOAD_CONST               2 (None)
              RETURN_VALUE

So those seem like very few instructions for the complex 4 steps that I've just described!. Well, that's because all the magic happens in a builtin function __build_class, that is loaded by the LOAD_BUILD_CLASS bytecode instruction. The article makes a great job explaining these opcodes.

When we create a class dynamically using type() (or any other metaclass), we are directly at step 4, we are skipping the first 3 steps. Obviously it's us who choose the metaclass to use, and there's not class body to execute. And as we create ourselves the namespace object to pass to the metaclass the __prepare__ method that helps prepare that namespace is not executed. That's maybe the main difference then, that in the dynamic class creation the metaclass __prepare__ method does not intervene. That's interesting, cause indeed I was not familiar with that __prepare__ method (also referred as hook).

When talking about metaclasses I always think about __new__ and __init__ (and __call__ that intervenes when instances of a class created by the metaclass are created), I've talked about them in different posts, one of the most interesting being this, but was unfamiliar with __prepare__. We've seen that it allows us to prepare the namespace, OK, but when can we need that? Well, very rarely (this is particularly dark metaclass stuff). That can be material for another post, for now I'll just say that enum.EnumMeta makes use of it.

SQL, NULL, Unknown

Lately I've been revisiting the rather particular behaviour of NULL in SQL, and it has led me into a better understanding of how different SQL is from General Programming Languages

- Binary Logic vs Ternary Logic

General Programming Languages (Python, JavaScript, ruby, Java...) use binary logic (Boolean logic in particular, and indeed that's the only logic I was aware of). Conditions are either True or False.
SQL uses a ternary logic (kleene logic), where we have TRUE, FALSE, and UNKNOWN

- The meaning of Missing Data

Both in General Programming Languages and in SQL we use null (None in Python) to represent missing data. There are 2 reasons for missing data, either it does not apply to that object, or we don't know it. Let's say we have an instance of a ShopItem class. Its expirationDate attribute can be null either because this object is a Book, and books do not expire, or because the printed date on this beans can is blurry (or we've had no time to read it yet) and then we don't know it, it's unknown.

In general Programming Languages null is a value (that represents that there's nothing here, there is no value here, for whatever the reason, either because it does not apply or because we don't know it), and with binary logic comparing a value to to another value is either true or false. So "a" == null is false, and null == null is true.

In SQL we have a sort of mismatch. On one hand we have ternary logic with that additional UNKOWN concept, but on the other hand we still have a single value, NULL, to represent both that it does not apply or that we don't know it. So how should NULL behave in comparisons? SQL designers decided to treat NULL as a marker that represents that the value is unknown (so we can not express that the value does not apply).

Once we have understood that, the apparent odd behaviour of NULL in comparisons suddenly makes sense. Any comparison using the standard operators (=, !=, <, >, <>) involving a NULL value will return UNKNOWN, even NULL = NULL or NULL != NULL return UNKNOWN. The negation of UNKNOWN (NOT UNKNWON) is also UNKNOWN.

What is odd is what I've just said, that SQL lacks a way to indicate that the value does not apply. It seems one of the main influences in the design of SQL ended realizing this was a serious problem, but too late:

Codd actually realized this flaw later in his life and proposed that SQL should have two different kinds of NULLs: A-Values (Absence) and I-Values (Information Unknown). Sadly, by then, SQL was already set in stone.

Context Managers Part 1

Context Managers have existed in Python since version 2.5, while Assignment Expressions (walrus operator) were added in version 3.8. Somehow recently I came up to wondering if we can replace the "as" by a ":=" assignment. I mean, can we do this?:

with it := MyContextManager():
	# do whatever with it

rather than this:

with MyContextManager() as it:
	# do whatever with it

The answer is NO, or well, more accurately, sometimes yes, sometimes no, but you should always avoid it. To understand this we have to review what a Context Manager is and how they work. Notice that they're part of a broader concept: Automatic Resource Management that also includes Garbage Collecion and RAII (Resource Acquisition Is Initialization).

A Python Context Manager handles the setup and cleanup of resources in your programs. A context manager is any object that implements:

__enter__(self)
__exit__(self, exc_type, exc_value, traceback)

And it's used like this:

with EXPR as target:
    BODY

And now the important part. Conceptually, Python does something roughly like this (as explained by a GPT):

resource_manager = EXPR
resource = resource_manager.__enter__()
try:
    target = resource
    BODY
finally:
    manager.__exit__(...)

So the key point is: The object used to manage the context and the object bound after as do not have to be the same object. That is exactly why __enter__() is allowed to return anything.

Some Context Managers are implemented so that __enter__ returns the context manager itself, while others return a different object. Basically, in the first case the resource being managed and the Resource Manager (Context Manager) are the same object, in the second case they are different as the management responsability has been moved away from the resource itself, to a different object.

Another interesting topic. We know that in Python a single with block can include multiple context managers. I mean:

with open('a.txt', 'r') as fr, open('b.txt', 'w') as fw:
    do_something(fr, fw)

I was wondering if cases where the second context manager makes use of the first context manager, and that I think is more common to find written like this:

with ContextManager1("aaa") as ctx1:
	with ContextManager2(ctx1) as ctx2:
    		do_something(ctx1, ctx2)

Could be written with a single with (avoiding the additional nesting level):

with ContextManager1("aaa") as ctx1, ContextManager2(ctx1) as ctx2:
	do_something(ctx1, ctx2)

The answer is YES. The first

Python evaluates multiple context managers in a single with statement sequentially from left to right. The moment the first context manager is entered, its return value is bound to the as variable, making it immediately available for the next context manager on the same line.

Type Hints Notes 2026

Type hints are more and more prevalent in recent Python code. I'm still not too severe about them, but my level of strictness continues to grow over time. I've lately learnt a couple of things:

Variadic Parameters. When typing functions that have packed (variadic) parameters in its signature (*args, **kwargs), we put the type of the individual parameter, we don't have to type it as a collection or dictionary (save if each parameter is really a collection), I mean, for a pipe function we should do:

# this is RIGHT
def pipe(val: Any, *fns: Callable) -> Any:

# this is WRONG
def pipe(val: Any, *fns: list[Callable]) -> Any:

Tuples. In Python we use tuples for "groups" of a fixed number of elements, a pair, a trio... We express it in the signature like this: tuple[str, str] or tuple[int, str, str]... But how to express that a function returns (or receives) a "group" of an unknown number of elements? We also use tuples, combined with ellipsis (...), like this:

tuple[int, ...]         # any number of ints: (), (1,), (1, 2, 99), ...
tuple[int | str, ...]   # any number of elements, where each element can be an int or a str (), ("a", 1), ("a", "b", "c"), (1, 2, 1) ...
  
tuple[int, str, bool]   # exactly 3 elements: an int, a str, a bool
  

An important detail that I've learnt thanks to a typing issue. We know that in a Python try-except block, the except clause can manage multiple exception types, I mean: except RuntimeError, TypeError, NameError:. Those multiple exceptions are a tuple, not just any iterable. Let's see an example (the last line is what is WRONG):

# multiple exceptions
def multiple_exceptions(exceptions: tuple[type[Exception], ...]) -> None:
    try:
        raise ValueError("This is a ValueError")
    except exceptions as e:
        print(f"Caught an exception: {e}")

multiple_exceptions((ValueError, TypeError))  
# Caught an exception: ValueError

# important, this is WRONG, we have to pass a tuple, not just any collection
multiple_exceptions([ValueError, TypeError])
# TypeError: catching classes that do not inherit from BaseException is not allowed

Indeed, an equivalent function with a variadic signature feels more natural and idiomatic than the above (and furthermore prevents the confusion of passing over any collection rather than exactly a tuple):

# this variadic signature feels more natural
def multiple_exceptions2(*exceptions: type[Exception]) -> None:
    if not exceptions:
        raise ValueError("pass at least one exception type")
    try:
        raise ValueError("This is a ValueError")
    except exceptions as e:
        print(f"Caught an exception: {e}")

multiple_exceptions2(ValueError, TypeError) 

And notice also how I've added a guard against the empty-call case (except () is invalid at runtime).

Python partial and placeholders

When Python 3.14 was released I had already read about some of its main features (those that involve a PEP and that have been discussed in the Python discussion forums), like Lazy Annotations and Template Strings. When reading in depth recently the release notes I came across a small feature added to functools.partial (and partialmethod) that I find particularly useful:

functools:
Add the Placeholder sentinel. This may be used with the partial() or partialmethod() functions to reserve a place for positional arguments in the returned partial object. (Contributed by Dominykas Grigonis in gh-119127.)

Just a reminder of what partial function application is (don't confuse it with the related concept of curried functions):

In computer science, partial application (or partial function application) refers to the process of fixing a number of arguments of a function, producing another function of smaller arity.

Indeed I already talked about functools.partial some time ago

The "basic" approach to partial function application is that we can just fix (pre-fill) arguments from left to right. This is what we have also in JavaScript with function.prototype.bind (that binds as first argument the "this" value). As Python supports named arguments, functools.partial already supported fixing named arguments.

def format_geo_info(country, region, city, population):
    return f"{city}, {region.upper()} ({country}) - {population}"
    
bound_format = functools.partial(format_geo_info, "France")
print(bound_format("Occitanie", "Toulouse", 500_000))
# Toulouse, OCCITANIE (France) - 500000
print(bound_format("Occitanie", city="Toulouse", population=500_000))
# Toulouse, OCCITANIE (France) - 500000

What was not possible until this version was fixing some intermediate non-named argument, but this is possible since version 3.14 thanks to the Placehodler sentinel value:

format_french_city_with_unknown_population = partial(format_geo_info, "France", Placeholder, Placeholder, 0)
print(format_french_city_with_unknown_population("Ile de France", "Saint Denis"))
# Saint Denis, ILE DE FRANCE (France) - 0

Not a revolutionary feature, but one that I've missed occasionally. A trivial implementation could be something like this:

# supports positional and keyword arguments, but not placeholders
def my_basic_partial(func, *args, **kwargs):
    return lambda *fargs, **fkwargs: func(*args, *fargs, **(kwargs | fkwargs))
    
# add support for placeholders in the arguments
PLACEHOLDER = object()
def my_complete_partial(func, *args, **kwargs):
    def new_func(*fargs, **fkwargs):
        merged_args = []
        fargs_iter = iter(fargs)
        for arg in args:
            if arg is PLACEHOLDER:
                merged_args.append(next(fargs_iter))
            else:
                merged_args.append(arg)
        merged_args.extend(fargs_iter)
        return func(*merged_args, **(kwargs | fkwargs))
    return new_func

format_french_city_with_unknown_population = my_complete_partial(format_geo_info, "France", PLACEHOLDER, PLACEHOLDER, 0)
print(format_french_city_with_unknown_population("Ile de France", "Saint Denis"))
# Saint Denis, ILE DE FRANCE (France) - 0

format_2 = my_complete_partial(format_geo_info, "France", city="Toulouse")
print(format_2("Occitanie", population=500_000))
# Toulouse, OCCITANIE (France) - 500000

In my aforementioned previous post about partial in Python I gave some reasons for using partial over directly trapping the variables with a closure (of course internally partial has to use either closures or a callable class). I've just realised that I was missing the main reason, partial is more semantic.

- Intent-Revealing Code: partial(func, arg) explicitly states your intent to partially apply arguments, improving readability and self-documentation. - Declarative Style: It focuses on the result (a new specialized function) rather than the imperative mechanics of capturing lexical scope.

Lodash, the excellent JavaScript library, also features placeholders in its implemention of partial.

Glibc

Compiling to native code, and furthermore for a Linux system... wow, sounds scary, and very, very far away from what I've been doing in the last decade(s). Well, the thing is that my employer decided some months ago that we had to compile to native code some of our Python applications. It's not something performance related, it's for preventing access to the source code of these applications. We were looking into cython, but we settled on Nuitka, an amazing piece of software that has been serving us so well.

Normally almost every native application compiled for a Linux system has been dynamically linked against glibc. OK, and, what's glibc?

The GNU C Library, commonly known as glibc, is the GNU Project implementation of the C standard library. It provides a wrapper around the system calls of the Linux kernel and other kernels for application use. Despite its name, it now also directly supports C++ (and, indirectly, other programming languages).

So when a Linux native application (using glibc) starts, the dynamic linker (libdl.so) will dynamically load the shared objects (SO, .so files, the equivalent to windows DLL's) needed by the application (like glibc.so) and link the callsites to the functions imported from those libraries.

Obviously glibc evolves over time, so, what about versions? First, what glibc version is installed on my system? You can check the SO's loaded by a running process by doing: lsof -p PID | grep .so. Normally you'll see that it's using: libc.so.6 (in Ubuntu it located here: /usr/lib/x86_64-linux-gnu/libc.so.6). That 6 is not the version number (libc.so.6 is the name for the library since 1997!), the version number is something like 2.XX (2.39 in my ubuntu 24.04). You find it by using: ldd --version

So, what happens if I compile my application in a system with one version of glibc and try to run it in a system with a different version? Well, the situation is quite more fine-grained that I thought. Version numbers are not checked at the glibc level, but at the function level. This is so because glibc uses symbolic versioning

The "Symbol Versioning" Approach (Advanced)

This is what glibc uses. It is also used by heavy-hitters like OpenSSL, Qt, and libgcc.

    The Logic: The filename stays the same (e.g., libc.so.6 has been the name since 1997), but individual functions inside the file are tagged with versions.

    The Result: Multiple versions of the same function can coexist in one file. This allows for extreme backward compatibility without breaking the system every time a single function is updated.
    
    The primary goal of symbol versioning is backward compatibility. It allows a single library file to provide multiple versions of the same function so that:

    Old binaries compiled against v2.10 continue to use the v2.10 implementation.

    New binaries compiled against v2.11 use the new v2.11 implementation.

So multiple versions of the same function live inside glibc, and your binary will dynamically link against the one it was compiled for. And, when does the version number of a function change? Normally it only changes if the function interface (the contract, the ABI) changes, but not if its only its internal implementation that changes. So if we compare symbolic versioning to semantic versioning (SemVer, a more familiar versioning schema), we could say that in symbolic versioning a version change corresponds to a Major version in semantic versioning.

You are exactly right: A new Symbol Version is functionally equivalent to a Major Version bump for that specific function. It signals to the linker that the "Contract" for that specific symbol has changed, and old programs should look for the previous contract elsewhere in the same file.

Notice how symbolic versioning is used for functions inside a library, while semantic versioning (when used) is normally used for libraries.

The glibc version (that one obtained with ldd --version) has no importance in terms of loading the library in memory (the dynamic linker will load libc.so.6 regardless of its "internal" version), the important part is the specific version of each function that we try to link.

I guess when you program in C you are aware of the version of each function that you are using, as you have to adapt your code to the ABI of the function if it has changed, but when that happens behind the scenes, that's quite different. In our case, we just write Python code, and the beautiful Nuitka takes care of transforming it to C and then compiling it to native. So it's Nuitka who takes care of writing the C code in accordance to the function versions inside the glibc in the system. So if then you run that binary in a system with an older glibc version it could happen that your binary is "pointing" to a function with a symbolic version (let's say openEncryptedFile@GLIBC_2.12) higher than the one in the older glibc (let's say openEncryptedFile@GLIBC_2.10) present in the current system, and your application will crash. Basically this means that you have to compile your Python application in a system with a glibc version <= that the glibc version in the target system. It feels odd at first, as the starting point is just the same Python code, and if in one system it can just use openEncryptedFile@GLIBC_2.10 why doesn't it compile it always with that 2.10 even if a bigger version (openEncryptedFile@GLIBC_2.12) is present? Well, that's how things work by default, when compiling, code will be linked to the highest version of that function present in the glibc in the compilation machine.

If you wonder if other .so libraries (SO, ELF libraries) also use symbolic versioning, it depends. For smaller, simpler libraries what is usually used is the SONAME approach, the library (.so file) name changes with each version (this is a coarse grained approach).

Symbolic versioning is the technically superior approach, but it is not the universal standard for all ELF libraries. It depends entirely on the library maintainers and their commitment to long-term ABI stability.

In the Linux ecosystem, there are two primary ways to manage library changes:  
1. The "SONAME" Approach (Common)

Most smaller or simpler libraries use the SONAME mechanism. 
You’ve likely seen files like libfoo.so.1 and libfoo.so.2.

    The Logic: If the developers change the interface, they increment the "Major" version number in the filename itself.  

    The Result: Programs linked against libfoo.so.1 will refuse to start 
    if only libfoo.so.2 is present. This is a "heavy-handed" fix because 
    it requires recompiling every program that uses the library even if 
    the specific function they use didn't actually change.

2. The "Symbol Versioning" Approach (Advanced)

This is what glibc uses. It is also used by heavy-hitters like OpenSSL, Qt, and libgcc.

    The Logic: The filename stays the same (e.g., libc.so.6 has been the name since 1997), 
    but individual functions inside the file are tagged with versions.

    The Result: Multiple versions of the same function can coexist in one file. 
    This allows for extreme backward compatibility without breaking the system every time a single function is updated.

To complete this post, I'll add some useful, related commands:

• To check the SO's used by a given program.
  For a binary on disk: ldd /usr/bin/program_name
  For a running process: lsof -p [PID] | grep '\.so'
• To view the symbols used by a program (the specific functions imported from SO's)
  All imported symbols: nm -Du
  Symbols + Versions: objdump -T | grep '*UND*'
  Only glibc symbols: objdump -T | grep 'GLIBC_'
  Library Version Map: readelf -V
  
• To view the symbols/functions exported by glibc in your system: objdump -T /usr/lib/x86_64-linux-gnu/libc.so.6

Some additional findings related to the last command. For example I want to see the versions of pthread_spin_init present in my glibc: objdump -T /usr/lib/x86_64-linux-gnu/libc.so.6 | grep pthread_spin_init

That gives me:

0000000000a4130 g DF .text 000000000000000d GLIBC_2.34 pthread_spin_init 00000000000a4130 g DF .text 000000000000000d (GLIBC_2.2.5) pthread_spin_ini

Which is very interestinng as it shows us that a symbol version is not a sequential counter for that specific function. Instead, it is a timestamp or a marker of the glibc release that defined that specific version of the function's ABI. From a GPT:

How glibc handles ABI changes with symbol versioning

Original version: Suppose foo() was introduced in GLIBC_2.2.5. That version is tagged as foo@GLIBC_2.2.5.

ABI change in glibc 2.32: If glibc developers change the ABI of foo() in version 2.32 (e.g., change its behavior, arguments, or return type in a way that breaks compatibility), they will:

Keep the old implementation as foo@GLIBC_2.2.5.
Add a new implementation as foo@GLIBC_2.32.

At runtime:

A binary linked against glibc 2.2.5 will request foo@GLIBC_2.2.5, and the dynamic linker will resolve it to the old implementation.
A binary linked against glibc 2.32 will request foo@GLIBC_2.32, and get the new implementation.

This mechanism ensures backward compatibility while allowing glibc to evolve.

Python Sentinel Values

Every now and then we need a flag or sentinel value. A unique value that we can distinguish from the normal values that we are processing and that has a particular meaning. a special, unique value used in programming to signal the end of data processing, a loop, or an operation. For example in my recent post about adding null-safety to a pipe function I was using 2 sentinels/flags: NULL_SAFE and COALESCE.

The essential function a sentinel value has to accomplish is to have a unique identity, so that comparing it by identity (Python: is, JavaScript: ===) with any other value/object in our system has to return false. So the most simple approach is just using a new object for each of our sentinels.

NO_INVEST = object()  # Sentinel value

def invest(amount: int | None | object) -> str:
	if amount is NO_INVEST:
		return "No investment"
	else:
		amount = amount or 0
		return f"we've invested {amount}"

print(invest(NO_INVEST))  # Output: No investment
print(NO_INVEST)  # Output: object object at 0x...

That simple approach works fine, but it's missing a few things. Printing the value is messy (we get an "object object at 0x..." representation, it would be nice to get NO_INVEST) and its rather typing unfriendly. Saying that invest can receive an object, apart from int or None lacks any meaning. What kind of object is that?. A sentinel value should (mainly) have these features:

• A unique identity (is comparison)
• A meaningful repr (debuggability)
• Clear typing (especially for static type checkers)

Our basic sentinel only provides the first one (identity). That's why some smart guy came up with a very interesting PEP 661 proposing a new Sentinel class. Unfortunately that PEP is in deferred status. The document also presents different techniques commonly used for Sentinel values, like the simple object() that I've just shown, using an enum or using a class. Using a class is the best approach to me, I'll show several iterations until getting what I think is the best we can get so far.

Approach 1. Classes are objects in Python. We can use a class for each sentinel object, and in order to get a nice representation we can give it a metaclass with a custom __repr__. The missing piece is having some typing friendliness, we're still stuck with the Any signature. Additionally, declaring a class for something that is not intended to work as an object factory, but to be used as an object in itself is rather unnatural.

    class SentinelMeta(type):
        def __repr__(cls):
            return cls.__name__
        
    class Sentinel(metaclass=SentinelMeta):
        pass
        
    class NO_INVEST(Sentinel): pass


    #def invest(value: int | None | Type[NO_INVEST]) -> str: # this signature feels a bit strange, but it works
    #def invest(value: int | None |Literal[NO_INVEST]) -> str: # this one feels better, but only works with mypy, not with pylance
    def invest(amount: int | None | Any) -> str:
        if amount is NO_INVEST:
            return "No investment"
        else:
            amount = amount or 0
            return f"we've invested {amount}"

    print(invest(NO_INVEST))  # Output: Using NO_INVEST sentinel in match statement
    print(NO_INVEST)  # Output: NO_INVEST

Approach 2. We can make the usage quite more natural by hiding the class creation behind a function. That also allows us to skip the Sentinel base class.

    class SentinelMeta(type):
        def __repr__(cls):
            return cls.__name__

    def sentinel(name: str):
        return SentinelMeta(name, (), {})
    
    NO_INVEST = sentinel("NO_INVEST")


    #def invest(value: int | Type[NO_INVEST]) -> str: # pylance doesn't like it, using a variable as type
    #def invest(value: int | Sentinel) -> str: # we don't have a Sentinel class... so forget it
    def invest(amount: int | None | Any) -> str:
        if amount is NO_INVEST:
            return "No investment"
        else:
            amount = amount or 0
            return f"we've invested {amount}"
        
    print(invest(NO_INVEST))  # Output: Using NO_INVEST sentinel
    print(NO_INVEST)  # Output: NO_INVEST

This one feels quite natural to use, but we still have the problem with typing. We can leverage the Generic types/class subscripting that we saw in my previous post for getting something like this (Approach 3)

    class SentinelMeta(type):
        def __repr__(cls):
            return cls.__name__
    
    class Sentinel(metaclass=SentinelMeta):      
        def __class_getitem__(cls, item):
                return cls
    
    def sentinel(name: str):
        return SentinelMeta(name, (Sentinel,), {})
    
    NO_INVEST = sentinel("NO_INVEST")


    #def invest(value: int | Sentinel) -> str:
    def invest(amount: int | None | Sentinel[NO_INVEST]) -> str: # at a typing level it's equivalent to the above, but it's provides extra meaining      
        if amount is NO_INVEST:
            return "No investment"
        else:
            amount = amount or 0
            return f"we've invested {amount}"

    print(invest(NO_INVEST))  # Output: Using NO_INVEST sentinel      
    print(NO_INVEST)  # Output: NO_INVEST
	

Hey, this one feels rather good to me. The Sentinel[NO_INVEST] looks pretty nice in that signature. The type checking is mainly off, cause our sentinel() function is not typed, so it's considered as returning Any, and when we pass Any to a function it disables type checking. This means that for the type system any sentinel that we create with sentinel() is just an Any, so the type checker will allow passing any sentinel to a function that expects a specific sentinel. It's not a problem for me, what I'm mainly interested in is the semantics, the clarity, that this type annotation provides to the function signature.

Given that we are using Sentinel classes as objects, not as object factories, we can make these classes to be more object like, by preventing instantiation. Additionally, Sentinels do not have attributes and are not intended to be expanded with attributes, so we can prevent them from getting attributes added dynamically. All in all we get:

    class SentinelMeta(type):
        def __repr__(cls):
            return cls.__name__
        
        # prevent sentinel classes from being instantiated
        def __call__(cls, *args, **kwargs):
            raise TypeError(f"{cls.__name__} is a sentinel and cannot be instantiated") 
        
        # prevent sentinel classes from being modified
        def __setattr__(cls, name, value):
            raise AttributeError(f"Cannot modify sentinel {cls.__name__}")


    class Sentinel(metaclass=SentinelMeta):      
        def __class_getitem__(cls, item):
                return cls

    
    def sentinel(name: str):
        return SentinelMeta(name, (Sentinel,), {})

    
    NO_INVEST = sentinel("NO_INVEST")

There's something missing in this implementation, support for our sentinels to be pickled (particularly important if we plan to use them in multiprocessing scenarios). That complicates the design and I've deliberately left it aside for the moment. Maybe we'll see it in another post.

Python Type Checkers and Type Expressions

In this previous post I explained how Python allows the usage of any object, not just type objects, for its annotations "Annotations can be any valid Python expression". Annotations are used to provide metadata, and while normally that metadata is just typing information, we can provide any sort of metadata for custom use at runtime (and as we saw we have the Annotated mechanism to combine both typing and custom metadata. While doing some tests at that time I noticed that VS Code (the Pylance extension) would warn (with: "Call expression not allowed in type expression" or "Variable not allowed in type expression") against using annotations like this (that as I've said is perfectly valid):

# metadata for parameters
@dataclass
class ValueRange:
    lo: int
    hi: int

# pylance warning: Call expression not allowed in type expression
def create_post_1(
    title: ValueRange(5, 20), 
    content: ValueRange(5, 100),
) -> dict:
    return {"title": title, "content": content}


val_range = ValueRange(1, 10)
# pylance warning: Variable not allowed in type expression
def fn2(a: val_range) -> None:
    pass
    
# but it's stored OK in the function annotations 
print(annotationlib.get_annotations(fn2))
# {'a': ValueRange(lo=1, hi=10), 'return': None}

So if this works fine at runtime (you can see that the annotation is stored along with the function) why pylance warns against it? Because we have to differentiate what is valid for runtime use vs what is valid at type checking time. From a GPT:

The runtime accepts arbitrary expressions. Static type checkers do not. This is so because static typing tools parse Python, but they don’t execute it, and they don’t compile it to bytecode either. Type checkers rely on syntactic patterns, not runtime behavior

This is something I had never thought about before. Python type checkers analyze your code without executing anything (they are static). So the types in the type annotations that they are going to analyze have to be expressed in a direct, static form, not as the result of executing an expression (as they are not going to execute that expression). The different Python type checkers (mypy, Pylance, pyre...) parse python source code into an AST (normally different from CPython AST) and analyze it, they do not run code, indeed they do not even compile the code to bytecodes to create code objects. That's why they can only work with type expressions (annotation expressions), that follow a specific syntax, not with any expression. From here: Note that while annotation expressions are the only expressions valid as type annotations in the type system, the Python language itself makes no such restriction: any expression is allowed.

Type checkers operate on expressions that syntactically denote types, which basically is a type name or a generic type. And this has sparked my curiosity about how generic types (MyClass[T]) work. For the type checker it's simple, as it does not execute anything, it just has to parse that particular syntax. But what's the runtime meaning of such generic expression?

Well, it's subscripted access to an object (to a class). When we do my_instance["x"] this searches for a __getitem__ method in my_instance's type. So MyClass["x"] should just search __getitem__ in MyClass's type (that is, its metaclass). That's correct, but given that Python designers have always considered metaclasses like particularly complex and/or exotic, they decided to introduce (via PEP560) a hook to make easier to implement subscripted access to classes. Rather than having to define a metaclass for MyClass, we can directly define a __class_getitem__ method in MyClass.

Pipe Operator and Null Safety

I've talked a couple of times [1] and [2] about how beautiful it's having a pipe operator in a language, though it's not particular common, and ways to simulate it in Python. Having a pipe operator makes applying functions to a value as convenient as chaining methods. When chaining methods we can leverage (if available) the safe navigation/optional chaining/elvis (?.) operator, to deal with null values. So, I've been thinking about null safety and pipes (not applying a function if the value is null, and coalescing to a default value).

In my previous post I mentioned that JavaScript had 2 different proposals for a pipe operator, but one of them has been discarded. I've been checking if this proposal includes null safety and the answer is not. It was discussed in the early stages, apart from the normal |> operator, having an additional ?|> operator for null safe cases, but it was discarded

// not null-safe, active proposal
user
  |> getProfile(%)
  |> formatProfile(%)
  
// null-safe, has been discarded
value ?|> fn
value |> fn ?? default

It was rejected on the basis that Pipelines should be pure syntax for data flow, not control flow.

To my surprise (I was not aware php continues to be used and evolve) PHP has recently added a pipe operator to the language, and for the moment it also lacks a null-safe version.

For Python decision makers adding a pipe operator seems "making the language too complex for beginners"... (you can't imagine how much I hate that so common kind of "pythonic" reflections...), but as I explained in my previous post we can easily add a pipe function that makes the trick (what has also been requested multiple times is adding such kind of function to functools, but no luck so far). An implementation is so simple as this:

def pipe(val: Any, *fns: Callable[[Any], Any]) -> Any:
    """
    pipes function calls over an initial value
    """
    def _call(val, fn):
        return fn(val)
    return functools.reduce(_call, fns, val)

And we can use it like this:

@dataclass
class Post:
    id: str
    title: str
    author: str

def get_post(post_id: str) -> Post | None:
    # simulate a function that may return None
    if post_id == "1":
        return Post(id="1", title="First post", author="1")
    else:
        return None

def get_address(person_id: str) -> str | None:
    # simulate a function that may return None
    if person_id == "1":
        return "Rue de La Nation, Paris"
    else:
        return None

pipe("1",
    get_post,
    lambda post: get_address(post.author),
	str.upper,
    print,
)	

# RUE DE LA NATION, PARIS

Creating a null aware equivalent is quite simple. The idea I came up with is having pipe accept not just a sequence of callables, but a sequence of callables or flag and callable or flag and value, with the flag indicating the we have to check for null before applying the Callable, or that we have to coalesce it to a value. Let's see the code:

// sentinel values
NULL_SAFE = object()
COALESCE = object()

def pipe(val: Any, *steps: Callable[[Any], Any] | tuple[Any, Callable[[Any], Any] | Any]) -> Any:
    """
    pipes function calls over an initial value, with support for null safety and coalescing:
    """
    def _call(val, step: Callable[[Any], Any] | tuple[Any, Callable[[Any], Any] | Any]) -> Any:
        if callable(step):
            return step(val)
        else:
            option = step[0]
            if option is NULL_SAFE:
                fn = step[1]
                return None if val is None else fn(val)
                
            elif option is COALESCE:
                default_val = step[1]
                return default_val if val is None else val
            else:
                raise ValueError(f"Invalid option: {option}")
    
    return functools.reduce(_call, steps, val)

pipe2("2",
    (NULL_SAFE, get_post),
    (NULL_SAFE, lambda post: get_address(post.author)),
    (COALESCE, "Not found"),
    str.upper,
    print,
)

# NOT FOUND

The function is quite minimal. We should add it proper error handing, throwing meaningful exceptions for each potential incorrect usage. You can just ask a GPT to add it and you'll end up with something like this:

def pipe(val: Any, *steps: Union[Callable[[Any], Any], Tuple[object, Any]]) -> Any:
    """
    Pipe value through callables or option-tuples.
    Steps can be:
      - a callable: called as fn(acc)
      - null_safe(fn): tuple (NULL_SAFE, fn) — only call fn if acc is not None
      - coalesce(default): tuple (COALESCE, default) — replace None with default

    Raises TypeError or ValueError for invalid steps.
    """
    def _call(val: Any, step: Union[Callable[[Any], Any], Tuple[object, Any]]) -> Any:
        if callable(step):
            return step(val)

        if not (isinstance(step, tuple) and len(step) == 2):
            raise TypeError("pipe2 steps must be callables or 2-tuples from null_safe/coalesce")

        option, payload = step
        if option is NULL_SAFE:
            if val is None:
                return None
            if not callable(payload):
                raise TypeError("NULL_SAFE payload must be callable")
            return payload(val)

        if option is COALESCE:
            default = payload
            return default if val is None else val

        raise ValueError(f"Unknown pipe2 option: {option!r}")

    return functools.reduce(_call, steps, val)