Python, functions, codeobjects and closures

In this post I mentioned that when a Python function traps variables from an outer scope (free vars), that is, when the function is a closure, it gets added a __closure__ attribute that points to a tuple that contains a cellvar object for each trapped variable. Each cellvar object is a wrapper for the trapped value, adding one level of indirection. That's how multiple closures that have trapped the same value can share it, and modifications done by one of them is seen by all others, including the original funcion where the value was defined. That's important, because it means that once a value is trapped by a closure (and wrapped in a cellvar object), the function where the value was defined no longer access to it directly, but also through the cellvar object. I've been investigating a bit how the compiler and the runtime manage all this via code objects, __closure__, co_freevars, co_cellvars... and here it goes:

In dynamic, interpreted languages the border between compilation (if any) and execution can feel a bit blurry. In Python when the runtime is executing a file, every time it finds a class declaration or a function declaration it creates an object for it. If that declaration happens inside a loop (we can put a class or function declaration wherever we want) a new object for that class gets created in each iteration. OK, that's for objects, but what about their code? On one side we know that CPython compiles each python module to bytecodes before running it. On the other hand we probably know that a function object has a __code__ attribute that points to its code object. A code object has a bunch of attributes with information about the code, and the code itself (an array of bytes referenced by the co_code attribute). This article contains tons of information about codeobjects.

When I said that compilation and execution are a bit blurred it's because both of them happen at runtime. In a first stage code objects are created when the runtime imports a module and compiles it to bytecodes before executing it (those bytecodes will be saved to the __pycache__ for future reuse). Code objects are created not only for functions but also for modules (containing the top-level code in the module) and for classes (for the class-level code, that is, the code that defines class fields and member functions, that in turn will have their own code objects). Once this compilation-codeobject creation is done, we enter the seconde stage, that is execution proper. Execution starts by running the code in the codeobject for a module, running this code will find class declarations and function declarations. Each of these declarations create objects for those classes and functions (and for member functions contained in the classes). It's interesting that neither modules nor classes have a __code__ attribute. I guess this is so because the code in a module is executed only once (when it's imported), and the code in a class (class-level code) is executed only each time the class is defined, while for functions, when the function is defined its codeobject is not executed, but attached to the function object created at that time, so that it can be executed later each time the function object is invoked.

Let's dive deeper now into this relation between function objects, codeobjects and closures. It's all better explained here but I had managed to understand the process myself, so I'll write it here. When compiling a function a codeobject is created for that function and also for each other function nested inside it. When compiling the outer function (let's say outer_fn), it takes note of variable declarations and when an internal function is found (let's say inner_fn), the function is in turn compiled. The compiler takes note of variable declarations in inner_fn, and those variables used in inner_fn but not declared in it (nor received as parameters) are candidates to be free-vars. It checks the outer functions to see if these variables are declared there, and if so, they are confirmed as free-vars and set in the inner_fn.__code__.co_freevars tuple. Furthermore, these variables will be set in the co_cellvars of the outer function where they are defined (outer_fn.__code__.co_cellvars in this case). So at compilation time information about the variable names ("normal", freevar, cellvar) is stored in the codeobjects so that at execution time, when functions are created, they can assist the runtime in creating closures. What is very, very important, is that the compiler uses different bytecode instructions for accessing variables defined as co_cellvars or co_freevars than for accessing "normal" variables. The values for those variables are wrapped in cellvar objects, and as accessing its values invokes an extra indirection, specific bytecode instructions are emitted (this is so both for the outer function where the variable has been defined and for the inner function that traps it in its closure), instructions like LOAD_DEREF and STORE_DEREF.

At execution time, when a function declaration is found a function object is created, with its __code__ attribute pointing to its codeobject. In outer_fn as its codeobject has co_cellvars the corresponding variables are wrapped in cellvar objects, and when the inner_fn declaration is found, as its codeobject has co_freevars, a __closure__ attribute is set in the inner_fn object, with references to the cellvar objects that we created in outer_fn.

I think all this partially explains why code blocks executed by the compile()/exec() functions can not create closures. Let's see an example:

def test_closure_1():
    fn_st = "\n".join([
    "def format(msg):", 
    "    print(f'{dec}msg{dec}')"
    ])

    # this does not work. The function that I define inside exec can not find the "dec" variable
    d = {}
    dec = "||"
    exec(fn_st + "\n" + "d['fn'] = " + "format")
    format = d["fn"]
    format("a")

test_closure_1()
# name 'dec' is not defined

When test_closure_1 is compiled it can not see that sometime in the future we'll dynamically create a nested function, because that nested function is defined in a string and will be compiled in the future when test_closure_1 is executed, not now while test_closure_1 is being compiled. This means that it can not know that "format" would want to have "dec" as a freevar, so "dec" in test_closure_1 will be compiled as a normal variable, not as a cellvar. Because of that, format can not trap it as a freevar, cause it would be working on a cellvar while test_closure is working on a normal variable.

If the variable to be trapped were declared in the block itself, it would be possible for the compiler when invoked by exec() to treat it as a cellvar and trap it, but for whatever the reason Python does not support that, as we can see in this code:

def test_closure_2():
    fn_st = "\n".join([
    "dec = '||'",
    "def format(msg):", 
    "    print(f'{dec}msg{dec}')"
    ])

    # this does not work. The function that I define inside exec can not find the "dec" variable
    d = {}
    exec(fn_st + "\n" + "d['fn'] = " + "format")
    format = d["fn"]
    format("a")

test_closure_2()
# name 'dec' is not defined

There's another case that could be supported, but it's not. When we have a variable trapped by a "normal" closure (format1) and we try to trap it by a function (format2) defined inside an exec block. That variable is already managed as a cellvar by the surrounding function and the normal closure, so the exec() could see that and add it to format2.__closure__ but it does not.

def test_closure_3():
    # create a normal closure with dec freevar
    dec = "||"
    def format1(msg):
        print(f'{dec}msg{dec}')
    
    # and try to create another closure in an exec block
    fn_st = "\n".join([
        "def format2(msg):", 
        "    print(f'{dec}msg{dec}')"
    ])
    d = {}
    exec(fn_st + "\n" + "d['fn'] = " + "format2")
    format2 = d["fn"]
    
    format1("a")
    format2("b")
    
test_closure_3()
# ||msg||
# name 'dec' is not defined