Python Execution Unit

In a previous post about Python class bodies and lexical scopes I mentioned that the code in a class body is executed by creating a "synthetic" function around it. When Python compiles (at runtime) one module it basically creates a tree of code objects (let's think of a code object as an object containing bytecodes and metadata). A code object for the module level code, a code object for each function or method, a code object for each class body... Then at runtime it'll create a class object when it comes across the LOAD_BUILD_CLASS bytecode instruction, and a function object when it comes across the MAKE_FUNCTION bytecode instruction. Functions reference the code object that contains their code, and as I've said when a class is created a synthetic, one-shot function is created at runtime in order to execute the code object corresponding to the body of that class. Based on this, one could think that in order to execute the code object corresponding to the module level code, Python we'll also create a "synthetic" function on the fly. I've been discussing this with some GPT's and that's not the case. In order to execute some Python code it's not strictly necessary to have a function.

At the lowest level code execution of interpreted code (bytecodes) in CPython involves invoking the _PyEval_EvalFrameDefault function with a Frame object (as we saw in some previous post this is an optimized "internal Frame" object, not the user-friendly Frame object that gets materialized when we use sys._getframe or inspect.stack()[n].frame). This Frame points to a code object, the current instruction position, variables (locals (normal locals, parameters, closure variables...), globals, builtins). That information is normally gathered from invoking a function (CALL instruction that goes through the vector-call protocol). The function object contains the globals (the module level variables of the module where the function was defined), closure variables, the code object. Let's see the precise description given by a GPT of the relation between Frames, Functions and Code Objects:

A useful way to understand Python execution is to separate several layers that are often mentally collapsed into one:

Code object: the compiled instructions and static metadata.
Function object: a callable package containing a code object and persistent execution context.
Frame: the state of one concrete execution of a code object.
Evaluator: the interpreter loop that executes the frame, represented in CPython by machinery such as _PyEval_EvalFrameDefault().
For an ordinary function call, the flow is approximately:

function object
   │
   │ supplies:
   │   code object
   │   globals
   │   closure cells
   │   defaults and keyword defaults
   │
   │ call supplies:
   │   positional and keyword arguments
   ▼
frame
   │
   ▼
_PyEval_EvalFrameDefault()

This leads to an important principle:

The evaluator fundamentally executes a frame. A function object is one mechanism for packaging the information needed to construct and initialize that frame, but bytecode execution does not intrinsically require a function object.

2. The role of each object
- Code object
A code object describes what is to be executed. It contains, among other things:

bytecode instructions
constants
referenced names
local-variable names
cell-variable names
free-variable names
argument-count metadata
source-position information
A code object does not normally contain the runtime values of its globals, free variables, or arguments.

- Function object
A function object packages a code object with persistent context, notably:

function.__code__
function.__globals__
function.__closure__
function.__defaults__
function.__kwdefaults__
Arguments belong to a particular invocation, so they are supplied when the function is called rather than being permanently stored in the function object.

- Frame
A frame represents one particular execution. It brings together the code and the concrete runtime state required by the evaluator, including:

instruction position
evaluation stack
locals
globals
builtins
arguments
cell and free-variable storage
exception state
Thus, multiple invocations of the same function use the same function and code objects but create distinct frames.

A Function stores closure cells and globals, defaults, receiving its parameters in each invokation. But what if we don't need any of these? When executing the code object for module level code, we have no closure cells (there's not a lexical outer scope from which to trap free vars, the module itself is the outermost scope), there are no parameters to be provided... so indeed a function object is not necessary. This is different from the execution of a class body, where as seen in my aforementioned post, we can trap free vars from an outer scope (the class being defined inside a function) and also have globals (those of the module where the class is defined).

So contrary to the execution of a class body, the execution of module level code does not need the creation of a function for it. Conceptually we can think module execution as: exec(module_code, module_dict, module_dict)

Indeed, there are 2 other mechanisms to execute code without having a function, exec and eval. So module level code execution, exec and eval follow this route:

C-Level Evaluation: Instead of executing a function-based call path, the subsystem calls the internal C-API entry point: PyObject *PyEval_EvalCode(PyObject *co, PyObject *globals, PyObject *locals)

Frame Allocation: PyEval_EvalCode sets up a new PyFrameObject. Crucially, both globals and locals map directly to the single __dict__ dictionary of the module.

Conceptual Flow of exec("x = 10")
[String Input] -> compile(text, "", "exec") -> PyEval_EvalCode() -> PyFrameObject -> Interpreter Loop

And to complete the post with more GPT wisdom:

Final mental model
The cleanest formulation is:
_PyEval_EvalFrameDefault() executes a frame. A frame represents one concrete execution of a code object. A function object is a higher-level package that helps turn a call into such a frame by supplying the code object, globals, closure cells, defaults, and related metadata.

From that model:

Module bodies bypass the function-object layer because the import or exec machinery already has the code object and namespace, and module code has no enclosing lexical scope from which to capture free variables.
Ordinary functions use function objects to package reusable code together with persistent context and to support repeated calls with different arguments.
Class bodies in CPython use temporary function objects because the class-body code may need globals and closure cells, while __build_class__ separately supplies a prepared class-local namespace.

As you can see my posts are becoming more and more a summary of my investigation on some topic with a GPT. There are few external links and more and more text copy-pasted from the final output of a (sometimes rather long) GPT discussion (or multiple ones). That's how life is now. Anyway, during this "investigation" I've also come across an excellent explanation of how the Python bytecode interpreter works.