Python Dynamic Class Creation part 2

In a recent post I mentioned the rough steps that Python executes to create a class when it comes across a class statement (we can call it the "class creation pipeline")

• Step 1 — Determine the metaclass. Python calls __build_class__ (a builtin), which inspects the bases and the explicit metaclass= kwarg to resolve which metaclass to use (with MRO-based metaclass conflict resolution).
• Step 2 — Prepare the namespace. The metaclass's __prepare__ classmethod is called: namespace = Meta.__prepare__('Foo', (Base,), **kwargs). This returns the dict (or dict-like object) that will serve as the class namespace. For type, this is just a plain dict. For enum.EnumMeta, for example, it returns a special _EnumDict.
• Step 3 — Execute the class body. The compiled code object for the class body is executed as a function, with the namespace from step 2 as its locals(). This is the key insight: it's essentially exec(body_code, globals(), namespace). After this, namespace contains {'x': 1, '__module__': ..., '__qualname__': ...}.
• Step 4 — Call the metaclass. Meta('Foo', (Base,), namespace) is called — which, for the default type, invokes type.__call__ → type.__new__ → type.__init__. This is where the actual class object is constructed.

And explained how when we create a class dynamically using type() (or any other metaclass), we are directly at step 4. Obviously we have to take care of deciding what metaclass to use and prepare the namespace ourselves, which means that we should invoke the metaclass __prepare__ method (hook) ourselves, something that I was not aware of. Deciding what metaclass to use may seem pretty straightforward, and it is if we intend to use a particular metaclass, but if we just want to use the one that corresponds based on the inheritance chain it's not that evident. Most times, when all base classes just use type, it's just type, but we really would have to take care of analysing the metaclasses of the classes in that inheritance chain.

So indeed, correctly building a class dynamically is more complex that it seemed to me... Well, not really, cause I had missed that since python 3.3 the types module comes with a nice function just for that, types.new_class(name, bases=(), kwds=None, exec_body=None). This function follows the 4 aforementioned steps in the class creation process. It will select the metaclass to use based on whether we provide a metaclass in kwds or the parent classes in bases. It will invoke __prepare__ to create and prepopulate the class namespace, and pass it over to the exec_body function for it to continue populating it (that exec_body function corresponds to the code that we will place in the class body of a normal class statement). Finally it will invoke the selected metaclass to create the class.

def populate(namespace):
	for field in fields:
		namespace[field.name] = create_descriptor(field)

Generated = types.new_class(
	"Generated",
	(Base,),
	{"metaclass": Meta},
	populate,
)

There's another function in types that should call your attention, types.prepare_class(name, bases=(), kwds=None). It returns the metaclass to be used and the namespace created by metaclass.__prepare__, so that then you can further populate that namespace (with the code that you would put in the exec_body callback if you were using new_class), and invoke that metaclass.

meta, namespace, remaining_keywords = types.prepare_class(
	"Generated",
	(Base,),
	{"metaclass": Meta},
)

for field in fields:
	namespace[field.name] = create_descriptor(field)

Generated = meta(
	"Generated",
	(Base,),
	namespace,
	**remaining_keywords,
)

So while new_class performs the 4 steps in the class creation pipeline, prepare_class performs the 2 first steps and leaves under your responsibility performing steps 3 and 4. We could say that create_class follows a callback-based style while prepare_class moves us into an imperative, phase-oriented style.

Sometimes it can be not possible to decide what metaclass should be used based on the base classes combination (and the explicit metaclass if we happen to have provided one). If the base classes drive us to different, uncompatible (not in the same inheritance chain), metaclasses, we get an error:

>>> class MetaA(type): pass
... 
>>> class MetaB(type): pass
... 
>>> class A(metaclass=MetaA): pass
... 
>>> class B(metaclass=MetaB): pass
... 
>>> class C(A, B): pass
... 
Traceback (most recent call last):
  File "python-input-4", line 1, in module
    class C(A, B): pass
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

If the metaclass obtained from the base classes and the one we provide explicitly do no belong to the same inheritance chain,

>>> class C(B, metaclass=A): pass
... 
Traceback (most recent call last):
  File "python-input-6", line 1, in module
    class C(B, metaclass=A): pass
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases