AsyncExecutor

It's interesting how depending on what you have been working on lately you can come up with different solutions for the same issue. 3 years ago I posted about a solution to limit the number of asynchronous calls running in parallel in JavaScript. Basically I was asking permission to a TicketManager that was like a sort of C#'s SemaphoreSlim.

In the last months I've been working mainly with Python, and I've discovered the so nice functionality provided by ThreadPoolExecutor and ProcessPoolExecutor, and before the prospect of limiting async calls in JavaScript again (e.g. limiting the number of http requests in progress to the same server at a given time), I've naturally come up with the idea of implementing an AsyncExecutor.

So we have a class with a max number of slots (actions), and we submit to it asynchronous functions and their parameters. If it has some free slot it will run the asynchronous function and return the Promise returned by that call. If there are no slots, it puts the funtion in a queue, and returns a Promise that will get resolved once the function has been invoked and resolved. That's all:

class AsyncAction{
    constructor(fn, args, resolveFn, rejectFn){
        this.fn = fn;
        this.args = args;
        this.resolveFn = resolveFn;
        this.rejectFn = rejectFn;
    }
}

export class AsyncExecutor{
    constructor(maxRunningActions){
        this.maxRunningActions = maxRunningActions;
        this.runningCounter = 0;
        this.pendingActions = [];
    }

    submit(fn, ...args){
        let resolveFn, rejectFn;
        let pr = new Promise((resFn, rejFn) => {
            resolveFn = resFn;
            rejectFn = rejFn;
        });
        let action = new AsyncAction(fn, args, resolveFn, rejectFn);

        if (this.runningCounter < this.maxRunningActions){
            this._runAction(action);
        }
        else{
            this.pendingActions.push(action);
        }

        return pr;
    }

    async _runAction(action){
        this.runningCounter++;
        let result;
        try{
            result = await action.fn(...(action.args));
        }
        catch (ex){
            this._processActionResult(action, false, ex);
            return;
        }
        this._processActionResult(action, true, result);
    }

    _processActionResult(action, successful /*bool*/, result){
        this.runningCounter--;
        if (successful){
            action.resolveFn(result);
        }
        else{
            action.rejectFn(result); //here result is an exception
        }
        if (this.pendingActions.length){
            this.runningCounter++;
            this._runAction(this.pendingActions.shift());
        }
    }
}

You can test it with this:

import {AsyncExecutor} from "./asyncExecutor.js"


function asyncSleep(interval){
    return new Promise(resFn => {
        setTimeout(resFn, interval);
    })
}

async function mockDownload(url, delay){
    console.log(`starting download: ${url}`);
    await asyncSleep(delay);
    return url.toUpperCase();
}

let executor = new AsyncExecutor(3);
let asyncDownloads = []
for (let i=0; i<= 6; i++){
    asyncDownloads.push(executor.submit(
        mockDownload, 
        `www.resistence.fr/post_${i}`, Math.floor(Math.random() * 4000)
    ).then(result => {
        console.log(`download ${i} finished`);
        return result;
    }));
    console.log("after submit");
}

let results = await Promise.all(asyncDownloads);
console.log(`results: ${JSON.stringify(results, null, "\t")}`);

/*
starting download: www.jesoutienslapolice.fr/post_0
after submit
starting download: www.jesoutienslapolice.fr/post_1
after submit
starting download: www.jesoutienslapolice.fr/post_2
after submit
after submit
after submit
after submit
after submit
starting download: www.jesoutienslapolice.fr/post_3
download 2 finished
starting download: www.jesoutienslapolice.fr/post_4
download 3 finished
starting download: www.jesoutienslapolice.fr/post_5
download 0 finished
starting download: www.jesoutienslapolice.fr/post_6
download 1 finished
download 5 finished
download 6 finished
download 4 finished
results: [
        "WWW.JESOUTIENSLAPOLICE.FR/POST_0",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_1",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_2",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_3",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_4",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_5",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_6"
]

*/

As usual, I've uploaded it to a gist.

Python Descriptors

Properties (aka getters-setters) in python do not seem like any particularly "advanced" feature. In principle it's just adding a getter and setter method to access a given attribute. Well, properties are a specific case of descriptors. In short a descriptor is an object with a __get__ method, and optionally __set__ and __delete__ methods. Adding a property to a class is not just directly adding a getter (setter, deleter) method to the class, it's adding an object that is an instance of the property class. There's a very nice explanation here

descriptors are a low-level mechanism that lets you hook into an object's attributes being accessed. Properties are a high-level application of this; that is, properties are implemented using descriptors. Or, better yet, properties are descriptors that are already provided for you in the standard library.

When we do an attribute lookup on an object and the found attribute is a descriptor (and it finds it in the class of that object, not directly in the object.__dict__) the runtime invokes its __get__ method (or __set__ or __delete__). Yes, descriptor objects have to be in the class, not directly in the instance, you can read an explanation here

What is really interesting of descriptors is that they are used to implement much of the python behaviour we like so much. For example, I used to think that in order to return a bound method when you access a method from an instance the python runtime would have some very specific code for that. A function (something that we define with "def" or "lambda") is an instance of the function class, that happens to be a descriptor. When we define a method in a class, we are adding that function object to that class __dict__. So when invoke that method in an instance (my_instance.method1("aaa")), we are first looking up the "method1" attribute in the my_instance object. It will be found in the __dict__ of my_instance class, and as it has a __get__ method (and has not been found directly in my_instance), that __get__ method is executed, and it takes care of returning an object that is an instance of the method class, bound to my_instance (and finally, for the call itself, the ("aaa") part, as the retrieved object is callable, __call__ method in the method class is invoked) So we have a language feature that is not implemented by some specific runtime/interpreter code, but just using a standard python feature.

The python documentation about descriptors is really, really good, as it helps you understand pretty important elements of the language, like attribute lookup, bound methods, class methods (a decorator applied to a function descriptor)... It provides python implementations of these functionalities, so it's really worth a read.

Attribute look-up is implemented in object.__getattribute__ and type.__getattribute__. When looking up an attribute of an object, the runtime searches for the __getattribute__ method in that object's class. So for an instance of a "normal class" (let's say an instance p1 of class Person:, p1.instance_attribute) the runtime searches for __getattribute__ in Person.__dict__, where unless that we have redefined it for the class, it won't find it, so it will continue to search in the inheritance chain and will end up finding it in object.__dict__["__getattribute__"]. If we are looking up an attribute in a class itself (Person.my_class_attribute), it will search in Person's class, that is either type or a class derived from type (if Person were using another metaclass (I mean: class Person(metaclass=CustomMetaclass):). So we'll end up with type.__dict__["__getattribute__"].
If the __getattribute__ that we've executed finds a descriptor, the invocation of __get__ in that descriptor is done with: desc.__get__(a, type(a)) from the object.__getattribute__, and with: desc.__get__(None, A) from the type.__getattribute__.

When you check the class of an object using the __class__ attribute, it's also implemented as a descriptor (same as __name__, as I already mentioned here), so __class__ is not in the object dictionary, but in the class, particularly in its inheritance chain, in object.__dict__['__class__']. There's a good explanation here

It's interesting that properties are implemented to match the behaviour of object.__getattribute__ and type.__getattribute__ so that when accessing a property from an instance we obtain the value, while accessing it from the class returns the proprerty object.

In [70]: class Book:
    ...:     @property
    ...:     def price(self):
    ...:         return 4
    ...:         

In [71]: b1 = Book()

In [72]: b1.price
Out[72]: 4


In [74]: b1.__class__.__dict__["price"]
Out[74]: <property> at 0x1c31f647ce0>

In [75]: b1.__class__.price
Out[75]: <property> at 0x1c31f647ce0>

This is a bit different in JavaScript, a language that also has the concept of descriptors (they are used for any attribute, not just for getters-setters aka "access descriptors"), but where to get the descriptor object rather than the value, we have to use a specific method: Object.getOwnPropertyDescriptor

class Person{
    constructor(name, lastName){
        this.name = name;
        this.lastName = lastName;
    }

    get fullName(){
        return `${this.name} ${this.lastName}`; 
    }

    sayHi(toX){
        return `Hi ${toX} I am ${this.name}`
    }

}

let p1 = new Person("Francois", "Belmondo")
console.log(p1.fullName); //Francois Belmondo

// the fullName access descriptor is in Person.prototype (just as the sayHi function)
//these 2 execute the "getter", but with the Person.prototype as "this", so gets undefined name and lastName
console.log(Person.prototype.sayHi("aa")); //Hi aa I am undefined
console.log(Person.prototype.fullName); //undefined undefined

let d1 = Object.getOwnPropertyDescriptor(Person.prototype, "fullName"); //the descriptor

let d2 = Object.getOwnPropertyDescriptor(p1, "fullName"); // undefined
let d3 = Object.getOwnPropertyDescriptor(Person, "fullName"); //undefined

Young Ahmed

Last Saturday Tve2 gifted us with the screening of the absolutely excellent Belgian film Young Ahmed - Le jeuene Ahmed. In my opinion Belgium is the European country where the Muslim invasion and the process of replacement of the European culture by a hostile and less evolved (in moral terms, in terms of achievements, etc) culture is in its most advanced stage. Well, I think it's worse than in France in terms of percentages, though in France the invaders are even more hostile, cause they are motivated not only by their interpretation of religion and by their racist (anti-white and anti-EastAsian racism) and ultranationalist "culture" but also by the colonial hatred and that senseless victimization feeling that the far-left (from Sartre to Melenchon) has instilled on them for the last decades.

Ahmed is a 13 years old teenager of maghrebian descent that gets radicalised by his Imam to a level that you can only believe if you've experience the failed multicultural experience of countries like France or Belgium, otherwise you would think that such level of hatred and fanatism can not exist in our century. I appreciate a lot the bold and realistic stance of the directors, that do not try to find any sort of justification for this madness. The guy's family does not live in poverty and is not a victim of "discrimination" from the evil "white Belgians". The mother is a well integrated woman that behaves and looks like a normal person (I don't ask immigrants to become exemplary citizens, I just ask them to be normal people (with "normal" being based on the standards of our European culture) that just wants the best for her kids, and his sister looks also totally normal, but his brother seems to be starting to move to the wrong side. There was a cousin that died as a yihadist, that is a dishonor for the mother, but a role-model for this Ahmed scumbag.

This film shows us fundamentalist Islam in all its cruelty. A teenager that refuses to shake hands with his teacher (that had become a family friend after much helping him years ago with different problems), because she is a woman. She is a working woman of Muslim descent, well integrated (and we can guess that she still considers herself a Muslim, as she tries to explain to Ahmed that her Coran does not say what his Imam is teaching him), that because of that is no longer considered a Muslim by the Imam or Ahmed, but an apostate (yes, if you are not as radical as us, you are no longer a Muslim). Strict (or fundamentalist, I'm not sure what term to use) interpretations of Islam consider Apostasy to be punishable by death... so well, Ahmed brings yihad home by trying to kill this brave woman!!!

This film shows 2 essential elements of radical Islam without directly naming/explaining them. The first one is the fundamentalist interpretation of Taqiya. Allegedly, Taqiya means that Muslims can dissimulate their faith (or just lie, saying that they are not Muslims) to save their lives. Fundamentalist views consider that you can (and must) lie to protect and spread Islam. So Ahmed lies all the time, saying that his Imam has nothing to do with his radical views, that he regrets his actions and wants to ask for forgiveness to his teacher...
The second essential element is salvation through yihad. When Ahmed considers to have committed a terrible sin (he has kissed a girl!!!) he goes like crazy in a new attempt to kill his teacher, as this action to "protect" Islam will clean his sins and save him. This view is the basis of how many drug dealers and criminals of "muslim culture" suddenly get radicalised and become yihadists/terrorists. After doing wrong things for years, now you kill and die in the name of God and all your sins get cleaned and you go to heaven to enjoy your well deserved Harem... cool...

An additional point of interest is that after having sinned (kissing that girl!!!) Ahmed first tries to convert her to Islam to marry him (buff, that's love at first sight), because that way his sin could be forbidden (I guess it's not just that kissing a Muslim is not so bad as kissing an infidel, it's that if you are converting her you are spreading the faith, so that earns you points). As she rejects such a beautiful proposal and prefers to remain a Kuffar, the only way to salvation, as I've just mentioned in the previous paragraph, is doing "something big", like killing an apostate.

To sum up, a true must-see film.

functools Partial and __name__

I've recently come across a small issue with python functools and a simple solution. First, let's start with some theory.

You can get the name of a class by means of __name__. Maybe you think that's an attribute of the class (I mean, given a class Person, you could think that you have the __name__ attribute in Person.__dict__). That's wrong. __name__ is a descriptor in Person.__class__ (Person's metaclass), hence, a descriptor in Person.__class__.__dict__. Notice that if __name__ were directly an attribute in the class, you could do person_instance.__name__, something that is not possible

class Person:
	pass
Person.__name__
Out[52]: 'Person'

Person.__dict__["__name__"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [53], in ()
----> 1 Person.__dict__["__name__"]

KeyError: '__name__'

Person.__class__.__dict__["__name__"]
Out[54]: <attribute '__name__' of 'type' objects>

type(Person.__class__.__dict__["__name__"])
Out[55]: getset_descriptor

p1 = Person()

p1.__name__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [57], in ()
----> 1 p1.__name__

AttributeError: 'Person' object has no attribute '__name__'

You can also get the __name__ of a function. Again, this is not an attribute directly in each function dictionary, but a descriptor in the function's __class__.__dict__. Python functions are objects that are instances of the class "function", so it's interesting how instances of the function class has a __name__, by means of the __dict__[__name__] in the function class, but as we've seen above, a normal class like Person does not have such __name__ in its own __dict__.

def say_hi(who):
    print(f"hi {who}")
    

say_hi.__name__
Out[60]: 'say_hi'

say_hi.__dict__["__name__"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [61], in ()
----> 1 say_hi.__dict__["__name__"]

KeyError: '__name__'

say_hi.__class__.__dict__["__name__"]
Out[62]: <

The functools module provides us with all (or most) "functional helper" functions that we can be used to from other languages. functools.partial is used to create partially applied funtions, so it's similar to JavaScript Function.prototype.bind. Interestingly, functions.partial does not return another function (a closure trapping the parameters being bound and the original function itself), but an instance of the functools.partial class (as such class has a __call__ method, that instance is callable and is mainly like a function to us). The problem I've found with that partial object is that it lacks a __name__ attribute.

def format_message(wrap_str, msg):
    return f"{wrap_str}{msg}{wrap_str}"

format_with_x = functools.partial(format_message, "x")
print(format_with_x("Hi"))

print(type(format_message))
print(format_message.__name__)

print(type(format_with_x))
try:
    print(format_with_x.__name__)
except BaseException as ex:
    print(f"Exception: {ex}")

# xHix
# <class 'function'>
# format_message
# <class 'functools.partial'>
# Exception: 'functools.partial' object has no attribute '__name__'

I've found here that functools provides a fix for this, the functools.updatewrapper function, that "Updates a wrapper function to look like the wrapped function". This means that we can just do this:

def create_wrapped_partial(func, *args, **kwargs):
    partial_func = functools.partial(func, *args, **kwargs)
    functools.update_wrapper(partial_func, func)
    return partial_func
	
format_with_z = create_wrapped_partial(format_message, "z")
print(format_with_z("Hi"))
print(type(format_with_z))
try:
    print(format_with_z.__name__)
except BaseException as ex:
    print(f"Exception: {ex}")
	
# zHiz
# <class 'function'>
# format_message
# <class 'functools.partial'>
# format_message	

Run Function with Timeout

I've recently needed to be able to stop a function if it had not finished after a timeout period. I'm not talking about an eventloop and an asynchronous function returning an awaitable in Python (or a Promise in Javascript). Indeed, years ago I already posted about expirable promises in JavaScript. What I'm talking about is a synchronous Python function (normally doing some CPU bound calculation) that we want to stop if it has not finished after a given amount of time. I say stop its operation, not just stop waiting for it but let it run in the background uselessly consuming resources.

A solution that easily comes to mind is running the function in a separate Thread and joining that thread with the timeout option. If the timeout expires our calling code will stop blocking and continue on, but the problem is that the thread doing that costly CPU operation will continue to run and we have no way to kill it. This is something that really bothers me. Neither Python, nor .Net nor Java let you kill a thread "from outside" (I mean from another thread, what is still possible I think is killing the thread you are running in). I know that the reason for discouraging and not providing this functionality is that when you kill another thread it could be holding resources or doing something critical and killing it could mess things...) OK, but we are grown ups, so if you know that such thread can be killed withoug causing problems, you should be allowed... Well, anyway...

In order to overcome the limitations imposed by the GIL to parallelism, python provides an extremely powerful library to run your code in separate processes as easily as you run it in separate threads in other platforms: multiprocessing (and the higher level ProcessPoolExecutor in concurrent.futures). Contrary to threads, you are allowed to kill (terminate) processes, so I've implemented the timeout functionality running the function in a separate process.

So I have a create_func_with_timeout factory function, that receives a function fn and a timeout value and returns a new function that takes care or running fn in a separate process, waiting for it for a timeout time and throwing a TimeoutException if it expires. If fn itself were to throw an exception we want to have it available in the calling process, so to manage this we wrap the execution of fn in the mp_function_wrapper function. This function returns to the calling process the normal result of running fn (or an exception it that were the case) by writing it to a multiprocessing queue.

Reading the code should make it clear:

import multiprocessing
import queue
import threading
from time import sleep
from typing import Any, Dict, TextIO, List, Callable, Tuple, Optional, AnyStr, Match, cast


class TimeoutException(Exception):
    pass

def create_func_with_timeout(fn: Callable, timeout: int) -> Callable:
    """
    returns a new function that runs the provided fn in a separate process that either finishes normally or times out
    """
    def mp_function_wrapper(fn: Callable, queue: multiprocessing.Queue, *args, **kwargs):
        """
        this function has been started in a separate process
        it "returns" the result of fn to the main process by writing it to the queue
		we could take fn and timeout as closure values, but it seems more clear to pass them as parameters
        """
        try:
            result = fn(*args, **kwargs)
        except BaseException as ex:
            result = ex
        queue.put(result)
        
    def func_with_timeout(*args, **kwargs):
        """
        has the fn to run trapped as closure value
        """
        try:
            multiprocessing.set_start_method('fork')
            # multiprocessing.set_start_method('spawn')
        except RuntimeError as ex:
            print(f"set_start_method: {ex}")
        print(f"func_with_timeout {fn.__name__} {timeout}")
        response_queue = multiprocessing.Queue()
        extended_args = [fn, response_queue, *args]
        proc = multiprocessing.Process(target=mp_function_wrapper, args=extended_args, kwargs=kwargs)
        proc.start()
        try:
            result = response_queue.get(block=True, timeout=timeout)
        except queue.Empty:
            # the process is still running, so finish it off
            proc.terminate()
            raise TimeoutException()
        if result and isinstance(result, BaseException):
            raise result
        return result
  
    return func_with_timeout


def format_items(items):
    #print(f"format_items")
    results = []
    for item in items:
        sleep(1)
        results.append(item.upper())
        #print(item)
    return ",".join(results)

format_with_timeout = create_func_with_timeout(format_items, 5)

def run_test(items):
    print("- test")
    try:
        res = format_with_timeout(items)
        print(f"result: {res}")
    except BaseException as ex:
        print(f"exception: {type(ex).__name__} {ex}")


print("- starting")

run_test(["a", "b", "c"])

run_test(["a", "b", "c", "d", "e", "f", "g"])

run_test(["a", None])

print("- ending")

# - starting
# - test
# format_items
# result: A,B,C
# - test
# format_items
# exception: TimeoutException
# - test
# format_items
# exception: AttributeError 'NoneType' object has no attribute 'upper'
# - ending

I've also uploaded it to a gist

Creating classes dynamically

Python classes are profoundly dynamic. We can add new methods to a class or an instance, modify the inheritance chain, switch one method implementation for another one, or dynamically create a new class... Remember from my previous post that classes are instances of the type metaclass or of a custom metaclass.

Given these methods and data attributes that we want in our dynamic class:

# initializer
def init(self, name):
    self.name = name
  
# method
def say_hi(self, to):
    return f"{self.name} says hi to {to}"
  
# class method
@classmethod
def my_class_method(cls, arg):
    return f"{cls.__name__}.{cls.my_class_method.__name__} invoked with arg: {arg}, {cls.class_attribute}"

class_items = {
    # constructor
    "__init__": init,
      
    # class data
    "class_attribute": "AAAA",
      
    # methods
    "say_hi": say_hi,
    "my_class_method": my_class_method
}

def test_dynamic_class(cls):
    p1 = cls("Francois")
    print(p1.say_hi("Antoine"))
    print(cls.my_class_method("BB"))
    print(cls.class_attribute)

We can create a class with type as its metaclass by invoking type like this:

# creating class dynamically
Person = type("Person", (object, ), class_items)

test_dynamic_class(Person)

# Francois says hi to Antoine
# Person.my_class_method invoked with arg: BB, AAAA
# AAAA

An alternative would be creating an "empty" class and adding the methods and data attributes like this:

class Person2:
    pass

for key, value in class_items.items():
    setattr(Person2, key, value)

test_dynamic_class(Person2)

# Francois says hi to Antoine
# Person2.my_class_method invoked with arg: BB, AAAA
# AAAA

If we want to dynamically create a class using a custom metaclass we have to use types.new_class. Notice that rather than passing a dictionary with the attributes, we pass a function that adds those attributes to the "class namespace"

class Meta1(type):
    pass

def fill_namespace(ns):
    ns.update(class_items)

Person3 = types.new_class(
    name="Person3",
    #bases=(B, C),
    kwds={"metaclass": Meta1},
    exec_body=fill_namespace
)

test_dynamic_class(Person3)

# Francois says hi to Antoine
# Person3.my_class_method invoked with arg: BB, AAAA
# AAAA

We can also use the alternative technique of creating an "empty class" (but with our custom metaclass): class Person3(metaclass=Meta1) and then adding the methods and data attributes to it.

Python Metaclasses

Metaclasses are a very interesting topic. The basic idea is In object-oriented programming, a metaclass is a class whose instances are classes.. That means that basically all Object Oriented languages have metaclasses. In C# all classes are instances of Type, in Java all classes are instances of Class... So the interesting thing comes when we can define custom metaclasses, as we can do for example in Groovy and Python.

I talked about Groovy metaclasses so many years ago. Custom metaclasses in Grooy are frequently used, while that's not the case in Python. The reason for this is that most of the features that require metaclasses in Groovy (expanding and object or class, interception...) are present in python through different means (__getattribute__, __getattr__...)

This means that custom metaclasses are rarelly necessary in Python, but anyway, I find it useful to have an (at least partial) understanding of how they work and their relation with callables and the attribute lookup mechanism.

This is probably the most in depth article I've found about metaclasses. I can not say I've fully understood it... but with it and other readings I think I have the basic idea of how things work. Let's go

type
Classes and functions are objects. object is an instance of type. type is an instance of type and inherits from object. A class is an instance of type. A metaclass is an instance of type and inherits from type. To find the metaclass of a class A, we just do type(A).

Callables
When python comes across this: x() it checks what class x is an instance of, and will call the __call__ method in that class with x as first parameter.

Creation of an Instance of a class
When creating an instance of a class A (a = A()) it invokes its metaclass .__call__ (so for "normal" classes it's type.__call__ and for classes with a custom metaclass it's CustomMetaclass.__call__). Then type.__call__(cls, *args, *kwargs) basically does this: invokes cls.__new__(cls) (that unless overriden is object.__new__) and with the returned instance object it invokes cls.__init__(instance)

Creation of a class
When creating a class A (class A:) it's like doing A = type("A", bases, attributes). So it invokes __call__ in type's metaclass, that happens to be type itself, so type.__call__(type). Then this invokes type.__new__ and type.__init__ (by the way, in case you are wondering whether type.__new__ is just the same as object.__new__, no, they are not)

Custom metaclass
- When creating a class with a custom metaclass (class A(metaclass=MetaB):), we are creating an instance of MetaB, so we end up calling MetaB.__new__ and MetaB.__init____.
- When creating an instance of a class A that has a metaclass MetaB (a = A()), this calls MetaB.__call__ (that could have been overriden in MetaB, or otherwise will be type.__call__)

From all the above, we can get to the 2 main uses of metaclasses:
- Defining __new__ and/or __init__ in the metaclass we can manage how instances of that metaclass (classes) are created. - Defining a custom __call__ method in the metaclass we can manage how instances of classes of that metaclass are created (singleton?).

This article makes a good read about the above. Just to make the signatures clear:

class MyMeta(type):
    def __new__(self,name,base,ns):
        print("meta new")
        return type.__new__(self,name,base,ns)
    def __init__(self,name,base,ns):
        print("meta init") self.cloned = False
        type.__init__(self,name,base,ns) class 

metaclasse and method lookup

Metaclasses do not add extra layers to the attribute lookup mechanism. Let's say we have "class A" and "a = A()". When looking up an attribute at1 in object a (a.at1), python searches first in a's __dict__ (or slots) and if not there, it'll check what class a is an instance of (that is, A), and will search in A's __dict__ and in that of its base classes. It will not use A's metaclass in this lookup mechanims at all.

If we search an attribute in a class (A.att1), python searches first in A.__dict__, then as it sees that is an instance of type (or a derived metaclass) or maybe just because it sees that it has a __bases__ attribute, it checks in the __dict__ of its parents. If not there, as A is an instance of type, it will search in type.__dict__. So yes, in this case it's using the metaclass, but just because it's the first "instance of", but it does not go deeper with the "instance of" of "instance of" of...

metaclasses and inheritance Custom metaclasses are inherited from your base classes. With multiple inheritance it becomes a bit complex, it's perfectly explained in the superb article that I mentioned above.

Immutable
You can not switch the metaclass of a class for a different metaclass once it has been created (and contrary to what some posts say, it makes sense to want to do that, as changing the metaclass of a class can change how instances of the class are created). However, there's an easy workaround, similar to what I do in my previous post. Create a new class that inherits from the one we want to modify and assign it the new metaclass. I mean:

class Meta1(type):
	pass
	
class A(metaclass=Meta1):
	pass

class Meta2(type):
	pass	

# now, let's "change" the metaclass of A from Meta1 to Meta2

class A2(A, metaclass=Meta2):
	pass
	

And then use A2 rather than A for creating new instances. We've previously seen that for already existing instances the metaclass of it's class does not play any role, so not being able to change it does not pose any problem.