Monday 29 August 2022

Safe Dereferencing in Python

There's a beautiful feature that I first came across with in Groovy many years ago and that over time has made it into C# first and TypeScript/JavaScript recently. It goes under different names: Safe Navigation, Save Dereferencing, Optional Chaining. Well, I'm refering to the "?" operator when accessing an attribute, that will return null (undefined in JavaScript) if the attribute is missing, and will not continue with ensuing attribute look ups if we were in a chain. I mean:

country?.capital?.location?.xCoordinate;

Notice that optional chaining also works for method invokation, using in JavaScript this odd syntax: ob.method?.()

It's one of those few features that I miss in Python. There is proposal PEP 505 to add it to the language, but for the moment it's in deferred state. So, same as I did 10 years ago when JavaScript was missing the feature and I sort of implemented a workaround, I've written a simple python function to sort of simulate it. We pass the chain as a List of strings (representing the attributes) or tuples (representing parameters for a method invokation):

UPDATE: The below implementation is quite a crap, for a better one check this follow-up post.


def safe_access(ob, chain):
    """
    sort of "safe accessor" (javascript's x?.attr?.attr2...)
    this version includes method invocation, so it's quite different from my getattr_chain.py
    """
    cur_value = ob
    for item in chain:
        if isinstance(item, str): #accessing attribute
            cur_value = getattr(cur_value, item, None)
        else: #invoking
            cur_value = cur_value(*item)
        if cur_value is None:
            return None
    return cur_value


class Person:
    def __init__(self, name, age, profession):
        self.name = name
        self.age = age
        self.profession = profession

class Profession:
    def __init__(self, name, salary):
        self.name = name
        self.salary = salary
    
    def format_profession(self, st1, st2):
        return f"{st1}-{self.name}-{st2}"

p1 = Person("Francois", 4, Profession("programmer", 50000))

print(safe_access(p1, ["name"]))
print(safe_access(p1, ["city"]))
print(safe_access(p1, ["city", "name"]))
print(safe_access(p1, ["profession", "salary"]))
print(safe_access(p1, ["profession"
    , "format_profession", ("¡¡", "!!")
    , "upper", ()
]))

In JavaScript, if we try to invoke a method, but it exists as a data attribute, not as a function, the optional chaining will throw an error: TypeError: xxx is not a function


> ob = {name: "aa"};
> ob.name?.();
Uncaught TypeError: ob.name is not a function

I'm also doing that in my python code (you'll get a TypeError, xxx is not callable). I think it could make sense to return None rather than throwing an exception, for that we would update the invokation part in the above function with a callable check, like this:


            # cur_value = cur_value(*item)

            # if not callable, return None also
            if callable(cur_value):
                cur_value = cur_value(*item)
            else:
                return None


Safe dereferencing/Optional Chaining is related to other 2 features, also present in C# and recently in JavaScript (when I wrote this post it had not made it into JavaScript yet), the null coalescing operator and the null coalescing assignment. I think the aforementioned deferred PEP should also address these 2 things. For the moment, depending on the data that you are expecting, you could check with (as we used to do in JavaScript until recently) or. I mean a = b or c (rather than an hypothetycal: a = b ?? c). You can take a look at this discussion to see the possible issues with this approach.

Friday 19 August 2022

AsyncExecutor

It's interesting how depending on what you have been working on lately you can come up with different solutions for the same issue. 3 years ago I posted about a solution to limit the number of asynchronous calls running in parallel in JavaScript. Basically I was asking permission to a TicketManager that was like a sort of C#'s SemaphoreSlim.

In the last months I've been working mainly with Python, and I've discovered the so nice functionality provided by ThreadPoolExecutor and ProcessPoolExecutor, and before the prospect of limiting async calls in JavaScript again (e.g. limiting the number of http requests in progress to the same server at a given time), I've naturally come up with the idea of implementing an AsyncExecutor.

So we have a class with a max number of slots (actions), and we submit to it asynchronous functions and their parameters. If it has some free slot it will run the asynchronous function and return the Promise returned by that call. If there are no slots, it puts the funtion in a queue, and returns a Promise that will get resolved once the function has been invoked and resolved. That's all:


class AsyncAction{
    constructor(fn, args, resolveFn, rejectFn){
        this.fn = fn;
        this.args = args;
        this.resolveFn = resolveFn;
        this.rejectFn = rejectFn;
    }
}

export class AsyncExecutor{
    constructor(maxRunningActions){
        this.maxRunningActions = maxRunningActions;
        this.runningCounter = 0;
        this.pendingActions = [];
    }

    submit(fn, ...args){
        let resolveFn, rejectFn;
        let pr = new Promise((resFn, rejFn) => {
            resolveFn = resFn;
            rejectFn = rejFn;
        });
        let action = new AsyncAction(fn, args, resolveFn, rejectFn);

        if (this.runningCounter < this.maxRunningActions){
            this._runAction(action);
        }
        else{
            this.pendingActions.push(action);
        }

        return pr;
    }

    async _runAction(action){
        this.runningCounter++;
        let result;
        try{
            result = await action.fn(...(action.args));
        }
        catch (ex){
            this._processActionResult(action, false, ex);
            return;
        }
        this._processActionResult(action, true, result);
    }

    _processActionResult(action, successful /*bool*/, result){
        this.runningCounter--;
        if (successful){
            action.resolveFn(result);
        }
        else{
            action.rejectFn(result); //here result is an exception
        }
        if (this.pendingActions.length){
            this.runningCounter++;
            this._runAction(this.pendingActions.shift());
        }
    }
}

You can test it with this:


import {AsyncExecutor} from "./asyncExecutor.js"


function asyncSleep(interval){
    return new Promise(resFn => {
        setTimeout(resFn, interval);
    })
}

async function mockDownload(url, delay){
    console.log(`starting download: ${url}`);
    await asyncSleep(delay);
    return url.toUpperCase();
}

let executor = new AsyncExecutor(3);
let asyncDownloads = []
for (let i=0; i<= 6; i++){
    asyncDownloads.push(executor.submit(
        mockDownload, 
        `www.resistence.fr/post_${i}`, Math.floor(Math.random() * 4000)
    ).then(result => {
        console.log(`download ${i} finished`);
        return result;
    }));
    console.log("after submit");
}

let results = await Promise.all(asyncDownloads);
console.log(`results: ${JSON.stringify(results, null, "\t")}`);

/*
starting download: www.jesoutienslapolice.fr/post_0
after submit
starting download: www.jesoutienslapolice.fr/post_1
after submit
starting download: www.jesoutienslapolice.fr/post_2
after submit
after submit
after submit
after submit
after submit
starting download: www.jesoutienslapolice.fr/post_3
download 2 finished
starting download: www.jesoutienslapolice.fr/post_4
download 3 finished
starting download: www.jesoutienslapolice.fr/post_5
download 0 finished
starting download: www.jesoutienslapolice.fr/post_6
download 1 finished
download 5 finished
download 6 finished
download 4 finished
results: [
        "WWW.JESOUTIENSLAPOLICE.FR/POST_0",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_1",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_2",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_3",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_4",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_5",
        "WWW.JESOUTIENSLAPOLICE.FR/POST_6"
]

*/

As usual, I've uploaded it to a gist.

Friday 12 August 2022

Python Descriptors

Properties (aka getters-setters) in python do not seem like any particularly "advanced" feature. In principle it's just adding a getter and setter method to access a given attribute. Well, properties are a specific case of descriptors. In short a descriptor is an object with a __get__ method, and optionally __set__ and __delete__ methods. Adding a property to a class is not just directly adding a getter (setter, deleter) method to the class, it's adding an object that is an instance of the property class. There's a very nice explanation here

descriptors are a low-level mechanism that lets you hook into an object's attributes being accessed. Properties are a high-level application of this; that is, properties are implemented using descriptors. Or, better yet, properties are descriptors that are already provided for you in the standard library.

When we do an attribute lookup on an object and the found attribute is a descriptor (and it finds it in the class of that object, not directly in the object.__dict__) the runtime invokes its __get__ method (or __set__ or __delete__). Yes, descriptor objects have to be in the class, not directly in the instance, you can read an explanation here

What is really interesting of descriptors is that they are used to implement much of the python behaviour we like so much. For example, I used to think that in order to return a bound method when you access a method from an instance the python runtime would have some very specific code for that. A function (something that we define with "def" or "lambda") is an instance of the function class, that happens to be a descriptor. When we define a method in a class, we are adding that function object to that class __dict__. So when invoke that method in an instance (my_instance.method1("aaa")), we are first looking up the "method1" attribute in the my_instance object. It will be found in the __dict__ of my_instance class, and as it has a __get__ method (and has not been found directly in my_instance), that __get__ method is executed, and it takes care of returning an object that is an instance of the method class, bound to my_instance (and finally, for the call itself, the ("aaa") part, as the retrieved object is callable, __call__ method in the method class is invoked) So we have a language feature that is not implemented by some specific runtime/interpreter code, but just using a standard python feature.

The python documentation about descriptors is really, really good, as it helps you understand pretty important elements of the language, like attribute lookup, bound methods, class methods (a decorator applied to a function descriptor)... It provides python implementations of these functionalities, so it's really worth a read.

Attribute look-up is implemented in object.__getattribute__ and type.__getattribute__. When looking up an attribute of an object, the runtime searches for the __getattribute__ method in that object's class. So for an instance of a "normal class" (let's say an instance p1 of class Person:, p1.instance_attribute) the runtime searches for __getattribute__ in Person.__dict__, where unless that we have redefined it for the class, it won't find it, so it will continue to search in the inheritance chain and will end up finding it in object.__dict__["__getattribute__"]. If we are looking up an attribute in a class itself (Person.my_class_attribute), it will search in Person's class, that is either type or a class derived from type (if Person were using another metaclass (I mean: class Person(metaclass=CustomMetaclass):). So we'll end up with type.__dict__["__getattribute__"].
If the __getattribute__ that we've executed finds a descriptor, the invocation of __get__ in that descriptor is done with: desc.__get__(a, type(a)) from the object.__getattribute__, and with: desc.__get__(None, A) from the type.__getattribute__.

When you check the class of an object using the __class__ attribute, it's also implemented as a descriptor (same as __name__, as I already mentioned here), so __class__ is not in the object dictionary, but in the class, particularly in its inheritance chain, in object.__dict__['__class__']. There's a good explanation here

It's interesting that properties are implemented to match the behaviour of object.__getattribute__ and type.__getattribute__ so that when accessing a property from an instance we obtain the value, while accessing it from the class returns the proprerty object.


In [70]: class Book:
    ...:     @property
    ...:     def price(self):
    ...:         return 4
    ...:         

In [71]: b1 = Book()

In [72]: b1.price
Out[72]: 4


In [74]: b1.__class__.__dict__["price"]
Out[74]: <property> at 0x1c31f647ce0>

In [75]: b1.__class__.price
Out[75]: <property> at 0x1c31f647ce0>

This is a bit different in JavaScript, a language that also has the concept of descriptors (they are used for any attribute, not just for getters-setters aka "access descriptors"), but where to get the descriptor object rather than the value, we have to use a specific method: Object.getOwnPropertyDescriptor


class Person{
    constructor(name, lastName){
        this.name = name;
        this.lastName = lastName;
    }

    get fullName(){
        return `${this.name} ${this.lastName}`; 
    }

    sayHi(toX){
        return `Hi ${toX} I am ${this.name}`
    }

}

let p1 = new Person("Francois", "Belmondo")
console.log(p1.fullName); //Francois Belmondo

// the fullName access descriptor is in Person.prototype (just as the sayHi function)
//these 2 execute the "getter", but with the Person.prototype as "this", so gets undefined name and lastName
console.log(Person.prototype.sayHi("aa")); //Hi aa I am undefined
console.log(Person.prototype.fullName); //undefined undefined

let d1 = Object.getOwnPropertyDescriptor(Person.prototype, "fullName"); //the descriptor

let d2 = Object.getOwnPropertyDescriptor(p1, "fullName"); // undefined
let d3 = Object.getOwnPropertyDescriptor(Person, "fullName"); //undefined


Sunday 7 August 2022

Young Ahmed

Last Saturday Tve2 gifted us with the screening of the absolutely excellent Belgian film Young Ahmed - Le jeuene Ahmed. In my opinion Belgium is the European country where the Muslim invasion and the process of replacement of the European culture by a hostile and less evolved (in moral terms, in terms of achievements, etc) culture is in its most advanced stage. Well, I think it's worse than in France in terms of percentages, though in France the invaders are even more hostile, cause they are motivated not only by their interpretation of religion and by their racist (anti-white and anti-EastAsian racism) and ultranationalist "culture" but also by the colonial hatred and that senseless victimization feeling that the far-left (from Sartre to Melenchon) has instilled on them for the last decades.

Ahmed is a 13 years old teenager of maghrebian descent that gets radicalised by his Imam to a level that you can only believe if you've experience the failed multicultural experience of countries like France or Belgium, otherwise you would think that such level of hatred and fanatism can not exist in our century. I appreciate a lot the bold and realistic stance of the directors, that do not try to find any sort of justification for this madness. The guy's family does not live in poverty and is not a victim of "discrimination" from the evil "white Belgians". The mother is a well integrated woman that behaves and looks like a normal person (I don't ask immigrants to become exemplary citizens, I just ask them to be normal people (with "normal" being based on the standards of our European culture) that just wants the best for her kids, and his sister looks also totally normal, but his brother seems to be starting to move to the wrong side. There was a cousin that died as a yihadist, that is a dishonor for the mother, but a role-model for this Ahmed scumbag.

This film shows us fundamentalist Islam in all its cruelty. A teenager that refuses to shake hands with his teacher (that had become a family friend after much helping him years ago with different problems), because she is a woman. She is a working woman of Muslim descent, well integrated (and we can guess that she still considers herself a Muslim, as she tries to explain to Ahmed that her Coran does not say what his Imam is teaching him), that because of that is no longer considered a Muslim by the Imam or Ahmed, but an apostate (yes, if you are not as radical as us, you are no longer a Muslim). Strict (or fundamentalist, I'm not sure what term to use) interpretations of Islam consider Apostasy to be punishable by death... so well, Ahmed brings yihad home by trying to kill this brave woman!!!

This film shows 2 essential elements of radical Islam without directly naming/explaining them. The first one is the fundamentalist interpretation of Taqiya. Allegedly, Taqiya means that Muslims can dissimulate their faith (or just lie, saying that they are not Muslims) to save their lives. Fundamentalist views consider that you can (and must) lie to protect and spread Islam. So Ahmed lies all the time, saying that his Imam has nothing to do with his radical views, that he regrets his actions and wants to ask for forgiveness to his teacher...
The second essential element is salvation through yihad. When Ahmed considers to have committed a terrible sin (he has kissed a girl!!!) he goes like crazy in a new attempt to kill his teacher, as this action to "protect" Islam will clean his sins and save him. This view is the basis of how many drug dealers and criminals of "muslim culture" suddenly get radicalised and become yihadists/terrorists. After doing wrong things for years, now you kill and die in the name of God and all your sins get cleaned and you go to heaven to enjoy your well deserved Harem... cool...

An additional point of interest is that after having sinned (kissing that girl!!!) Ahmed first tries to convert her to Islam to marry him (buff, that's love at first sight), because that way his sin could be forbidden (I guess it's not just that kissing a Muslim is not so bad as kissing an infidel, it's that if you are converting her you are spreading the faith, so that earns you points). As she rejects such a beautiful proposal and prefers to remain a Kuffar, the only way to salvation, as I've just mentioned in the previous paragraph, is doing "something big", like killing an apostate.

To sum up, a true must-see film.