Wednesday, 10 April 2024

Python callable

In most languages you can only invoke functions/methods, and in some languanges (python, javascript) they happen to be objects ("first class object" is the common way to say it). Normally those "invokable objects" have to be of a specific type (for example the Function type in JavaScript), but Python allows us to make any object invokable (callable) by adding a __call__ method to the class (the type) of the object. The other language I know with a similar feature is Kotlin, with the invoke operator function (we can define an invoke operator member function in a class and call it just by doing ob() on its instances, no need to do ob.invoke()). Notice that some time ago I wrote a post about my failed attempt to simulate this callable functionality in JavaScript.

The feature in Python has a quirk. When python sees a "function invocation" it'll search for the __call__ attribute in the object and invoke it. But __call__ is a dunder method, and contrary to what it does with normal attribute lookups, when searching "implicitly" a dunder method Python does not start the search in the object itself, but in its type. What I mean is that when coming across with a "person1()" call Python will search for a __call__ method in type(person1) (e.g. Person), not in person1. This means that if you want to do callable a specific instance of a class, not all its instances, adding a __call__ attribute to that instance won't work.

We can add a __call__ attibute to the class (Person) but that will make all persons invokable, and that's not what we want. Well, the solution is simple. We can create a new class (let's say CallablePerson) that inherits from Person and has that __call__ method, and change the class of p1 from Person to that new class, I mean: p1.__class__ = CallablePerson. I've thought it would be a nice feature to add to the ObjectLiteral class that we saw in a recent post a function do_callable() to make that literal object callable. Something like this:


class ObjectLiteral(SimpleNamespace):
    def bind(self, name: str, fn: Callable) -> "ObjectLiteral":
        setattr(self, name, MethodType(fn, self))
        return self
    
    def extend(self, cls, *args, **kwargs) -> "ObjectLiteral":
        parent_cls = types.new_class("Parent", (self.__class__, cls))
        self.__class__ = parent_cls
        cls.__init__(self, *args, **kwargs)
        return self

    # - Option 1
    def do_callable(self, fn: Callable) -> "ObjectLiteral":
        # don't add it directly to ObjectLiteral, as it would make all inatances callable, we only want this one to be it
        parent_cls = types.new_class("Parent", (self.__class__, ))
        parent_cls.__call__ = fn
        self.__class__ = parent_cls
        return self

p1 = ObjectLiteral(
    name="Xuan",
    age="50",
)

p1.do_callable(lambda self: print(f"hey {self.name} is callable!"))
p1()
# hey Xuan is callable!
    

That's the normal approach for when turning an instance callable is a decision taken after its class was designed. But if we know that at some point some instances of a class will want to become callable, we can design our class with this in mind beforehand. Define a __call__() in the class that will invoke a _call_imp() (for example) method in the instance if it exists, and otherwise thrown the same exception that we get if we try to invoke a not callable object. The do_callable() method will just add a _call_imp function to the object. What I mean is something like this:


class ObjectLiteral(SimpleNamespace):
    def bind(self, name: str, fn: Callable) -> "ObjectLiteral":
        setattr(self, name, MethodType(fn, self))
        return self
    
    def extend(self, cls, *args, **kwargs) -> "ObjectLiteral":
        parent_cls = types.new_class("Parent", (self.__class__, cls))
        self.__class__ = parent_cls
        cls.__init__(self, *args, **kwargs)
        return self

    # - Option 2
    def __call__(self, *args, **kwargs):
        if hasattr(self, "_call_imp"):
            return self._call_imp(self, *args, **kwargs)
        raise TypeError(f"'{type(self).__name__}' object is not callable")

    def do_callable(self, fn: Callable) -> "ObjectLiteral":
        self._call_imp = fn
        return self

Wednesday, 3 April 2024

Python Dynamism Oddities

I've always been fascinated by features provided in a greater or lesser level by dynamic languages. Adding new attributes/properties to an object (either instance or class), changing the class (prototype in JavaScript) of an existing object, modifying the inheritance chain, hooking into the getting/setting of attributes... There's no doubt that Python is not short of these beautiful features (adding also the concept of metaclasses). I'll share here some quirks, rules, limitations... of some of these features that I've learnt about in the last weeks.

Freely adding new attributes to an object is what I consider one of the main features of a dynamic language. In Python we can do that with our custom classes and instances of those classes, but not with an instance of the object class itself, which throws an "'object' object has no attribute" Exception. This happens because an instance of object does not have a __dict__ attribute, that is where instance attributes (unless we are using slots) are kept. If we create an empty class inheriting from object, its instances have __dict__ and there's no problem in adding attibutes to it. We can read more about this here.


o1 = object()
print(dir(o1))
# o1 does not have __dict__
# ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']

try:
    o1.city = "Paris"
except Exception as ex:
    print(f"Exception: {ex}")
#Exception: 'object' object has no attribute 'city'

class Ob(object):
    pass
    
o2 = Ob()
print(dir(o2))
# o2 has __dict__
#['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__']

o2.city = "Paris"
print(o2.city)
#Paris

A Python object has a __class__ attribute pointing to its class (let's say Class1), and that attribute can be set to another class (let's say Class2), changing dynamically the behaviour of that object (now it has Class2 methods rather than Class1 ones). A class has a __bases__ attribute that points to a tuple containing its inheritance chain. We can set that attribute to a different tuple, hence changing its inheritance chain, but there's a gotcha with that. If you class just derives from object, trying to reassign its __bases__ will throw an exception:
__bases__ assignment: 'XX' deallocator differs from 'object'
However, if your class derives from another class (for example just an empty class deriving from object) the reassignment to __bases__ works without a problem. The exception seems to be a bug that is known and remains open since 2003!!! (which is pretty odd to say the least). When the __bases__ reassignment works fine it also causes the MRO (method resolution order, __mro__ attribute) of the class to be recalculated.


class A1:
    pass

class A2:
    pass

class B:
    pass

# for classes that directly inherit from object __bases__ reassignment fails
try:
    B.__bases__ = (A1, A2)
except Exception as ex:
    print(f"exception! {ex}")
# exception! __bases__ assignment: 'A1' deallocator differs from 'object'

# but if we inherit from any other class (for example this "empty" one) it works
class Empty:
    pass

class C (Empty):
    pass

print(f"C.__mro__ before __bases__ reassignment: {C.__mro__}")
# C.__mro__ before __bases__ reassignment: (, , )
C.__bases__ = (A1, A2)
print(f"C.__mro__ before __bases__ reassignment: {C.__mro__}")
# C.__mro__ after __bases__ reassignment: (, , , )


Classes do not have a __metaclass__ attribute, which makes sense. As the metaclass is the class used to create a class, it's the __class__ attribute of the class object which points to the metaclass (same as the __class__ object in an instance of a class points to its class). This said, we could think that we can change the metaclass of a class by reassigning its __class__ attribute, but this reassignment works for instances, not for classes, for which we get an exception: __class__ assignment only supported for mutable types or ModuleType subclasses


class Meta1(type):
    pass

class A2(metaclass=Meta1):
    def __init__(self,  name):
        self.name = name

print(f"A2 metaclass: {A2.__class__}")
# A2 metaclass: 
print(f"A2 metaclass: {type(A2)}")
# A2 metaclass: 

try:
    A2.__class__ = type
except BaseException as ex:
    print(f"Exception! trying to change the metaclass: {ex}")
# Exception! trying to change the metaclass: __class__ assignment only supported for mutable types or ModuleType subclasses


Metaclasses are mainly used to manage how classes (instances of that metaclass) are created, so changing the metaclass of an existing class is useless for that. However, metaclasses can also have an effect on how instances of a class are created, by providing a custom __call__ method in the metaclass. (a = A() will invoke __call__ in A's metaclass). So we can change the __call__ method of a metaclass to modify how instances of the class are created



def _custom_call(cls, *args, **kwargs):
    print(f"invoking {cls.__name__} construction")
    instance = type.__call__(cls, *args, **kwargs)
    # Additional customization if needed
    return instance

Meta1.__call__ = _custom_call
a2 = A2("Iyan")
# invoking A2 construction
print(f"a2 type: {type(a2).__name__}")
# a2 type: A2
print(f"a2.name: {a2.name}")
# a2.name: Iyan

There's an additional use of metaclasses that was unknown to me, defining a different MRO (method resolution order, stored in the __mro__ attribute) for classes of that metaclass. I discovered this feature in this post. Defining a custom mro() method in the metaclass changes the MRO of a class at class creation type. You can not change the MRO of an existing class by reassigning its __mro__ attribute, as it's readonly. There's a crazy dicussion about a hack for that here.

By overriding mro() on the metaclass we can define a custom __mro__ for our class. Python will then traverse it instead of the default implementation, which is provided by type.mro().

Sunday, 24 March 2024

Python Object Literals

Kotlin object expressions are pretty nice. They can be used as the object literals that we like so much in JavaScript, but furthermore you can extend other classes or implement interfaces. In Kotlin for the JVM these object expressions are instances of (anonymous) classes that the compiler creates for us.

Python does not provide syntax for object literals, but indeed it's rather easy to get something similar to what we have in Kotlin. For easily creating a "bag of attributes" we can leverage the SimpleNamespace class. To add methods to our object we have to be careful, cause if we just set an attribute to a function, when we later invoke it it will be called without the "receiver-self". We have to simulate the "bound methods" magic applied to functions declared inside a class (that are indeed descriptors that return bound-methods). We just have to use types.MethodType to bind the function to the "receiver". Of course, we also want to allow our "literal objects" to extend other classes. This turns out to be pretty easy too, given that in Python we can change the class of an existing object via the __class__ attribute (I tend to complain about Python syntax, but its dynamic features are so cool!) a feature that we combine with the sometimes disdained multiple inheritance of classes. So we'll create a new class (with types.new_class) that extends the old and the new class, and assign this class to our object.
So, less talk and more code! I ended up with a class extending SimpleNamespace with bind and extend methods. Both methods modify the object in place and return it to allow chaining.


class ObjectLiteral(SimpleNamespace):
    def bind(self, name: str, fn: Callable) -> "ObjectLiteral":
        setattr(self, name, MethodType(fn, self))
        return self
    
    def extend(self, cls, *args, **kwargs) -> "ObjectLiteral":
        parent_cls = types.new_class("Parent", (self.__class__, cls))
        self.__class__ = parent_cls
        cls.__init__(self, *args, **kwargs)
        return self

We'll use it like this:



class Formatter:
    def __init__(self, w1: str):
        self.w1 = w1
    
    def format(self, txt) -> str:
        return f"{self.w1}{txt}{self.w1}"
    

p1 = (ObjectLiteral(
        name="Xuan",
        age="50",
    )
    .extend(Formatter, "|")
    .bind("format2", lambda x, wr: f"{wr}{x.name}-{x.age}{wr}")
    .bind("say_hi", lambda x: f"Bonjour, je m'appelle {x.name} et j'ai {x.age} ans")
)

print(p1.format("Hey"))
print(p1.format2("|"))
print(p1.say_hi())
print(f"mro: {p1.__class__.__mro__}")
# let's add new attributes
p1.extra = "aaaa"
print(p1.extra)

# let's extend another class
class Calculator:
    def __init__(self, v1: int):
        self.v1 = v1
    
    def calculate(self, v2: int) -> str:
        return self.v1 * v2
    
p1.extend(Calculator, 2)
print(f"mro: {p1.__class__.__mro__}")
print(p1.format("Hey"))
print(p1.format2("|"))
print(p1.say_hi())
print(p1.calculate(4))

print(f"instance Formatter: {isinstance(p1, Formatter)}")
print(f"instance Formatter: {isinstance(p1, Calculator)}")


# |Xuan-50|
# Bonjour, je m'appelle Xuan et j'ai 50 ans
# |Hey|
# |Xuan-50|
# Bonjour, je m'appelle Xuan et j'ai 50 ans
# mro: (, , , , )
# aaaa
# mro: (, , , , , , )
# |Hey|
# |Xuan-50|
# Bonjour, je m'appelle Xuan et j'ai 50 ans
# 8
# instance Formatter: True
# instance Formatter: True

Notice how after the initial creation of our object we've continued to expand it with additional attributes, binding new methods and extending other classes.

Tuesday, 19 March 2024

Named Arguments Differences

In 2 previous posts [1] and [2] we saw that Python, JavaScript and Kotlin have some differences (python limitations we can say) in how they deal with Default arguments. I was wondering if there are any differences in how they handle Named Arguments.

As for JavaScript, given that it does not support named arguments (which is rather surprising), there's not much to say.

The main difference between Python and Kotlin Named Arguments (also knows as Keyword Arguments in Python) has to do with mixing positional and named arguments in the same call. In Python the named arguments can be provided in any order, but you can not use a not-named argument (positional) after a named argument. In Kotlin, you can also provide named arguments in any order when you are not providing not-named arguments after them. So yes, that "when" means that you are also allowed to provide not-named arguments after named arguments, but if you do this, all the arguments (positional ones and named ones) have to be provided in just the same order they were defined in the function signature.

Let's see some examples in Python:


def format(w1, w2, txt):
    return f"{w1}{w2}{txt}{w2}{w1}"

print(format("-", "|", "Bonjour"))

print(format("-", "|", txt="Bonjour"))

print(format("-", w2="|", txt="Bonjour"))

print(format("-", txt="Bonjour", w2="|"))

# Positional argument cannot appear after keyword arguments
#print(format("-", txt="Bonjour", "|"))
#print(format("-", l2="|", "Bonjour"))


And in Kotlin:


package namedParameters


fun format(w1: String, w2: String, txt: String): String {
    return "$w1$w2$txt$w2$w1"
}


fun main() {
    println(format("-", "|", "Bonjour"))

    println(format("-", "|", txt = "Bonjour"))
    
    println(format("-", w2 = "|", txt = "Bonjour"))
    
    println(format("-", txt = "Bonjour", w2 = "|"))

    //error: Mixing named and positioned arguments is not allowed
    //println(format("-", txt="Bonjour", "|"))
    
    //If I'm going to provide an unnamed parameter after a named one (not available in Python), all the parameters in the call have to be passed in order
    
    println(format("-", w2 = "|", "Bonjour"))

}

From what I've read here named arguments in C# have the same behaviour as in Kotlin (you can provide not-named arguments after named ones, but this forces you to provide all of them in order). At first sight this feature seemed of little use to me. Once you provide some named argument, why would you skip the name of an ensuing argument if that's going to force you to provide all the arguments in the order defined in the signature? Well, indeed it makes sense, let me explain. When we have variables that have the same names as function parameters it can seem a bit redundant to use named arguments, so let's say we are passing as arguments some "inline values" and some variables with a different name from the parameter. This is a case where even if we are keeping the same order as in the function signature, using named arguments for those "inline values" and variables will do our code more clear. But if after those arguments we are going to pass a variable that has the same name as the parameter, using a named argument is redundant, so it's nice not to be forced to name it.

A bit related to this it has come to my mind a discussion in Python (there's a recently created PEP draft for it) about a syntax for shortening the use of named arguments when a variable and a parameter have the same name. The idea seems to be inspired by ruby and would look like this:


#For example, the function invocation:

my_function(my_first_variable=, my_second_variable=, my_third_variable=)

#Will be interpreted exactly equivalently to following in existing syntax:

my_function(
  my_first_variable=my_first_variable,
  my_second_variable=my_second_variable,
  my_third_variable=my_third_variable,
)


This looks like an excellent idea to me that would make unnecessary implementing the Kotlin/C# feature that we've been discussing.

There's one feature of named arguments in Python that I mentioned in my previous post. In Python a function can force us to provide certain (or all) arguments as named ones by means of "*", you can read more here and here. It seems there's been some discussion about adding this to Kotlin, but nothing has been done so far.

Sunday, 10 March 2024

Default Arguments differences

After my previous post about late-bound default arguments/parameters (again I continue to be confused about saying "argument" or "parameter", and indeed the Kotlin documentation also seems to use them indistinctively in some occasions) I realised that there's another limitation of Python default arguments when compared to JavaScript or Kotlin. In python once you define a default argument, all the ensuing arguments also have to be defined as default ones, otherwise you'll get a Syntax Error.

This is not the case in Kotlin (and it seems in Ruby either), where:

If a default parameter precedes a parameter with no default value, the default value can only be used by calling the function with named arguments.


fun foo(
    bar: Int = 0,
    baz: Int,
) { /*...*/ }

foo(baz = 1) // The default value bar = 0 is used

So it's OK not to provide one default argument that has non-default arguments after it, but in that case you have to invoke the funcion naming the remaining arguments. There's an exception to this, if the last of those argument is a lambda (trailing lambda), you can just use the nice syntax of passing it outside the parentheses:

If the last argument after default parameters is a lambda, you can pass it either as a named argument or outside the parentheses

As all this ends up being a bit confusing, I think it's generally adviced to put all the default arguments at the end of the function signature (if the last one is a lambda this one is excluded from the rule).

In JavaScript you can also define them as in Kotlin, but given that there's no support for named arguments the behaviour is that if you pass less parameters than the function defines in its signature, it gets them in order, so it seems like it's not particularly useful. Well, it's useful if we explicitely pass undefined as the value for a default parameter, as in that case the default argument is used, rather than value (that's not the case if we pass null)


const format = (l1, l2 = "_", txt) => `${l1}${l2}${txt}${l2}${l1}`;

// not what we would like
console.log(format("|", "Iyan"));
//|IyanundefinedIyan|

console.log(format("|"));
// |_undefined_|

console.log(format("|", "_", "Iyan"));
// |_Iyan_|

// the thing of having declared a default parameter before a non default parameter is only useful if we decide to pass undefined, that will be replaced by the default value
console.log(format("|", undefined, "Iyan"));
// |_Iyan_|

Well there is a syntax "trick" in Python that sort of allows the kotlin behaviour as explained here. It leverages a Python feature that was unknown to me so far, Keyword-Only Arguments. So you can declare non default arguments after default arguments by placing a ", *" between them, as it forces you to pass by name the remaing non default arguments (so you get the same behaviour as in Kotlin). The disadvantage here is that indeed you are forced to pass those parameters by name always, even when you are providing all the values and not using any of the defaults.


# We're not allowed to write this:
# Non-default argument follows default argumentPylance
# def format(l1, l2 = "_", txt):
#     return f"{l1}{l2}{txt}{l2}{l1}"

# but we can use the * feature
def format(l1, l2 = "_", *, txt):
    return f"{l1}{l2}{txt}{l2}{l1}"


# both fail:
# format() missing 1 required keyword-only argument: 'txt'
#print(format("|", "Iyan"))
#print(format("|"))

# works fine
print(format("|", txt="Iyan"))
# |_Iyan_|

# the disadvantage is that we are forced to always pass txt by name, even when we are passing all values
#TypeError: format() takes from 1 to 2 positional arguments but 3 were given
#print(format("|", "-", "Iyan"))

print(format("|", "-", txt="Iyan"))

Tuesday, 5 March 2024

Destructuring, astuple, attrgetter

I already talked about destructuring in JavaScript/Python/Kotlin in this previos post. So, to sum up, we can use destructuring with any iterable Python object (that's equivalent to JavaScript Array destructuring). That's nice, but it would be event more cute to have something that we could use with any object (a bit in the vein of JavaScript Object destructuring) with no need to make it iterable. I've been looking into some possibilities:

If you are working with simple dataclasses the astuple() function comes pretty handy. Notice though, that it will recursivelly retrieve elements of other dataclasses, lists, dictionaries... which probably is not what you want./p>


@dataclass
class Person:
    name: str
    city: str
    post_code: int
    age: int
    job: str

p1 = Person("Iyan", "Xixon", 33200, 49, "Coder")

name, _, pc, job, _ =  astuple(p1)
print(name, pc, job)
#Iyan 33200 Coder


That approach works in a positional way. We know the order of the the attributes of the class and we take those positions we want and discard others (with the _ convention).

Another option, this one more nominal, as we take attributes based on their name, is using operator.attrgetter, like this:


name, pc, job = operator.attrgetter("name", "post_code", "job")(p1)
print(name, pc, job)
#Iyan 33200 Coder


It looks ok, but using attribute names in its string form is a huge risk. If you rename an attribute refactoring tools will know nothing about your string, and you have to remember that you are accessing that attribute as a string to manually fix it. It's easy to make a mess... With that in mind, I think I prefer the redundancy of writing the object name multiple times:


name, pc, job = p1.name, p1.post_code, p1.job
print(name, pc, job)
#Iyan 33200 Coder


Related to this, what if we want to transform each of the values we are destructuring, but with a different transformation for each value? I'm not talking about a normal map() call where we apply the same funcion, I'm talking about applying different functions. For that, a transform function like this will do the trick:


def transform(funcs, items):
    return [func(item) for func, item in zip(funcs, items)]


That way we can write something like this:



name, pc, job = transform(
    (str.upper, lambda x: x, str.lower), 
    attrgetter("name", "post_code", "job")(p1), 
)
print(name, pc, job)
#IYAN 33200 coder


name, pc, job = transform(
    (str.upper, lambda x: x, str.lower), 
    (p1.name, p1.post_code, p1.job), 
)
print(name, pc, job)
IYAN 33200 coder

Monday, 19 February 2024

Yielding aka Allowing others to Run

At the end of my previous post I talked about the yield() Kotlin function, and how we use it to cooperate with other coroutines by asking if another coroutine wants to run, and if so, suspending the caller coroutine so that another coroutine gets scheduled. We don't have a specific yield function in JavaScript or Python, we just leverage another more generic function for that purpose, let's see.

In Python we use await asyncio.sleep(0) for that. asyncio.sleep() will suspend the current coroutine in all cases, regardless of the interval provided (from the documentation: "sleep() always suspends the current task, allowing other tasks to run."). The event loop will take control and will check if another coroutine wants to run, and if not, if the interval is 0, will resume the sleeping coroutine immediatelly. So it's the same as yield in Kotlin

Javascript follows the same strategy as Python for this. The setTimeout() function always returns control to the event loop, even when the interval is 0, with the callback being added to the macrotask queue (from the documentation: "If this parameter is omitted, a value of 0 is used, meaning execute "immediately", or more accurately, the next event cycle". If there's nothing in the microtask queue or the macrotask queue, the callback we've just added will run immediatelly (in the next event loop iteration), as we've provided a 0 wait interval. OK, I'm talking about callbacks, which seems a bit odd, so notice that we can easily "promisify" the setTimeout function getting an equivalent to asyncio.sleep(), like this:


async function asleep(timeout) {
    return new Promise(res => setTimeout(res, timeout));
}

Additionally, in JavaScript we can await any value, not just a Promise. We can just write await "aa" or await Promise.resolve("aa") (both are equivalent, await "aa" is indeed transformed in await Promise.resolve("aa")). Awaiting for an already resolved Promise will add the "then callback" for that promise (in an async function that's the invokation to the next state of the state-machine for that function) to the microtask queue. When the event loop gets back the control it will schedule the next task in the microtask queue. So if there was some previous task it will be executed, else our "then callback" task will be executed, so this is equivalent to yielding.

So all in all in JavaScript we have 2 levels of yielding. We can yield to tasks in the microtask queue by awaiting a resolved Promise, or yield to tasks in the macrotask queue, by using setTimeout(0). You can read more about this here and here.

We know that Kotlin has a suspend delay() function, but we can not use it as a replacement for yield(), because delay() only suspends when the interval is bigger than 0 (from the documentation: "If the given timeMillis is non-positive, this function returns immediately". So delay(0) has no effect at all, the next line will run sequentially, no suspension and chances for other coroutines to run will happen.

I must say that I pretty like the Kotlin approach. Having a specific yield function rather than leveraging a particular case of another function is more semantic