Tuesday, 2 July 2024

Python Modules and Inheritance

We can say that Python is a "very object oriented language" because not only functions are objects, but also modules are objects. Each module is an instance of the module class. That's one of those builtin types that can be referenced through the types module, using types.ModuleType in this case.


import types

# get a reference to the current module
m1 = sys.modules[__name__]

print(f"module type: {type(m1)}, name: {__name__}")
#module type: class 'module', name: module1


print(type(m1) is types.ModuleType) #True
print(m1.__class__ is types.ModuleType) #True


With modules being objects one could think about having in them features that we have in "normal objects", like intercepting attribute access, being callable... These features are based on dunder methods that are defined not in the object itself, but in its class (__getattr__ for the "missing attribute functionality", __setattr__ and __getattribute__ for intercepting attribute setting or getting, __call__ for callables), but modules are just instances of types.ModuleType not of custom classes where we can define those dunder methods, so it seems like there's not much we can do...

Well, indeed there is. First, PEP-562 was implemented some years ago to provide the __getattr__ and __dir__ functionality to modules. You can define these methods directly in your module and it will work. So it seems like the standard behaviour of these two dunder methods has been modified so that for module objects they work just being in the object itself rather than in its class.

There was a proposal, PEP-726) to add that same support for __setattr__ and __delattr__, but it was rejected. But hopefully, reading the PEP itself we find a simple alternative for it, that leverages the beautiful python dynamism. If we want a module with "super-powers", we can just define in our module a class that inherits from module and provides those dunder methods we need, and make our module inherit from it just by setting the __class__ attribute. Cool! We can use this technique not only for dunder methods, but also for adding getters-setters (descriptors) to our module. Let's see an example of a callable module with also a get descriptor:


# module2.py
import sys
import types

def format_msg(msg: str) -> str:
    return f"[[{msg.upper()}]]"

# define a new Module type (inheriting from the standard Module)
class MyModule(types.ModuleType):
    # make our module callable
    def __call__(self, msg):
        return self.format_msg(msg)
    
    # define a property
    @property
    def formatted_name(self):
        return f"-||{self.__name__}||-"

# set the class of the current module to the new MyModule Module type    
sys.modules[__name__].__class__ = MyModule


###########################

# main.py
import module2

print(module2.format_msg("aaa"))

# our module is callable:
print(module2("aaa"))

# property (getter)
print(module2.formatted_name)


It's important to notice something that is mentioned in the PEP regarding performance and this technique:

But this variant is slower (~2x) than the proposed solution. More importantly, it also brings a noticeable speed regression (~2-3x) for attribute access.

Tuesday, 25 June 2024

Deferring Log Message Construction-Formatting

When I started to dive into Kotlin, among many surprising pieces of code I remember coming across some strange loggin code like this:


logger.debug { "Person location is: $formatLocation(p1.location)" }

So they are passing a lambda expression (that returns a string) to the debug() function rather than passing the string itself. OK, this makes good sense if formatting that string is expensive, cause if because of our log-level the message is not going to be printed, performing its expensive formatting is a waste of resources. So wrapping the string generation in a lambda is deferring the operation to the logger itself, so if it's not needed it won't be performed. The above code is using kotlin-logging a wrapper over sfl4j. You can read a nice article here.

I have not found any Python logging wrapper allowing that, but indeed the logging standard module partially provides the feature, and implementing the full feature is pretty simple. Let's see.

I was not aware that the different logging methods in the logger object can receive not just an string, but also a string and arguments. If that's the case, the library will apply "printf style" to that string and those arguments. That's nice and clever, but has some limitations. First, we have to use the old "printf style" syntax, rather than modern f-strings. Second, that's fine if we are just using the string formatting mini-language for the formatting, and if the arguments to be logged do not have to be calculated, cause in both cases that operation would be performed before invoking the logger, so not deferred and always executed.


logger.info("{%s} - {%s}", 24, 28)

So let's see how to overcome those limitations. Python has lambda expressions, so to get the same fully deferred behaviour that we have in Kotlin we would need a specialization of the logging.logger class with overriden methods accepting callables rather than strings. But that also involves some specialization of the logger creation itself, to return this logger rather than the standard one. While searching if such kind of wstuff already exists in some nice pip module (that does not seem so) I came across a simple, clever and very nice solution by means of a "LazyMessage" class. Well, it's a bit strange cause the code in that stackoverflow discussion is overriding the __format__ method in that class, that seems wrong, what you really have to override is the __str__ method. So we just need to wrap our messages in a lambda and an instance of LazyMessage


import logging

class LazyMessage(object):
    def __init__(self,func):
        self.func=func
    def __str__(self):
        return self.func()


logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("log")

def get_price():
	print("getting price")
	return 1111

# these 2 calls won't be logged due to the log level
# but here the call to get get_price() is performed anyway
logger.debug(f"current price is {get_price()}")
# getting price

# while here thanks to LazyMessage we avoid the call to get_price()
logger.debug(LazyMessage(lambda: f"current price is {get_price()}"))

print("...........")

# these 2 calls get logged. We just verify that LazyMessage __str__ is invoked correctly
logger.info(f"current price is {get_price()}")
# getting price
# INFO:log:current price is 1111

logger.info(LazyMessage(lambda: f"current price is {get_price()}"))
# getting price
# INFO:log:current price is 1111

There's another related topic I'll mention here. f-strings are evaluated just at the point where they are declared. If we want to use them as a template that we declare somewhere (even in a template file) an we use later in differet places, we have to use a trick. We can use a normal string that mimics an f-string, and evaluate it later on just by using eval(). This is the python equivalent to what I discuss in this JavaScript and C# post from years ago.


def lazy_fstring(fst: str, **kargs) -> str:
    """receives a pseudo-fstring and performs the evaluation"""
    fst = 'f"""' + fst + '"""'
    return eval(fst)


# notice that this is not a real f-string, it lacks the initial f
fst = "{kargs['p1']} says hi to {kargs['p2']}"
msg = lazy_fstring(fst, p1="Xose", p2="Francois")
print(msg)

Monday, 17 June 2024

No Man's Land

In the last 2 years I've watched tons of very good series and films on arte.tv. Some of them were excellent and would have well deserved a post, but I did not find the time, save for Les Papillons Noirs. 2 months ago I watched another series, very different but as exciting and gorgeous as that one, I'm talking about No Man's Land.

The fight of the Kurdish people against 2 hideous Islamist and Fascist organizations, ISIS and the Turkish state is a topic that really touches me. After the YPG/YPJ had managed to defeat the ISIS monster in a demonstration of honor and courage that seems so alien to this century, when in 2018 Erdogan launched his assault against Afrin I almost cried of anger and hatred.

I have to admit that in these last years I've been quite disconnected from what's been happening in Rojava/Northern Syria, as we, Europeans, are involved in our own war against Islamists, decolonialists, anti-western ideologists and their native collaborators/woke lunatics (with La France Insoumise in France being the biggest example of a gathering of all our enemies). Maybe it's also that the situation of the Kurdish people being abandoned and betrayed by all other nations and massacred by the Turkish scum had turned so unbearable that I just prefered not to know.

So when I came across a French/Belgian/Isreli series revolving around the Kurdish struggle in 2014 I felt immediately attracted. These were the "good times" when the Kurds were fighting only against ISIS (of course Turkey was supporting ISIS, but in a partially covered way, not the all out war that they declared to the Kurds later) and it seemed possible that they would manage to get the Freedom and the State that this People deserve so much.

The story is mesmerizing. Spoiler warningThere's more to it than this, but the essential part of the story is that of a young French archeologist that because of odd life circunstances (one love, one execution, one family broke up) ends up aiding the Iranian resistence against the Theocratic Iranian State, with the Mossad pulling the strings without her knowledge. After a failure she will end up fighting along the Kurdish YPJ against the ISIS monster, as a form of redemption and catharsis for someone that no longer has anything to lose. That's enough, you just have to find this series and watch it.

The actress playing that role, Mélanie Thierry makes such an amazing interpretation. Apart from being one of the most beautiful women on Earth, the way she conveys all the pain and despair that she has so inside is just amazing.

Sunday, 9 June 2024

Kotlin (vs Python) Type System

I recently read this amazing article about the Kotlin type system. Though focused on Kotlin, it's useful to get a better understanding of Type Systems at a general level. In the past I would have thought of types just as classes, interfaces and functions, but particularly with advanced type systems like those in Kotlin, Python typing and TypeScript, we have to understand types in a broader sense (with nullable types, union and interception types where supported, Callables-Function Types, Nothing-Never...). I think we should think of Types as contracts of what an object can do, descriptions of what an object is. As the article says:

A class is a template for creating objects. A type defines expectations and functionalities.

I wrote in the past about Top and Bottom types in TypeScript and Python typing. I've realised that what I say in both articles about Any as a bottom type is not fully correct (at least "academically"). In both languages Any behaves as if it were at the bottom of the Type Hierarchy because it disables the type-checker, not because of allegedly sitting at the bottom of that Type Hierarchy. In TypeScript and Python the Any type allows us to combine the full dynamism of both languages and the convenience, safety and error preventing powers of types, cause by disabling the type-checker it's as if it were derived from any other type, and accessing to any method or attribute in it is valid. Casting an object to Any will allow us to add or access any attibute in that object without any complain from the type-checker.

This said, in Kotlin there's not that "as if", in Kotlin Any? (and Any) is just a Top type, it's not a "sort of" bottom type, and in Kotlin there's no way to disable the type-checker. The bottom type in Kotlin is Nothing. We can not cast an object to Nothing, and Nothing is mainly used to express cases where a function will never return (infinite loop, exception). As Nothing derives from any other type, if we mark a function as returning a string, but in some conditions it returns the string but in others it throws an exception, the compiler will be happy with it. Notice that we can not create an instance of Nothing or cast an object to Nothing. We can not create instances of Nothing, indeed, I think type-theory says that bottom types can not be instantiated, but I'm not 100% sure.


// can not create instances of Nothing (private constructor)
//var n = Nothing()

println("null is Nothing: ${null is Nothing}") //False

In Kotlin for the JVM the relation between Any and Object is a bit particular. While in Java all objects inherit from Object, as we've seen in Kotlin all objects inherit from Any? and Object does not show up in the type hierarchy diagrams. We can create instances using the Any and Object constructors, and at a Kotlin level both constructors will return an instance of Any (if we check its Kclass with instance::class). But in reality, at the JVM level, both instances are instances of Object (as we can see with instance::class::java). We can cast an Object instance to Any and viceversa, but the compiler will warn us each time we use Object with a: "This class shouldn't be used in Kotlin. Use kotlin.Any instead" message.


class Animal {}

var any1 = Any()
println("any1::class: ${any1::class}") //kotlin.Any
println("any1::class.java: ${any1::class.java}") //java.lang.Object
println("any1 is Any: ${any1 is Any}") // true
println("any1 is Object: ${any1 is Object}") // true
var ob: Object = any1 as Object


var o1 = Object()
println("o1::class: ${o1::class}") //kotlin.Any
println("o1::class.java: ${o1::class.java}") //java.lang.Object
println("o1 is Object: ${o1 is Object}") // true   
println("o1 is Any: ${o1 is Any}") // true       
any1 = o1 as Any //warning: no cast needed
any1 = o1

var a2: Animal = Animal()

// this does not compile: error: incompatible types: Object and Animal
println("a2 is instance of [Object]?: ${a2 is Object}")
    
// warning: this class shouldn't be used in Kotlin. Use kotlin.Any instead.
o1 = a1 as Object


As I've said, in Python, Any is a top type, and a sort of bottom type. We have to notice that Any only exists at the type-checking level, at runtime we can not check if an object is an instance of Any, and indeed we can not create an instance of Any. Python has a real bottom type, Never (that is equivalent to NoReturn)

A special kind of type is Any. A static type checker will treat every type as being compatible with Any and Any as being compatible with every type.[1]
The Any type is both a top type (so you can assign anything to Any) and a bottom type (so you can assign Any to anything). In effect it turns off the type system whenever you use Any.[2]

The behaviour of Any with regards to nullability is a bit strange. While for other type declarations, e.g. a1: Animal that declaration is a not nullable one (in Python it's common to say Optional rather nullable), and to get its nullable equivalent we have to use any of this forms: a1: Optional[Animal] or a1: Animal | None, for Any this is different. Any means any value, including None, so you can assign None to a variable declared as Any (None is compatible with Any), so Python's Any is different from Kotlin's Any. Though an Any varible could point to None, the type-checker will not apply on it the restrictions that it applies on nullables (Optional's), and we can access any attribute on it freely, cause indeed as I aforementioned, "Any disables the type checker". I we want something equivalent to Kotlin's Any? we should use Any | None. There's some discussion about this Any vs Optional[Any] here

We used to simplify any union containing Any to Any, but @ddfisher found out this is incorrect. E.g.if something is either an int or Any, when it is an int, it definitely doesn't have an append() method.

Saturday, 1 June 2024

Expressions, Declarations, Statements

"What's de difference between expression and statement?" is a typical question with a simple answer: An expression evaluates to a value (produces a value). A statement does something. All expressions are (can be used as) statements, but not all statements are (can be used as) expressions. As expressions produce a value you can use them on the right side of an assignment, as a parameter to a function, in a condition...

There's an additional item to take into account declarations. I think that we can say that declarations are statements that "declare" something (declare a class, a function, an enum, a variable...)

As modern commonly used languages feature richer syntaxes, there are more things to take into account. Things that in older languages were just statements, not expressions, in these rich languages are expressions. In Kotlin we have if-else expressions (well, that's nothing too impressive, it's just the equivalent to the ternary operator), when (switch) expressions, try-catch expressions, throw and return expressions.

When I first came across this "almost everything is an expression" Kotlin feature I was pretty amazed, and I read here something that explains it, but that I've now realised that is not fully accurate.

In Kotlin nothing is just a statement. Every statement is either a declaration or a expression.

But as I've said, that's not fully correct. In Kotlin there are 2 things that are not expressions and are not declarations either: loops and assignments. This is a great read.

Kotlin does not explicitly distinguish between statements, expressions and declarations, i.e., expressions and declarations can be used in statement positions.

It's interesting to note that JavaScript lacks these nice try-catch, switch, throw, return expressions... but on the other hand features class expressions (named and unnamed) and function expressions (named and unnamed) while in Kotlin we don't have class expressions at all, and we have anonymous function expressions, but not named ones.

It's also interesting that the 2 things that in Kotlin are neiter expressions nor declarations, but only statements (loops and assignments), in Python (that unfortunately has a much less rich syntax) can be used as expressions. for expressions correspond to list-generator comprehensions, and assignment expressions correspond to the walrus operator :=.

Friday, 17 May 2024

Kotlin Interface 'Invokation'

In general I like Kotlin syntax a real lot (mainly the "almost everything is an expression" thing) but there are things that I find surprising/confusing (time ago we saw one of them). Recently, when reading a bit about Ktorm, the object-relation mapper, I came across some rather surprising syntax. The article itself mentions it and explains what's going on (otherwise I don't think I had manages to understand how that was working), but I think it'll be useful to me to put it here.


interface Department : Entity {
    companion object : Entity.Factory()
    val id: Int
    var name: String
    var location: String
}

val department = Department()


So the above code seems as if we were somehow creating an instance of an interface (Department), which is obviously not possible. I can think of 2 other situations where we write code that looks similar, but with a different explanation.

One case is the special syntax for functional interfaces/SAM conversions


val runnable = Runnable { println("This runs in a runnable") }

And the other case are object expressions with the object (of the anonymous class) implementing an interface. But the object: syntax, and the "class body" makes clear that we are not "instantiating that interface".


interface Writable {
    fun write(): String
}

//object expressions implementing interface, there's not () after the interface name
val o2 = object: Writable {
    var name = "Francois"
    override fun write() = name
}


The case we're dealing with in this post is a different thing. The original article explains it:

The Entity.Factory class overloads the invoke operator, so we can use brackets to call the companion object as it’s a function. The code creating a department object:

val department = Department()

That’s the charm of Kotlin, Department is an interface, but we can still create its instances, just like calling a constructor function.

What that means is that in the end that code that seemed as if we were invoking a sort of "interface constructor" indeed was calling the invoke method of the companion object of that interface, that is:


val department = Department()

// is right the same as:

val department = Department.Companion.invoke()

Friday, 10 May 2024

Kotlin iteration, toList vs asIterable

In this previous post I talked about Iterables and Sequences in Kotlin, about the eager vs deferred and lazy nature of their extension functions (this StackOverflow discussion is pretty good). As I read somewhere "Collections (that are classes that implement Iterable) contain items while Sequences produce items". What I did not mention in that post is that the fact that Sequences are not Iterables feels quite strange to me (and I see no reason for the Sequence[T] interface not to implement the Interface[T] interface, it would not interfere with using the right extension function). In Python and JavaScript (where we don't have an Iterable class or interface) an iterable is a sort of "protocol". Anything that can be iterated (we can get an iterator from it) is iterable. In Kotlin we have an Iterable interface, but there are things (Sequences) that can be iterated (both the Iterable and Sequence interfaces have an iterator() method) without implementing that Interface. I guess what is confusing for me is saying "X is iterable" cause it seems very similar to saying "X is (implements) an Iterable". Saying "X can be iterated" seems more appropriate, a Sequence can be iterated but it's not an Iterable.

Well, let's put semantic discussions aside and let's see something more concrete. We can obtain a Sequence from an Iterable by means of the asSequence() extension function or the Sequence() function. We'll do that if we want to leverage the lazy extension functions for mapping, filtering, etc... In the other direction we can "materialize" a sequence into a Collection using the toList() extension function. That way we force the sequence to "produce" all its items and we store them in a collection. Additionally we also have the Sequence[T].asIterable() extension function. When/why would we use asIterable() rather than toList()?

Let's say we have a sequence on which we will be applying some filtering/mapping, and we want those functions to be applied in one go, eagerly, as soon as one item would be needed in an ensuing iteration. BUT we don't want to do it now. We can use asIterable() to get an object that implements the Iterable interface but contains no elements (so we are deferring the execution). Then, when we first invoke the map/filter extension function (on Iterable) the whole original sequence will be iterated and processed in one go (egarly). If we use .toList() the whole iteration of the sequence occurs immediatelly, before calling map/filter to process the items. I'm using the deferred/immediate execution and lazy/eager evaluation terms as explained in this excellent post.



fun createCountriesSequence() = sequence {
    println("Before")
    yield("France")
    println("next city")
    yield("Portugal")
    println("next city")
    yield("Russia")
}

    var countriesSequence  = createCountriesSequence()
    // converting the sequence to list immediatelly iterates the sequence
    var countries = countriesSequence.toList()
    println("after converting to list")

    // Before
    // next city
    // next city
    // after converting to list

    println()

    // what's the use of calling .asIterable() rather than directly .toList()?
    // with .asIterable we obtain a deferred collection (nothing runs until we first need it)
    // but as the extension functions in Iterable are eager, first time we apply a map or whatever the whole iteration is performed 
 
    countriesSequence = createCountriesSequence()
    //converting to iterable does not perform any iteration yet
    var countriesIterable = countriesSequence.asIterable()
    println("after creating iterable")

    println("for loop iteration:") //no iteration had happenned until now
    for (country in countriesIterable) {
        println(country)
    }

    // after creating iterable
    // for loop iteration:
    // Before
    // France
    // next city
    // Portugal
    // next city
    // Russia
    
    println()

    countriesSequence  = createCountriesSequence()
    countriesIterable = countriesSequence.asIterable()
    println(countriesIterable::class)
    println("after creating iterable") //no iteration has happenned yet (deferred execution)

    //applying an Iterable extension function performs the whole iteration (eager evaluation)
    val upperCountries = countriesIterable.map({ 
        println("mapping: $it")
        it.uppercase() 
    })
    println("after mapping")
    println(upperCountries::class.simpleName)


    // after creating iterable
    // Before
    // mapping: France
    // next city
    // mapping: Portugal
    // next city
    // mapping: Russia
    // after mapping
    // ArrayList


One more note. The iterator() method of an Iterable or Sequence has to be marked as an operator, why? This is the explanation I found in the documentation and in some discussion:

The for-loop requires the iterator(), next() and hasNext() methods to be marked with operator. This is another use of the operator keyword besides overloading operators.

The key point is that there is a special syntax that will make use of these functions without a syntactically visible call. Without marking them as operator, the special syntax would not be available.

Remember than in Kotlin (as in Python) we can not define custom operators, we can just overload the predefined ones.