Python enums

Every now and then I go through the same basic doubts with enums in Python (how did I do that?, can I do that?) so it sounds sensible to write here those basic things (though I guess won't be more useful than what you can easily find at here). Python enums is another example (like abstract classes) of clever use of Metaclasses to provide features that in other languages have to be managed especifically by the compiler and extra syntax.

We create enums by inheriting from the enum.Enum class, that has EnumType (in older versions it was called EnumMeta) as metaclass. Then, when you define an enum by inheriting from Enum (let's say: class Color(Enum)), EnumType becomes its metaclass, and the __new__ method in the EnumType metaclass magically takes care of traversing the attributes that you have defined in your class (let's say: RED, BLUE...) and assigns to them instances of Color with the corresponding value. You can check the source code and see that it uses for that intermediate instances of the _proto_member class.

From the documentation:

The EnumType metaclass is responsible for providing the __contains__(), __dir__(), __iter__() and other methods that allow one to do things with an Enum class that fail on a typical class, such as list(Color) or some_enum_var in Color. EnumType is responsible for ensuring that various other methods on the final Enum class are correct (such as __new__(), __getnewargs__(), __str__() and __repr__()).

Thanks to having those dunder methods in the metaclass we can iterate an enum class, use the "in" operator and so on.

I guess we can think of an enum as a class with a limited number of instances, each one having a name and a value and where each instance is accessible as a static attribute of the class. Given a Query enum:

class Query(Enum):
    SELECT = "select"
    INSERT = "insert"
    UPDATE = "update"
    DELETE = "delete"
    
my_query = Query.INSERT
print(my_query.name) # INSERT
print(my_query.value) # insert

We can obtain the enum member corresponding to a name like this

# Obtain an enum member from its name:
ins1 = Query["INSERT"]

And the enum member corresponding to a value like this

# Obtain an enum member from its value:
ins2 = Query("insert")

And remember that we have a unique member for each value, so:

print(ins1) # Query.INSERT
print(ins2) # Query.INSERT
print(ins1 == ins2) # True
print(ins1 is ins2) # True

The operations above will cause an error if the name or value do not exist, so we should better write something like this:

try:
    ins1 = Query["INSSSEERRRRT"]
except KeyError as ex:
    print(f"error: {ex}")
    
# or:
name = "INSSSEERRRRT"
ins1 = Query[name] if any(name == query.name for query in Query) else None
print(f"ins1: {ins1}")

# ---------------------------

try:
    ins2 = Query("insssseeerrtt")
except ValueError as ex:
    print(f"error: {ex}")

# or:    
value = "insssseeerrtt"
ins2 = Query(value) if any(value == query.value for query in Query) else None
print(f"ins2: {ins2}")
    

The above makes me wonder why there is not a pair of static methods in the Enum class, something like names() and values() that would return me just that, the possible names and values of an enum.

The standard Python enums are enough in most cases, but if you want something more advanced like the Java/Kotlin enums, where we can have multiple attributes as values and have instance methods (the archetypical Java Planets example), you can use the powerful aenum module (advanced enums), that just implements the Planets example.

Multiple "this" in Kotlin

I wrote a post some months ago about the use of this in different languages. I explained there how Kotlin comes with this surprising Qualified this feature that allows access to the this of enclosing scopes (each of those this is the original receiver in that scope). Summarizing:

If this has no qualifiers, it refers to the innermost enclosing scope. To refer to this in other scopes, label qualifiers are used

I mentioned in that article that I do not particularly appreciate the feature in C#, Java, Kotlin of having an implicit this. I find it confusing and prefer to use an explicit this. To my surprise and greater confusion, I've learnt that Kotlin allows for multiple implict this. You can have access to the this of enclosing scopes (for which you would use a qualified this) in an implicit way, skippig the "this@scope". Let's see an example:

class Book (var title: String) {}

class Person (var name: String) {
    fun doTest(book: Book) {
        val fn: Book.(String) -> Unit = { how ->
            // I have access to 2 implicit "receivers": Book and Person
            println("${name} is reading ${title} with ${how}")
        }
        book.fn("interest")
    }
}

fun main() {
    val p1 = Person("Francois")
    p1.doTest(Book("Atomka"))
    //
}

The function with receiver fn, has access to 2 implict this. One, an instance of Book, for its receiver, and another, an instance of Person, for the receiver of the doTest member function. We can rewrite the above function using explicit, qualified, this:

class Person (var name: String) {
    fun doTest(book: Book) {
	val fn2: Book.(String) -> Unit = { how ->
            println("${this@Person.name} is reading ${this.title} with ${how}")
        }
        book.fn2("interest")
    }
}

If you wonder how the function has access to those this of other scopes, well it's basically the same as how closures are implemented. The function is desugared into a class that has those this of other scopes (and other variables that it could be trapping in a closure) as properties.

This feature is particularly useful for Type-safe builders for DSL's.

Apart from nested functions having access to the this of the enclosing scopes, there's another pretty odd case where a function has access to multiple this, when an extension function is declared as a member of another class. You can read about it here

You can declare extensions for one class inside another class. Inside such an extension, there are multiple implicit receivers - objects whose members can be accessed without a qualifier. An instance of a class in which the extension is declared is called a dispatch receiver, and an instance of the receiver type of the extension method is called an extension receiver.

BrokenProcessPool Exception

Using some code at work pretty similar to this code from last December I've come across an exception that was confusing me quite a lot. So I have some code more or less like this:

async def parse_blocks(blocks):
	parser = BlockParser()
	pending_blocks = []
	for block in blocks:
		future_result = executor.submit(parser.parse, block, 0)
		pending_blocks.append(future_result)) 
	while len(pending_blocks):
		done_blocks, pending_blocks = await asyncio.wait(pending_blocks, return_when=asyncio.FIRST_COMPLETED)
		for done_block in done_blocks:
			#this is equivalent to the below: parsed_block = await done_block
			parsed_block = done_block.result()
			# do whatever
	print("all blocks parsed")

The parser.parse function that I submit to the pool has a try-except wrapping its content, I mean, something like this:

class Parser:
	def parse(self, block):
		try:
			#whatever
			result = NormalResult()
		except Exception as ex:
			result = FailedResult()
		return result
				

So there's no reason for a process in the process-pool of the ProcessPoolExecutor to crash. But to my surprise the main process was crashing in parse_blocks(), with a BrokenProcessPool exception happening in parsed_block = done_block.result(). When you access the result of a "rejected" Future (OK, yes, resolved-rejected is JavaScript terminology, I mean a Future that does not complete with a result, but with an exception), the exception that you have set with set_exception() is thrown. But, how can that Future finish with an exception if as I've said the code (running in the Process-Pool) that will "resolve-reject" that Future is in a try-except?

Well, that can happen if someone kills that child process. I guess the ProcessPoolExecutor detects that one of its process has died and sets an exception in the corresponding Future (that then is propagated to the asyncio.Future, remember from my december post that I'm wrapping the concurrent.futures.Future in an asyncio.Future). That way when you try to access the Future's result you get an exception.

As for who was killing that pool process another previous post comes into play. In some occasions some of the tasks that I submit to the pool involves some massive processing, and we end up using all the RAM and swap. At that point the OOM Killer comes into play and kills the process that more memory is using, that is one of the processes of the pool. Checking the kernel ring buffer with dmesg I could find an "Out of memory: Kill process" that corresponded with one of the processes of the Pool (I was writing to my application log the id's of these processes)

While writing this post I've been thinking again about why we have 2 kinds of Futures: concurrent.futures.Future and asyncio.Future. Given that the asyncio.Future is intended for use in an eventloop, you can not block on it to get a result. When you call to result() on it if the Future has not completed yet you'll get an exception. This is different in concurrent.futures.Future, where the result() method will block waiting for the Future to complete.

Python, abc's vs Protocols

In languages like Kotlin or C# the differences between interfaces and abstract classes are quite clear. Both constructs have different features. Interfaces can not have state (save using very arcane Kotlin techniques) but abstract classes can. We can implement multiple interfaces but only extend one abstract class. In the past interfaces could not have "real" methods, just method declarations, but kotlin and modern C# provide default interface methods. Because of the multiple inheritance from interfaces but single inheritance from classes there's normally a semantic difference. Interfaces normally define capabilities (and a class can have multiple capabilities by implementing multiple interfaces), while using an abstract class implies an is arelation. This said, I've worked on projects where the architecs had defined interfaces without default methods, that were not intended for multiple inheritance (they were not capabilites, they were a "main responsability") and I think pure abstract classes would have made more sense.

In Python, given its dynamic nature and that it features multiple inheritance from classes, it didn't seem necessary to add interfaces to the language. I think we can say that abstract classes in Python (abc's, that were not initially present either) play the role of both Abstract classes and interfaces in other languages. Comparing the different capabilities of Python, Kotlin and C# the only reason that occurs to me for having a specific interface mechanism in Python would be for having a restricted form of abc's, lacking state. Well, there's < href="https://pypi.org/project/python-interface/">a module in pip for declaring interfaces, but reading the documentation on how they differ from abc's, the motivation is throwing errors earlier and making the methods signature part of the contract...

As we've seen in a previous post, a few releases ago Python introduced a new feature, Protocols. The main motivation was adding structural typing, but when you see the non basic examples and learn that they are a particular form of abc´s, it seems like Protocols are abc's with extra powers (structural typing) and that unless that you don't want the structural typing we should move from abc's to Protocols. Well, in python3.10 (and below) there was an important difference, protocols could not have an __init__ method, which means that they could not have useful state (you could add them attributes when executing other methods, but in most use cases state is needed at construction/inizialitation time). This lack of state indeed made them feel like interfaces. This limitation seemed more like a byproduct than like an intended feature. It was the way to prevent Protocols from being instantiated. This was implemented by replacing the __init__ method that you could have defined in the Protocol with an "empty" _no_init method. Problem with this was that it not only prevented you from instantiating the protocol (which is good) but prevented you from defining common state and initialization logic that could be invoked from child classes./p>

This limitation made sense when using protocols in the basic way, just for structural typing, but when you use them combining structural typing an abc's features, decorating the protocol methods with @abstractmethod, this __init__ replacement is not necessary, as having abstractmethods will already prevent instantiation of the protocol (same as they prevent instantiation of abstract classes).

You can read this stackoverflow discussion about the replacement of __init__ for _no_init and how it's considered by many as a problem

In python 3.11 this thing of __init__ being replaced by _no_init no longer happens (there's a long discussion about it here), so you can write code like this:

from abc import abstractmethod
from typing import Protocol

class Formatter(Protocol):
    # in python3.10 __init__ gets replaced by _no_init, so you can not instantiate  a protocol
    # but in python3.11 it's no longer the case
    def __init__(self, name):
        print("Formatter.__init__")
        self.name = name

    # If I don't declare it as abstract I'll be able to instantiate this Protocol, that makes not sense
    # so declare it as @abstract, and so I'll get the "Can't instantiate abstract class Formatter with abstract method format"
    @abstractmethod
    def format(self, txt):
        pass


class WrapFormatter(Formatter):
    def __init__(self, name, wrap):
        print("WrapFormatter.__init__")       
        # the __init__ method in the Protocol has been replaced by that _no_init thing
        super().__init__(name)
        self.wrap = wrap

    def format(self, txt):
        return f"{self.wrap}{txt}{self.wrap}"


try:
    f = Formatter("abstract formatter")
except Exception as ex:
    print(f"Exception: {ex}")
    # as I have an abstract method (format) I get: "Can't instantiate abstract class Formatter with abstract method format"

f1 = WrapFormatter("wrapper", "||")
# it prints:
# WrapFormatter.__init__
# Formatter.__init__
print(f1.format("Francois"))

With all the above my conclusion is that unless that for some reason you don't want the structural typing feature, we should use Protocol's instead of abc's.

Sorting and Comparison

Comparison for sorting/ordering (that is, determining if one value is less than, equal to or greater than another value) in Python works slightly different from what I was used to in other languages. For example in C# and Kotlin the design is almost the same, so let's see that first.

Classes in C# can implement the IComparable interface to define how to sort them. Additionally you can define comparer objects (implementing the IComparer interface), so that you can define different comparison strategies for a same class. The CompareTo or Compare methods of those interfaces return an integer to indicate if one value is less, equal or greater. Basically the same goes for Kotlin, where you have the Comparable interface and the Comparator interface, which methods also return an integer.

So When we want to sort/order a collection in C# (with the order method) we have 2 options: either our collection is made up of IComparable objects, or we provide an IComparer object. There's another sorting method, OrderBy, that receives a function that for each element in the collection returns a "key", that is an IComparer object (this is a bit odd to me, it would seem more natural returning an IComparer).

The philosophy in Kotlin is almost the same (but it allows both sorting in place, with sort, and returning a new collection with sorted). So Collections have a sort method that can be used for sorting a collection of Comparable items, or any collection by providing a Comparer object. We also have a sortBy method that receives a "selector" function that for each object in the collection returns a Comparable object. As for operator overloading, the different comparison operators work on Comparable objects invoking the compareTo method.

In Python we make objects comparable by defining the Rich Comparison operators. Well, that makes a lot of methods, but indeed for sorting your objects you only need to define the __lt__ method, as explained in the documentation.

The sort routines use < when making comparisons between two objects. So, it is easy to add a standard sort order to a class by defining an __lt__() method

There are some advanced cases (not just sorting comparisons) where defining the additional Rich methods is useful. Apart from its usage in numpy, I've read somewhere that SqlAlchemy does some use of them.

Same as in kotlin we can sort in place (using the sort method) or return a new collection (using the sorted function). If the collection contains comparable objects (implementing the __lt__ method) that's all, else we'll have to provide a key function (aka selector). This function receives a collection item and must return a comparable object (so it's like the sortBy - selector in Kotlin). I've never needed nothing more than this in Python, but if you compare it with C# - Kotlin where you can provide a Comparer object (apart from the key-selector function), it could seem like we are missing something. It would seem as if a Comparer object/function could better deal with complex comparison scenarios (for example compare based on several properties and several combinations of values and ranges in those properties). No, not really, but let's clarify this.

Let's say I have a Country class that I want to compare by default based on its population. It's just a matter of adding a __lt__ method to the class:

class Country:
    def __init__(self, name: str, population: int):
        self.name = name
        self.population = population
    
    def __lt__(self, other: "Country"):
        return self.population < other.population

countries = [
    Country("Germany", 80),
    Country("Spain", 48),
    Country("France", 68),
    Country("Russia", 150),
    Country("Portugal", 10),
    Country("China", 1000),
]

print("standard sorting (by population):")
sorted_countries = sorted(countries)
print([country.name for country in sorted_countries])

#standard sorting (by population):
#['Portugal', 'Spain', 'France', 'Germany', 'Russia', 'China']
       

Now let's say I want to sort it by name. OK, it's just using as key function one that returns the name.

print("sorting by name:")
sorted_countries = sorted(countries, key=lambda x: x.name)
print([country.name for country in sorted_countries])

#sorting by name:
#['China', 'France', 'Germany', 'Portugal', 'Russia', 'Spain']

And now the interesting part. I want to sort it by name, but giving the maximum weight to the France name, so it's not just a matter of returning as sorting key one property of the object. This could seem more natural to be managed with a sort of Comparer object/function, but we can do just the same with a key/selector function that returns a comparable object that implements the comparison logic that we would have set in the Comparer object. Let's see:

class FrenchNameCentricKey:
    """Think of this Key as a Comparer object that compares Country objects in an "odd" way"""
    def __init__(self, country: Country):
        self.country = country

    def __lt__(self, key: "FrenchNameCentricKey") -> bool:
        if self.country.name == "France" and key.country.name == "France":
            return False
        if self.country.name == "France":
            return False
        if key.country.name == "France":
            return True
        return self.country.name < key.country.name
        
print("French centric sorting:")
sorted_countries = sorted(countries, key=FrenchNameCentricKey)
print([country.name for country in sorted_countries])       

#French centric sorting:
#['China', 'Germany', 'Portugal', 'Russia', 'Spain', 'France']  	

The thing to bear in mind is that a comparer object/function receives 2 objects to compare, while a key-selector function returns a "comparable" object. This "comparable" object will get its comparison method invoked passing it over another instance of another "comparable" object.

In the past Python used compare functions (following the 1, 0, -1 school of thought) rather than key/selector functions. Because of that Python provides a cmp_to_key function that transforms a compare function into a key function. At first the idea seemed like magic to me, but with what I've explained above the implementation seems pretty clear to me now (notice that it returns a class not a function).

def cmp_to_key(mycmp):
    """Convert a cmp= function into a key= function"""
    class K(object):
        __slots__ = ['obj']
        def __init__(self, obj):
            self.obj = obj
        def __lt__(self, other):
            return mycmp(self.obj, other.obj) < 0
        def __gt__(self, other):
            return mycmp(self.obj, other.obj) > 0
        def __eq__(self, other):
            return mycmp(self.obj, other.obj) == 0
        def __le__(self, other):
            return mycmp(self.obj, other.obj) <= 0
        def __ge__(self, other):
            return mycmp(self.obj, other.obj) >= 0
        __hash__ = None
    return K