Thursday 28 July 2022

functools Partial and __name__

I've recently come across a small issue with python functools and a simple solution. First, let's start with some theory.

You can get the name of a class by means of __name__. Maybe you think that's an attribute of the class (I mean, given a class Person, you could think that you have the __name__ attribute in Person.__dict__). That's wrong. __name__ is a descriptor in Person.__class__ (Person's metaclass), hence, a descriptor in Person.__class__.__dict__. Notice that if __name__ were directly an attribute in the class, you could do person_instance.__name__, something that is not possible


class Person:
	pass
Person.__name__
Out[52]: 'Person'

Person.__dict__["__name__"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [53], in ()
----> 1 Person.__dict__["__name__"]

KeyError: '__name__'

Person.__class__.__dict__["__name__"]
Out[54]: <attribute '__name__' of 'type' objects>

type(Person.__class__.__dict__["__name__"])
Out[55]: getset_descriptor

p1 = Person()

p1.__name__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [57], in ()
----> 1 p1.__name__

AttributeError: 'Person' object has no attribute '__name__'

You can also get the __name__ of a function. Again, this is not an attribute directly in each function dictionary, but a descriptor in the function's __class__.__dict__. Python functions are objects that are instances of the class "function", so it's interesting how instances of the function class has a __name__, by means of the __dict__[__name__] in the function class, but as we've seen above, a normal class like Person does not have such __name__ in its own __dict__.


def say_hi(who):
    print(f"hi {who}")
    

say_hi.__name__
Out[60]: 'say_hi'

say_hi.__dict__["__name__"]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [61], in ()
----> 1 say_hi.__dict__["__name__"]

KeyError: '__name__'

say_hi.__class__.__dict__["__name__"]
Out[62]: <

The functools module provides us with all (or most) "functional helper" functions that we can be used to from other languages. functools.partial is used to create partially applied funtions, so it's similar to JavaScript Function.prototype.bind. Interestingly, functions.partial does not return another function (a closure trapping the parameters being bound and the original function itself), but an instance of the functools.partial class (as such class has a __call__ method, that instance is callable and is mainly like a function to us). The problem I've found with that partial object is that it lacks a __name__ attribute.


def format_message(wrap_str, msg):
    return f"{wrap_str}{msg}{wrap_str}"

format_with_x = functools.partial(format_message, "x")
print(format_with_x("Hi"))

print(type(format_message))
print(format_message.__name__)

print(type(format_with_x))
try:
    print(format_with_x.__name__)
except BaseException as ex:
    print(f"Exception: {ex}")

# xHix
# <class 'function'>
# format_message
# <class 'functools.partial'>
# Exception: 'functools.partial' object has no attribute '__name__'


I've found here that functools provides a fix for this, the functools.updatewrapper function, that "Updates a wrapper function to look like the wrapped function". This means that we can just do this:


def create_wrapped_partial(func, *args, **kwargs):
    partial_func = functools.partial(func, *args, **kwargs)
    functools.update_wrapper(partial_func, func)
    return partial_func
	
format_with_z = create_wrapped_partial(format_message, "z")
print(format_with_z("Hi"))
print(type(format_with_z))
try:
    print(format_with_z.__name__)
except BaseException as ex:
    print(f"Exception: {ex}")
	
# zHiz
# <class 'function'>
# format_message
# <class 'functools.partial'>
# format_message	

Friday 22 July 2022

Run Function with Timeout

I've recently needed to be able to stop a function if it had not finished after a timeout period. I'm not talking about an eventloop and an asynchronous function returning an awaitable in Python (or a Promise in Javascript). Indeed, years ago I already posted about expirable promises in JavaScript. What I'm talking about is a synchronous Python function (normally doing some CPU bound calculation) that we want to stop if it has not finished after a given amount of time. I say stop its operation, not just stop waiting for it but let it run in the background uselessly consuming resources.

A solution that easily comes to mind is running the function in a separate Thread and joining that thread with the timeout option. If the timeout expires our calling code will stop blocking and continue on, but the problem is that the thread doing that costly CPU operation will continue to run and we have no way to kill it. This is something that really bothers me. Neither Python, nor .Net nor Java let you kill a thread "from outside" (I mean from another thread, what is still possible I think is killing the thread you are running in). I know that the reason for discouraging and not providing this functionality is that when you kill another thread it could be holding resources or doing something critical and killing it could mess things...) OK, but we are grown ups, so if you know that such thread can be killed withoug causing problems, you should be allowed... Well, anyway...

In order to overcome the limitations imposed by the GIL to parallelism, python provides an extremely powerful library to run your code in separate processes as easily as you run it in separate threads in other platforms: multiprocessing (and the higher level ProcessPoolExecutor in concurrent.futures). Contrary to threads, you are allowed to kill (terminate) processes, so I've implemented the timeout functionality running the function in a separate process.

So I have a create_func_with_timeout factory function, that receives a function fn and a timeout value and returns a new function that takes care or running fn in a separate process, waiting for it for a timeout time and throwing a TimeoutException if it expires. If fn itself were to throw an exception we want to have it available in the calling process, so to manage this we wrap the execution of fn in the mp_function_wrapper function. This function returns to the calling process the normal result of running fn (or an exception it that were the case) by writing it to a multiprocessing queue.

Reading the code should make it clear:


import multiprocessing
import queue
import threading
from time import sleep
from typing import Any, Dict, TextIO, List, Callable, Tuple, Optional, AnyStr, Match, cast


class TimeoutException(Exception):
    pass

def create_func_with_timeout(fn: Callable, timeout: int) -> Callable:
    """
    returns a new function that runs the provided fn in a separate process that either finishes normally or times out
    """
    def mp_function_wrapper(fn: Callable, queue: multiprocessing.Queue, *args, **kwargs):
        """
        this function has been started in a separate process
        it "returns" the result of fn to the main process by writing it to the queue
		we could take fn and timeout as closure values, but it seems more clear to pass them as parameters
        """
        try:
            result = fn(*args, **kwargs)
        except BaseException as ex:
            result = ex
        queue.put(result)
        
    def func_with_timeout(*args, **kwargs):
        """
        has the fn to run trapped as closure value
        """
        try:
            multiprocessing.set_start_method('fork')
            # multiprocessing.set_start_method('spawn')
        except RuntimeError as ex:
            print(f"set_start_method: {ex}")
        print(f"func_with_timeout {fn.__name__} {timeout}")
        response_queue = multiprocessing.Queue()
        extended_args = [fn, response_queue, *args]
        proc = multiprocessing.Process(target=mp_function_wrapper, args=extended_args, kwargs=kwargs)
        proc.start()
        try:
            result = response_queue.get(block=True, timeout=timeout)
        except queue.Empty:
            # the process is still running, so finish it off
            proc.terminate()
            raise TimeoutException()
        if result and isinstance(result, BaseException):
            raise result
        return result
  
    return func_with_timeout


def format_items(items):
    #print(f"format_items")
    results = []
    for item in items:
        sleep(1)
        results.append(item.upper())
        #print(item)
    return ",".join(results)

format_with_timeout = create_func_with_timeout(format_items, 5)

def run_test(items):
    print("- test")
    try:
        res = format_with_timeout(items)
        print(f"result: {res}")
    except BaseException as ex:
        print(f"exception: {type(ex).__name__} {ex}")


print("- starting")

run_test(["a", "b", "c"])

run_test(["a", "b", "c", "d", "e", "f", "g"])

run_test(["a", None])

print("- ending")

# - starting
# - test
# format_items
# result: A,B,C
# - test
# format_items
# exception: TimeoutException
# - test
# format_items
# exception: AttributeError 'NoneType' object has no attribute 'upper'
# - ending


I've also uploaded it to a gist

Friday 15 July 2022

Creating classes dynamically

Python classes are profoundly dynamic. We can add new methods to a class or an instance, modify the inheritance chain, switch one method implementation for another one, or dynamically create a new class... Remember from my previous post that classes are instances of the type metaclass or of a custom metaclass.

Given these methods and data attributes that we want in our dynamic class:


# initializer
def init(self, name):
    self.name = name
  
# method
def say_hi(self, to):
    return f"{self.name} says hi to {to}"
  
# class method
@classmethod
def my_class_method(cls, arg):
    return f"{cls.__name__}.{cls.my_class_method.__name__} invoked with arg: {arg}, {cls.class_attribute}"

class_items = {
    # constructor
    "__init__": init,
      
    # class data
    "class_attribute": "AAAA",
      
    # methods
    "say_hi": say_hi,
    "my_class_method": my_class_method
}

def test_dynamic_class(cls):
    p1 = cls("Francois")
    print(p1.say_hi("Antoine"))
    print(cls.my_class_method("BB"))
    print(cls.class_attribute)

We can create a class with type as its metaclass by invoking type like this:


# creating class dynamically
Person = type("Person", (object, ), class_items)

test_dynamic_class(Person)

# Francois says hi to Antoine
# Person.my_class_method invoked with arg: BB, AAAA
# AAAA


An alternative would be creating an "empty" class and adding the methods and data attributes like this:


class Person2:
    pass

for key, value in class_items.items():
    setattr(Person2, key, value)

test_dynamic_class(Person2)

# Francois says hi to Antoine
# Person2.my_class_method invoked with arg: BB, AAAA
# AAAA


If we want to dynamically create a class using a custom metaclass we have to use types.new_class. Notice that rather than passing a dictionary with the attributes, we pass a function that adds those attributes to the "class namespace"


class Meta1(type):
    pass

def fill_namespace(ns):
    ns.update(class_items)

Person3 = types.new_class(
    name="Person3",
    #bases=(B, C),
    kwds={"metaclass": Meta1},
    exec_body=fill_namespace
)

test_dynamic_class(Person3)

# Francois says hi to Antoine
# Person3.my_class_method invoked with arg: BB, AAAA
# AAAA

We can also use the alternative technique of creating an "empty class" (but with our custom metaclass): class Person3(metaclass=Meta1) and then adding the methods and data attributes to it.

Saturday 9 July 2022

Python Metaclasses

Metaclasses are a very interesting topic. The basic idea is In object-oriented programming, a metaclass is a class whose instances are classes.. That means that basically all Object Oriented languages have metaclasses. In C# all classes are instances of Type, in Java all classes are instances of Class... So the interesting thing comes when we can define custom metaclasses, as we can do for example in Groovy and Python.

I talked about Groovy metaclasses so many years ago. Custom metaclasses in Grooy are frequently used, while that's not the case in Python. The reason for this is that most of the features that require metaclasses in Groovy (expanding and object or class, interception...) are present in python through different means (__getattribute__, __getattr__...)

This means that custom metaclasses are rarelly necessary in Python, but anyway, I find it useful to have an (at least partial) understanding of how they work and their relation with callables and the attribute lookup mechanism.

This is probably the most in depth article I've found about metaclasses. I can not say I've fully understood it... but with it and other readings I think I have the basic idea of how things work. Let's go

type
Classes and functions are objects. object is an instance of type. type is an instance of type and inherits from object. A class is an instance of type. A metaclass is an instance of type and inherits from type. To find the metaclass of a class A, we just do type(A).

Callables
When python comes across this: x() it checks what class x is an instance of, and will call the __call__ method in that class with x as first parameter.

Creation of an Instance of a class
When creating an instance of a class A (a = A()) it invokes its metaclass .__call__ (so for "normal" classes it's type.__call__ and for classes with a custom metaclass it's CustomMetaclass.__call__). Then type.__call__(cls, *args, *kwargs) basically does this: invokes cls.__new__(cls) (that unless overriden is object.__new__) and with the returned instance object it invokes cls.__init__(instance)

Creation of a class
When creating a class A (class A:) it's like doing A = type("A", bases, attributes). So it invokes __call__ in type's metaclass, that happens to be type itself, so type.__call__(type). Then this invokes type.__new__ and type.__init__ (by the way, in case you are wondering whether type.__new__ is just the same as object.__new__, no, they are not)

Custom metaclass
- When creating a class with a custom metaclass (class A(metaclass=MetaB):), we are creating an instance of MetaB, so we end up calling MetaB.__new__ and MetaB.__init____.
- When creating an instance of a class A that has a metaclass MetaB (a = A()), this calls MetaB.__call__ (that could have been overriden in MetaB, or otherwise will be type.__call__)

From all the above, we can get to the 2 main uses of metaclasses:
- Defining __new__ and/or __init__ in the metaclass we can manage how instances of that metaclass (classes) are created. - Defining a custom __call__ method in the metaclass we can manage how instances of classes of that metaclass are created (singleton?).

This article makes a good read about the above. Just to make the signatures clear:


class MyMeta(type):
    def __new__(self,name,base,ns):
        print("meta new")
        return type.__new__(self,name,base,ns)
    def __init__(self,name,base,ns):
        print("meta init") self.cloned = False
        type.__init__(self,name,base,ns) class 

metaclasse and method lookup

Metaclasses do not add extra layers to the attribute lookup mechanism. Let's say we have "class A" and "a = A()". When looking up an attribute at1 in object a (a.at1), python searches first in a's __dict__ (or slots) and if not there, it'll check what class a is an instance of (that is, A), and will search in A's __dict__ and in that of its base classes. It will not use A's metaclass in this lookup mechanims at all.

If we search an attribute in a class (A.att1), python searches first in A.__dict__, then as it sees that is an instance of type (or a derived metaclass) or maybe just because it sees that it has a __bases__ attribute, it checks in the __dict__ of its parents. If not there, as A is an instance of type, it will search in type.__dict__. So yes, in this case it's using the metaclass, but just because it's the first "instance of", but it does not go deeper with the "instance of" of "instance of" of...

metaclasses and inheritance Custom metaclasses are inherited from your base classes. With multiple inheritance it becomes a bit complex, it's perfectly explained in the superb article that I mentioned above.

Immutable
You can not switch the metaclass of a class for a different metaclass once it has been created (and contrary to what some posts say, it makes sense to want to do that, as changing the metaclass of a class can change how instances of the class are created). However, there's an easy workaround, similar to what I do in my previous post. Create a new class that inherits from the one we want to modify and assign it the new metaclass. I mean:


class Meta1(type):
	pass
	
class A(metaclass=Meta1):
	pass

class Meta2(type):
	pass	

# now, let's "change" the metaclass of A from Meta1 to Meta2

class A2(A, metaclass=Meta2):
	pass
	

And then use A2 rather than A for creating new instances. We've previously seen that for already existing instances the metaclass of it's class does not play any role, so not being able to change it does not pose any problem.

Wednesday 6 July 2022

Intercept instance access

As of lately I've been reviewing some Groovy code, mainly to remember how metaclasses are used in Groovy (they work quite differently from Python). Both instances and classes have a metaclass property, and among other things they can be used to intercept access to a specific instance or to all instances of a class. You can check it here



class InterceptionThroughMetaClassTest extends GroovyTestCase {

    void testPOJOMetaClassInterception() {
        String invoking = 'ha'
        invoking.metaClass.invokeMethod = { String name, Object args ->
            'invoked'
        }

        assert invoking.length() == 'invoked'
        assert invoking.someMethod() == 'invoked'
    }

    void testPOGOMetaClassInterception() {
        Entity entity = new Entity('Hello')
        entity.metaClass.invokeMethod = { String name, Object args ->
            'invoked'
        }

        assert entity.build(new Object()) == 'invoked'
        assert entity.someMethod() == 'invoked'
    }
}


In Python we don't use metaclasses for that. We define a __getattribute__ method (you can check this previous post) in one class and this way we intercept the access to all instances of that class. So. what if we wanted to intercept the access only to one specific instance? Defining __getattribute__ at the instance level won't work.

But given Python's versatility, there's an easy trick to achieve just that. Given an instance that we want to intercept we can create a new class that inherits from that instance's class. Add a __getattribute__ method to that class, and then change the class of the instance from the original class to the derived class that includes the __getattribute__ method. Nice!


# In Groovy we can add interception to an instance (not to a whole class) by means of adding it to the instance metaclass
# we can do something like that in Python by creating a new class that inherits the original class of the instance, and changing the class of the instance
# in that new class we implement __getattribute__

class Person:
    def __init__(self, name):
        self.name = name

    def say_hi(self, person):
        #print("inside say_hi")
        return f"Hi person, I'm {self.name}"


def add_interception_to_instance(obj, interceptor_fn):
    class Interceptor_class(obj.__class__):
        def __getattribute__(self, name):
            interceptor_fn(self, name)
            return object.__getattribute__(self, name)
    obj.__class__ = Interceptor_class

p1 = Person("Xuan")
print(p1.say_hi("Francois"))
# Hi person, I'm Xuan

def logger_interceptor_fn(obj, attr):
    print(f"intercepting access to {attr}")

add_interception_to_instance(p1, logger_interceptor_fn)
print(p1.say_hi("Francois"))
#interception happens:
# intercepting access to say_hi
# intercepting access to name
# Hi person, I'm Xuan
print(isinstance(p1, Person))
# True

# other instances of Person are not intercepted
p2 = Person("Iyan")
print(p2.say_hi("Francois"))
# Hi person, I'm Iyan

I've uploaded it to a gist