Intercept instance access

As of lately I've been reviewing some Groovy code, mainly to remember how metaclasses are used in Groovy (they work quite differently from Python). Both instances and classes have a metaclass property, and among other things they can be used to intercept access to a specific instance or to all instances of a class. You can check it here

class InterceptionThroughMetaClassTest extends GroovyTestCase {

    void testPOJOMetaClassInterception() {
        String invoking = 'ha'
        invoking.metaClass.invokeMethod = { String name, Object args ->
            'invoked'
        }

        assert invoking.length() == 'invoked'
        assert invoking.someMethod() == 'invoked'
    }

    void testPOGOMetaClassInterception() {
        Entity entity = new Entity('Hello')
        entity.metaClass.invokeMethod = { String name, Object args ->
            'invoked'
        }

        assert entity.build(new Object()) == 'invoked'
        assert entity.someMethod() == 'invoked'
    }
}

In Python we don't use metaclasses for that. We define a __getattribute__ method (you can check this previous post) in one class and this way we intercept the access to all instances of that class. So. what if we wanted to intercept the access only to one specific instance? Defining __getattribute__ at the instance level won't work.

But given Python's versatility, there's an easy trick to achieve just that. Given an instance that we want to intercept we can create a new class that inherits from that instance's class. Add a __getattribute__ method to that class, and then change the class of the instance from the original class to the derived class that includes the __getattribute__ method. Nice!

# In Groovy we can add interception to an instance (not to a whole class) by means of adding it to the instance metaclass
# we can do something like that in Python by creating a new class that inherits the original class of the instance, and changing the class of the instance
# in that new class we implement __getattribute__

class Person:
    def __init__(self, name):
        self.name = name

    def say_hi(self, person):
        #print("inside say_hi")
        return f"Hi person, I'm {self.name}"


def add_interception_to_instance(obj, interceptor_fn):
    class Interceptor_class(obj.__class__):
        def __getattribute__(self, name):
            interceptor_fn(self, name)
            return object.__getattribute__(self, name)
    obj.__class__ = Interceptor_class

p1 = Person("Xuan")
print(p1.say_hi("Francois"))
# Hi person, I'm Xuan

def logger_interceptor_fn(obj, attr):
    print(f"intercepting access to {attr}")

add_interception_to_instance(p1, logger_interceptor_fn)
print(p1.say_hi("Francois"))
#interception happens:
# intercepting access to say_hi
# intercepting access to name
# Hi person, I'm Xuan
print(isinstance(p1, Person))
# True

# other instances of Person are not intercepted
p2 = Person("Iyan")
print(p2.say_hi("Francois"))
# Hi person, I'm Iyan

I've uploaded it to a gist

Create Child Instance from Parent Instance

Recently I was thinking for a while about how better design this case (in python). I have a Base class and a Child class and I want to create a Child instance from an existing Base instance. I ended up with this design

import json

class City:
    def __init__(self, name, population):
        self.name = name
        self.population = population
        self.neighbourhoods = []

    def add_neighbourhood(self, name):
        self.neighbourhoods.append(name)

class FrenchCity(City):
    def __init__(self, name, population, urban_area_population):
        super().__init__(name, population)
        self._initialize(urban_area_population)

    def _initialize(self, urban_area_population):
        self.urban_area_population = urban_area_population
        self.related_cities = []

    def add_related_city(self, city):
        self.related_cities.append(city)

    @classmethod
    def create_from_city(cls, city, urban_area_population):
        # I'm doing object.__new__ rather than calling FrenchCity constructor cause in other classes 
        french_city = object.__new__(cls)
        french_city.__dict__.update(city.__dict__)
        french_city._initialize(urban_area_population)
        return french_city


city = City("Paris", 2000000)
city.add_neighbourhood("Belleville")
f_city = FrenchCity.create_from_city(city, 12000000)
f_city.add_related_city("Saint Mande")
print(json.dumps(f_city, indent=4, default=lambda x: x.__dict__))

So as you can see I have added a create_from_[parent] static method to the child class. In it I assign the data in the parent instance __dict__ to the child instance __dict__. Yes, I know that direct access to __dict__ should be avoided as it's mainly an implementation detail, but at the same time it's well known and I don't think it's going to change in the near future.

The other interesting part in the create method is that rather than invoking the child constructor, I'm calling object.__new__ (that's also why I've split the initialization functionality between the normal __init__ method and an _initialize method) rather than the normal Child "constructor" (that ends up calling both __new__ and __init__). In this particular class I could use that "constructor" cause it's clear what parameters to pass from the parent instance fields, but this is not always the case, so the generic way is to create an instance of the child class using __new__, then initialize the child properties with an _initialize method (that we also use when creating the instance from scratch rather than from an existing parent instance), and then set the properties from the parent instance using __dict__

There's a gotcha. We could have a class that were using __slots__ rather than __dict__. Adapting to that should not be a big deal, but I would have to review how __slots__ work.

Measuring Time

In some recent script at some point I wanted to measure the time consumed in different sections of code (in a basic manner, it's not profiling information to store somewhere or anything, just to print on screen and probably end up removing it). I ended up with this:

start = time.time()
# do things
print(time.time()-start)
start = time.time()
# do something
print(time.time()-start)
start = time.time()
# do something
print(time.time()-start)
start = time.time()

Those 2 lines for each time I want to print and start the next "snapshot" add quite a bit of noise to the code. Oddly, I did not find an immediate, ready-made solution for this. so I came up with a nice use for a generator. We can have a generator function like this:

def elapsed_generator_fn():
    start = time.time()
    yield # we need a first next() call to perform this initialization
    while True:
        now = time.time()
        elapsed = now - start
        start = now
        yield elapsed

And write our code this way:

    elapsed = elapsed_generator_fn()
    # init it
    next(elapsed)

	# do something
    print(f"elapsed: {next(elapsed)}")

	# do something
    print(f"elapsed: {next(elapsed)}")

	# do something
    print(f"elapsed: {next(elapsed)}")

In the above code the first next() call is for initialization (setting the initial start value). It looks a bit odd in the code, so we could fix it by adding a function that creates the generator-iterator, does this init call, and returns it.

def elapsed_generator_factory():
    generator = elapsed_generator_fn()
    next(generator)
    return generator
	
elapsed = elapsed_generator_factory()
# do something
print(f"elapsed: {next(elapsed)}")

# do something
print(f"elapsed: {next(elapsed)}")

# do something
print(f"elapsed: {next(elapsed)}")

As I've said, it seems to me like a cool use for generators. If we want something more complete (that gives us the total time, that stores the different snapshots...) we should move on and end up with a simple class like this.

class Elapsed:
    def __init__(self):
        self.initial_start = None
        self.snapshots = {}

    def start(self):
        self.initial_start = time.time()
        self.start = self.initial_start

    def take_snapshot(self, name):
        now = time.time()
        self.snapshots[name] = now - self.start
        self.start = now
        return self.snapshots[name]
    
    def total(self):
        return time.time() - self.initial_start

    def retrieve_snapshots(self):
        return self.snapshots
		
    elapsed = Elapsed()
    elapsed.start()
    time.sleep(2)
    print(f"elapsed: {elapsed.take_snapshot('first')}")

    time.sleep(1)
    print(f"elapsed: {elapsed.take_snapshot('second')}")

    time.sleep(3)
    print(f"elapsed: {elapsed.take_snapshot('third')}")

    print(f"total elapsed: {elapsed.total()}")

    print(f"snapshots: {elapsed.retrieve_snapshots()}")

Python Type Hints and DI

After some initial reluctance I've started to use Python type hints for any non trivial code. I'm not strict at all (usually I don't declare all types, only when they are meaningful to me), and most times I don't enable the type checker (in vscode-pylance), I just want the sort of documentation that type declarations provide, and the improved intellisense experience. Somehow type hints in python seem much better to me than using typescript over javascript.

I've found something really interesting. We know that type hints are not enforced by the Python runtime (so Python remains as dynamic as always), they are intended to be used by third party tools before you run your code. This said, the interesting discovery to me is that the type information is available at runtime in form of annotations. So while the runtime won't impose any resctrictions based on these annotations, your code can use this information. In python 3.10 it's available through inspect.get_annotations(). I guess this is how the diffeent Dependency Injection libraries that have flourished in recentyears work.

This reminds me of how Dependency Injection works in Angular with TypeScript. So far, In JavaScript we don't have a standard way (like annotations) to add information (metadata) to our objects, but there is a proposal, the reflect-metadata api, and angular-polyfill.js takes care of adding it to our browser. If you add to your tsconfig.json file the emitDecoratorMetadata option (that also needs the experimentalDecorators option), TypeScript will save type information as metadata in the generated JavaScript (it generates code that adds this information to Reflect.metadata if the API exists). Example of generated JavaScript code:

var __metadata = (this && this.__metadata) || function (k, v) {
    if (typeof Reflect === "object" && typeof Reflect.metadata === "function") return Reflect.metadata(k, v);


__decorate([
    deco,
    __metadata("design:type", Function),
    __metadata("design:paramtypes", [String]),
    __metadata("design:returntype", void 0)

As explained here TypeScript will only do this for classes, functions... that have been decorated (with any decorator, you can just create a dummy one). You have a pretty nice explanation of how DI works in Angular-TypeScript here.

Some Python Tricks

I've been writing a lot of python code these last months and there are some simple tricks that were not so obvious to me the first weeks and that now have become quite essential in my day to day. Let's see:

Dictionary access and default Initially, when getting a value from a dictionary and assigning a default value if the key was missing, I was using the "sort of" ternary operator, like this:

my_value = (my_dict["key"] if key in my_dict
	else "default")

Well, there's a much cleaner way to do this, the get method:

my_value = my_dict.get("key", "default")

First item matching condition I'm pretty used to the .Net First(predicate) and FirstOrDefault(predicate) Linq Extension Methods. It seems odd to me that python lacks a "first(predicate)" function (either builtin or in itertools), but well, I've managed to interiorize the pythonic way to do this. Use a generator expression to apply the filter (rather than a list comprehension, so that you get an iterator that lazily advances as you call it) and then call next on it.

def cities_generator():
    cities = ["Toulouse", "Paris", "Porto", "Prague", "Xixon"]
    for city in cities:
        print(f"reading: {city}")
        yield city

first_p = next(city 
    for city in cities_generator()
    if city.startswith("P")
)

That's equivalent to First(). As for FirstOrDefault(), the next() function happens to accept a default value to return if the iterator is finished. Nice:

first_o = next((city 
    for city in cities_generator() 
    if city.startswith("O")
    ), None
)

itertools.takewhile and itertools.dropwhile. These are (particularly the former) pretty useful functions. They are the equivalents to .Net TakeWhile and SkipWhile, nothing more to say.

I guess almost everyone agrees that slicing is one of the most beautiful python features. Slicing does not work with iterators, so the obvious solution is to convert to iterator to a list and apply the slicing. That's fine with small iterables, but if the iterator is going to return many items and you just need an intermediate slice, processing all of them with the list conversion is a real waste. itertools.islice comes to the rescue. Obviously, it will have to process all the previous items to those in the slice that you are requesting, but it won't process the ones that come after the slice, and furthermore it won't store in memory all the previous (not needed) elements, so you are also saving memory space, not just CPU cycles. An important notice, islice does not support negative indexes.

To get the last item in an iterator that matches a predicate I think the best that we can do is to convert the iterator to a list, create a reverse iterator with reversed, and then call next. I use the reversed() function rather than reversing the list with list.reverse() cause the second one would do extra work by traversing the whole list to create the (in place) reverse list.

last_p = next(city 
    for city in reversed(list(cities_generator()))
    if city.startswith("P")
)

Dictionary Comprehensions and groupby

Dictinary comprehensions have been part of Python (2.7 and 3.1) since 2009-2010, so more or less when my long python hiatus started, meaning that I have not discovered them until recently. You won't need them with such frequency as List Comprehensions, but every now and then you'll have a chance to make your code more cute thanks to them. You can check several use cases here

I've recently combined them with itertools.groupby, and it seems worth to me to post it here. Let's say I have several cities that I want to group them in a countries dictionary.

class City:
    def __init__(self, name, country):
        self.name = name
        self.country = country

cities = [
    City("Toulouse", "France"),
    City("Prague", "Czech Republic"),
    City("Paris", "France"),
    City("Lisbon", "Portugal"),
    City("Porto", "Portugal")
]

A first approach to group them in a dictionary would be something like this:

def traditional_approach(cities):
    countries = {}
    for city in cities:
        if not city.country in countries:
            countries[city.country] = [city]
        else:
            countries[city.country].append(city)
    return countries

countries = traditional_approach(cities)
print(json.dumps(countries, indent=4, default=lambda x: x.__dict__))

Using group_by and dictionary comprehensions we have this:

def group_by_approach(cities):
    key_fn = lambda city: city.country
    cities.sort(key=key_fn)
    city_groups = itertools.groupby(cities, key_fn)   # iterable[str, grouper]
    return {key:list(group) for key,group in city_groups}

countries = group_by_approach(cities)
print(json.dumps(countries, indent=4, default=lambda x: x.__dict__))

Maybe the second code is not clearer than the first one, but it looks so nice :-) Notice that itertools.groupby has a huge gotcha if you are used to sql group by, or .Net Linq GroupBy: you have to previously sort your list by the key that you'll use later on to group:

It generates a break or new group every time the value of the key function changes (which is why it is usually necessary to have sorted the data using the same key function). That behavior differs from SQL’s GROUP BY which aggregates common elements regardless of their input order.

Update 2022/08/17. I've found out today that there's another way to do this, leverage the setdefault dictionary method:

def traditional_approach_2(cities):
    countries = {}
    for city in cities:
        countries.setdefault(city.country, []).append(city)
    return countries

Python Lazy Object Initialization

Over the years I've posted several times about "Lazy objects" (Lazy initialization) in different languages. When implementing it in Python these days I've come up with a solution that I think is really interesting.

Of course the Lazy object is transparent. I mean, the client does not know if the object is being lazily initialized or not, "laziness" is an implementation detail that does not affect the interface. This has nothing to do with things like C#'s Lazy<T>. The class of the object that we're going to lazily initialize does not require any changes either. We design our class normally and then when we see it fit to use lazy initialization we create a lazy object from it.

Both in C# and in JavaScript I've used proxy objects to implement lazy initialization. I think this is the common approach and looks quite good to me, but it has some performance implications. Once the object has been initialized the proxy mechanism remains in place and it still checks in each property access if you already have done the initialization or not (so that you use the initialized instance or invoke the initialization method).

Python has an amazing combination of features that allows the clean lazy initialization technique that I'm going to show here:

• The __getattribute__ and __setattr__ methods. When defined in a class these methods intercept the access to properties in instancess of that class or instances of derived classes. They intercept the access both "directly" (I mean, instance.attribute) and through the getattr and setattr functions. Notice that adding these methods directly to an instance of a class rather than the class, will have no effect.
• Same as JavaScript Python allows us to create a class inside a function. This class can inherit from the value provided in one variable (so it can dinamically inherit from one class or another). We can return this dynamic class from our method for further use outside it.

I leverage these features to create lazy objects by means of creating a new class that inherits from the class of the instance that I want to behave lazily. In this class I define the __getattribute__ and __setattr__ hooks/traps, and in the __init__ method I store the parameters for later reuse. I carefully invoke the initialization method through the class rather than the instance, to avoid further triggering of the traps. What is very interesting is that once I do the object initialization, I remove both hooks from the class, so they will no longer interfere with ensuing attributes access, and hence no performance penalties.
So I have a factory function that creates instances of a dynamic _Lazy class:

def lazy(cls, *args, **kwargs):
    class _Lazy(cls):
        def __init__(self, *args, **kwargs):
            _Lazy.original_args = args
            _Lazy.original_kwargs = kwargs


        def _lazy_init(self):
            print(f"_lazy_init")
            # remove the traps so that they do not interfere in the next accesses
            del _Lazy.__setattr__
            del _Lazy.__getattribute__
            # invoke the __init__ of the "target" class
            super().__init__(*self.original_args, *self.original_kwargs)
            #change the parent class so that when we do a "type()" we no longer get "_Lazy", but the "real" class
            self.__class__ = _Lazy.__bases__[0]
            
        
        def __setattr__(self, name, value):
            print(f"setting attribute: {name}")
            #self._lazy_init() # can't do this as it will trigger the traps again
            # however, traps do not have effect on accesses to attributes through the class itself rather than through instances
            _Lazy._lazy_init(self)
            setattr(self, name, value)

        def __getattribute__(self, name):
            print(f"getting attribute: {name}")
            _Lazy._lazy_init(self)
            return getattr(self, name)
    
    return _Lazy(*args, **kwargs)

And I use it like this:

class Person:
    def __init__(self, name, age):
        print(f"{Person.__name__}.__init__")
        self.name = name
        self.age = age

    def say_hi(self, to_someone):
        print(f"{self.name} with age {self.age} says Bonjour to {to_someone}")

def test_1():
    lazy_p1 = lazy(Person, "Lazy Francois", 14)
    print(f"type: {type(lazy_p1).__name__}")
    # initialization takes place
    lazy_p1.say_hi("Xose") 
    
    # trap has been deactivated
    print(lazy_p1.name)
    lazy_p1.say_hi("Xose") 
    print(f"type: {type(lazy_p1).__name__}")

test_1()
# output:
# type: _Lazy

# getting attribute: say_hi
# _lazy_init
# Person.__init__

# Lazy Francois with age 14 says Bonjour to Xose
# Lazy Francois
# Lazy Francois with age 14 says Bonjour to Xose

# type: Person

As the icing of the cake, once the initialization is done I also change the __class__ of the instance (yes, one more Python powerful feature), to point again to the original class rather than the derived _Lazy one. This way, if we check the type with type we get the original type, meaning that after the initialization we no longer can know if the object was initialized normally or lazily.

I've uploaded the code to this gist.