Python altinstall and WSL2

When working with multiple Python versions installed on the same machine, on Windows I use pyenv, but on Linux I prefer to do an altinstall. This means downloading the Python source code and building it. It may sound scary, but it's pretty simple. Once done you'll have a new Python installation in /usr/local/bin - /usr/local/lib that will not interfere with the system/default one (the one that came "from factory" with your Linux distribution, that is used by the Operating System itself and that should not be modified) that is installed on /usr/bin - /usr/lib. Just use use python3.xx to invoke that newly installed version, and python3 to invoke the system Python.

What feels odd is that it's not something that is explained in so many places. The official Python documentation just mentions this. The more detailed instructions that I've always followed are here, and as it explains installation is that simple as this:

sudo apt install build-essential zlib1g-dev \
libncurses5-dev libgdbm-dev libnss3-dev \
libssl-dev libreadline-dev libffi-dev 

wget https://www.python.org/downloads/release/python-3xxx/Python-3.xx.x.tar.xz
tar xf Python-3.xx.x.tar.xz
./configure
make altinstall

The first step is particularly important. Python comes with python modules that use native modules, and to compile those native modules they need some -dev packages installed on your system (these source packages contain mainly C header files), otherwise the compilation of those modules will fail and you'll have an incomplete installation that will cause errors when trying to import those missing modules. If you plan to use sqlite on your system, given that the sqlite module that is part of the Python distribution depends on a native module, in order to compile it you must add this: libsqlite3-dev to the list of dependencies to install that I listed above.

My work laptop (the one provided by my employer I mean) is still a Windows one. I have no problem with that, I used to have good knowledge of Windows internals, and even now that I'm more of a Linux person (all my personal computers are Linux based) I still consider that Windows architecture is really good (though I've come to distaste the updates system, the restore points, the UI...). That said, I'm using WSL2 more and more these days. I have Python3.13 installed as an altinstall on it and it's been working perfectly fine for testing on linux stuff that I develop on Windows. The other day I went one step further and wanted to debug that code on linux. Your Windows VS Code can work with folders on your WSL2 installation just in the same way it works with code on a remote linux machine. The WSL extension works in combination with the Remote SSH extension, installing to your $HOME/.vscode-server/ folder in WSL2 the code it needs on the linux side (same as it does when working with any remote Linux server). I think all this remote development is something that a few years back one could not dream about.

With VS Code and the WSL extension combined, VS Code’s UI runs on Windows, and all your commands, extensions, and even the terminal, run on Linux. You get the full VS Code experience, including autocomplete and debugging, powered by the tools and compilers installed on Linux.

The thing is that when trying to run under the debugger my code on WSL2 I was confronted with this

debugpy/launcher/debuggee.py", line 6, in module
    import ctypes
  File "/usr/local/lib/python3.10/ctypes/__init__.py", line 8, in module
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'

Initially I was thinking it would be some problem of the debugger itself, some issue with the amazing "remote development experience" that was making it fail to find that module, but just jumping into a WLS2 terminal, opening a Python3.13 REPL and trying to import _ctypes was causing the same error. So that _ctypes module was really missing on my Python3.13 WSL2 altinstallation.

Jumping to my main Ubuntu personal laptop, with also a Python3.13 altinstallation and importing _ctypes I got:

$ python3.13
Python 3.13.0 (main, Nov  9 2024, 16:10:52) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _ctypes
>>> _ctypes.__file__
'/usr/local/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so'

lib-dynload seems to be where native modules get installed (I can see also for example the sqlite.so), si if the ctypes.so is missing is that some necessary -dev package was missing when I did my altinstall on WSL2. For the sake of experiment I decided to just copy the _ctypes___.so from my laptop to the windows WSL laptop. Doing that, I got another missing module, libffi, that is imported by _ctypes. Doing a sudo apt list --installed | grep libffi I see that there's not a libffi-dev package installed on my WSL2, so somehow when a time ago I installed the different -dev packages needed to compile Python I missed to install it (so the Python compilation could not create that libffi.so into lib-dynload), and the issue had not hit me until now. To fix the problem I installed libffi-dev, uninstalled python3.13 and did a new altinstall. It works like a charm now.

There does not seem to be an automatic mechanism to uninstall a Python version installed as altinstall (a Python install takes little space and indeed I assume that I could just have do a new install without removing the existing one and it would get correctly updated), but anyway, as explained here I removed this list of folders/files:

    directory /usr/local/lib/python3.13
    directory /usr/local/include/python3.13
    file /usr/local/lib/libpython3.13.a
    file /usr/local/lib/pkgconfig/python-3.13-embed.pc
    6 files /usr/local/bin/*3.13*

While checking this thing of the missing native module (.so) I also used these commands:
lsof -p [PID] | grep .so to see the shared objects loaded by a process (lsof was an old friend of mine)
readelf -d (this was new to me. It gives you information about an elf binary file (executable or shared object, the equivalent to a windows PE file), and among that information you can see the shared objects needed by that binary, eg:

readelf -d _ctypes.cpython-313-x86_64-linux-gnu.so
$ readelf -d _ctypes.cpython-313-x86_64-linux-gnu.so

Dynamic section at offset 0x21cf8 contains 25 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libffi.so.8]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x7000

Python Generators Close and Return Enhancement

In my previous post I mentioned how I find it strange that the value returned by generator.close() (if it returns a value) is not made available in the next call to generator.next() (or generator.send()) as the value of the StopIteration exception. A few months ago I wrote about the problems with for loops for getting access to the value returned by a generator, and provided a solution. Well, I've improved such solution to have generator object that wraps the original generator and addresses both issues. If we close the generator and it returns a value, the next time we call next() or send() such value is available in StopIteration.value. Additionally, if a generator returns a value (either if it's closed or it finishes normally) that return value is made accessible in a result attribute of our Generator wrapper. OK, much talk, show me the code:

import inspect
from typing import Generator, Any

def cities_gen_fn():
    try:
        yield "Xixon"
        yield "Paris"
        yield "Lisbon"
        yield "Bilbao"
    except GeneratorExit:
        pass    
    # return this value both if closed or in a normal execution
    return "XPLB"


# wraps a generator in an "extended generator" that stores the value returned by close() to "return it" it the next call to .next() or .send() 
# it also stores the returned value if the original generator returns something
# that stored return-close value is only returned as StopIteration.value in the first call to .next()-.send(), ensuing calls return StopIteration.value as None

class GeneratorWithEnhancedReturnAndClose:
    def __init__(self, generator_ob: Generator[Any, Any, Any]):
        self.generator_ob = generator_ob
        self.result = None
        self.just_closed = False
        self.closed = False
        
    def __iter__(self):
        return self
    
    def _do_iterate(self, caller: str, val: Any) -> Any:
        if self.just_closed:
            self.just_closed = False
            ex = StopIteration()
            ex.value = self.result
            raise ex
        try:
            if caller == "__next__":
                return next(self.generator_ob)
            else:
                return self.generator_ob.send(val)
        except StopIteration as ex:
            if self.result is None:
                self.result = ex.value
            raise ex
            
    def __next__(self):
        return self._do_iterate("__next__", None)

    def send(self, val):
        return self._do_iterate("send", val)       

    def close(self):
        if not self.closed:
            self.closed = True
            self.just_closed = True 
            self.result = self.generator_ob.close()
            return self.result
        
    def throw(self, ex):
        return self.generator_ob.throw(ex)

print("- getting return value after for-loop")
cities = GeneratorWithEnhancedReturnAndClose(cities_gen_fn())
for city in cities:
    print(city)
print(f"return value: {cities.result}")

print("------------------------")
print("- using next() and close()")

cities = GeneratorWithEnhancedReturnAndClose(cities_gen_fn())
print(next(cities))
print(next(cities))
print(f"closing generator: {cities.close()}")
# first iteration after closing it returns the close-value in the StopIteration.value
try:
    next(cities)
except Exception as ex:
    print(f"generator finished {ex.value}")

# next iteration returns StopIteration with value = None
try:
    next(cities)
except Exception as ex:
    print(f"generator finished {ex.value}")

print(f"return value: {cities.result}")

print("------------------------")
print("- using send() and close()")
# test now that send() also works OK

def freak_cities_gen():
    try:
        w = yield "Xixon"
        w = yield f"{w}Paris{w}"
        w = yield f"{w}Lisbon{w}"
        yield f"{w}Bilbao{w}"
    except BaseException: #GeneratorExit:
        pass    
    # return this value both if closed or in a normal execution
    return "XPLB"
 
cities = GeneratorWithEnhancedReturnAndClose(freak_cities_gen())
print(next(cities))
print(cities.send("||"))
print(f"closing generator: {cities.close()}")
# first iteration after closing it returns the close-value in the StopIteration.value
try:
    next(cities) #it's the same using next or send
except Exception as ex:
    print(f"generator finished {ex.value}")

# next iteration returns StopIteration with value = None
try:
    cities.send("|") #it's the same using next or send
except Exception as ex:
    print(f"generator finished {ex.value}")

print(f"return value: {cities.result}")



# - getting return value after for-loop
# Xixon
# Paris
# Lisbon
# Bilbao
# return value: XPLB
# ------------------------
# - using next() and close()
# Xixon
# Paris
# closing generator: XPLB
# generator finished XPLB
# generator finished None
# return value: XPLB
# ------------------------
# - using send() and close()
# Xixon
# ||Paris||
# closing generator: XPLB
# generator finished XPLB
# generator finished None
# return value: XPLB

Notice that we could inspect the call stack to get from what method we are being called and rewrite the code this way:

    def _do_iterate_slow(self, val: Any) -> Any:
        # this is very cool with the use of introspection to check the caller name, but that's pretty slow
        if self.just_closed:
            self.just_closed = False
            ex = StopIteration()
            ex.value = self.result
            raise ex
        try:
            if inspect.stack()[1][3] == "__next__":
                return next(self.generator_ob)
            else:
                return self.generator_ob.send(val)
        except StopIteration as ex:
            if self.result is None:
                self.result = ex.value
            raise ex
           
    def __next__(self):
	return self._do_iterate_slow(None)

    def send(self, val):
        return self._do_iterate_slow(val)

But other than being very cool, that stack access is rather slow, so we better avoid such technique.

Closing Python Generators

This post is about some rarely used features of Python generators (JavaScript generators are pretty similar, but with some differences that would deserve its own post).

First of all, the internals of Python generators is pretty interesting. They are quite different from C# generators or Kotlin suspend functions, where the function is converted by the compiler into a class with a "state machine method" with labels for each "suspension point" and properties for the current label and local variables. In Python, the generator object created from a generator function points to that function as such, and holds a frame object with the variables and the next instruction to run. Each time the generator function is resumed it gets this frame object (gi_frame) (rather than starting with an unitialized one) containing its state and position of its next instruction (gi_frame.f_lasti). It's very nicely explained here. We can see with this simple code that the gi_frame and the frame taken (via inspect) from the stack in the generator function execution are indeed the same object, not a copy:

import inspect

def cities_gen_fn():
    print(f"frame id: {id(inspect.stack()[0].frame)}")    
    yield "Xixon"
    print(f"frame id: {id(inspect.stack()[0].frame)}")
    yield "Paris"
    yield "Lisbon"
    yield "Bilbao"

cities = cities_gen_fn()
print(next(cities))
print(f"gi_frame id: {id(cities.gi_frame)}")
print(next(cities))

# frame id: 2550405375184
# Xixon
# gi_frame id: 2550405375184
# frame id: 2550405375184

Python generator objects have a close() method that allows us to set the generator as finished. One common use case is when looping over a generator and at some point a condition tells us to stop. Of course you can leave the loop using the break statement, but that's a bit different, break will leave the loop immediatelly, not in the next iteration, and as the generator has not been finished, we still can continue to iterate it after the loop.

def cities_gen_fn():
    yield "Xixon"
    yield "Paris"
    yield "Lisbon"
    yield "Bilbao"

print("- using break")
cities = cities_gen_fn()
for city in cities:
    if (city := city.upper())[0] == "L":
        break
    print(city)
print(next(cities))

print("- using .close()")
cities = cities_gen_fn()
for city in cities:
    if (city := city.upper())[0] != "L":
        cities.close()
        print(city)
try:
    print(next(cities))
except StopIteration as ex:
    print("generator is finished")

# - using break
# XIXON
# PARIS
# Bilbao

# - using .close()
# XIXON
# generator is finished

I can think of some situation in the past where this .close() method would have come handy. Let's say we have a main function that creates a generator and delegates on other functions certain tasks involving iterating that generator. Each of those functions could determine based on its own logic that the generator is finished, so it would close it, and then the main function would no longer invoke the remaining functions with it. Unaware of this .close() functionality I was returning a "is_finished" boolean from each of those functions.

The documentation on .close() shows that it's a quite interesting and complex beast. Raises a GeneratorExit at the point where the generator function was paused. Wow, that's quite a bit mind blowing. So it's as if when the generator function is resumed somehow the interpreter injects a raise GeneratorExit() sentence in the place where the gi_frame.f_lasti is pointing to! If the generator does not catch the exception the generator finishes (the next iteration attempt will throw a StopIteration) and the close() call returns None (that's the behaviour in the examples above). Python3.13 has introduced a new feature, the generator can catch the exception and return a value to the close() method. The main effect, finishing the generator is the same, but we have this extra of returning a value to the caller. Let's see:

def cities_gen_fn():
    yield "Xixon"
    yield "Paris"
    yield "Lisbon"
    yield "Bilbao"


def cities2_gen_fn():
    try:
        yield "Xixon"
        yield "Paris"
        yield "Lisbon"
        yield "Bilbao"
    except BaseException: #GeneratorExit:
        return "No City"
        #this returned value in case of a close() is returned by close(), but not as return value of the generator (StopIteration.value is None)


for cities_gen in [cities_gen_fn(), cities2_gen_fn()]:
    print(next(cities_gen))
    print(f"close result: {cities_gen.close()}")
    print("generator has been closed")
    try:
        next(cities_gen)
    except Exception as ex:
        print(f"Exception: {type(ex)}, value: {ex.value}")
    print("--------------------------")

# Xixon
# close result: None
# generator has been closed
# Exception: 'StopIteration', value: None
# --------------------------
# Xixon
# close result: No City
# generator has been closed
# Exception: 'StopIteration', value: None
# --------------------------

What feels a bit odd to me is that the value returned by the generator to .close() is not considered as a generator return value and made available as the .value property of the next StopIteration exception.

We have another related method, generator.throw(). It's also used to finish a generator, but throwing exceptions, for which I don't see any clear use case.

Raises an exception at the point where the generator was paused, and returns the next value yielded by the generator function. If the generator exits without yielding another value, a StopIteration exception is raised. If the generator function does not catch the passed-in exception, or raises a different exception, then that exception propagates to the caller.

I'll show some example, but honestly I don't see when this method can be useful.

def cities_gen_fn():
    yield "Xixon"
    yield "Paris"
    yield "Lisbon"
    yield "Bilbao"

cities_gen = cities_gen_fn()
print(next(cities_gen))
try:
    print(f"throw result: {cities_gen.throw(Exception())}")
    print("after generator throw")
except Exception as ex:
    print(f"Exception: {ex}")
try:
    print("next iteration attempt")
    next(cities_gen)
except Exception as ex:
    print(f"Exception in next() call: {type(ex)}, value: {ex.value}")

# Xixon
# Exception: 
# next iteration attempt
# Exception in next() call: 'StopIteration', value: None


print("--------------------------")


def cities2_gen_fn():
    try:
        yield "Xixon"
        yield "Paris"
        yield "Lisbon"
        yield "Bilbao"
    except Exception: 
        yield "Except City"


cities_gen = cities2_gen_fn()

print(next(cities_gen))
print(f"throw result: {cities_gen.throw(Exception())}")
print("after generator throw")
try:
    print("next iteration attempt")
    next(cities_gen)
except Exception as ex:
    print(f"Exception in next() call: {type(ex)}, value: {ex.value}")


# Xixon
# throw result: Except City
# after generator throw
# next iteration attempt
# Exception in next() call: 'StopIteration', value: None

Python locals(), f_locals, local namespace

The freshly released Python 3.13 mentions some updates to the locals() behaviour. Reading those notes, confirms to me (as I have outlined here) that trying to create new variables in exec()/compile() will have no effect outside of the "block" executed in exec-compile itself (reassigning an "external" variable will not have effect either) the code will always run against an independent snapshot of the local variables in optimized scopes, and hence the changes will never be visible in subsequent calls to locals(), and also opens the door to some really interesting stuff: FrameType.f_locals now returns a write-through proxy to the frame’s local and locally referenced nonlocal variables in these scopes.

Let's go a bit deeper into the above statements. Each time we execute a function, a "local namespace" object is created for that function (it's a sort of dictionary), where local variables and parameters are stored (and also free vars if the function is a closure). I guess we can think of this local namespace object as JavaScript's Activation Object. Let's see:

def create_fn():
    trapped = "aaa"
    def fn(param1):
        nonlocal trapped
        trapped = "AAAA"
        local1 = "bbb"
        print(f"fn local namespace: {locals()}")
    return fn

fn1 = create_fn()
fn1("ppppppp")

# fn local namespace: {'param1': 'ppppppp', 'local1': 'bbb', 'trapped': 'AAAA'}

As aforementioned, code executed by the exec()/compile() functions receives a snapshot of the local namespace of the invoking function, meaning that adding a variable or reassigning a variable in that snapshot will not have effect outside the exec() itself. I mean:

def declare_new_variable(param1):
    # creating a new variable or setting an existing variable in exec will not crash,, but it in the local namespace snapshot that it receives
    # but will not have effect in the original local namespace
    print(f"- {declare_new_variable.__name__}")
    # create new variable
    exec(
        "a = 'Bonjour'\n"
        "print('a inside exec: ' + a)\n"
    )
    # a inside exec: Bonjour

    p_v = "bbb"
    # assign to existing variable
    exec(
        "p_v = 'cccc'\n"
        "print('pv inside exec: ' + p_v)\n"
    )
    # pv inside exec: cccc
    
    print(f"locals: {locals()}")
    # locals: {'param1': '11111', 'p_v': 'bbb'}
    # the new variable "a" has not been created in the local namespace, and p_v has not been updated
	

And now the second part of the first paragraph, the FrameType.f_locals. I've been playing with it to learn that from a Python function we can traverse its call stack, getting references to a write-through proxy of the local namespace of each stack frame. This means that from one function we have access (read and write) to any variable in any of its calling functions (any function down in the stack), and even "sort of" add new variables. I'm using inspect.stack() to get access to the stack-chain, then freely move through it, get the stack-frame I want, and use f_locals to get that "write-through proxy" to its local namespace.

def child2():
    print("- enter child2")
    c2_v1 = "child2 v1"
    c2_v2 = 200
    print("child2")
    parent_locals = inspect.stack()[2].frame.f_locals
    print(f"parent_locals viewed from child2: {parent_locals}")
    print("modify existing parent variable, p_v1")
    parent_locals["p_v1"] = parent_locals["p_v1"] + "_modified"
    print("add variable p_v3 to parent")   
    parent_locals["p_v3"] = "extra var"   
    # remove variable this way fails:
    #del parent_locals["p_v2"] 
    # TypeError: cannot remove variables from FrameLocalsProxy
    print("- exit child2")


def child1():
    print("- enter child1")
    c1_v1 = "child1 v1"
    c1_v2 = 20
    child2()
    print("- exit child1")


def parent():
    p_v1 = "parent v1"
    p_v2 = 2
    print("before calling child")
    print(f"parent: {locals()}")
    child1()
    print("after calling child")
    # p_v1 has been updated and p_v3 has been added:
    print(f"parent: {locals()}")

    # I can see the updated value of this var
    print(f"p_v1: {p_v1}")

    #but trying to acces the new variable like this will fail:
    try:
        print(f"p_v3: {p_v3}")
    except Exception as ex:
        print(f"Exception: {ex}")


parent()

# before calling child
# parent: {'p_v1': 'parent v1', 'p_v2': 2}
# - enter child1
# - enter child2
# child2
# parent_locals viewed from child2: {'p_v1': 'parent v1', 'p_v2': 2}
# modify existing parent variable, p_v1
# add variable p_v3 to parent
# - exit child2
# - exit child1
# after calling child
# parent: {'p_v1': 'parent v1_modified', 'p_v2': 2, 'p_v3': 'extra var'}
# p_v1: parent v1_modified
# Exception: name 'p_v3' is not defined

As you can see at the end of the above code, adding new variables to a function through the f_locals has an odd behaviour. A new entry is created in the local namespace corresponding to that f_locals. We can see the variable with locals() (regardless of whether it was added by code deeper in the stack chain) but trying to access it directly by its name will fail. The new variable exists in the local namespace, but it seems as if the variable name does not exist, and yes it's just that, as explained by this post:

Functions are special, because they introduce a separate local scope. The variables inside that scope are fixed when the function is compiled (the number and names of the variables, not their values). You can see that by inspecting a function's .__code__.co_varnames attribute.
That fixed registry of variable names is what is used when names are looked up from inside the function. And that registry is not updated when you're calling exec.

FIC Xixón 2023

While going through the Program of the 2024 FICXixon edition I've realised that I had forgotten to finish and publish my post about the previous edition, indeed, publishing this kind of post 1 year later has become as much a tradition as the fact of writing a post about the festival. OK, here it goes:

One more year and one more FICXixon edition, number 61, from November 17th to November 25th, 2023. Once again I feel that sort of pride when I see how my hometown manages to organize such an excellent event, a middle size city (and a small region) in one corner of our continent, that after striving to survive to different waves of massive economical crisis and population losses, now seems to be on the way to stabilization and even recovery (IT sector, wind turbines factories, small private shipyards, some home workers moving/returning here).

As always I'm a bit constrained by work and by not feeling like going to sessions in those cinemas far away from my flat, (Yelmo in La Calzada, Laboral) and in the end I only attended to 4 sessions (I could not watch Baltimore, by Moldy and Lawlor, as when I finally made up my mind to get a ticket for it it was already sold out). I pretty nailed it with these 4 films, as all of them were good or very good

• Shoshana. Saturday 18, Teatro Jovellanos. Based on a true story, excellent, absolutely excellent, so much that I should write a separate post for it, but I think that is not happening, so I better write some lines directly here. I've watched quite a bunch of films (normally very good) dealing with the complexities of life in Israel-Palestine, from different angles and perspectives, but I think this is the first one I've watched about the (recent) beginning of all that mess. Set in the 1930's/1940's, in the British colony of Palestine, where European jews have been settling recently, escaping the fascist threat in Europe and pursuing the dream of thriving in the land of their ancestors. Many of them are left-wing, not only the ones in the Kibbutz, also those in Tel-Aviv. They dream of having their own independent state, a homeland, and they buy lands and work hard. Most of them think it will be possible to get along with Arabs, but as the film ends it's sadly clear even for those that believed in "living together" not just in "live separate and not kill each other", that coexisting with Muslims in equality (not in dhimmitude) is for the most part impossible.
• Day of the Tiger (Tigru). Wednesday 22, OCine. An interesting Romanian film inspired by true events. A woman working in a zoo is going through a bad time and in an error she lets a tiger escape the zoo. A patrol sets off to track the animal and a fucking mother fucker whose main pride and fun in life consists of hunting animals becomes the leader. There was an encounter with the director at the end, where he explained that his love for animals encouraged him to do this film. I was intrigued about where do you find wild animals like these (the tiger role was played by 2 different animals) for a film, if there is a "tiger's casting agency" :-) Well, it's a bit like that. These tigers come from Northern France, where a guy has a huge farm where he takes care of a bunch of wild animals. When the animals are "hired" you also hire the guy, as of course some expert has to handle them during the recording.
• Disco Boy. Thursday 23, Teatro Jovellanos. Very powerful French drama. A tough story. One Belarusian young man enters illegally in France and decides to join the Foreign Legion, where if he manages to serve for 3 years he will be granted with French citizenship. The way he entered in France is so hard as the way he chose to try to stay. If you know how France works it's revolting. It's revolting that for a guy with a similar culture to ours and that sure would work hard and integrate into the French society without a problem, the Far-Left (anti)French system makes this as hard as possible, while anyone coming from an incompatible culture, with a hatred and distaste for our values, and with a firm desire of profiting from our welfare system, will find all sort of support from all kind of Far-Left associations. I say "ours" in this paragraph cause though I don't have French citizen, I consider it my second home/country/identity/belonging (and I've paid taxes there for years, not like all that scum that is destroying the country).
• Matronas (Sages-femmes). Saturday 25, OCine. I think I was not particularly expectant about this film, but it was the last day of Festival and I wanted to attend one more screening, and the thing is that it was excellent. I had never thought about how intense and demanding is the work of bringing new lives to this world. This film takes us through the lives of several French women that do just that, working as midwives (sage-femme in Francais). An unattended surprise. When leaving the screening I heard some women that seemed to be midwives themselves praising how they felt reflected on the film.

Python Functions Disallowing Named Arguments

There are some builtin Python functions (written in C) that do not accept named (keyword) arguments (so they force the use of positional ones), for example (range, map, filter...) If you try to pass them a keyword argument you'll get an error: [function] takes no keyword arguments. This feels like an odd limitation to me, and furthermore, very confusing, as when looking into the documentation (for example for functions filter and map) I see no indication of this restriction, so you "learn by trying".

I've been trying to find information about the reason for this odd behaviour, and have not found much. There is this discussion that says that some builtin functions are defined with a "fast header" that does not accept keyword arguments.

This is because they use only a fast simple header ...(... PyObject *args) and therefore all named parameters are rejected automatically. (Python can never introspect into names of arguments in C source. :-)

Many other C functions have a header ...(... PyObject *args, PyObject *kwds) and they support exact list of names explicit by implementing much more complicated validation PyArg_ParseTupleAndKeywords. These names must be written to docs strings manually.

This is a bit confusing to me, but here they go some of my ideas on this (comparing it to other platforms). From what I've read the JVM has no notion of named arguments, meaning that the Kotlin compiler takes care of them when producing the bytecodes for the function call, and has a minimum performance implications at runtime:

Using named arguments produces the same bytecode as using positional arguments. However, changing the order of the arguments leads to the creation of temporary variables. This is because the arguments need to be evaluated in the order they appear, but passed to the method in the original order.

On the other hand, Python bytecode has the notion of Keyword arguments at the bytecode level, so it's the interpreter at runtime who takes care of them, which I guess has some minor performance implication. I would say that when writing native functions that you want to be as fast as possible, preventing a function from being invoked with keyword arguments provides some "nano-improvement".

Thanks to this searching I've found a surprising feature. I already knew that you can use the * symbol in your function signature to force all parameters after it to be passed as named (keyword) arguments, but I did not know that python 3.8 (PEP-570) introduced Positional Only Parameters. Using the / symbol in your function signature, parameters defined before it can only be provided as positional arguments, not as keyword ones.

At first sight it seems like an odd addition. That some builtin functins behave like that is a disturbance for me, so why would I want to disturb others by using that in some of my own functions? Well, there's a pretty good explanation here. Of the 4 reasons given there, the one that really makes this feature useful and valuable to me is the last one:

Since the parameters to the left of / are not exposed as possible keywords, the parameters names remain available for use in **kwargs:

Honestly I'd never thought about that potential problem when using variable number of keyword arguments. When you receive in your packed **kwargs an argument with a name that you already use for your other parameters you get an exception (Exception: [function] got multiple values for argument '[xxx]'), as the interpreter does not know to what parameter that named argument refers to. Let's see an example with the problem and the solution using /

def format_msg(msg: str):
    return f"[[{msg}]]"

def log_call(msg, fn, *args, **kwargs):
    print(f"{fn.__name__} {msg}")
    return fn(*args, **kwargs)
    

# no keywork argument, so it works fine
print(log_call("invoked", format_msg, "hi"))
#format invoked
#Out[11]: 'hi'

# but here we have a problem
try:
    print(log_call("invoked", format, msg="hi"))
except Exception as ex:
    print(f"Exception: {ex}")

# TypeError                                 Traceback (most recent call last)
# Input In [12], in ()
# ----> 1 log_call("invoked", format, msg="hi")
# TypeError: log_call() got multiple values for argument 'msg'


# that we can prevent by redefining the function like this:
def log_call(msg, fn, /, *args, **kwargs):
    print(f"{fn.__name__} {msg}")
    return fn(*args, **kwargs)
    

print(log_call("invoked", format_msg, msg="hi"))
# format_msg invoked
# [[hi]]'

Now that I'm aware of this potential problem in Python a further question arises, how do they manage this in Kotlin? Well, the thing is that they don't have this problem, cause kotlin supports variable number of unnamed arguments, but not of named arguments. The vararg modifier used in a function signature denotes an array of arguments, but there's not an additional modifier for denoting a dictionary of parameters (like python's **). Related to this, the spread operator * only applies to arrays, there's not an equivalent to Python's ** (packing/unpacking) for dictionaries.

Javascript Function.bind

In my previous post I talked about how in Python we have 2 different functions (functools.partial and functools.partialmethod) for using partial function application to functions or methods, due to how callables and bound methods work. A natural question is, what about JavaScript?

JavaScript comes with a Function.prototype.bind function. It allows us to bind a "this" value (thisArg) and other arguments (arg1, arg2...) to a function. When binding values to a function that does not use the "this" value (or just uses it for the 'global this') we can pass null or undefind as thisArg to bind. So Function.prototype.bind will serve us well for binding to a plain function, and for "extracting" a method from a class to use it as a function. I mean:

function format(it1, it2, it3, it4) {
    return `${it1} - ${it2} _ ${it3} . ${it4}`;
}

// function.bind works good for functions 
let fn1 = format.bind(null, "aa", "bb");
console.log(fn1("cc", "dd"));



class Formatter {
    constructor(name) {
        this.name = name;
    }

    format(it1, it2, it3, it4) {
        return `${this.name} + ${it1} - ${it2} _ ${it3} . ${it4}`;
    }
}

//and for a method that we want to "extract" from the class to use as a function 
let format2 = Formatter.prototype.format.bind({name: "Xuan"}, "aa");
console.log(format2("bb", "cc", "dd"));
 

Notice that arrow functions do not have a dynamic 'this', but what used to be called a lexical 'this' (the term seems to have fallen into disuse). They'll search for "this" in their scope chain [1]. This means that trying to bind a "this" value to an arrow function is useless, it will always search "this" in its scope chain. On the other side, binding other parameters to an arrow function is perfectly fine.

// function.bind works fine with arrow functions when binding normal parameters

let greet = (msg, fromX, toY) => `${msg} from ${fromX} to ${toY}`;
console.log(greet.name); //greet
let sayHi = greet.bind(null, "Hi");
console.log(sayHi.name); //bound greet
console.log(sayHi("Xose", "Francois"));
//Hi from Xose to Francois

// but binding "this" will not work. We get a bound function, but the "this" value will not be used by the arrow, it will continue to use the lexical this
let greet2 = (fromX, toY) => `${this.msg} from ${fromX} to ${toY}`;
let info = {msg: "Hi"};
let sayHi2 = greet2.bind(info); 
console.log(sayHi2("Xose", "Francois"));
//undefined from Xose to Francois

Using Function.prototype.bind for binding values to a method (that we'll continue to use as method, not as a function) rather than a normal function is a bit particular. So we have a class with a method that receives several parameters (apart from "this") and we want to bind some of them. Because of Function.prototype.bind expecting us to bind something as "this" we have to invoke bind passing that instance as thisArg and then attach the bound function to the instance itself. That's a bit odd, but OK for binding for a particular instance:

let formatter = new Formatter("Francois");
formatter.format = formatter.format.bind(formatter, "aa", "bb")
console.log(formatter.format("cc", "dd"));

But what if we want the parameter binding for all instances of the class, so that it works for any new instance that we create. For that our best bet is implementing our own partial function.

function partial(fn, ...trappedArgs) {
    return function(...args) {
        return fn.call(this, ...trappedArgs, ...args);
    }
}

// it works fine used with methods
Formatter.prototype.format = partial(Formatter.prototype.format, "aa", "bb")
let formatter2 = new Formatter("Francois");
console.log(formatter2.format("cc", "dd"));

//and when used with plain functions
let fn2 = partial(format, "aa", "bb");
console.log(fn2("cc", "dd"));

Python Decorator Classes applied to Methods

When writing my post about functools.partial I noticed the presence of a functools.partialmethod function to apply partial to functions that will be used as methods. I had not realised of this before, but knowing how methods are implemented (functions implement the descriptor protocol, so when a function attached to a class is accessed through an object (receiver) the __get__ of the function gets invoked and returns a bound method) the issue becomes clear. From the documentation:

To support automatic creation of methods, functions include the __get__() method for binding methods during attribute access. This means that functions are non-data descriptors that return bound methods during dotted lookup from an instance.

functools.partial does not return another function (a Function is both a callable and descriptor), but a callable object that is an instance of a class other than Function (that has a __call__ method but is not a descriptor). Because of that, you can not use it as a method cause not having a __get__ you are losing the bound-method part. That's why a separate partialmethod function is needed.

We have this same problem with decorators that have been implemented as classes rather than as functions. When you apply one of these decorators, defined lets say as class MyDecorator, you create an instance of that MyDecorator class, and that instance is a callable object. If you apply MyDecorator to a method, obtaining a callable is not enough, we also need it also to have a __get__ that returns a bound method. Taking inspiration from here and here I've set up an example:

import types

class LoggedCall:
    def __init__(self, f):
        self.func = f
        # for the __name__, __doc__, etc attributes to be set to those of the wrapped function
        functools.update_wrapper(self, f)
        

    def __call__(self, *args, **kwargs):
        print(f"{self.func.__name__} invoked")
        return self.func(*args, **kwargs)

    def __get__(self, instance, cls):
        # the reason for adding the "if instance is" is not in case we apply it to an independent function (that would work fine)
        # is in case we decide to call it through the class and not an instance (see example below)           
        return types.MethodType(self, instance) if instance is not None else self
    

class Person:
    def __init__(self, name: str):
        self.name = name

    @LoggedCall
    def say_hi(self, someone: str):
        print(f"{self.name} says hi to {someone}")


p1 = Person("Iyan")
p1.say_hi("Francois")

# it for this odd way to invoke it for which we need to add the "if instance" in the __get__
Person.say_hi(p1, "Antoine")

Notice that this example is quite a bit contrived. Rather than using a decorator class we could have used a decorator function, that feels way more natural, and needs no extra bound-method creation from our part:

def log_call(func):
    functools.wraps(func)
    def wrapper(*args, **kwargs):
        print(f"{func.__name__} invoked")
        return func(*args, **kwargs)
    return wrapper


class Person2:
    def __init__(self, name: str):
        self.name = name

    @log_call
    def say_hi(self, someone: str):
        print(f"{self.name} says hi to {someone}")


p1 = Person("Iyan")
p1.say_hi("Francois")

Notice how in both cases I'm using functools.wraps of functools.update_wrapper for the wrapper function to look like the wrapped function (attributes: __module__, __name__, __qualname__, __annotations__, __type_params__, and __doc__)

So all in all, when adding a callable object that is not a function to a class as a method we need that callable to behave like a function, returning a bound-method when being accessed. For that we have to make the class of our callable object a descriptor also, with a __get__ method that returns a bound-method (just as functions do).

While writing this post I've gone throug a couple discussions [1] and [2] about using decorator functions vs decorator classes. Even if we want to keep state, I'm much more inclined to use a decorator function returning a closure than to use a decorator class.

Zarréu, Rain and Colomines

I love rainy weather and I love rainy places, it's been like that for most of my life. For European standards Asturies is a rainy region, 1000 mms per year in the main cities, above 1500 mms in mountain areas, allegedly 2000 mms in Picos d'Europa (but there are no meteo stations in the right places to measure that). One of the rainiest places in Asturies is on its southwest corner, L.leitariegos and Degaña allegedly should be well around 1700 mms, but in years (like the last one) when rains enter more from the SW than from the NW they can be well above that. From early September 2024 to October 10th there's been a combination of NW, W and SW situations that has left 450 mms in 40 days in Degaña (Zarréu)! Wow, that's massive and the best is that those have been strong but well sustained rains allowing the soil to absorb the water and not causing any sort of destructive floodings. All this being said, in the last year I've got a fascination with that hidden corner of Asturies called Degaña.

I've never set foot there as reaching the place in public transport is very, very complex, so I've done some virtual travel via google. Degaña has a coal mining past that brought it so many jobs and prosperity that are now long gone (as in neighbouring L.laciana (Vil.lablinu) in Llión, and in the other Asturian coal mining basins "cuenques mineres"). Because of that when you take a virtual-look into Zarréu you see one part of the village made up of traditional housing and another parts consisting of housing blocks built in the 60's (I guess) to house all the population growth gifted by the coal mining activities. It's the same kind of housing blocks that you'll find all over Asturies (frequently known as "colomines"), that in the outside look pretty similar to Soviet Krushevkas.

Like those, they were built with a max height of 4 floors, to avoid the need for elevators and save money. The ones that you see in Zarréu range from 2 to 4 floors, and have not seen any sort of restoration, so they look pretty old (but with the charm of a glorious past). To my astonishment, on one side of those blocks a very nice piece of modern architecture pops out. In the past I wrote this post expressing my delighment with some modern public housing buildings that I had just found in Mieres. As I say in that post, for Asturies that kind of construction is absolutely amazing, but it's something that you can find in so many places in France (except Toulouse, of course, the "Architectural redneck capital of Europe"). But this building in Zarréu is amazing in itself, not just in comparison to what is common in Asturies. The combination of modernity (shapes and black facades) and tradition (those slate tiles and those small balconies that are like a reinterpretation of the magnificent Asturian "corredor") is just unparallel here.

After the initial shock I investigated a bit to see how such a beautifully surprise came into existence (read: [1] and [2]). They were completed in 2009, as modern public housing intended for young people and coal miners. At that time there were still coal mines going on, and though the activity was clearly in decline, I think nobody could think at that time that in 10 years the Central Government would have closed them all (in the whole country) and that 2 years later it would close all the coal-fired power stations in the country!!!??? in their stupid plan of "achieving" the fastest energetic transition in the world, with a total despite for all those that will be left behind by this pseudo-ecologic delirium. We are talking of definitive closing, with the coal mines in Degaña and Ibias closing from one day to another, and with coal-fired power stations being dismantled almost immediatelly, so that following the German approach of putting some of them back to work to try to compensate the energy crisis due to the NATO war against Russia were totally impossible... So in retrospect that building was quite probably totally unnecessary (indeed it seems a twin building was also planned, but of course the idea was abandoned), as all that area (Ibias, Degaña, L.laciana...) has suffered a massive population decline, with the few young locals having mainly moved away in their search for survival (finding a job), and I guess that probably quite a few retired coal miners have also moved (like many in Mieres and Llangréu previously did, moving to Uviéu and Xixón)

The masterminds behind this gorgeous construction are the Asturian architect Nacho Ruiz Allén and the Navarrese architect Jose Antonio Ruiz Esquiroz. The've worked together in several other projects, among them this mesmerizing viewpoint also in Asturies: Walkway-Lookout over Rioseco Mines.

Do a google search for images of the place, it's outstanding:

Create Class Instance given a String with its Name

Reflection has always fascinated me, from basic introspection to creating code at runtime, to more crazy ideas like modifying at runtime how the language behaves (how loops work, how inheritance works, how attribute lookup works). The other day, working with Python, I needed a not particularly unusual reflective feature, create instances of a class which name we have in a string. Think of a plugin system, you don't know at development time which plugins you'll have available, and at runtime you are provided with the name of a particular plugin class (a class that someone has made available in the modules search path) that has to be instantiated.

This is a pretty easy task. Classes live inside modules, so the first step is loading (if it's not already loaded) the corresponding module for that class. Once we have a reference to the module we can access to the class object using getattr() and invoke it (classes are callable) to create an instance.

# we have  dblib.oradb.py module, with an OracleConnector class in it
full_class_name = "dblib.oradb.OracleConnector"
parts = param["param_type"].split(".")
mod_name_parts, class_name = (parts[:-1], parts[-1])

# load module
mod = importlib.import_module(".".join(mod_name_parts))

# retrieve a class object
connector_cls = getattr(mod, class_name)

# invoke the "constructor" with the corresponding parameters
connector = connector_cls("geodata", "user1")

# obviously the module is now loaded in sys.modules
mod == getattr(sys.modules, "mod_name")
# True

If we know that the module for the class is already loaded and with the class added to the global namespace, we could obtain a reference to the class using eval(), but that's not a particularly good approach. I mean:

from dblib.oradb.OracleConnector import *

class_name = "OracleConnector"
connector = eval(class_name)

What about Kotlin (for the JVM) / Java? There's not an extra syntax or library in Kotlin for this, so it's just the same as in Java. While in Python modules are the "loading unit", in Java that "loading unit" is the class (classes are loaded from the file system by the Class Loader as needed). The equivalent to the 2 steps process in Python: load module and retrieve class object, is a single step in Java, get a Class object by calling Class.forName(classNameString). The next part is more complicated in Java than in Python. Python classes are callables, so we just have to invoke the class object to obtain the class instance. In Java we have to obtain the constructor function from the class, and the thing that I had almost forgotten is that due to overloading we can have multiple constructors (or multiple methods with the same name), so to retrieve the correct constructor we have to provide the types of the different parameters that we are going to use in the ensuing invokation. The main idea is this (simplest possible case, as we are using a parameterless constructor)

val classToLoad: Class = Class.forName(classNameString)
//simplest possible case, just get a parameterless constructor
val ob = classToLoad.getDeclaredConstructor().newInstance();

And now a more complex/verbose case using a constructor that expects parameters (taken from here)

class MyClass {
    public MyClass(Long l, String s, int i) {

    }
}

Class classToLoad = Class.forName(classNameString);

Class[] cArg = new Class[3]; //Our constructor has 3 arguments
cArg[0] = Long.class; //First argument is of *object* type Long
cArg[1] = String.class; //Second argument is of *object* type String
cArg[2] = int.class; //Third argument is of *primitive* type int

Long l = new Long(88);
String s = "text";
int i = 5;

classToLoad.getDeclaredConstructor(cArg).newInstance(l, s, i);

I think I had never thought until today about the fact that though Kotlin supports method overloading we use it less frequently than in Java, and that's just because optional parameters eliminates some of the cases where we would be using method overloading.

Partial Function Application in Python

I won't explain here the difference between partial function application and currying, there are millions of articles, posts and discussions about it everywhere. As well explained by wikipedia, partial application refers to the process of fixing a number of arguments of a function, producing another function of smaller arity. JavaScript comes with function.bind, and Python comes with functools.partial. I rarely use it though. Most times I just need to bind a parameter to a function expecting few parameters, and for that I tend to just use a lambda, I mean:

from functools import partial

def wrap_msg(wrapper: str, msg: str) -> str:
    return f"{wrapper}{msg}{wrapper}"

wrap1 = partial(wrap_msg, "||")
print(wrap1("aaa"))

wrap2 = lambda msg: wrap_msg("||", msg)
print(wrap2("aaa"))

In the above case it's a matter of taste, but for other use cases there are clear differences. First, let's see what functools.partial exactly does:

Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords. If more arguments are supplied to the call, they are appended to args. If additional keyword arguments are supplied, they extend and override keywords. Roughly equivalent to:

So if we want to bind a parameter that is not the first one we'll have to bind it as a named parameter, and then we will have to invoke the resulting function passing all the parameters located after it as named ones. That's a constraint that probably makes more convenient using a lambda. On the other side, given the they extend and override keywords feature, binding a parameter as a named one gives as a very nice feature, we can use it to turn parameters into default parameters. We use partial to bind those parameters for which we've realised we want a default value, and then we still can invoke the resulting function providing those values if we want. I mean:

cities = ["Xixon", "Paris", "Lisbon"]
def join_and_wrap(items: Iterable[Any], jn: str, wr: str) -> str:
    return f"{wr}{jn.join(items)}{wr}"

join_and_wrap_with_defaults = partial(join_and_wrap, jn=".", wr="||")

# make use of both defaults
print(join_and_wrap_with_defaults(cities))

# "overwrite" one of the defaults
print(join_and_wrap_with_defaults(cities, jn=";"))
print(join_and_wrap_with_defaults(cities, wr="--"))

I was checking what others had to say about all this, and this post is an interesting one. They mention how functools.partial is pretty convenient to prevent the Closure in a loop issue. As you can see in the code below it's less verbose than the strategy of creating a wrapper function that works as scope. Notice that indeed with partial we are not creating a closure (it return a callable object that is different from a function, and traps the bound arguments as attributes of the object, not in the cells of a __closure__).

greetings = ["Bonjour", "Bon día", "Hola", "Hi"]

# remember the "closure in a loop issue" as the the variable has not a "iteration scope"

print("'- Closure in a loop' issue")
greet_fns = []
for greeting in greetings:
    def say_hi(name):
        return f"{greeting} {name}"
    greet_fns.append(say_hi)

for fn in greet_fns:
    print(fn("Francois"))

#Hi Francois
#Hi Francois
#Hi Francois
#Hi Francois

print("------------------------------")

print("- Fixed 1")
greet_fns = []
def create_scope(greeting):
    def say_hi(name):
        return f"{greeting} {name}"
    return say_hi
for greeting in greetings:
    greet_fns.append(create_scope(greeting))

for fn in greet_fns:
    print(fn("Francois"))

#Bonjour Francois
#Bon día Francois
#Hola Francois
#Hi Francois

print("------------------------------")

print("- Fixed 2")
greet_fns = []
def say_hi(greeting, name):
    return f"{greeting} {name}"
for greeting in greetings:
    greet_fns.append(partial(say_hi, greeting))

for fn in greet_fns:
    print(fn("Francois"))

#Bonjour Francois
#Bon día Francois
#Hola Francois
#Hi Francois

Reading about this stuff I've come across the better partial module, that implements an interesting approach to partial application, that feels like something in between the "classic" partial and currying. It creates partial functions that create other partial functions. You invoke them with real values and place-holders, until you have provided all the parameters. Sure this explanation is rather unclear, so I better copy-paste here an example from the module github.

from better_partial import partial, _
# Note that "_" can be imported under a different name if it clashes with your conventions

@partial
def f(a, b, c, d, e):
  return a, b, c, d, e

f(1, 2, 3, 4, 5)
f(_, 2, _, 4, _)(1, 3, 5)
f(1, _, 3, ...)(2, 4, 5)
f(..., a=1, e=5)(_, 3, _)(2, 4)
f(..., e=5)(..., d=4)(1, 2, 3)
f(1, ..., e=5)(2, ..., d=4)(3)
f(..., e=5)(..., d=4)(..., c=3)(..., b=2)(..., a=1)
f(_, _, _, _, _)(1, 2, 3, 4, 5)
f(...)(1, 2, 3, 4, 5)

Using Suspend Functions to Prevent Stack Overflows in Kotlin

This post is the Kotlin counterpart to these 2 previous JavaScript and Python posts. Avoiding stack overflows by means of using suspendable functions (or async-await in JavaScript, Python). This technique does not have much of a real practical purpose, the standard mechanism to prevent stack overflows is using trampolines, so consider this post as a brain exercise that has helped me to better understand how coroutines and suspendable functions work (same as the aformentioned posts helped me better understand async/await).

When a kotlin suspend function gets suspended at a suspension point it returns to the calling function, so the corresponding stack frame is released (the function will be later resumed, in the sentence following that suspension point, by "something" calling the resume method of the continuation object corresponding to that function. If we are not stacking things in the stack, there's no risk of a stack overflow. So if we add in our recursive function a call to a trivial suspend function before the next call to the recursive function, we should avoid the stack overflow. The main thing here is clarifying what "trivial" suspend function we can use for this.

Let's start by showing a recursive function that causes a stack overflow:

fun wrapText1(txt: String, times: Int): String = 
    if (times <= 0) txt
    else wrapText1("[$txt]", times - 1)
	
try {
	println(wrapText1("b", 10000))
}
catch (ex: StackOverflowError) {
	println("StackOverflow!!!")
}

// StackOverflow!!!

First of all, that case could be fixed by leveraging Kotlin's compiler support for Tail Call Optimizations. That function is Tail Recursive, so we can just mark it with the tailrec modifier and the compiler will transform that recursion into a loop, so no more overflows.

tailrec fun wrapText1(txt: String, times: Int): String = 
    if (times <= 0) txt
    else wrapText1("[$txt]", times - 1)

That said, let's forget about the tailrec power. That works for tail recursive functions, while the idea of leveraging suspendable functions to prevent stack overflows will work the same for tail and non-tail recursion. Notice that I´ll run my recursive function in a coroutine created with the runBlocking builder using an event loop dispatcher, so everything runs in a single thread, as in JavaScript and Python asyncio.

Let's first try using a suspend function that indeed does nothing. It does not invoke another "real" suspendable function. As you can see in my post from 2021, in JavaScript awaiting on a resolved Promise (or just on a normal object) is enough to avoid the stack overflow, as it causes the function to be suspended, returning and releasing its stack frame. The "then" part to run after the await is not executed inmmediatelly, but put on the microtask queue, meaning that the function returns to the event loop (and this will execute the "then handler" taken from the microtasks queue). Asynchrony in Kotlin is implemented in a quite different way (continuations...) and I was quite convinced that this "doNothing" function would not cause a suspension, but I had to confirm it.

suspend fun doNothingSuspend(): String {
    return "a"
}

// as expected this one DOES NOT prevent Stack Overflows. Calling a function marked as suspend but that is not really suspending is useless for this
// there won't be suspension. So it's like Python asyncio
suspend fun wrapTextAsync1(txt: String, times: Int): String {
    return if (times <= 0) txt
    else {
        doNothingSuspend()
        wrapTextAsync1("[$txt]", times - 1)
    }
}

try {
	println(wrapText("b", 10000))
}
catch (ex: StackOverflowError) {
	println("StackOverflow!!!")
}

//StackOverflow!!!

Yes, we get an ugly stack overflow. Calling that suspend function does not cause a suspension cause the complex code in which a suspend function is transformed by the compiler will not return an COROUTINE_SUSPENDED value, that is what the calling function checks in turn for returning to its calling function (and so on down to the event loop) or continuing with the next sentence (that is what happens in this case). This is a bit related to what happens in Python, where marking a function that "does not really do something asynchronous" as async (a sort of "fake async function") will not really suspend the function. Because of how coroutines are implemented in Python (similar to generators), the chain of async calls will get down to the Task that controls the chain of coroutine calls, and as what has been returned is not an awaitable, the Task will just invoke send() in the first coroutine in the chain, which will go through the different coroutines in the chain recreating their stacks, up to the coroutine from which we had invoked the "fake async function", that will continue, hence running sequentially, not interrupted by a suspension, without having returned the control to the event loop.

So if we want to have a real suspension in our recursive function, returning from the function and avoiding a stack overflow we'll have to invoke a function that really suspends. The immediate option that comes to mind is using a sort of sleep (delay, in the Kotlin coroutines universe) with a minimum value. I say "minimum value" cause passing just 0 to kotlinx.coroutines.delay will return immediatelly without causing any suspension (it won't return a COROUTINE_SUSPENDED value).

suspend fun wrapTextAsync2(txt: String, times: Int): String {
    return if (times <= 0) txt
    else {
        // using a delay(0) is NOT an option, as it returns immediatelly and no suspension happens
        delay(1)
        wrapTextAsync2("[$txt]", times - 1)
    }
}

That works fine. delay(1) returns to the event loop and after that delay the corresponding continuation will resume the recursive function in the next sentence in the function (the next call to that wrapTextAsync2) with a clean stack. But obviously there's an important performance penalty due to the delay.

If you've read this previous post you'll be also familiar with kotlinx.coroutines.yield. Reading the implementation details I was a bit dubious as to whether running my code in an environment consisting of an event loop created by runBlocking with no multiple suspend functions running concurrently (so no items waiting in the event loop queue) would be enough for yield() to cause a suspension, but if you invoke the function below, with whatever huge times value, you'll see that no stack overflow happens. So yes, yield() prevents a stack overflow.

suspend fun wrapTextAsync3(txt: String, times: Int): String {
    return if (times <= 0) txt
    else {
        yield()
        wrapTextAsync3("[$txt]", times - 1)
    }
}

Recursive IIFE's

I guess most of my hobbyist programming revolves around stuff with no real use other than making me think. A good example of this is that the other idea it came to my mind such an existential question like this "can we write recursive Immeditelly Invokable Function Expressions?"

In modern JavaScript we have Function Expressions and Arrow Function Expressions. As Funtion Expressions can be named, making it recursive is not a problem at all. For example:

let txt = function fn(txt, times) {
    return times <= 0 
        ? txt
        : fn(`[${txt}]`, --times);
}("aa", 3);
    
console.log(txt);
// [[[aa]]]

Arrow functions do not have a name, so they have no way to call themselves save if they have been assigned to a variable. So you have to assign it to a variable and then invoke it, and for running that you'll have to wrap both things in an additional function (an IIA(rrow)E.

txt = (() => {
    const fn = (txt, times) => times <= 0 
        ? txt
        : fn(`[${txt}]`, --times);
    return fn("aa", 3);
})();
console.log(txt);
// [[[aa]]]

That's way more verbose and unintuitive than a Function Expression, so the only reason for using it that I can think of is if that code was running inside a method and the recursive function wanted to use/trap the "this" of that method (arrow functions use the lexical "this"). That would be more convenient than the old-times "self = this;" trick.

class A {
    constructor(logger) {
        this.logger = logger;
    }

    doSomething() {
        // 2 nested arrow functions trapping the "lexical this"
        return (() => {
            const fn = (txt, times) => {
                this.logger.log("in call");
                return times <= 0 
                    ? txt
                    : fn(`[${txt}]`, --times);
            };
            return fn("aa", 3);
        })();        
    }
}

let a = new A(console);
txt = a.doSomething();
console.log(txt);
// [[[aa]]]

Lambda functions in Python are anonymous, but the Assignment Expressions (aka walrus operator) comes to rescue so we can write something so nice like this:

txt = (fn := lambda txt, times: txt 
    if times <= 0
    else fn(f"[{txt}]", times - 1)
)("aa", 3)

print(txt)
# [[[aa]]] 

So in the above code we have an expression that defines a lambda, assigns it to the 'fn' variable and retuns it. Then we immediatelly invoke the returned function. That function has trapped the 'fn' variable in its closure (it's a freevar) and it can invoke it recursivelly.

Function Currying Revisited (fixed)

For whatever the reason some days ago I was looking into adding a curry function to a python micro-library with some utility functions that I use in some small projects. I took a look to this post that I wrote some years ago, and I realized that my javascript implementation was wrong. I'll start by writing it again with the same logic, but supporting calls with several parameters at one time, that was not supported in that old version.

//This one is WRONG
function buggyCurry(originalFunc) {
    let providedArgs = [];
    return function curriedFn(...args) {
        providedArgs.push(...args);
        return providedArgs.length >= originalFunc.length
            ? originalFunc(...providedArgs)
            : curriedFn;
    };
}

Well, the idea is that to create a curried version of a provided function we have to create a function with state, where that state is the original function and the list of already provided parameters. As we invoke the curried function these parameters are appended to that list and the same curried function is returned. Once we have provided all the paremeters the original function is invoked and its return value returned. OK, but the above implementation is wrong. Keeping the list of provided parameters as state of the curried function means that when we start another round of calls to the curried function (a "curried-flow"), those parameters are already there and it fails. I mean:

function formatMessages(msg1, msg2, msg3) {
    console.log(`${msg1}-${msg2}-${msg3}`);
}

let curriedFormat = buggyCurry(formatMessages);

try {
    curriedFormat("a")("b")("c");
    curriedFormat("d")("e")("f");
}
catch (ex) {
    console.log(`Error: ${ex}`);
}
// a-b-c
// a-b-c
// c:\@MyProjects\MyJavaScriptPlayground\currying\curry.js:35
// curriedFormat("d")("e")("f");
//TypeError: curriedFormat(...) is not a function

The first invocation is fine. But the second invocation fails, cause rather than starting again with an empty list of parameters it already starts with those of the previous execution.

To fix that, each time we start a new invocation of the curried function (a "curried-flow") we have to start with a new parameters list. We have to separate the curried function, from the function that stores the state and returns itself until all parameters are provided (the curry logic). The function that starts the "curried-flow" is a sort of bootstrapper or factory. This is what I mean:

//This one is better, but still not fully correct
function curry(originalFunc) {
    // no difference between using an arrow function or a named function expressions, save that the named expression makes clear to me that it's a bootstrapper (or factory)
    //return (...args) => {
    return function curryBootStrap(...args) {
        let providedArgs = [];
        //II(named)FE
        return (function curriedFn(...args) {
            providedArgs.push(...args);
            return providedArgs.length >= originalFunc.length
                ? originalFunc(...providedArgs)
                : curriedFn;
        })(...args);
    };
}


function formatMessages(msg1, msg2, msg3) {
    console.log(`${msg1}-${msg2}-${msg3}`);
}

let curriedFormat = curry(formatMessages);
console.log(curriedFormat.name);

curriedFormat("a")("b")("c");
curriedFormat("d")("e")("f");
curriedFormat("g", "h")("i");
curriedFormat("j", "k", "l");

//a-b-c
//d-e-f
//g-h-i
//j-k-l

That (apparently) works!. But indeed I've just realised that it's not fully correct. It works fine in that use case where we only invoke multiple times the initial function, but it could be that we want to store a reference to some of the intermediate functions and then invoke it multiple times. In that case we would face the same problem as with the first version. That explains whay when checking some articles I always came across with a different implementation that seemed less natural to me, but that is the good one. Based on what I've read here (excellent article where he also talks about the bug that I've just shown), I've implemented it "my way":

// THIS ONE IS CORRECT
function curry(fn) {
    function saveArgs(previousArgs) {
        return (...args) => {
            const newArgs = [...previousArgs, ...args];
            return newArgs.length >= fn.length 
                ? fn(...newArgs)
                : saveArgs(newArgs);
        };
    }
    // Start with no arguments passed.
    return saveArgs([]);
}

In this version, on each invocation of the "curried-flow" either we have reached the end and the original function is invoked, or a new function that traps the existing parameters and joins them in a new array with the ones that it receives, and contains this "curry-logic", is created. The creation of that function is done through another function that I've called saveArgs (the original article named it accumulator). saveAs receives the existing parameters, allowing its child function to trap them. Notice that saveArgs is not a recursive function (neither direct nor indirect), the next invocation to saveArgs is done from a different function (the child function) that is called from the next "curried-flow" invocation, so these calls do not get stacked.

In the above implementations I'm using named function expressions in several places where I could use an arrow function. I'm aware of the differences between normal functions and arrow functions (dynamic "this" vs lexical "this"), that in these cases do not play any role as none of those functions is making use of "this". Using a named function is useful to me cause I try to use semantic names (it's sort of like adding a comment). Additionally there is some trendy use of arrows for factories of other functions that I don't find particularly appealing. I mean, rather than this compact version:

let formatFactory = (pre, pos) => (msg) => `${pre}${msg}${pos}`;

let format1 = formatFactory("-- ", " ..");
console.log(format1("name"));

I prefer writing this more verbose version:

function formatFactory(pre, pos) {
    return (msg) => `${pre}${msg}${pos}`;
}
let format = formatFactory2("-- ", " ..");
console.log(format("name"));

I don't write much JavaScript these days, and I was getting some problems cause I was mixing Kotlin behaviour with JavaScript behaviour. In Kotlin lambdas, if you don't explicitly return a value using the qualified return syntax, the value of the last expression is implicitly returned. This is so regardless of how many expression, declarations or statements it contains. In JavaScript, arrow functions only have an implicit return when they are made up of just one expression.

Literal Syntax for Iterable with Lazy values

The Python discussion forum is a great source of interesting ideas (and also of frustration when some ideas are rejected in a rather harsh way based on the "it's not pythonic" bullshit). The other day one guy was proposing something that was like a sort of "generator literal" or "literal syntax for an iterable with lazy values". I mean, a syntax to define an iterable with values that won't be calculated until each of them is needed (that is, lazily). A straightforward way to express a "collection" made up of values where we need that at least some of them gets generated by a function call just at the moment when the value is needed by the iteration. These are "lazy values"

The syntax he was proposing was not feasible as it would clash with other existing uses, but what is particularly interesting is the workaround that he's using (I'm simplifying one of the lines to make it more clear). It took me a bit to understand it:

values: Iterator[str] = (lambda: (
    (yield "abc"),
    (yield expensive_func("bbb")),
    (yield "something else"),
))()

So, what's that? First he's working around the painful Python's lambda limitations (they can't contain statements) with the technique that I mentioned in this post, the expression (a tuple) with nested expressions (each element of the tuple). We can yield from each nested expression (notice that we have to wrap yield in parentheses so that it's considered as an expression, not as a statement). The above code is equivalent to this:

def gen_fn():
    yield "abc"
    yield expensive_func("bbb")
    yield "something else"
	return (None, None, None)

values: Iterator[str] = gen_fn()

There's another simple, obvious approach. A helper class that wraps the values and "lazy values" (with the lazy values represented as functions that it has to invoke). Something like this:

class LazyIterable:
    def __init__(self, items: Iterable):
        self.items = items

    def __iter__(self):
        for item in self.items:
            yield item() if callable(item) else item


gen_ob = LazyIterable([
    "abc",
    lambda: expensive_func("bbb"),
    "something else",
])

There's not this literal syntax in JavaScript either, so we would have to use the same techniques I've just described, for example:

let genOb = (function*() {
    yield "abc";
    yield expensiveFunc("bbb");
    yield "something else";
})();

console.log("genOb created")

for (let it of genOb) {
    console.log(it);
}

Notice that we can not use arrow functions to define a generator (there's a proposal for that from 5 years ago...), so we are using a "traditional" function expression.

JVM Tiered Compilation

Over the years I've written multiple posts with basic information about JIT's and interpreters (I'll just link a few of them: [1], [2]). We could say that the combination of an interpreter, multi-tier JIT compilation, profiling, optimization and deoptimization... really fascinates me. The truffle interpreters and Graal JIT compilation combination [1], [2] is one of the last advances in the area that has kept me dreming with better understanding how all this works. While rethinking a bit about this topic I've realised that I had some misconceptions and doubts (kind of: "I think it's like that, but indeed I'm not sure about having read it or just being assuming it"). For example, I had got to wrongly think that the cycle of profiling-optimizing a given method was sort of neverending, which is not the case (at least in the JVM, either with C2 or Graal). So as of late I've been reading some pretty interesting material on this topic, mainly about the JVM, and I'll summarize here my findings.

When we have an interpreter, a "simple" and fast JIT and a more advanced and slower (in compilation time, of course the code it generates is faster) JIT, the base principle that drives the interaction between these 3 components is that the interpreter counts how many times it invokes a method and when it exceeds a threshold (the method is hot) it compiles it with the "simple" JIT compiler. This compiler adds "counting logic" to the code it generates, so that the method invokation counting continues and when another threshold is exceeded the method is invoked with the advanced JIT compiler. OK, things are more complex than that in the JVM. This article gives an excellent explanation (just notice that if we are using the Graal VM in not AOT mode we don't have C2, but Graal)

First, the interpreter is not just counting method invokations, it's also gathering profiling information. Well, from what I've read, the interpreter will only start to gather profiling information after a certain threshold of invokations. The "simple" C1 compiler also adds logic to gather profile info to the native code that it generates, so that in the future the "advanced" C2 compiler can use that profiling information to generate hyper-optimized code. It's important to note that profiling has a cost. When your code is not just doing its job but also gathering profile information it's doing extra work. That's why the C2 compiler is the ending point, it will not do further optimizations. It does not generate code with "invokation counting" and profile gathering logic so that in the future another more optimized version can be compiled. The code generated by C2 is just as optimized as "necessary", and that's all. Well, indeed there's an additional player, deoptimization. Both C1 and C2 do assumptions when compiling, with the intent to get optimized code, but those assumptions could be proven wrong in the future (the generated code contains assertions/guards to verify this), so in that case the code has to be deoptimized, returning to the interpretation stage and starting over. This was another source of doubts for me, I thought deoptimization would jump to a previous native code version, but no, it's mentioned in multiple sources that it just returns to the interpreted form. These 2 images are crystal clear:

Furthermore, there are 3 levels of C1 compilations. It's well explained in that article, and also in this one.

So the levels of compilation are as follows:

• 0: Interpreted code
• 1: Simple C1 compiled code
• 2: Limited C1 compiled code
• 3: Full C1 compiled code
• 4: C2 compiled code

C1 and C2 compilers run in separate threads (even several threads for each compiler depending on your number of cores). Each compiler has a queue where it receives compilation requests (so your program does not suddenly "stop" waiting for a compilation). Trivial methods that would not benefit of further profiling and C2 compilation are sent from the interpreter to C1 for a level 1 compilation, that won't add any profiling to the generated code (it won't even add the invokation-counting logic to it). That code is "final" unless it has to be deoptimized. Non trivial methods are sent to C1 level 3 (that adds full profiling) or C1 level 2 (that adds light profiling). It's not clear to me yet when it sends to level2 rather than level3. Finally, when a method is hot enough it's sent to the C2 compiler along with all the profiling information.

There's another point that I should detail a bit more. When I say that the interpreter and the C1 generated code are counting how many times a method is invoked, it's indeed more complex. It also counts how many loop iterations run inside that method. Invoking a method once, but then running a loop for 1000 iterations in that method makes it hotter than invoking 100 times a method that has no loops.

Compilation is based on two counters in the JVM: the number of times the method has been called, - and the number of times any loops in the method have branched back.

As one methods evolves from its bytecode form (to be interpreted) to one of several native forms (and then back to bytecode form if it has been deoptimized), how does the code that invokes that method know where to jump to? We are not going to be patching the many possible callsites to that method, so the solution seems simple (once you read it from someone smarter than you), adding an extra level of indirection. So calls to a method will go through a table that points to the actual code (the native code version or code that continues interpreting bytecodes). So it's that table who gets patched to point to the right address as the method evolves. We already have this kind of tables (i-tables and v-tables) for virtual methods, so I wonder if this patching is just applied on such v-table, or entries in that v-table point to an additional "bytecode vs native" table on which this patching happens. Wow, VM's are amazingly complex.

Another point that seems amazingly complex to me is the deoptimization part. So you are executing the Jitted compiled version of one method and an assertion/guard fails and we have to go back to the bytecode interpreted version. This is not only that future invokations of that method will have to jump into the interpreted version (the indirection table I've just talked about), it's that we are in the middle of the execution of one method (well, this sound also like the on stack replacement thing...). There's an article where they talk about this in the truffle-graal world.

The first problem is how to jump from compiled code back into an interpreter. First of all, we need to be able to jump from compiled code back into the interpreter. This isn’t as simple as a goto. We not only need to jump from one place to another, but we also need to recreate the state that the interpreter expects. That means that we need to recreate the stack frames for every method that was compiled but now is being interpreted. We then just have to restart execution in the interpreter in the AST node that corresponds to the machine code where we deoptimized - so we maintain a mapping between the two for all generated code.

Reading the different articles cited above I've been amazed by the amount of options that the JVM accepts (parameters that you pass to the java process in the command line). I was familiar with the -Xms and -Xmx settings to set the Heap size, and also the -Xss option to set the Thread stack size. But there are so many more! For example:

• XX:CompileThreshold=invocations. Sets the number of interpreted method invocations before compilation. By default, in the server JVM, the JIT compiler performs 10,000 interpreted method invocations to gather information for efficient compilation.
• -XX:-TieredCompilation. To disable tiered compilation (it's enabled by default)
• -XX:+PrintCompilation. When enabling this optione every time a method is compiled the JVM prints out a line with information about what has just been compiled

I won't close this post without linking these 2 articles about meta-compilation, tracing and partial evaluation, [1] and [2]

Meta-compilation reduces the individual implementation effort by making just-in-time compilers reusable and independent of any particular language. RPython and GraalVM are the main systems in this space. Both approach the problem by compiling the executing program together with the interpreter that is executing it. This combination provides the necessary specialization opportunities to produce efficient native code that rivals custom-built state-of-the-art JIT compilers.