Default Arguments differences

After my previous post about late-bound default arguments/parameters (again I continue to be confused about saying "argument" or "parameter", and indeed the Kotlin documentation also seems to use them indistinctively in some occasions) I realised that there's another limitation of Python default arguments when compared to JavaScript or Kotlin. In python once you define a default argument, all the ensuing arguments also have to be defined as default ones, otherwise you'll get a Syntax Error.

This is not the case in Kotlin (and it seems in Ruby either), where:

If a default parameter precedes a parameter with no default value, the default value can only be used by calling the function with named arguments.

fun foo(
    bar: Int = 0,
    baz: Int,
) { /*...*/ }

foo(baz = 1) // The default value bar = 0 is used

So it's OK not to provide one default argument that has non-default arguments after it, but in that case you have to invoke the funcion naming the remaining arguments. There's an exception to this, if the last of those argument is a lambda (trailing lambda), you can just use the nice syntax of passing it outside the parentheses:

If the last argument after default parameters is a lambda, you can pass it either as a named argument or outside the parentheses

As all this ends up being a bit confusing, I think it's generally adviced to put all the default arguments at the end of the function signature (if the last one is a lambda this one is excluded from the rule).

In JavaScript you can also define them as in Kotlin, but given that there's no support for named arguments the behaviour is that if you pass less parameters than the function defines in its signature, it gets them in order, so it seems like it's not particularly useful. Well, it's useful if we explicitely pass undefined as the value for a default parameter, as in that case the default argument is used, rather than value (that's not the case if we pass null)

const format = (l1, l2 = "_", txt) => `${l1}${l2}${txt}${l2}${l1}`;

// not what we would like
console.log(format("|", "Iyan"));
//|IyanundefinedIyan|

console.log(format("|"));
// |_undefined_|

console.log(format("|", "_", "Iyan"));
// |_Iyan_|

// the thing of having declared a default parameter before a non default parameter is only useful if we decide to pass undefined, that will be replaced by the default value
console.log(format("|", undefined, "Iyan"));
// |_Iyan_|

Well there is a syntax "trick" in Python that sort of allows the kotlin behaviour as explained here. It leverages a Python feature that was unknown to me so far, Keyword-Only Arguments. So you can declare non default arguments after default arguments by placing a ", *" between them, as it forces you to pass by name the remaing non default arguments (so you get the same behaviour as in Kotlin). The disadvantage here is that indeed you are forced to pass those parameters by name always, even when you are providing all the values and not using any of the defaults.

# We're not allowed to write this:
# Non-default argument follows default argumentPylance
# def format(l1, l2 = "_", txt):
#     return f"{l1}{l2}{txt}{l2}{l1}"

# but we can use the * feature
def format(l1, l2 = "_", *, txt):
    return f"{l1}{l2}{txt}{l2}{l1}"


# both fail:
# format() missing 1 required keyword-only argument: 'txt'
#print(format("|", "Iyan"))
#print(format("|"))

# works fine
print(format("|", txt="Iyan"))
# |_Iyan_|

# the disadvantage is that we are forced to always pass txt by name, even when we are passing all values
#TypeError: format() takes from 1 to 2 positional arguments but 3 were given
#print(format("|", "-", "Iyan"))

print(format("|", "-", txt="Iyan"))

Destructuring, astuple, attrgetter

I already talked about destructuring in JavaScript/Python/Kotlin in this previos post. So, to sum up, we can use destructuring with any iterable Python object (that's equivalent to JavaScript Array destructuring). That's nice, but it would be event more cute to have something that we could use with any object (a bit in the vein of JavaScript Object destructuring) with no need to make it iterable. I've been looking into some possibilities:

If you are working with simple dataclasses the astuple() function comes pretty handy. Notice though, that it will recursivelly retrieve elements of other dataclasses, lists, dictionaries... which probably is not what you want./p>

@dataclass
class Person:
    name: str
    city: str
    post_code: int
    age: int
    job: str

p1 = Person("Iyan", "Xixon", 33200, 49, "Coder")

name, _, pc, job, _ =  astuple(p1)
print(name, pc, job)
#Iyan 33200 Coder

That approach works in a positional way. We know the order of the the attributes of the class and we take those positions we want and discard others (with the _ convention).

Another option, this one more nominal, as we take attributes based on their name, is using operator.attrgetter, like this:

name, pc, job = operator.attrgetter("name", "post_code", "job")(p1)
print(name, pc, job)
#Iyan 33200 Coder

It looks ok, but using attribute names in its string form is a huge risk. If you rename an attribute refactoring tools will know nothing about your string, and you have to remember that you are accessing that attribute as a string to manually fix it. It's easy to make a mess... With that in mind, I think I prefer the redundancy of writing the object name multiple times:

name, pc, job = p1.name, p1.post_code, p1.job
print(name, pc, job)
#Iyan 33200 Coder

Related to this, what if we want to transform each of the values we are destructuring, but with a different transformation for each value? I'm not talking about a normal map() call where we apply the same funcion, I'm talking about applying different functions. For that, a transform function like this will do the trick:

def transform(funcs, items):
    return [func(item) for func, item in zip(funcs, items)]

That way we can write something like this:

name, pc, job = transform(
    (str.upper, lambda x: x, str.lower), 
    attrgetter("name", "post_code", "job")(p1), 
)
print(name, pc, job)
#IYAN 33200 coder


name, pc, job = transform(
    (str.upper, lambda x: x, str.lower), 
    (p1.name, p1.post_code, p1.job), 
)
print(name, pc, job)
IYAN 33200 coder

Yielding aka Allowing others to Run

At the end of my previous post I talked about the yield() Kotlin function, and how we use it to cooperate with other coroutines by asking if another coroutine wants to run, and if so, suspending the caller coroutine so that another coroutine gets scheduled. We don't have a specific yield function in JavaScript or Python, we just leverage another more generic function for that purpose, let's see.

In Python we use await asyncio.sleep(0) for that. asyncio.sleep() will suspend the current coroutine in all cases, regardless of the interval provided (from the documentation: "sleep() always suspends the current task, allowing other tasks to run."). The event loop will take control and will check if another coroutine wants to run, and if not, if the interval is 0, will resume the sleeping coroutine immediatelly. So it's the same as yield in Kotlin

Javascript follows the same strategy as Python for this. The setTimeout() function always returns control to the event loop, even when the interval is 0, with the callback being added to the macrotask queue (from the documentation: "If this parameter is omitted, a value of 0 is used, meaning execute "immediately", or more accurately, the next event cycle". If there's nothing in the microtask queue or the macrotask queue, the callback we've just added will run immediatelly (in the next event loop iteration), as we've provided a 0 wait interval. OK, I'm talking about callbacks, which seems a bit odd, so notice that we can easily "promisify" the setTimeout function getting an equivalent to asyncio.sleep(), like this:

async function asleep(timeout) {
    return new Promise(res => setTimeout(res, timeout));
}

Additionally, in JavaScript we can await any value, not just a Promise. We can just write await "aa" or await Promise.resolve("aa") (both are equivalent, await "aa" is indeed transformed in await Promise.resolve("aa")). Awaiting for an already resolved Promise will add the "then callback" for that promise (in an async function that's the invokation to the next state of the state-machine for that function) to the microtask queue. When the event loop gets back the control it will schedule the next task in the microtask queue. So if there was some previous task it will be executed, else our "then callback" task will be executed, so this is equivalent to yielding.

So all in all in JavaScript we have 2 levels of yielding. We can yield to tasks in the microtask queue by awaiting a resolved Promise, or yield to tasks in the macrotask queue, by using setTimeout(0). You can read more about this here and here.

We know that Kotlin has a suspend delay() function, but we can not use it as a replacement for yield(), because delay() only suspends when the interval is bigger than 0 (from the documentation: "If the given timeMillis is non-positive, this function returns immediately". So delay(0) has no effect at all, the next line will run sequentially, no suspension and chances for other coroutines to run will happen.

I must say that I pretty like the Kotlin approach. Having a specific yield function rather than leveraging a particular case of another function is more semantic

Cancelling Kotlin Coroutines

Kotlin coroutines come with a powerful cancellation mechanism. The first thing to make clear is that Cancellation is Cooperative. This is not like killing a process with kill -9, a coroutine has to cooperate with the cancellation mechanism by checking if a cancellation has been requested.

When we create a coroutine with a CoroutineBuilder like launch() or async() we obtain a Job (Deferred derives from Job) that we can cancel by invoking its cancel() method, that makes it transition into the cancelling state. The CoroutineContext has a reference to the Job, and a suspend function has always access to its context through the "magical" coroutineContext property defined in the kotlin.Coroutines package. I call it magical because its source code looks like this:

@SinceKotlin("1.3")
@Suppress("WRONG_MODIFIER_TARGET")
@InlineOnly
public suspend inline val coroutineContext: CoroutineContext
    get() {
        throw NotImplementedError("Implemented as intrinsic")
    }

That "implemented as intrinsic" means this:

Intrinsic basically means that the implementation is internal to the compiler.
...
In general intrinsic thus means that it is something that is built in to the “translation” system rather than provided in other ways (library) but is not a specified element of the language syntax (the language itself is always built in to the compiler).

If we compile to bytecodes some Kotlin suspend function using that property and then transpile to Java code, the Java code looks like this: ((Continuation)$continuation).getContext()), which makes pretty much sense. We know that each suspend function is associated to a continuation object, and a continuation object references the CoroutineContext of the coroutine (that as I've said in turn references a Job, a Dispatcher...). So the Kotlin implementation of coroutines and suspend makes Cancellation very simple. As we know, every suspend function in a chain of suspend calls in a coroutine receives as parameter the continuation of the caller suspend function (we end up with a chain of continuations), and that continuation gives access to the CoroutineContext, hence the Job and hence checking if the Job has been cancelled. Notice that JavaScript promises do not support cancellation (though there are alternative implementations like bluebird that do so).

In JavaScript the chain of Promises of an "async call stack" is created from the deepest Promise outwards, while in Kotlin the chain of Continuations of an "async call stack" is created from the outer caller to the deepest callee, and I guess that this makes implementing cancellation more straightforward in Kotlin.

I mentioned above that "a coroutine has to cooperate with the cancellation mechanism by checking if a cancellation has been requested" but indeed most times we don't have to do anything, this is implicitly done for us, as stated in the documentation:

Coroutine cancellation is cooperative. A coroutine code has to cooperate to be cancellable. All the suspending functions in kotlinx.coroutines are cancellable. They check for cancellation of coroutine and throw CancellationException when cancelled.

So in most occasions your suspend functions will just call to other suspend functions that already take care of cancellation (not only the ones in kotlinx.coroutines, but for example also Ktor requests. Let's think of a case where that's not the case. For example a function performing several sequential CPU-bound operations. We'll run that function in the ThreadPool (in its own Coroutine with the Dispatchers.Default dispatcher) and it will check after each operation if a cancellation has been requested. We use for that coroutineContext.ensureActive()

class Message(
    val header: String,
    val content: String,
    val footer: String,
    ) {}

// Example 1, as our suspend functions invoke delay, that is cancellable, our functions are also cancellable
suspend fun decryptHeader(txt: String): String {
    println("decryptHeader started")
    Thread.sleep(2000)
    return "[[$txt]]"
}

suspend fun decryptContent(txt: String): String {
    println("decryptContent started")
    Thread.sleep(2000)
    return "[[$txt]]"
}

suspend fun decryptFooter(txt: String): String {
    println("decryptFooter started")
    Thread.sleep(2000)
    return "[[$txt]]"
}

suspend fun decryptMessage(): String {
    val message = Message("Title", "Main", "notes")
    println("decryptMessage started")
    val header = decryptHeader(message.header)
    println("header obtained: $header")
    coroutineContext.ensureActive()

    val content = decryptContent(message.content)
    println("content obtained: $content")
    coroutineContext.ensureActive()

    val footer = decryptFooter(message.footer)
    println("footer obtained: $footer")
    coroutineContext.ensureActive()

    return "$header - $content - $footer"
}

suspend fun cancelComputation(c1: Deferred) {
    val elapsed = measureTimeMillis {
        delay(1000)
        c1.cancel()
        println("after invoking cancel")
        val res = try {
            c1.await()
        } catch (ex: Exception) {
            ex.message
        }
        println("result: $res")
    }
    println("elapsed time: $elapsed")
}

fun runCancellableComputation() {
    println("started")
    runBlocking {
        // this runs in the eventloop
        val c1 = async (Dispatchers.Default) {
            // this runs in the ThreadPool
            decryptMessage()
        }
        cancelComputation(c1)

    }
    println("Finished, current thread: ${Thread.currentThread().name}")
}

/*
started
decryptMessage started
decryptHeader started
after cancelling
header obtained: [[Title]]
result: DeferredCoroutine was cancelled
elapsed time: 2023
Finished, current thread: main
*/

After running the first CPU bound function we check in one step with ensureActive() if a cancellation has been requested and exit the coroutine in that case, pretty nice. I've seen some articles where they mention using yield() rather than ensureActive(). yield will also work, as it's a "cancel aware" suspend function but it's intended for something different. With yield we are cooperating, telling other coroutines in my same dispatcher to run if they need. In cases like this where we have less coroutines (just 1) than threads in the threadpool, that yielding would have no effect, but in cases where many coroutines fight for some thread, using yield is normally a good thing (if you are coopperative), but could be different from what you want, as on some occasions you'll want that task to finish as soon as possible in detriment of other tasks. Of course, do not confuse this yield with the SequenceScope.yield used for the "generator-like functionality".

There's a pretty nice explanation about this in stackoverflow:

I would answer the question in the context of 4 related things:

Sequence yield(value: T) is totally unrelated to coroutine yield()

isActive is just a flag to identify if the coroutine is still active or cancelled. You can check this flag periodically and decide to stop current coroutine or continue. Of course, normally, we only continue if it's true. Otherwise don't run anything or throws exception, ex. CancellationException.

ensureActive() checks the isActive flag above and throws CancellationException if it's false.

Coroutine yield() not only calls ensureActive() first, but then also politely tells other coroutines in the same dispatcher that: "Hey, you guys could go first, then I will continue." The reason could be "My job is not so important at the moment." or "I am sorry to block you guys for so long. I am not a selfish person, so it's your turn now." You can understand here exactly like this meaning in dictionary: "yield (to somebody/something): to allow vehicles on a bigger road to go first." SYNONYM: give way.

An additional note. I guess the check for cancellation could have been implemented implicitly after each call to a suspend function. When a suspension function finishes the coroutine invokes continuation.resume (or resumeWith) to continue on, so that resume() method could perform that check each time it's invoked.

Kotlin Coroutines and CPU and IO bound code

Time for a short follow-up to this recent post that showed my first non-theorical fiddling with Kotlin Coroutines. In that post I was running several suspend functions concurrently, by starting several coroutines concurrently (with the async() coroutine builder), but everything was running in the same Thread, in an event-loop created by the coroutine created by the runBlocking() coroutine builder. Obviously that's great for I/O bound code, but what if we have CPU bound code that we want to run in parallel in separate threads? Particularly let's say we have functions that both do CPU crushing and IO bound tasks. We can use for this coroutines with a dispatcher that leverages the ThreadPool.

If we have a CPU bound function and we launch 2 invokations using the async coroutine builder just as we did in my previous post, the code will run sequentially. This is so because we have no suspension points inside the CPU bound function, so once the first coroutine is launched by async() it will run without suspending and it's not until when it finishes that the second coroutine is launched (everyting running in the event-loop thread)

fun doCalculation(id: Int, duration: Int): Int{
    var count = 0
    while (count++ < duration) {
        println("calculation: $id, step: $count, thread: ${Thread.currentThread().name}")
        Thread.sleep(500)
    }
    println("Finishing calculation: $id, step: $count, thread: ${Thread.currentThread().name}")
    return id
}

fun performCalculationsSequentially() {
    runBlocking {
        val time = measureTimeMillis {
            val c1 = async { doCalculation(id = 1, duration = 3) }
            val c2 = async { doCalculation(id = 2, duration = 6) }
            val results = listOf(c1, c2).awaitAll()
            println("tasks finished, ${results.joinToString(", ")}")
        }
        println("Time taken: $time") 
    }
}

/*
Started, cpubound
calculation: 1, step: 1, thread: main
calculation: 1, step: 2, thread: main
calculation: 1, step: 3, thread: main
Finishing calculation: 1, step: 4, thread: main
calculation: 2, step: 1, thread: main
calculation: 2, step: 2, thread: main
calculation: 2, step: 3, thread: main
calculation: 2, step: 4, thread: main
calculation: 2, step: 5, thread: main
calculation: 2, step: 6, thread: main
Finishing calculation: 2, step: 7, thread: main
tasks finished, 1, 2
Time taken: 4591
*/

So running coroutines with an event-loop is great for cooperating between multiple suspend functions that really have suspension points (yes, the equivalent to JavaScript async code or Python asyncio) but is useless when the function is just doing processing without suspending.

If we invoke the async coroutine builder with Dispatchers.Default. I mean: async (Dispatchers.Default), the coroutine will run in a ThreadPool, so in the below code, the calculations will run in parallel, each of them in a Thread from the ThreadPool.

fun doCalculation(id: Int, duration: Int): Int{
    var count = 0
    while (count++ < duration) {
        println("calculation: $id, step: $count, thread: ${Thread.currentThread().name}")
        Thread.sleep(500)
    }
    println("Finishing calculation: $id, step: $count, thread: ${Thread.currentThread().name}")
    return id
}

fun performCalculations3() {
    runBlocking {
        println("coroutineContext $coroutineContext, thread: ${Thread.currentThread().name}")
        val time = measureTimeMillis {
            val c1 = async (Dispatchers.Default){
                println("inside async, coroutineContext $coroutineContext")
                doCalculation(id = 1, duration = 3)
            }
            val c2 = async (Dispatchers.Default){
                println("inside async, coroutineContext $coroutineContext")
                doCalculation(id = 2, duration = 6)
            }
            val results = listOf(c1, c2).awaitAll()
            println("tasks finished: ${results.joinToString(", ")}, coroutineContext $coroutineContext, thread: ${Thread.currentThread().name}")
        }
        println("Time taken: $time") // Time taken: 3079
    }
}

/*
coroutineContext [BlockingCoroutine{Active}@46f5f779, BlockingEventLoop@1c2c22f3], thread: main
inside async, coroutineContext [DeferredCoroutine{Active}@3a286e59, Dispatchers.Default]
inside async, coroutineContext [DeferredCoroutine{Active}@5e11571d, Dispatchers.Default]
calculation: 2, step: 1, thread: DefaultDispatcher-worker-2
calculation: 1, step: 1, thread: DefaultDispatcher-worker-1
calculation: 1, step: 2, thread: DefaultDispatcher-worker-1
calculation: 2, step: 2, thread: DefaultDispatcher-worker-2
calculation: 1, step: 3, thread: DefaultDispatcher-worker-1
calculation: 2, step: 3, thread: DefaultDispatcher-worker-2
Finishing calculation: 1, step: 4, thread: DefaultDispatcher-worker-1
calculation: 2, step: 4, thread: DefaultDispatcher-worker-2
calculation: 2, step: 5, thread: DefaultDispatcher-worker-2
calculation: 2, step: 6, thread: DefaultDispatcher-worker-2
Finishing calculation: 2, step: 7, thread: DefaultDispatcher-worker-2
Time taken: 3079
*/

So the main coroutine is using the event-looop, but the 2 coroutines created with async (Dispatchers.Default) are using the ThreadPool. It's clear that the 2 coroutines are running in parallel as the code takes 3 seconds to finish (duration 6 * 500 milliseconds), otherwise it wold take 4.5 seconds (9 * 500).

In this example, given that our doCalculation function is just CPU-bound and never suspending, we could wonder why we are using coroutines at all if what we want is to run our code in a ThreadPool. Rather than that we could directly use the ThreadPoolExecutor. Well thanks to the async CoroutineBuilder we obtain Deferred objects that we can await very easily, while I guess with the ThreadPoolExecutor this is not so straight forward. Anyway, the need for coroutines with thread-pool dispatchers is way more evident when we have a suspend function that is both suspending at some IO call and running some CPU-bound code (like doing a http request to obtain a text and then encrypting it).

It is also possible to switch the kind of dispatcher that a coroutine is already using, by means of the withContext function. This does not create a new coroutine, it creates a new CoroutineContext (with that Dispatcher that we've passed to it), and the block will run with that Context and Dispatcher. Remember that when we create a coroutine it gets a CoroutineContext (containing among other things a Dispatcher). For each suspend function invoked from that coroutine a Continuation object is created, that also points to that CoroutineContext. When we invoke withContext(), the Continuation objects corresponding to the suspend functions invoked from the block passed to withContext will use that new CoroutineContext, rather than the one of the coroutine. This way we can start a coroutine running in an event-loop, and at some point invoke some suspend function that will run in the ThreadPool.

Python late-bound default parameters

Last year I wrote a post about the odd behaviour of default arguments/parameters in Python (by the way, I always get confused about whether I should say arguments or parameters in this case). This behaviour comes from the fact that default parameters are bound at function definition time rather than at call time (late bound). As explained in that post, JavaScript, Kotlin and Ruby behave differently, the value for a default parameter is evaluated each time the function is call. At that time I had not paid attention to how powerful such a feature is. Looking into the MDN documentation I've seen that parameters can use previous parameters in its definition:

function greet(name, greeting, message = `${greeting} ${name}`) {
  return [name, greeting, message];
}

Kotlin documentation does not stress that much these advanced uses, but of course it also comes with them:

fun read(
    b: ByteArray,
    off: Int = 0,
    len: Int = b.size,
) { /*...*/ }

What has led me to review what default parameters allow in these languages is that I recently came across with a draft for a PEP (671), that of course was not received with particular interest by part of the community (probably the same ones that tells us that optional chaining has no particular use case...), that proposes taking Python default arguments to the next level by making them late-bound (and allowing them access to other parameters). As interesting as the proposal is the fact that one smart guy has sort of implemented it in the late module, by means of decorators.

The way to implement such thing in pure Python is not so misterious (particularly after checking the source code :-D Given that parameters are bound at definition time, let's bind something that will produce a value, rather than a value itself. Then, in order to make that producer run each time the function is invoked, let's wrap that function with another one that will do the calling. Join to this the powerful inspect(fn).signature() method, and we are done. What the late guy has implemented is really nice, but it does not seem so powerful as what the PEP proposes (and that is the same that we have in JavaScript and Kotlin). It does not allow late-bound parameters to depend on other parameters (either also late-bounds or normal). So after having checked the source code I went ahead with implementing my own version of late-binding (or call-time binding) for default parameters. Here it is:

from dataclasses import dataclass
from typing import Callable, Any
import functools
import inspect

@dataclass
class LateBound:
	resolver: Callable

def _invoke_late_bound(callable: Callable, arg_name_to_value: dict[str, Any]) -> Any:
    """
    invokes a callable passing over to it the parameters defined in its signature
    we obtain those values from the arg_name_to_value dictionary
    """
    expected_params = inspect.signature(callable).parameters.keys()
    kwargs = {name: arg_name_to_value[name]
         for name in expected_params
    }
    return callable(**kwargs)


def _add_late_bounds(arg_name_to_value: dict[str, Any], late_bounds: list[str, Callable]):
    """resolves late-bound values and adds them to the arg_name_to_value dictionary"""
    for name, callable in late_bounds:
        val = _invoke_late_bound(callable, arg_name_to_value)
        #this way one late bound can depend on a previous late boud 
        arg_name_to_value[name] = val
    

def _resolve_args(target_fn: Callable, *args, **kwargs) -> dict[str, Any]:
    """returns a dictionary with the name and value all the parameters (the ones already provided, the calculated latebounds and the normal defaults)"""
    # dictionary of the arguments and values received by the function at runtime
    # we use it to be able to calculate late_bound values based on other parameters
    arg_name_to_value: dict[str, Any] = {}
    arg_names = list(inspect.signature(target_fn).parameters.keys())
    for index, arg in enumerate(args):
        arg_name_to_value[arg_names[index]] = arg
    arg_name_to_value = {**arg_name_to_value, **kwargs}
    
    # obtain the values for all default parameters that have not been provided
    # we obtain them all here so that late_bounds can depend on other (compile-time or late-bound) default parameters
    #late bounds to calculate (were not provided in args-kwargs)
    not_late_bounds  = {name: param.default 
        for name, param in inspect.signature(target_fn).parameters.items()
        if not isinstance(param.default, LateBound) and not name in arg_name_to_value
    }
    arg_name_to_value = {**arg_name_to_value, **not_late_bounds}

    # list rather than dictionary as order matters (so that a late-bound can depend on a previous late-bound)
    late_bounds = [(name, param.default.resolver) 
        for name, param in inspect.signature(target_fn).parameters.items()
        if isinstance(param.default, LateBound) and not name in arg_name_to_value
    ]

    _add_late_bounds(arg_name_to_value, late_bounds)
    return arg_name_to_value


#decorator function
def late_bind(target_fn: Callable | type) -> Callable | type:
    """decorates a function enabling late-binding of default parameters for it"""
    @functools.wraps(target_fn)
    def wrapper(*args, **kwargs):
        kwargs = _resolve_args(target_fn, *args, **kwargs)
        return target_fn(**kwargs)

    return wrapper

And you can use it like this:

from datetime import datetime
from dataclasses import dataclass
from late_bound_default_args import late_bind, LateBound

@late_bind
def say_hi(source: str, target: str, greet: str, 
    extra = LateBound(lambda: f"[{datetime.now():%Y-%m-%d_%H%M%S}]"),
    ):
    """"""
    return f"{greet} from {source} to {target}. {extra}"

@late_bind
def say_hi2(source: str, target: str, greet: str, 
    extra = LateBound(lambda greet: f"[{greet.upper()}!]"),
    ):
    """"""
    return f"{greet} from {source} to {target}. {extra}"

print(say_hi("Xuan", "Francois", "Bonjour"))
print(say_hi2("Xuan", "Francois", "Bonjour"))

#Bonjour from Xuan to Francois. [2024-02-02_002939]
#Bonjour from Xuan to Francois. [BONJOUR!]


# access to the "self" parameter in a late-bound method works also fine
@dataclass
class Person:
    name: str
    birth_place: str

    @late_bind
    def travel(self, by: str, 
        start_city: str = LateBound(lambda self: self.birth_place), 
        to: str = "Paris"
        ):
        """ """
        return(f"{self.name} is travelling from {start_city} to {to} by {by}")
    
p1 = Person("Xuan", "Xixon")
print(p1.travel("train"))
# Xuan is travelling from Xixon to Paris by train

I've uploaded it to a gist

Kotlin Coroutines Basic Example Explained

As I explained in this previous post the approach used in Kotlin to asynchronous programming (and "generators") is a bit different from what I'm used to in JavaScript and Python (and even C#). Once I've managed to mainly understand how suspend, continuation passing style, compiler magic... work together, it's time to fiddle with some real code. What should be clear is that asynchronous programming in kotlin is more customizable than in JavaScript and Python. I'm thinking mainly about the different dispatchers, with which your code can run in an eventloop or a thread pool. Indeed we have at least 2 different thread pools. We use Dispatchers.Default for a pool for "CPU bound tasks" (with as many threads as CPU's in your machine), and Dispatchers.IO for a pool for "IO bound tasks"). Read this excellent article if you want to know more.

So I've started with one of the most common tasks in asynchronous programming that I can think of, lauching several non blocking IO operations (a simulated http request to get a blog-post), wait for all of them to complete and do something with their results.

The Python code for that looks like this:

delays = {
    "A1": 3,
    "B1": 1,
    "C1": 0.1,
}

async def get_post(id: str) -> str:
    print(f"getting post: {id}")
    if not id in delays:
        await asyncio.sleep(0.5)
        raise Exception(f"Missing post: {id}")   
     
    await asyncio.sleep(delays[id])
    return f"POST: [[{id}]]"

# async def retrieve_posts_sequentially():
#     print("started")
#     post_ids = ["A1", "B1", "C1"]
#     retrieved_posts = []
#     for id in post_ids:
#         retrieved_posts.append(await get_post(id))
#     print("all posts retrieved")
#     for post in retrieved_posts:
#         print(f"post: {post}")

async def retrieve_posts():
    print("started")
    post_ids = ["A1", "B1", "C1", "D1"]
    requested_posts = [asyncio.create_task(get_post(id)) for id in post_ids]
    retrieved_posts = await asyncio.gather(*requested_posts, return_exceptions= True)
    print("all posts retrieved")
    for post in retrieved_posts:
        print(f"post: {post}")


asyncio.run(retrieve_posts())	

First of all we have to create an event loop (with asyncio.run) to run our asyncio code in it. Then we have an async function (called coroutine function in python). As we know, invoking a coroutine creates a coroutine object, but does not start running the code. To run it we have to either await it, or launch it through create_task. Awaiting the coroutine (await get_post()) would launch it and suspend the calling function, so we would not do the second call to get_post() until the first one finishes and the event loop resumes the calling function. So the requests would run sequentially one after another. That's what would happen in the commented retrieve_posts_sequentially(). On the other side, starting the coroutine through create_task, launches the function without forcing us to await it. So we can launch all our requests and then wait for all of them to complete (with await asyncio.gather()). This way the requests run concurrently in the event loop thread. That's what happens in retrieve_posts().

Let's see now the Kotlin equivalent to the above:

val idsToDelays = mapOf(
    "A1" to 1000,
    "B1" to 2000,
    "C1" to 500,
)

suspend fun getPost(id: String): String {
    println("getPost $id start, context: $coroutineContext Thread: ${Thread.currentThread().name}")
    delay(idsToDelays[id] ?: 500)
    println("getPost $id end, context: $coroutineContext Thread: ${Thread.currentThread().name}")
    return if (idsToDelays[id] !== null) "POST: [[${id}]]"
        else throw Exception("missing ID")
		
val postIds = listOf("A1", "B1", "C1")

//runBlocking {
//	println("context: $coroutineContext")
//	postIds.forEach { getPost(it) }
//}
	
runBlocking {
	// same as the previous test, but now causing an exception and handling it
	val futures = postIds.map { async {
		//this is a try-catch expression, so we don't need to write "return"
		try {
			getPost(it)
		}
		catch (ex: Exception) {
			ex.message
		}
	}}

	val posts = futures.awaitAll()
	posts.forEach(::println)
}

/*
getPost A1 start, context: [DeferredCoroutine{Active}@566776ad, BlockingEventLoop@6108b2d7] Thread: main
getPost B1 start, context: [DeferredCoroutine{Active}@1554909b, BlockingEventLoop@6108b2d7] Thread: main
getPost C1 start, context: [DeferredCoroutine{Active}@6bf256fa, BlockingEventLoop@6108b2d7] Thread: main
getPost D1 start, context: [DeferredCoroutine{Active}@6cd8737, BlockingEventLoop@6108b2d7] Thread: main
getPost C1 end, context: [DeferredCoroutine{Active}@6bf256fa, BlockingEventLoop@6108b2d7] Thread: main
getPost D1 end, context: [DeferredCoroutine{Active}@6cd8737, BlockingEventLoop@6108b2d7] Thread: main
getPost A1 end, context: [DeferredCoroutine{Active}@566776ad, BlockingEventLoop@6108b2d7] Thread: main
getPost B1 end, context: [DeferredCoroutine{Active}@1554909b, BlockingEventLoop@6108b2d7] Thread: main
POST: [[A1]]
POST: [[B1]]
POST: [[C1]]
missing ID
Finished, current thread: main
*/

So as expected we use the suspend keyword (there's no async keyword in Kotlin) to designate our suspendable (asynchronous) function. To run our suspend functions we have to do it from a coroutine (that is quite different from a Python coroutine), that we create through the runBlocking() function, that is a coroutine builder. Calling runBlocking without providing a CoroutineContext (as in this case) runs the code in an event loop, everything in the same thread. From the documentation:

Runs a new coroutine and blocks the current thread interruptibly until its completion. This function should not be used from a coroutine. It is designed to bridge regular blocking code to libraries that are written in suspending style, to be used in main functions and in tests.
The default CoroutineDispatcher for this builder is an internal implementation of event loop that processes continuations in this blocked thread until the completion of this coroutine.

If we call the suspend function directly (see the commented block) we would have the same behaviour as in the commented python block. The calling function would be suspended and would not resume until getPost is completed. It means that we would be gathering the posts sequentially. To run the suspend functions concurrently we use the async() function (another coroutine builder). This way a new coroutine is created to run each getPost(), with these coroutines running concurrently. Calling async() without providing a specific dispatcher means that the new coroutines will use the same dispatcher as the parent coroutine (the coroutine that we created with runBlocking), that is, the event loop dispatcher. async() returns a Deferred object (similar to a Python Task-Future) that will complete when the corresponding coroutine completes. We can wait for all these Deferred to complete with the awaitAll() function. So this code works almost exactly the same as the Python code, there are no additional threads, everything runs in an event loop in its single thread. Notice what I mentioned in the first paragraph, asynchronous programming in Kotlin is amazingly powerful, and we can "easily" write coroutines that use a thread pool rather than an event loop, but that will be for another post.

To summarize:
- asyncio.run() function == runBlocking() function == start an event loop
- asyncio.createTask() function == async() function
- asyncio.gather(list[Task]) function == List[Deferred].awaitAll() function

Well, as my other main language is JavaScript I think I should also write the JavaScript version:

let postIds = ["A1", "B1", "C1"];
let postPromises = postIds.map(getPost);
let retrivedPosts = await Promise.all(postPromises);
retrivedPosts.forEach(console.log);

As we know in JavaScript invoking an async function returns a Promise, and there's no need of using await for launching the execution of the function. If we await the Promise the calling function gets suspended, and hence we would be in the sequential case. So we first do all the getPost calls and gather its returned Promises, and then we await for the completion of all of them with an await Promise.all(). We don't need to start an event loop on our own, as that's the basic driver of any JavaScript runtime.