Tuesday, 17 September 2024

Function Currying Revisited (fixed)

For whatever the reason some days ago I was looking into adding a curry function to a python micro-library with some utility functions that I use in some small projects. I took a look to this post that I wrote some years ago, and I realized that my javascript implementation was wrong. I'll start by writing it again with the same logic, but supporting calls with several parameters at one time, that was not supported in that old version.



//This one is WRONG
function buggyCurry(originalFunc) {
    let providedArgs = [];
    return function curriedFn(...args) {
        providedArgs.push(...args);
        return providedArgs.length >= originalFunc.length
            ? originalFunc(...providedArgs)
            : curriedFn;
    };
}


Well, the idea is that to create a curried version of a provided function we have to create a function with state, where that state is the original function and the list of already provided parameters. As we invoke the curried function these parameters are appended to that list and the same curried function is returned. Once we have provided all the paremeters the original function is invoked and its return value returned. OK, but the above implementation is wrong. Keeping the list of provided parameters as state of the curried function means that when we start another round of calls to the curried function (a "curried-flow"), those parameters are already there and it fails. I mean:


function formatMessages(msg1, msg2, msg3) {
    console.log(`${msg1}-${msg2}-${msg3}`);
}

let curriedFormat = buggyCurry(formatMessages);

try {
    curriedFormat("a")("b")("c");
    curriedFormat("d")("e")("f");
}
catch (ex) {
    console.log(`Error: ${ex}`);
}
// a-b-c
// a-b-c
// c:\@MyProjects\MyJavaScriptPlayground\currying\curry.js:35
// curriedFormat("d")("e")("f");
//TypeError: curriedFormat(...) is not a function

The first invocation is fine. But the second invocation fails, cause rather than starting again with an empty list of parameters it already starts with those of the previous execution.

To fix that, each time we start a new invocation of the curried function (a "curried-flow") we have to start with a new parameters list. We have to separate the curried function, from the function that stores the state and returns itself until all parameters are provided (the curry logic). The function that starts the "curried-flow" is a sort of bootstrapper or factory. This is what I mean:



//This one is better, but still not fully correct
function curry(originalFunc) {
    // no difference between using an arrow function or a named function expressions, save that the named expression makes clear to me that it's a bootstrapper (or factory)
    //return (...args) => {
    return function curryBootStrap(...args) {
        let providedArgs = [];
        //II(named)FE
        return (function curriedFn(...args) {
            providedArgs.push(...args);
            return providedArgs.length >= originalFunc.length
                ? originalFunc(...providedArgs)
                : curriedFn;
        })(...args);
    };
}


function formatMessages(msg1, msg2, msg3) {
    console.log(`${msg1}-${msg2}-${msg3}`);
}

let curriedFormat = curry(formatMessages);
console.log(curriedFormat.name);

curriedFormat("a")("b")("c");
curriedFormat("d")("e")("f");
curriedFormat("g", "h")("i");
curriedFormat("j", "k", "l");

//a-b-c
//d-e-f
//g-h-i
//j-k-l

That (apparently) works!. But indeed I've just realised that it's not fully correct. It works fine in that use case where we only invoke multiple times the initial function, but it could be that we want to store a reference to some of the intermediate functions and then invoke it multiple times. In that case we would face the same problem as with the first version. That explains whay when checking some articles I always came across with a different implementation that seemed less natural to me, but that is the good one. Based on what I've read here (excellent article where he also talks about the bug that I've just shown), I've implemented it "my way":



// THIS ONE IS CORRECT
function curry(fn) {
    function saveArgs(previousArgs) {
        return (...args) => {
            const newArgs = [...previousArgs, ...args];
            return newArgs.length >= fn.length 
                ? fn(...newArgs)
                : saveArgs(newArgs);
        };
    }
    // Start with no arguments passed.
    return saveArgs([]);
}

In this version, on each invocation of the "curried-flow" either we have reached the end and the original function is invoked, or a new function that traps the existing parameters and joins them in a new array with the ones that it receives, and contains this "curry-logic", is created. The creation of that function is done through another function that I've called saveArgs (the original article named it accumulator). saveAs receives the existing parameters, allowing its child function to trap them. Notice that saveArgs is not a recursive function (neither direct nor indirect), the next invocation to saveArgs is done from a different function (the child function) that is called from the next "curried-flow" invocation, so these calls do not get stacked.

In the above implementations I'm using named function expressions in several places where I could use an arrow function. I'm aware of the differences between normal functions and arrow functions (dynamic "this" vs lexical "this"), that in these cases do not play any role as none of those functions is making use of "this". Using a named function is useful to me cause I try to use semantic names (it's sort of like adding a comment). Additionally there is some trendy use of arrows for factories of other functions that I don't find particularly appealing. I mean, rather than this compact version:


let formatFactory = (pre, pos) => (msg) => `${pre}${msg}${pos}`;

let format1 = formatFactory("-- ", " ..");
console.log(format1("name"));


I prefer writing this more verbose version:


function formatFactory(pre, pos) {
    return (msg) => `${pre}${msg}${pos}`;
}
let format = formatFactory2("-- ", " ..");
console.log(format("name"));

I don't write much JavaScript these days, and I was getting some problems cause I was mixing Kotlin behaviour with JavaScript behaviour. In Kotlin lambdas, if you don't explicitly return a value using the qualified return syntax, the value of the last expression is implicitly returned. This is so regardless of how many expression, declarations or statements it contains. In JavaScript, arrow functions only have an implicit return when they are made up of just one expression.

Wednesday, 11 September 2024

Literal Syntax for Iterable with Lazy values

The Python discussion forum is a great source of interesting ideas (and also of frustration when some ideas are rejected in a rather harsh way based on the "it's not pythonic" bullshit). The other day one guy was proposing something that was like a sort of "generator literal" or "literal syntax for an iterable with lazy values". I mean, a syntax to define an iterable with values that won't be calculated until each of them is needed (that is, lazily). A straightforward way to express a "collection" made up of values where we need that at least some of them gets generated by a function call just at the moment when the value is needed by the iteration. These are "lazy values"

The syntax he was proposing was not feasible as it would clash with other existing uses, but what is particularly interesting is the workaround that he's using (I'm simplifying one of the lines to make it more clear). It took me a bit to understand it:


values: Iterator[str] = (lambda: (
    (yield "abc"),
    (yield expensive_func("bbb")),
    (yield "something else"),
))()


So, what's that? First he's working around the painful Python's lambda limitations (they can't contain statements) with the technique that I mentioned in this post, the expression (a tuple) with nested expressions (each element of the tuple). We can yield from each nested expression (notice that we have to wrap yield in parentheses so that it's considered as an expression, not as a statement). The above code is equivalent to this:



def gen_fn():
    yield "abc"
    yield expensive_func("bbb")
    yield "something else"
	return (None, None, None)

values: Iterator[str] = gen_fn()


There's another simple, obvious approach. A helper class that wraps the values and "lazy values" (with the lazy values represented as functions that it has to invoke). Something like this:



class LazyIterable:
    def __init__(self, items: Iterable):
        self.items = items

    def __iter__(self):
        for item in self.items:
            yield item() if callable(item) else item


gen_ob = LazyIterable([
    "abc",
    lambda: expensive_func("bbb"),
    "something else",
])



There's not this literal syntax in JavaScript either, so we would have to use the same techniques I've just described, for example:


let genOb = (function*() {
    yield "abc";
    yield expensiveFunc("bbb");
    yield "something else";
})();

console.log("genOb created")

for (let it of genOb) {
    console.log(it);
}

Notice that we can not use arrow functions to define a generator (there's a proposal for that from 5 years ago...), so we are using a "traditional" function expression.

Thursday, 5 September 2024

JVM Tiered Compilation

Over the years I've written multiple posts with basic information about JIT's and interpreters (I'll just link a few of them: [1], [2]). We could say that the combination of an interpreter, multi-tier JIT compilation, profiling, optimization and deoptimization... really fascinates me. The truffle interpreters and Graal JIT compilation combination [1], [2] is one of the last advances in the area that has kept me dreming with better understanding how all this works. While rethinking a bit about this topic I've realised that I had some misconceptions and doubts (kind of: "I think it's like that, but indeed I'm not sure about having read it or just being assuming it"). For example, I had got to wrongly think that the cycle of profiling-optimizing a given method was sort of neverending, which is not the case (at least in the JVM, either with C2 or Graal). So as of late I've been reading some pretty interesting material on this topic, mainly about the JVM, and I'll summarize here my findings.

When we have an interpreter, a "simple" and fast JIT and a more advanced and slower (in compilation time, of course the code it generates is faster) JIT, the base principle that drives the interaction between these 3 components is that the interpreter counts how many times it invokes a method and when it exceeds a threshold (the method is hot) it compiles it with the "simple" JIT compiler. This compiler adds "counting logic" to the code it generates, so that the method invokation counting continues and when another threshold is exceeded the method is invoked with the advanced JIT compiler. OK, things are more complex than that in the JVM. This article gives an excellent explanation (just notice that if we are using the Graal VM in not AOT mode we don't have C2, but Graal)

First, the interpreter is not just counting method invokations, it's also gathering profiling information. Well, from what I've read, the interpreter will only start to gather profiling information after a certain threshold of invokations. The "simple" C1 compiler also adds logic to gather profile info to the native code that it generates, so that in the future the "advanced" C2 compiler can use that profiling information to generate hyper-optimized code. It's important to note that profiling has a cost. When your code is not just doing its job but also gathering profile information it's doing extra work. That's why the C2 compiler is the ending point, it will not do further optimizations. It does not generate code with "invokation counting" and profile gathering logic so that in the future another more optimized version can be compiled. The code generated by C2 is just as optimized as "necessary", and that's all. Well, indeed there's an additional player, deoptimization. Both C1 and C2 do assumptions when compiling, with the intent to get optimized code, but those assumptions could be proven wrong in the future (the generated code contains assertions/guards to verify this), so in that case the code has to be deoptimized, returning to the interpretation stage and starting over. This was another source of doubts for me, I thought deoptimization would jump to a previous native code version, but no, it's mentioned in multiple sources that it just returns to the interpreted form. These 2 images are crystal clear:

Furthermore, there are 3 levels of C1 compilations. It's well explained in that article, and also in this one.

So the levels of compilation are as follows:

  • 0: Interpreted code
  • 1: Simple C1 compiled code
  • 2: Limited C1 compiled code
  • 3: Full C1 compiled code
  • 4: C2 compiled code

C1 and C2 compilers run in separate threads (even several threads for each compiler depending on your number of cores). Each compiler has a queue where it receives compilation requests (so your program does not suddenly "stop" waiting for a compilation). Trivial methods that would not benefit of further profiling and C2 compilation are sent from the interpreter to C1 for a level 1 compilation, that won't add any profiling to the generated code (it won't even add the invokation-counting logic to it). That code is "final" unless it has to be deoptimized. Non trivial methods are sent to C1 level 3 (that adds full profiling) or C1 level 2 (that adds light profiling). It's not clear to me yet when it sends to level2 rather than level3. Finally, when a method is hot enough it's sent to the C2 compiler along with all the profiling information.

There's another point that I should detail a bit more. When I say that the interpreter and the C1 generated code are counting how many times a method is invoked, it's indeed more complex. It also counts how many loop iterations run inside that method. Invoking a method once, but then running a loop for 1000 iterations in that method makes it hotter than invoking 100 times a method that has no loops.

Compilation is based on two counters in the JVM: the number of times the method has been called, - and the number of times any loops in the method have branched back.

As one methods evolves from its bytecode form (to be interpreted) to one of several native forms (and then back to bytecode form if it has been deoptimized), how does the code that invokes that method know where to jump to? We are not going to be patching the many possible callsites to that method, so the solution seems simple (once you read it from someone smarter than you), adding an extra level of indirection. So calls to a method will go through a table that points to the actual code (the native code version or code that continues interpreting bytecodes). So it's that table who gets patched to point to the right address as the method evolves. We already have this kind of tables (i-tables and v-tables) for virtual methods, so I wonder if this patching is just applied on such v-table, or entries in that v-table point to an additional "bytecode vs native" table on which this patching happens. Wow, VM's are amazingly complex.

Another point that seems amazingly complex to me is the deoptimization part. So you are executing the Jitted compiled version of one method and an assertion/guard fails and we have to go back to the bytecode interpreted version. This is not only that future invokations of that method will have to jump into the interpreted version (the indirection table I've just talked about), it's that we are in the middle of the execution of one method (well, this sound also like the on stack replacement thing...). There's an article where they talk about this in the truffle-graal world.

The first problem is how to jump from compiled code back into an interpreter. First of all, we need to be able to jump from compiled code back into the interpreter. This isn’t as simple as a goto. We not only need to jump from one place to another, but we also need to recreate the state that the interpreter expects. That means that we need to recreate the stack frames for every method that was compiled but now is being interpreted. We then just have to restart execution in the interpreter in the AST node that corresponds to the machine code where we deoptimized - so we maintain a mapping between the two for all generated code.

Reading the different articles cited above I've been amazed by the amount of options that the JVM accepts (parameters that you pass to the java process in the command line). I was familiar with the -Xms and -Xmx settings to set the Heap size, and also the -Xss option to set the Thread stack size. But there are so many more! For example:

  • XX:CompileThreshold=invocations. Sets the number of interpreted method invocations before compilation. By default, in the server JVM, the JIT compiler performs 10,000 interpreted method invocations to gather information for efficient compilation.
  • -XX:-TieredCompilation. To disable tiered compilation (it's enabled by default)
  • -XX:+PrintCompilation. When enabling this optione every time a method is compiled the JVM prints out a line with information about what has just been compiled

I won't close this post without linking these 2 articles about meta-compilation, tracing and partial evaluation, [1] and [2]

Meta-compilation reduces the individual implementation effort by making just-in-time compilers reusable and independent of any particular language. RPython and GraalVM are the main systems in this space. Both approach the problem by compiling the executing program together with the interpreter that is executing it. This combination provides the necessary specialization opportunities to produce efficient native code that rivals custom-built state-of-the-art JIT compilers.

Thursday, 29 August 2024

Getting Source Code at Runtime

One surprising feature present both in JavaScript and Python is that we can get access at runtime to the source code of a function. The internals are a bit different, and from that stems the fact that in Python this feature is slightly more limited than in JavaScript. Let's see.

In JavaScript user defined functions have a toString() method that returns a function's source code (including comments).


function sayHi(name) {
    name = name.toUpperCase();
    console.log(`Hi ${name}`)
}

console.log(sayHi.toString())
// function sayHi(name) {
//     name = name.toUpperCase();
//     console.log(`Hi ${name}`)
// }


For "native" functions or bound functions (obtained with .bind()) toString will just return: function xxx() { [native code] }.


function prepend(pr, msg) {
    return `${pr}${msg}${pr}`;
}
let fn = prepend.bind(null, "||");

console.log(`- bound function: ${fn.toString()}`);
//- bound function: function () { [native code] }

Notice that toString() works perfectly fine for functions created at runtime via eval():


let fn4 = eval("() => console.log('hi');");

console.log(fn4.toString());
//() => console.log('hi')


toString also works for classes, returning the whole class. It feels a bit surprising to me, cause under the covers classes are syntax sugar, and for a class Person what we really have is a Person function that corresponds to the constructor. So indeed, I don't know how we would get just the constructor code (other than extracting its substring from MyClass.toString()).


class Person {
    constructor(name) {
        this.name = name;
    }

    walk() {
        console.log("I'm waking")
    }
}

console.log(Person.toString())
// class Person {
//     constructor(name) {
//         this.name = name;
//     }

//     walk() {
//         console.log("I'm waking")
//     }
// }

console.log(typeof Person); 
//function


So it seems that when we define a function its source code is stored in some internal property of the corresponding function object. Checking the Language Specification it seems function objects have [[SourceText]] internal slot for that.

Things in Python are a bit different. We can obtain the source code of non native functions, classes and modules, but with some limitations, basically: getsource only works if it can open the file the source code exists in. The functionality is provided by the inspect.getsource() function. inspect is a standard python module, but the implementation feels like a bit "hackerish", like a functionality that was not initially intended and was added by means of leveraging some low level details. I've just said that in JavaScript functions have an slot pointing to its source code. This is not so straightforward in Python.

In Python a Function object has an associated code object (attribute __code__) that gives us access to the code of that function (through the co_code attribute). But that's a bytes object containing the Python bytecodes, not the Python source code. The __code__ object has 2 extra attributes: co_filename (with the full path to the python module where the function is defined) and co_firstlineno (with the line in that file where the function starts) (this is well explained here. So if we have the file where that function was defined, inspect.getsource can extract its source code, like this:


def format(txt: str):
    return f"[[{txt.upper()}]]"

print(inspect.getsource(format))
print(f"filename: {format.__code__.co_filename}")
print(f"firstlineno: {format.__code__.co_firstlineno}")

# def format(txt: str):
#     return f"[[{txt.upper()}]]"

# filename: /media/ntfsData/@MyProjects/MyPython_3.10_Playground/inspect/inspect_tests.py
# firstlineno: 27

This technique won't work for functions defined dynamically with exec-eval. There's not a file from which to get the source, and we'll get an exception: OSError: could not get source code.


format2_st = """
def format2(txt: str):
    return f'[[{txt.upper()}]]'
"""

def create_function(fn_st, fn_name):
	exec(fn_st)
	return eval(fn_name)

format2 = create_function(format2_st, "format2")
print(format2("aaa"))
# [[AAA]]

try:
    print(inspect.getsource(format2))
except Exception as ex:
    print(ex)
    # OSError: could not get source code

print(f"filename: {format2.__code__.co_filename}")
print(f"firstlineno: {format2.__code__.co_firstlineno}")

# [[AAA]]
# could not get source code
# filename: 
# firstlineno: 2
-----------------

inspect.getsource can also get the source code of a class. Classes do not have an associated code object, so the technique used has to be a bit different. You can check the inspect.py source code if you feel much intrigued.


class Person:
    def __init__(self):
        super().__init__()
        print("Person.__init__")

    def say_hi_to(self, to: str):
        return f"{self.name} says Hi to {to}"
        
print(inspect.getsource(Person))

# class Person:
#     def __init__(self):
#         super().__init__()
#         print("Person.__init__")

#     def say_hi_to(self, to: str):
#         return f"{self.name} says Hi to {to}"

By the way, inspect.getsource() can retrieve its own source code! nice :-)


inspect.getsource(inspect.getsource)
Out[14]: 'def getsource(object):\n    """Return the text of the source code for an object.\n\n    The argument may be a module, class, method, function, traceback, frame,\n    or code object.  The source code is returned as a single string.  An\n    OSError is raised if the source code cannot be retrieved."""\n    lines, lnum = getsourcelines(object)\n    return \'\'.join(lines)\n'

Wednesday, 21 August 2024

Python Pipes and Infix Notation

I recently came across a discussion about adding support for UFCS to Python. As expected, such idea was dismissed, but it gave me some good food for thought, as I had never heard before about this Uniform Function Call Syntax thing. So the idea is:

Allows any function to be called using the syntax for method calls (as in object-oriented programming), by using the receiver as the first parameter and the given arguments as the remaining parameters.

No major language implements this feature. In Kotlin we have a couple of things slightly related: Extension Functions and Function Types with Receiver. I can get a reference to an existing function and type it as a Function Type with Receiver, and in that case I can invoke that function reference both through a receiver, or passing it over as the first parameter.

The main interest of UFCS for me is for chaining function calls (aka pipes) and for that I would prefer a pipe operator. In this previous post I mentioned a very interesting project where they use this a combination of 2 operators: |> as a pipe operator. For that to work functions using that new operator have to be decorated with a decorator that will rewrite its source. That's a pretty amazing and crazy approach, but there's a more simple one, using a wrapper object (to wrap the functions being piped) that overrides the binary operator | (__ror__). There's an amazing project that does that, and more! The main Pipe class is just like this:


class B:
    def __init__(self, f=None, *args, **kw):
        self.f = f
        self.args = args
        self.kw = kw


class Pipe         (B): __ror__ = lambda self, x: self.f(x, *self.args, **self.kw)


And we use it like this:


def transform(st: str) -> str:
    return st[0].upper() + st[1:]

def clean(st: str, to_remove: list[str]) -> str:
    for it in to_remove:
        st = st.replace(it, "")
    return st

def wrap(st, wrapper: str) -> str:
    return f"{wrapper}{st}{wrapper}"

print("this is not Asturies" 
    | Pipe(transform)
    | Pipe(clean, ["not"])
    | Pipe(wrap, "--")
)

# --This is  Asturies--

Let's see a typical use case with iterables, mapping, filtering...



cities = ["Paris", "Prague", "Lisbon", "Porto"]
print(cities 
    | Pipe(lambda items: map(str.upper, items))
    | Pipe(lambda items: filter(lambda x: x.startswith("P"), items))
    | Pipe(list)
)

# ['PARIS', 'PRAGUE', 'PORTO']

That's a bit verbose, so to improve that the (very) smart guy behind pipe21 added extra classes for all the common use cases (Filter, Map and many more). With that, we can rewrite the above code like this:


cities = ["Paris", "Prague", "Lisbon", "Porto"]
print(cities 
    | Map(str.upper)
    | Filter(lambda x: x.startswith("P"))
    | Pipe(list)
)

# ['PARIS', 'PRAGUE', 'PORTO']

I should mention that the almighty coconut language comes with support for pipes (indeed it supports multiple pipes operators).

Kotlin supports operator overloading, but it has a more limited set of operators that can be overloaded, and "|" is not among them, so it seems we can not port this approach.

Related to this I've come across another pretty amazing use of operator overloading (and the callable concept), enabling infix notation for function calls. I copy-paste below the code:


from functools import partial

class Infix(object):
    def __init__(self, func):
        self.func = func
        
    def __or__(self, other):
        return self.func(other)
        
    def __ror__(self, other):
        return Infix(partial(self.func, other))
        
    def __call__(self, v1, v2):
        return self.func(v1, v2)

>>> @Infix
... def add(x, y):
...     return x + y
...
>>> 5 |add| 6
11

>>> instanceof = Infix(isinstance)
>>>
>>> if 5 |instanceof| int:
...     print "yes"
...
yes

>>> curry = Infix(partial)
>>>
>>> def f(x, y, z):
...     return x + y + z
...
>>> f |curry| 3

>>> g = f |curry| 3 |curry| 4 |curry| 5
>>> g()
12        
        

I find particularly appealing the isinstance example. We are converting the isinstance function in a sort of instanceof operator that being used to javascript instanceof operator, and kotlin is operator (in Kotlin, as in JavaScript, reference equality is checked with ===) seems more natural to me. This idea is pretty amazing cause we are using operator overloading (that can only be applied to a predefined set of operators) to create sort of new operators!!!

Monday, 12 August 2024

Python Return From Generator

In my previous post I talked about returning values (along with yielding) from generators and said that it also applied to Python. While writing the Python equivalent to the code in that post I've realised that there are more differences than I expected.

Let's start by "normal" (synchronous) iterators. The most important difference is that Python iterators yield a value in each iteration (rather than a done-value pair as JavaScript does) and throw an StopIteration exception when they reach its end (while JavaScript yields a {done: true, value: undefined}). If we return a value from the generator it's set in the value attribute of the exception. We don't have access to that exception in for-in loop, and if we do an extra next() call once outside the loop, it throws an exception again, but this time with value set to None (so this behaviour is equivalent to the JavaScript one). This means that if we want access to that value, we either use a while loop, or use a class (that as in my previous post I called AsyncGeneratorWithReturn) that wraps that loop for us. So we end up with this code (notice how in __iter2__ we leverage the yield from construct)


from typing import Any, Callable

def citiesGen():
    yield "Paris"
    yield "Porto"
    return "Europe"


print("- for of loop")
cities = citiesGen()
for city in cities:
    print(city)

try:
    print(next(cities))
except StopIteration as ex:
    print(f"return value: {ex.value}")
    
# Paris
# Porto
# None

#that's the same behaviour as in JavaScript, once it has generated the "end" (first StopIteration Exception with the value attribute set)
#the next calls raise StopIteration with value as None

print("- while loop")
#this works fine:
cities = citiesGen()
while True:
    try:
        city = next(cities)
    except StopIteration as ex:
        print(f"return value: {ex.value}")
        break


print("- GeneratorWithReturn")

class GeneratorWithReturn:
    def __init__(self, generator_fn: Callable, strategy: int):
        self.generator_fn = generator_fn
        self.result = None
        self.strategy = strategy

    # this works fine
    def __iter1__(self):
        cities = self.generator_fn()
        while True:
            try:
                yield next(cities)
            except StopIteration as ex:
                self.result = ex.value
                break

    # but it can be refactored to just one line!
    def __iter2__(self):
        self.result = yield from self.generator_fn()        


    def __iter__(self):
        return getattr(self, f"__iter{self.strategy}__")()    


for strategy in (1, 2):
    cities = GeneratorWithReturn(citiesGen, strategy)
    for city in cities:
        print(city)
    print(f"return value: {cities.result}")


I said in my previous post that this idea of yielding and finally returning a value seemed mainly interesting to me for async iteration. We know that same as JavaScript Python features async generators (and async for loops). And the interesting thing is that they come with a limitation, we can not return values from them (we get a #SyntaxError: 'return' with value in async generator) and we can not use yield from (the equivalent to JavaScript's yield *). However, this is not a big deal, we can easily workaround the limitation by yielding an object of a particular class (that I've called GeneratorReturn) that wraps the return value, and wrapping the async for in an AsyncGeneratorWithReturn class that takes care of this return value (rather than taking care of a StopAsyncIteration exception).



import asyncio

class GeneratorReturn:
    def __init__(self, value):
        self.value = value

class AsyncGeneratorWithReturn:
    def __init__(self, agen_fn):
        self.agen_fn = agen_fn
        self.result = None

    async def __aiter__(self):
        agen = self.agen_fn()
        while True:
            if isinstance(res := await anext(agen), GeneratorReturn):
                self.result = res.value
                return
            else:
                yield res


async def designVacations():
    destinations = []
    await asyncio.sleep(1)
    destinations.append("Paris")
    yield "destination found: Paris"
    await asyncio.sleep(1)
    destinations.append("Porto")
    yield "destination found: Porto"
    	#return (" -> ").join(destinations)
    	#SyntaxError: 'return' with value in async generator
    yield GeneratorReturn((" -> ").join(destinations))


async def async_main():
    print("- AsyncGeneratorWithReturn");
    vacationsProcess = AsyncGeneratorWithReturn(designVacations)
    async for city in vacationsProcess:
        print(city)
    print(f"return value (travelPlan): ${vacationsProcess.result}")

    # destination found: Paris
    # destination found: Porto
    # return value (travelPlan): $Paris -> Porto

if __name__ == "__main__":
    asyncio.run(async_main())


It's interesting to notice that while Javascript for await can be used both with asynchronous and synchonous iterators, Python async for only works with asynchronous iterators. In Javascript the iteration method of asynchronous and synchronous iterators is called next, while in Python we have 2 different methods: anext() and next(). All this is related to the fact that while in JavaScript we can await for a Promise and for a normal value, in Python we can only await for an awaitable (coroutine, Future or Task).

Monday, 5 August 2024

JavaScript Return From Generator

Recently I came across a discussion about using the return value from a a generator. This applies both to Python and JavaScript, but in this post I'll focus on JavaScript. We know that generators yield values, but they can also return a value. So far I had only used return statements in generators to finish/close on condition, just a "return;" without a value. But you can do a "return x;" and that x will show up in the object returned in the last iteration (the one that indicates that the iteration has finished). That means that rather than the typical {done: true, value: undefined} you'll get a {done: true, value: x}. The thing is that the most common way to iterate a generator, the for-of loop, won't give us access to that value. Let's see an example:


function* citiesGen() {
    yield "Paris";
    yield "Porto";
    return "Europe";
}

console.log("- for of loop");

cities = citiesGen();
for (let city of cities) {
    console.log(city);
} 
console.log(cities.next())
// Paris
// Porto
// { value: undefined, done: true }

We iterate the yielded values, and the loop stops when the generator returns a {done: true, value: "Europe"}, but the loop does not give us acces to that value. If after the loop we invoke next() again, it will return this object: {done: true, value: undefined}, the value is undefined, no longer "Europe", so we've lost it.

To circunvent this problem we can use a less elegant while loop rather than a for-of, like this:


let city;
cities = citiesGen();
while (!city?.done) {
    city = cities.next();
    if (!city.done) {
        console.log(city.value);
    }
}
console.log(`return value: ${city.value}`) 

That works fine, but that while loop looks a bit ugly. We can wrap that logic into a class implementing the Iteration Protocols and providing access to the returned value. That way we can use a for-of loop, like this:


class GeneratorWithReturn {
    constructor(generatorFn) {
        this.generatorFn = generatorFn;
        this.result = undefined;
    }

    *[Symbol.iterator]() {
        let item;
        let generatorOb = this.generatorFn();       
        while (!item?.done) {
            item = generatorOb.next();
            if (!item.done) {
                yield item.value;
            }
        }
        this.result = item.value;        
    }
}

cities = new GeneratorWithReturn(citiesGen);
for (let city of cities) {
    console.log(city);
} 
console.log(cities.result);

//Paris
//Porto
//return value: Europe 

That looks much better, but we can rewrite our iterator in a very concise way by leveraging the yield* operator, that delegates to another iterable object. The very interesting thing is that with yield* we get acces to the yielded values, and also to the returned value, like this:


class GeneratorWithReturn {
    constructor(generatorFn) {
        this.generatorFn = generatorFn;
        this.result = undefined;
    }

    *[Symbol.iterator]() {
        this.result = yield* this.generatorFn();        
    } 
}

cities = new GeneratorWithReturn(citiesGen);
for (let city of cities) {
    console.log(city);
} 
console.log(cities.result);

//Paris
//Porto
//return value: Europe

This feature of returning a value from a generator does equally apply to async iterators. And this is really useful, cause while I don't see particular real use cases for returning a value in normal iterators, in async iterators it allows us to express very nicely the idea of an async function that produces (yields) intermediate values (for example completion percentages) and a final return value. Imagine we have an asynchronous process that calculates several destinations to return a travel plan. Apart from that final "travel plan" we want to get each destination as it gets "calculated. We can express it with an async generator like this:


function aSleep(delay) {
    return new Promise(resolve => setTimeout(resolve, delay))
}

async function* designVacations(){
    let destinations = []
    await aSleep(1000);
    destinations.push("Paris");
    yield "destination found: Paris";
    await aSleep(1000);
    destinations.push("Porto");
    yield "destination found: Porto";
    return destinations.join(" -> ");
}


We'll use it with an adapted version of the class that we've just seen, but that this time providing the Asynchronous Iteration


class AsyncGeneratorWithReturn {
    constructor(generatorFn) {
        this.generatorFn = generatorFn;
        this.result = undefined;
    }

    // notice that though I'm not using await inside the function I have to mark it as async
    // otherwise I get an error: TypeError: yield* (intermediate value) is not iterable
    async *[Symbol.asyncIterator]() {
        this.result = yield* this.generatorFn();        
    } 
}


(async function asyncMain() {
    vacationsProcess = new AsyncGeneratorWithReturn(designVacations);
    for await (let city of vacationsProcess) {
        console.log(city);
    } 
    console.log(`return value (travelPlan): ${vacationsProcess.result}`);
}

// destination found: Paris
// destination found: Porto
// return value (travelPlan): Paris -> Porto


Notice that we use yield* to delegate to another async iterator, there's not a await yield* kind of operator. This is pretty nice as the same operator works for delegating to normal or async iterators. Another point to notice is that though we don't have any await in our asyn generator method, it has to be marked as async: async *[Symbol.asyncIterator]().