Python walrus limitation

Sometime ago I talked about Python Assignment Expressions aka walrus operator, and overtime I've really come to appreciate it. Some weeks ago I came across and odd limitation of this operator, it can not be used for assigning to an attribute of an object (so you can use it only with variables), as you'll get an "SyntaxError: cannot use assignment expressions with attribute" error. I don't remember what I was trying when I hit this problem, but now I can think of an example like the verify method below:

class Country:
    def __init__(self, name):
        self.name = name
        self.cities = None
        self.last_verification = None
    
    def _lookup_cities(self):
        print("looking up cities")
        return ["Paris", "Toulouse", "Lyon"]

    def verify(self):  
	# [code to perform verification here]
        print(f"last verification done at: {(self.last_verification := datetime.now())}")
        # SyntaxError: cannot use assignment expressions with attribute
        

So the above throws a SyntaxError: cannot use assignment expressions with attribute. I can think of one technique to circunvent this limitation, leveraging an "assign()" custom function that I use sometimes to conveniently set several properties in one go).

def assign(val: Any, **kwargs) -> Any:
    for key, value in kwargs.items():
        setattr(val, key, value)
    return val

    def verify(self):
	# [code to perform verification here]
        print(f"last verification done at: {assign(self, last_verification=datetime.now()).last_verification}")

That syntax is cool, but having the print() call as the most visible part of the statement is probably confusing, as it makes us think that the important action in that line is print, while setting the last_verification attribute is the real deal in that line. So probably using the "traditional syntax" would make sense:

    def verify(self):
	# [code to perform verification here]
        self.last_verification = datetime.now()
        print(f"last verification done at: {self.last_verification}")

Another example for using this technique:

    def _lookup_cities(self):
        print("looking up cities")
        return ["Paris", "Toulouse", "Lyon"]
		
    def get_cities(self) -> list[str]:
        # return self.cities or (self.cities := self._lookup_cities())
        # SyntaxError: cannot use assignment expressions with attribute
        return self.cities or assign(self, cities=self._lookup_cities()).cities

Notice that this case could be rewritten using a lazy property via functools @cached_property

    @cached_property
    def cities(self):
        print(f"initializing lazy property")
        return self._lookup_cities()

That looks really neat, but notice that I think we should be careful with the use of cached/lazy properties. On one hand, cities represents data belonging to the object, it's part of its state, so using a property rather than a method feels natural. But on the other hand, to obtain those cities maybe we do a http or db request, an external request. This kind of external interaction can be considered as a side-effect, so in that sense we should use a method. In general, I think lazy properties should only be used for data that is calculated based on other data belonging to the object (and if that data is read-only or we observe it and update the property accordingly, and if that calculation is not too lenghty, as accessing a property should always be fast). This is an interesting topic and it has prompted me to revisit this stackoverflow question that I remember to have read several times over the last 15 years.

I have to add that these examples would look much nicer if we had a pipe operator (like in elixir) and we could write something like this:
return self.cities or (self | assign(cities=self._lookup_cities()) | .cities)

File Locking 2

One decade ago I wrote this post about file-locking (aka file-sharing) and I've been revisiting it lately. That post was focused on Windows, and indeed I had not realized that "the attitude" towards file-locking in Windows and Linux is pretty different. We can say that in Windows file-locking is an integral part of how the the OS manages files. The sharing mode (dwShareMode) parameter of the CreateFile function (used to open files) determines file-sharing.

On the other side, for Unix-like OS's file-locking does not seem to be a major concern. By default no locking of any kind is performed when opening a file (so multiple processes can read and write to the file at the same time). The open() system call does not have any parameter related to locking-sharing. It's true that there is support for file-locking (fcntl, flock, lockf), but it's rather loose, as we can say that it's cooperative:

File locks under Unix are by default advisory. This means that cooperating processes may use locks to coordinate access to a file among themselves, but uncooperative processes are also free to ignore locks and access the file in any way they choose. In other words, file locks lock out other file lockers only, not I/O.

The above was pretty reassuring cause at work we have one application (not developed by us) that appends information to a file, and we wanted to write some code that would be periodically reading that file, with both applications running on Linux. We could not afford opening the file for reading and that right at that moment the other application tried to open it for writing and failed (maybe crashing, as we don't know what kind of error handling, if any, the application has).

For full peace of mind I did a fast check. Open a file for reading in Python (fr = open("test.txt", "r")) and while it's open append lines to it from the terminal (echo hi >> test.txt). No crash and the file gets updated without a problem.

The surprising thing is that doing just the same test on Windows also works fine! Well, indeed it makes good sense, let me explain. The Python open() function does not provide any sort of "File Sharing" parameter, while the different .Net file opening methods do. One can easily think that this is because while both languages are multiplatform, Python was born in a more Linux oriented community, while .Net was for many years Windows-centric, so both libraries reflect what exists in their "favorite" OS. But at the same time, I guess Python developers decided that it should try to show the same behaviour on any OS. Given that Python's open() on Linux can not provide any locking behaviour (cause as I've already mentioned the underlying Linux open() system call does not), it should do the same on Windows, so when it invokes the underlying Windows API CreateFileW function, it does so requesting Read and Write sharing. From here:

Python’s builtin open() shares read and write access, but not delete access. If you need a different share mode, you’ll have to call CreateFile directly via ctypes or PyWin32’s win32file module.

JavaScript Arguments and Arrow Functions

After writing last week about how to sort of emulate JavaScript's arguments-object in Python, it seems a good idea to mention the special behaviour of the arguments-object in arrow functions. When arrow functions were added to JavaScript most articles promptly informed about the particular behaviour of the this-object in arrow functions (that is indeed one of its main features). The main idea that got stamped in my brain was "arrow functions do not have dynamic this, but lexical this". This means that they do not receive as "this" value the "receiver", the object on which they get invoked, but the "this" of their lexical environment. I used to think that such "this" value would get bound to the arrow function in a similar way to when we use function.bind(), but it's not like that. Arrow functions are not bound-functions with some particular property pointing to the "this", but they just look it up in its scope chain. From [1] and [2]:

[1] In essence, lexical this means that an arrow function will keep climbing up the scope chain until it locates a function with a defined this keyword. The this keyword in the arrow function is determined by the function containing it.

[2] Arrow functions lack their own this binding. Therefore, if you use the this keyword inside an arrow function, it behaves just like any other variable. It will lexically resolve to an enclosing scope that defines a this keyword.

What has prompted this post is another thing that I had not realized until recently (and it's not so hard as it comes in the MDN documentation) is that something similar happens with the arguments-object.

Arrow functions don't have their own bindings to this, arguments, or super, and should not be used as methods.

"don't have their own bindings" means that they'll be looked up in the scope chain (as any other varible). Arrow functions do not have its own arguments-object containing the arguments received by the arrow, but look up that arguments-object in the scope chain, so it will contain the arguments received by the first enclosing function (at the time of the arrow creation, not of the arrow invokation) that is not an arrow. Notice how in the example below, it prints "aaa" (from its scope chain) rather than "bbb" (that is what it receives as parameter).

// arrow functions do not have an "arguments" or "this" binding, they take them from their scope chain
function f2(a) {
    console.log("inside f");
    let fn1 = (b) => console.log(arguments);
    return fn1;
}

let arrow = f2("aaa")
arrow("bbb")
//[Arguments] { '0': 'aaa' }

I'll leverage this post to mention that changing where an entry in the arguments-object points to changes also where the variable itself points to. So this behaviour corresponds to what happens in Python with the FrameType.f_locals write-through proxy, not to what we have in Python via locals() (a snapshot through which we can not change where the original variables point to).

function test(user) {
    console.log(`user: ${user}`)
    console.log(`arguments[0]: ${arguments[0]}`);
    arguments[0] = "Iyán";
    console.log(`arguments[0]: ${arguments[0]}`);
    console.log(`user: ${user}`)
}

test("Xuan");

// user: Xuan
// arguments[0]: Xuan
// arguments[0]: Iyán
// user: Iyán

JavaScript vs Python: arguments vs locals()

It's not uncommon to have a function that acts as an adapter, wrapping another existing function, having its same parameters and performing some action before and/or after invoking the original function with those parameters. JavaScript provides a very convenient way to invoke the original function without having to write again that full list of arguments in the call, just using the arguments array-like object and the spread operator. I mean:

function doAction(a, b, c) {

}
function checkParams(a, b, c) {
	if (a === "undefined") {
		return null;
	}
	return doAction(...arguments);
}

In many cases the arguments object is not necessary, we can just use the rest operator in the adapter function rather than the list of arguments. That's great when the adapter function is not doing anything with the arguments, for example when creating generic wrappers like this:

function doAction(a, b, c) {

}

function createLoggingWrapper(fn) {
	return (...params) => {
		console.log("started");
		res = fn(...params);
		console.log("finished");
		return res;
	};
}

In Python the equivalent to the rest-spread operaters is the packing-unpacking operator (that given that Python features named parameters has extra support for that) . So we can write the second example just like this:

from functools import wraps

def doAction(a, b, c):
    print(f"{a}{b}{c}")


def createLoggingWrapper(fn):
    @wraps(fn)
    def logged(*args, **kwargs):
        print("started")
        res = fn(*args, **kwargs)
        print("finished")
        return res
    return logged

doAction = createLoggingWrapper(doAction)
doAction("AA", c="CC", b="BB")

But what about the first example? We don't have a direct equivalent to the arguments object, but we have a workaround that is almost equivalent, using locals(). I already talked about the locals() function in a previous post. It returns the list of variables in the "local namespace" at that moment (that is function arguments and local variables), so if we run it before any local variable is declared we can use it as the JavaScript arguments object. Let's see:

def format(name, age, city, country):
    return f"formatting: {name} - {age} [{city}, {country}]"

def format_with_validation(name, age, city, country):
    if city:
        print("validation OK")
        return format(**locals()) # equivalent to: format(name=name, age=age, city=city, country=country)
    else:
        print("validation KO")
        return None

print(format_with_validation("Francois", 40, "Paris", "France"))
print(format_with_validation("Francois", 40, "Lyon", "France"))

# validation OK
# formatting: Francois - 40 [Paris, France]
# validation OK
# formatting: Francois - 40 [Lyon, France]

There's one gotcha. locals() returns also those variables (free-vars) trapped by closures. So if our wrapping function happened to be working as a closure, this technique will not work, as we'll be passing extra parameters to the wrapped function (the "counter" variable in the example below).

def format(name, age, city, country):
    return f"formatting: {name} - {age} [{city}, {country}]"

def create_format_with_validation():
    counter = 0
    def format_with_validation(name, age, city, country):
        nonlocal counter       
        counter +=1
        print(f"invokation number: {counter}")
        if city:
            print("validation OK")
            return format(**locals()) # equivalent to: format(name=name, age=age, city=city, country=country)
        else:
            print("validation KO")
            return None
    return format_with_validation

format_with_validation = create_format_with_validation()
try:
    print(format_with_validation("Francois", 40, "Paris", "France"))
except Exception as ex:
    print(f"Error: {ex}")
    # TypeError: format() got an unexpected keyword argument 'counter'. Did you mean 'country'?

Well, we can easily fix that by removing from the object returned by locals the entries corresponding to trapped variables. Indeed, Python is so amazingly instrospective that we can automate that. The code object of one function has a co_freevars attribute that gives us the names of the freevars used by that function. So we can write a "remove_free_vars" function and do this:

def remove_free_vars(loc_ns: dict, fn) -> dict:
    return {
        key: value for key, value in loc_ns.items()
        if key not in fn.__code__.co_freevars
    }

def create_format_with_validation_closure_aware():
    counter = 0
    def format_with_validation(name, age, city, country):
        nonlocal counter       
        counter +=1
        print(f"invokation number: {counter}")
        if city:
            print("validation OK")
            return format(**(remove_free_vars(locals(), format_with_validation))) 
        else:
            print("validation KO")
            return None
    return format_with_validation

format_with_validation = create_format_with_validation_closure_aware()
print(format_with_validation("Francois", 40, "Paris", "France"))

There's one additional gotcha. As locals() returns a dictionary-like object, if the function to be invoked has been defined restricting that some of its arguments can not be provided as named arguments (that odd feature that I described here).

Python Generator with Calculated Values

I frequently take a look into the Python discussion forum, mainly to the "ideas" section, where people propose things they would like to see in the language or in libraries. Though sadly most of the time most requests for additions to the language are promptly rejected (sometimes with absurd reaons, as if given by people that do not use other modern languages...) the proposed alternatives and the ideas themselves can be pretty interesting. Sometime ago I came across an idea that I was not immediatelly grasping, Literal syntax for generatos.

We know generators generate values lazily, as requested. But if we had a literal syntax we would be already providing all values, so why not just use a list? Well, because the idea would be supporting that some values are generated by functions, that get invoked when the iteration requests it. Well, the guy showed the workaround that he's using, and it's pretty, pretty, interesting:

queries: Iterator[str] = (lambda: (
    (yield "abc"),
    (yield t if (t := expensive_func()) else None),
    (yield "something else"),
))()

Honestly it took me quite a while to understand what's going on there. So he's using an immediatelly invokable lambda with an expression containing multiple expressions (this is the trick that I explained some time ago), where each subexpression is yielding a value. In Python yield can be used both as a statement and as an expression. The clear case for the latter is when we write a = yield "whatever", for using with the send(value) method). In this example, for the yield to be considered as an expression it has to be wrapped with parentheses.

It looks pretty nice, and it would have never occurred to me. The alternative that I could have envisioned would be a generator factory like this:

def lazy_gen(values: Iterable) -> Generator:
    """
    returns a Generator object
    """
    def gen_fn_wrapper():
        for it in values:
            yield it() if callable(it) else it
    return gen_fn_wrapper()

Indeed, maybe this approach is easier to understand. Let's see an example:

def expensive_format(msg: str):
    sleep(2)
    return msg.upper()

print("started")
messages: Generator[str, None, None] = (lambda: (
    (yield "abc"),
    (yield expensive_format("my message")),
    (yield expensive_format("my message 2")),
    (yield "something else"),
))()

for message in messages:
    print(message)

print("- second approach")

messages = lazy_gen([
    "abc", 
    lambda: expensive_format("my message"),
    partial(expensive_format, "my message 2"),
    "something else",
])
for message in messages:
    print(message)

There's another approach that they mention in the discussion thread and that uses a trick that I find rather confusing. We can use a decorator to immediatelly invoke a function that we've just defined. This means that we can define a normal generator function that contains statements and invoke it to obtain a generator object that will get assigned to a variable with the name of the function that we've just decorated. As I've said, it feels terribly confusing, but I'll include it here for the sake of completion. I have to say that defining the decorator inline with a lambda feels pretty cool :-) Let's see:

@lambda x: x() # Or @operator.call
def messages():
    yield "abc"
    yield expensive_format("my message")
    yield expensive_format("my message 2")
    yield "something else"

for message in messages:
    print(message)          

yield expressions are, well, expressions, which means that in principle we can use them anywhere an expression can be used, for example as argument to a function. I mean

def f1():
    yield "a"
    print(str((yield "b")))
    yield "d"
  

gen = f1()
print(next(gen))
#'a'
print(next(gen))
#'b'
print(gen.send(11))
#11
#'d'

As Bestas

I had pending to watch As Bestas, a French-Spanish (Galician) film since quite a while ago. I wanted to watch it with my parents, particularly with my Galician father, but as it runs for more than 2 hours it was not easy to find the appropriate moment. This Christmas Eve has been the time. I didn't know that it was based on a true story that happened also in rural Galicia, to a Dutch couple that settled there, between 1996 and 2014. The film is particularly appealing to me cause though I am and feel so deeply Asturian, France is my second country, my second identity, and Galicia is the land of all my paternal ancestors, so I consider it my third country and identity. Dialogs in the film are in French, Galician and some Spanish I think also. It felt odd to me that I could understand better the parts in French than those in Galician, you can never disappoint your ancestors enough...

So in the film we have a French couple of "neo-rurals" that have settled somewhere in the rural interior of Galicia. These are not "digital nomads" teleworking from a small paradise, but 2 hard worker idealists that want to make a living working the land (in a traditional way it seems). From the beginning we see that they do not get on well with some of their neighbours, 2 of them in particular, 2 brothers. At first we can think that this hostility, this distrust, is because of some sort of xenophobia, which would be paradoxal in a land, Galicia, that in spite of its wild natural wealth has never managed to feed all its children, making Galicians (even more than Asturians) one of the most migrant people in the world (you can find Galicians anywhere, indeed the joke says that North Americans came across one Galician in the Moon when they landed there).

The permanent tension between the French couple and these 2 brothers, that bully them and keep a continuous threatening attitude to them creates a thick and oppressive atmosphere, accentuated by the Galician landscape and the backward feeling of the village. The "bar" where good part of the interaction between the French man and the 2 brothers takes place feels like taken out from another century. This reminds me of one discussion with one friend of mine (also an Asturian with Galician ancestry) when he told me about how poor and less developed some parts of rural Galicia feel when compared to their Asturian counterparts. It's strange, cause on the other side, Galician cities (particularly Vigo), that is the part of Galicia that I know, feel way more cosmopolitan and developed than Asturian cities.

As the film evolves, the reason for the resentment that the 2 brothers express for the French couple unveils. The couple were some of the main opponents that refused selling their lands to an energy company that wanted to set up an Eolian farm in the village, and needed for it the lands of all the villagers, so this refusal prevented other villagers from selling. We arrive then to the most intense moment of the film, the discussion where the main brother explains full of bitterness and hatred how his miserable life would have changed if he had sold his lands and left the village for the city. It's a point where you end up empathizing with a character that up to that point had been profoundly revolting.

Well, I think that's all, I can not tell more without fully spoiling the film. Just reserve 2 hours of your life (OK, a bit more if you have to search the torrent and download it) and watch it.

Closing Javascript Generators

In this previous post about closing Python generators I mentioned that JavaScript generators had a similar feature that would deserve a post on its own, so here it is.

JavaScript generators have a return() method. We can think of it as partially equivalent to Python's close() method. This is so for the simple (and main I guess) use cases that I explained in my previous post (use it as a replacement for break, and when you are passing the generator around to other methods and one of them can decide to close it). For example:

function* citiesGen() {
    yield "Paris";
    yield "Porto";
    return "Europe";
}

// using .return() rather than break
let cities = citiesGen();
for (let city of cities) {
    if (city == "Porto") {
        cities.return();
        console.log("closing generator");
    }
    console.log(city);
}
/*
Paris
closing generator
Porto
*/

Then we have the more advanced cases, for which indeed finding a use case seems not so apparent to me. Here it's where the differences with Python's close() are important. JavaScript return() accepts a value, that will be returned as part of the value-done pair returned when the generator is finished. This "when it's finished" is key, as a try-finally in the generator code can prevent the return() call from finishing the generator in that call. It will continue to produce values as instructed from the finally part, and once completed will return the value that we had passed in the return() call. The theory:

The return() method, when called, can be seen as if a return value; statement is inserted in the generator's body at the current suspended position, where value is the value passed to the return() method. Therefore, in a typical flow, calling return(value) will return { done: true, value: value }. However, if the yield expression is wrapped in a try...finally block, the control flow doesn't exit the function body, but proceeds to the finally block instead. In this case, the value returned may be different, and done may even be false, if there are more yield expressions within the finally block.

And the practice:

function* citiesGen2() {
    yield "Paris";
    try {
        yield "Lyon";
        yield "Porto";
        return "Stockholm";
    }
    finally {
        yield "Lisbon";
        yield "Berlin";
    }
}

cities = citiesGen2();
console.log(cities.next());
console.log(cities.next());
console.log(cities.return("Over"));
console.log(cities.next());
console.log(cities.next());
console.log(cities.next());


// { value: 'Paris', done: false }
// { value: 'Lyon', done: false }
// { value: 'Lisbon', done: false }
// { value: 'Berlin', done: false }
// { value: 'Over', done: true }
// { value: undefined, done: true }

If this feels odd to you, you're not alone :-D. This is quite different from Python, where a call to close() always finishes the generator, even if we are catching the Exception and returning something from it.

generator.close()

Raises a GeneratorExit at the point where the generator function was paused. If the generator function catches the exception and returns a value, this value is returned from close(). If the generator function is already closed, or raises GeneratorExit (by not catching the exception), close() returns None. If the generator yields a value, a RuntimeError is raised. If the generator raises any other exception, it is propagated to the caller. If the generator has already exited due to an exception or normal exit, close() returns None and has no other effect.

Same as Python, JavaScript generators also have a throw() method, and again, I see no much use for it.