Tour Triangle

Warning: Please, notice that I consider France as my second country and I love it, so don't take the beginning of this post as an attack against France, I just criticise a common attitude in one part of the French population that, as someone that loves architecture, rather bothers me, and that for someone that has lived most of his life in a city (Xixón) where 15 stories buildings are just everywhere and people do not particularly complain about them, is pretty difficult to understand (notice that it goes far beyond the grand-ensemble trauma, and anyway, it's not HLM buildings that cause troubles, it's some of the people living there who do it.

The irrational hatred for High Rise Buildings could almost be considered as an essential part of the modern French identity. It's not that everyone hates them, indeed I guess part of the population does not care too much about them, but the percentage of people that vehemently hates any building higher than 50 meters and that is willing to create collectives, flood the media with lies about the "evils" of such buildings, go to court or whatever... in order to stop its construction is absolutely perplexing. Probably Marseille and Lyon are the 2 cities less affected by this redneck attitude, but unfortunately, Toulouse is right on the other side of the spectrum. This is the city where these collectives of "rednecks against the modern urban landscape" has an uncomprehensible power, furthermore, they even redefine the definition of high-rise, for these idiots which main interest in life is to hinder the development of the city, any building beyond 30 meters is a skyscrapper....

Things in Paris have been looking better in these last years. In 2018 Paris intra-muros saw the completion of the first sky-scrapper built in this century, the new TGI (notice of course that La Defense, that continues to thrive with interesting developments, is outside the limits of the "ville de Paris"), and the "Tours Duo" by Jean Nouvelle are well under construction (and there are plans for another skyscrapper in this same area). Furthermore, and this is what motivates this post, I've read recently that finally, the Tour Triangle has managed to overcome the last 2 (of many) appeals presented by some redneck association, and is now confirmed to become a new addition to the Parisian sky line. The images in in this article are just amazing, it looks massively beautiful!

IQueryable and Default Interface Methods

The discussion in the .Net world as to whether it's OK to have repositories returning IQueryable rather than IEnumerable has been rather present all these years. The common consensus is that it's not, and your repositories should not return an IQueryable. There are multiple reasons for this, but from an architecture point, probably this is one of the best I've read.

The point is that using IQueryable, you're asking for a query builder and not for a model. A query builder specifies how the query should be, how to get the data. Then why are we using a repository? Isn't the repo's job to know how to get the thing we want? We're using the repository because we want to specify what to get, not how to get it.

This goes along the idea of repositories having methods for "fat queries", so that the repository itself does the filtering, rather than returning a "collection" on which you apply filters. A similar approach is combining Repository and Specification pattern, your repository no longer have specific methods for each fat query, but you pass it over a specification with the query itself and it's the repository who performs the query.

While I agree with all the above, the idea of returning an IEnumerable and filtering it later, does not always seem like such a sin to me, after all we use Linq to Objects all the time. The big problem with returning an IEnumerable is that we would lose all the magic of IQueryable transforming our filters into a query and running it for us. The Linq methods for IEnumerable and IQueryable have been implemented as extension methods, and as we know this is a compiler trick, and method override and runtime resolution do not apply. If my IPersonRepository.GetItems returns and IEnumerable, my PersonSqlServerRepository.GetItems has to return an IEnumerable also in its signature (as C# lacks of Method return type covariance). We can respect that signature but return an IQueryable anyway, I mean:

IEnumerable GetItems()
{
 ...
 return dbContext.Persons; //returns and IQueryable
}

But the problem is that then, when doing myPersonsRepository.GetItems().Where(.....); the C# compiler would set a call to System.Linq.Enumerable.Where (rather than System.Linq.Queryable.Where), so we would be retrieving all the items from the Database, and then performing the filtering.

Now that C# 8 has added Default Interface Methods, if all the linq methods where added as default methods to IEnumerable and IQueryable the whole thing would change. As the method resolution would be done at runtime, calling Where in the above case would end up going to the IQueryable.Where, and generating the consequent sql query. This would be really sweet, but I've not heard so far about any plans of "copy-pasting" (you can not just move them due to backwards compatibility) the methods in System.Linq.Queryable into IQueryable as default interface methods.

Promise.all with a Limit

A few months ago I wrote a post about how to limit the number of async operations that we run in parallel in JavaScript. Last week I came up with something related, but a bit different. What if we want to run n async operations and wait for all of them to complete, but we want to limit how many of them we run in parallel. A good example of this is if we want to do several http requests and combine their results, but we don't want to flood the server with all the requests at the same time (indeed browsers manage this themselves limiting how many requests to the same server they open in parallel, so we won't be overloading the server, but at JavaScript level the requests are already launched, and the timeout starts to count...). Waiting for all the async operations to complete is what we get with Promise.all, but it does not provide a limit option.

Well, the idea seems simple to implement, here it goes my take on it:

class PromiseAllWithLimit{
    //tasks: Array of functions that return a Promise
    constructor(tasks, maxItems){
        this.allTasks = tasks;
        this.maxItems = maxItems;
        this.taskResults = [];
        this.nextTaskId = 0;
        this.resolveAll = null; //points to the resolve function to invoke when all tasks are complete
        this.completedCounter = 0;
    }

    //tasks: Array of functions that return a Promise
    //returns a Promise that gets resolved when all tasks are done
    run(){
         while(this.nextTaskId < this.maxItems){
            this._runNextTask();
        }
        return new Promise((res) => this.resolveAll = res);
    }

    _runNextTask(){
        //we need this var to get it trapped in the closure to run in the task.then()
        let curTaskId = this.nextTaskId
        let nextTask = this.allTasks[this.nextTaskId];
        this.nextTaskId++;
        nextTask().then((result) => {
            this.completedCounter++;
            this.taskResults[curTaskId] = result;
            if (this.nextTaskId < this.allTasks.length){
                this._runNextTask();
            }
            else{
                //no more tasks to launch
                //if all tasks are complete, complete the All Promise
                if(this.completedCounter === this.allTasks.length){
                    this.resolveAll(this.taskResults);
                }
            }
        });
    }
}

That we can use like this:

function getRandomInt(max) {
    return Math.floor(Math.random() * Math.floor(max));
  }

  //returns Promise
function getPost(postId){
    console.log("getPost " + postId + " started");
    return new Promise(res => {
        setTimeout(() => {
            console.log("---> getPost " + postId + " finishing");
            res(`${postId} - content`)
        }, 2000 * (1 + getRandomInt(2)));
    });
}

(async () => {
    let tasks = [];
    for (let i=0; i<10; i++){
        tasks.push(() => getPost("postId_" + i));
    }
    let results = await new PromiseAllWithLimit(tasks, 5).run();
    console.log("all tasks are complete");
    console.log(results.join("\n"));
})();

There's an important point to notice. While Promise.all receives an array of promises, our PromiseAllWithLimit receives an array of functions that return a Promise. I'm calling these functions "tasks" in my code (that have little to do with .Net Tasks), let's think of them as "lazy-promises". I can not directly pass Promises, cause that would mean that we are already running all the functions, that is just what we want to limit. We return a Promise that will get resolved once all the "tasks" are complete (so we store its resolve "callback" for later invokation).

I've uploaded the code to this gist.

Tiered Compilation

.Net core 3.0 has come up with a huge surprise for me, Tiered Compilation (it exists since 2.1, but now it's enabled by default). Years ago I had written about how odd it seemed to me that contrary to Java HotSpot and to all modern JavaScript engines (that initially either interpret or JIT compile a method with a fast, lower-quality Jitter and then, if considered appropriate, will replace it, hot-swap, with a optimised Jitted version) .Net continued to use a single pass JIT compiler. Now things have changed.

Now .Net core comes with a fast, lower quality JIT compiler (tier-0) and with a slower, higher quality JIT compiler (tier-1). At runtime, methods are initially compiled to Native code with tier-0, and if they are executed more than 30 times, they'll get compiled again (by tier-1) and hot-swapped. This article provides some useful diagrams. Long in short, calls to compiled methods have now an additional level of indirection, where a count and jump to "initial, non-optimised native code" is done until the counter threshold is reached, then the "count and jump" is replace by a "jump to the new optimised native code". Several versions of the compiled native code get stored and over time could get hot-swapped again, but I have not found information about when this could be done.

Reading some other articles [1], [2] you'll find that .Net already had since the beginning 2 different quality JIT compilers. A faster one (equivalent to current tier-0) was used for debug builds, and an optimised one (equivalent to tier-1) for release builds. The thing is that you were stuck to one version or another, and no hot-swapping existed. There's also the question of why not using an interpreter rather than a not optimised JIT (and then the optimised JIT for frequently invoked methods), as done by Java Hot-Spot or some JavaScript engines. The main answer is that right now .Net does not have a good-enough interpreter (but it seems Mono has introduced a high-quality interpreter... so who knows if this could change in the future).

Finally, we have to take into account that .Net Core provides also Ready To Run images (R2R). This means that the compiler will compile your C# code (I guess this also exists for VB.Net and F#) to native code rather than CIL Bytecodes. This way when the application is run, the tier-0 compiler will not be used against that code, so start-up times are even faster, cause we already have the not-optimised native code, but the counting and recompiling if appropriate with the tier-1 JIT compiler still applies, so we can end up anyway with highly optimised code.

10 Years Post

It's now 10 years since I started this blog. If you read my first post, you'll see that I was not so sure I would manage to persist on it and make it one part of my life. I can say that I've succeeded. Posting here has become one part of my life, I've turned it into an obligation, sometimes I don't feel much like writing, but I force myself to do it, I force myself to continue, to focus on something. I've written some posts in some really difficult moments of my life, and when looking back it amazes me that certain month of certain year I were able to put certain things aside and focus on writing some paragraphs unrelated to that situation and that maybe no more than 10 people would ever read. It's nice when someone still manages to surprise himself in a positive way.

10 years have passed, and obviously many things have occurred. For someone moving from thirty-something to forty-something I guess I have not gone through some of the expected processes: finally finding a "life-partner", having children... Well, for an Asturian these processes are not so evident, this is probably the place on Earth with the lowest fertility rate, and I think the percentage of people my age that have not found (and many of us have never cared at all) a person to share life with is probably really, really high. It's not a surprise at all, 10 years ago I had no interest on finding "that person", and hopefully that interest has not emerged over these years (I say hopefully cause I'm quite sure this egocentric bastard would not manage to keep a steady relationship). As I said, I've gone through some bad moments, some of them pretty bad, but it has helped me to better appreciate the good moments, and better understand what makes up those moments: family, friends, places...

Not having taken those expected steps does not mean that things have been static, on the contrary, I've changed so much in these years. My political positions have turned like 160 degrees, my main interests, the things that I appreciate the most... all this has changed a real lot. Something that in 2009 was hardly an idea became a necessity, almost an obsession (in 2012-13), to live abroad for a long time. I finally made it and ended up in one country that had never appealed to me that much, France, and it has become my second identity, my second motherland... Several years there quenched my thirst for change, and for several reasons I realised that it was time to return to my primary home, that now I appreciate much more than before. I keep my strong links with France, and one could say that in a sense I've changed from being a "citizen of the world" to being a "citizen of two countries and one continent".
My musical interests have not remained static either, but after many evolutions and having broadened my taste pretty much, now I mainly favour the same styles I loved 20 years ago.
As for work, from an external perspective little has changed, but my internal attitude is now so different. I no longer dream of "doing something significant", I still enjoy learning and coding (well, sometimes at least), but regarding work all I aspire to is to get easy assignments and reduce stress.

Many people feel sad (even terrified) about getting old. I think it was the same for me in the past, but not now. It's not because I don't appreciate this life, just the contrary. As time goes by we are all aware that the end is closer, but not everyone is aware of how having reached this point in the path is an achievement. People die at any age, under any situation, so being able to look back and see how the traversed path has got longer should relieve us and make us feel fortunate, make us give thanks to whoever-whatever we think has helped us reach this point. Already quite a long time ago I said this sentence "Cada día vivido es una victoria frente a la muerte / Each day we live is a victory against death". Some friends thought of it as a very depressive conclusion, about an obsession with the idea of death stalking us I guess... but it's not like that really, or at least only partially. For me it's light coming out of darkness... We are our memories, and the more memories the more we are. The last pages of Serotonin by Houllebecq (a crazy, amazing book) quite shocked me (I read it in French, which is one of my most unexpected achievements of this decade). The main character, sank in a depression that is literally killing him, prints pictures of his life and wallpaper his room, what is intended to be his last room, with them. They say that when you're going to die you see all your life pass at high speed, this was like a low speed version that provided pleasures that the present no longer hosted. The more things you have to fill that sequence, I think the more sense you can say your life has had.

This has been a very personal post, and if you have read it to this point, most likely you are someone that is a part of my life. If that's the case, I want to say Thanks, for having been there all this time and for remaining there in the future. I plan to continue to write boring paragraphs here, and I hope we'll be lucky to share a "20 years post" in 10 years time. I also hope we'll be sharing passages of our lives face to face, in front of a coffee (or a beer for many of you) more frequently than we do. You know that I'm a socially lazy person, but I appreciate more and more those shared moments. On se voit!

Iterating multiple times

There's something in the behaviour of IEnumerables-IEnumerators created by an Iterator method in C# that is quite confusing. Let's start by the behaviour of JavaScript generators, that is the expected one for me, and let's see then the oddities of the C# version.

In JavaScript, a generator function returns a generator object, an object that is both an iterable and an iterator. So when asking it for an iterator (via [Symbol.iterator]) it returns itself. Because of this, if we try to iterate multiple times our iterable, the iteration will only happen the first time. During this first iteration the iterable-iterator object reaches its end, so next iterations will do nothing.

function* getCountries(){
 yield "France";
 yield "Belgium";
 yield "Portugal";
}

let countriesGenOb = getcountries();

let countriesIterator1 = countriesGenOb[Symbol.iterator]();
console.log(countriesGenOb === countriesIterator1 ? "same object" : "different object"); //same object


console.log("- first iteration:");
for (let country of countriesGenOb){
 console.log(city);
}

//France
//Belgium
//Portugal

console.log("-------------");

console.log("- second iteration:");
//no iteration is done, countriesGenOb[Symbol.iterator] is returning the same object
//that was already iterated to the end in the previous loop
//very interesting, this behaviour is different from C#, here the generator object (this is both iterable and iterator) is returning itself, rather than a copy
for (let country of countriesGenOb){
 console.log(city);
}

//Nothing gets printed

let countriesIterator2 = countriesGenOb[Symbol.iterator]();
let countriesIterator3 = countriesGenOb[Symbol.iterator]();
console.log(countriesGenOb === countriesIterator2 ? "Same reference" : "Different reference"); 
//same reference
console.log(countriesGenOb === countriesIterator3 ? "Same reference" : "Different reference"); 
//same reference

The behaviour in C# is different and surprising. An Iterator method returns an object that implements both IEnumerable and IEnumerator, but we can iterate it multiple times. This is like that because it seems that when we call GetEnumerator on and IEnumerable-IEnumerator that has already been enumerated, it does not return a reference to the same object, but to a new object implementing also IEnumerable and IEnumerator.

  var countries = GetCountries();
  Console.WriteLine(countries is IEnumerable); 
  //True
  Console.WriteLine(countries is IEnumerator); 
  //True

  Console.WriteLine(countries == countries.GetEnumerator() ? "Same reference" : "Different reference");
  //Same reference

  //the Iterator method is returning an IEnumerable/IEnumerator object, but the thing is that calling to GetEnumerator returns a new instance, rather than the object itself
  //Because of that the 2 loops do a whole iteration, and enumerator1 and 2 are different objects.
  Console.WriteLine("- first iteration:");
  foreach(string country in countries)
   Console.WriteLine(country);
  
  //France
  //Belgium
  //Portugal
  
  Console.WriteLine("-----------------");
  Console.WriteLine(countries == countries.GetEnumerator() ? "Same reference" : "Different reference");
  //Different reference
  
  Console.WriteLine("- second iteration:");
  foreach(string country in countries)
   Console.WriteLine(country);
  
  //France
  //Belgium
  //Portugal

  Console.WriteLine("-----------------");

  var enumerator1 = countries.GetEnumerator();

  Console.WriteLine((enumerator1 == countries) ? "Same reference" : "Different reference"); 
  //Different reference
  

  var enumerator2 = countries.GetEnumerator();
  Console.WriteLine((enumerator1 == enumerator2) ? "Same reference" : "Different reference"); 
  //Different reference

<-- CSharp and JavaScript current sources F:\Main\MyWebCreatures\deploytonenyures\SourceCode\GeneratorsIterators_IterateMultipleTimes -->

Deferred-Lazy Iterable

When in my last post I talked about how to provide lodash-like functionality to Async Iterables, I did not realise that indeed lodash (or underscore for the matter) do not work with normal Iterables. The Array methods work with arrays and the Collection methods work with Objects. Well, Iterables are objects, but if you check the source code for example for map you'll see that it checks if the object is an array (in which case it traverses it with a while-length loop) or just an Object, in which case it traverses it using Object.keys... so it won't work for non-Array iterables. The solution is converting your iterable to an Array, which in modern JavaScript is so clean as doing [...myIterable],

var _ = require('lodash');

function* citiesGeneratorFn(){
 let cities = ["Xixon", "Toulouse", "Lisbon", "Tours"];
 for (city of cities){
  yield city;
 }
}

let cities = citiesGeneratorFn();

//lodash methods do not work with iterables
console.log("with iterable");
console.log(_.map(cities, city => city.toUpperCase()).join(";"));
//nothing gets printed

//we have to convert the iterable to an array
console.log("with array");
console.log(_.map([...cities], city => city.toUpperCase()).join(";"));
//XIXON;TOULOUSE;LISBON;TOURS

Problem with this is that you are losing one of the main features of iterables, that they are deferred. First, if my generator function*: citiesGeneratorFn above were doing something like retrieving the city data via a server call, I would not need to do that retrieval until the moment when I were doing a next call to get that city, but when converting to array, we are doing all those calls in advance, even if then I'm only going to use the first city. Second, as lodash does not support iterables, all the map, filter... functions are applied to all the elements of the array, even if again we are only going to use the first one. Because of this is why a library like lazy.js (of which I talked recently) exists.

So, it came to my mind how would I go in order to try to implement something like that, a map, filter... functions that worked nicely with iterables and hence deferred. Thanks to generators it's pretty straightforward. I create an ExtendedIterable class that will wrap the original iterable. This class features map and filter methods. The important thing is that each of these methods returns a new ExtendedIterable (so I can chain the calls) that applies the function logic to the value to return, one by one, taking advantage again of generators. All in all, we have this:

class ExtendedIterable{
    constructor(iterable){
        this.iterable = iterable;
    }

    *[Symbol.iterator](){
        for (let it of this.iterable){
            yield it;
        }
    }

    map(fn){
        //I need to use the old "self" trick, cause we can not declare a generator with an arrow function
        //so as we are using a normal function, "this" is dynamic, so we have to trap it via "self"
        let self = this;
        function* _map(fn){
            for (let it of self.iterable){
                yield fn(it);
            }
        };
        let newIterable = _map(fn);
        return new ExtendedIterable(newIterable);
    }

    filter(fn){
        let self = this;
        function* _filter(fn){
            for (let it of self.iterable){
                if (fn(it))
                    yield it;
            }
        };
        let newIterable = _filter(fn);
        return new ExtendedIterable(newIterable);
    }
}

function* citiesGeneratorFn(){
 let cities = ["Xixon", "Toulouse", "Lisbon", "Tours"];
 for (city of cities){
  yield city;
 }
}
let cities = citiesGeneratorFn();

//I could just use the array, and array is iterable and the citiesGenerator is not doing any processing, just retrieving the value directly
//let cities = ["Xixon", "Toulouse", "Lisbon", "Tours"];

let extendedIterable = new ExtendedIterable(cities);

let result = extendedIterable.map(it => it.toUpperCase())
    .filter(it => it.startsWith("T"));

for (let it of result){
    console.log(it);
} 

//or just convert to array and join it
//console.log([...result].join(";"));

I've uploaded the code to a gist. This is just a POC, if you want something real, go for lazy.js.