Friday, 14 July 2017

The Autopsy of Jane Doe

It has passed almost one year since I watched The Witch, and my attitude towards horror films has not changed, I continue to be no particularly interested on "Paranormal" Horror, but again, as with The Witch, The American Cosmograph has given me the opportunity to watch a really good one, The Autopsy of Jane Doe.

One father and son run proudly and professionally an "Autopsy business". One evening the police brings them a new "client", a young, beautiful, unidentified (hence the "Jane Doe" thing) woman found dead in unknown circunstances. They start the examination, and as it advances more an more odd findings stack up until the first paranormal events start. So far, the film was a bit gore, now it turns violent and claustrophobic, with some good jump scares. To round it off they manage to give it a nice source to all this supernatural mess (OK, it's paranormal stuff, don't thik of a rational explanation, but the idea is cool). Indeed, this is one more point that makes me relate this film and The Witch

The acting is pretty good, particularly Brian Cox, and the 90 minutes of the film pass too fast. There is one of those moments of 5 seconds when the screen goes completely black (there's a blackout), that is particularly good when you are watching the film in the huge screen of a cinema.

All in all, a very good cinema, that for sure will be on top ten of 2017 Horror films and that you should watch.

Friday, 7 July 2017

ES6 Proxies and getPrototypeOf

The other day, somehow came to my mind an old post about static methods in JavaScript, and I thought that similar to the "avoid overriding" reason that I mention in that post, another use case would be to prevent operations from being intercepted by a proxy. I immediately thought of key actions like getPrototypeOf as an action that is better to be sure that the real method is being called and not a replacement, so it's much better to have it as a static method than as an instance method. So to my surprise I found out that the Proxy API provides a trap for getPrototypeOf, that indeed intercepts all these operations:

  • Object.getPrototypeOf()
  • Reflect.getPrototypeOf()
  • __proto__
  • Object.prototype.isPrototypeOf()
  • instanceof

Seeing that list I thought, OK, it intercepts all those cases, but will it intercept the case when the runtime is traversing the prototype chain of an object?.

I've done a short test, and seems like the traversal of the prototype chain when the runtime searches for an item is not affected by the proxy trap.


class Person{
 constructor(name){
  this.name = name;
 }
 
 sayHi(){
  return "Hello I'm " + this.name;
 } 
}
 
let p1 = new Person("Francois");

let proxiedP1 = new Proxy(p1, {
  getPrototypeOf(target) {
 return null;
  }
});

console.log("- Operations with proxiedP1");

console.log("proxiedP1 instanceof Person: " + (proxiedP1 instanceof Person));
//false

proto = Reflect.getPrototypeOf(proxiedP1);
console.log("proto is " + (proto === null ? "NULL" : "NOT NULL"));
//proto is NULL


//INTERESTING, proxying getPrototypeOf does not have an effect in how the runtime itself gets access to the prototype chain, so the sayHi method continues to work, even if instanceof is now telling me that proxiedP1 is not a Person anymore
console.log(proxiedP1.sayHi());
//works fine

So if you run the code above you'll get that the internal prototype [[Prototype]] of proxiedP1 is null and that it is not an instance of Person, but the call to sayHi, that is not directly attached to the p1 object, but to its [[Prototype]], will work fine! So officially you are not a Person, but in fact you are...

I find the existance of this trap like a source of confusion, but well, you can read about some use case here.

This has raised some questions in my head. From the programmer perspective the access to a method in JavaScript is not based on Method Dispatch or Message Sending, it's just a property lookup. That property lookup just returns a function and then it's invoked (if it it does not exist the runtime will follow up the prototype chain to find it). I assumed that at the lower level it would be implemented like that, but this would means that the addition of proxies would have introduced a sort of conditional to any property access in any object in the system. The runtime would have to check if the object is a proxy and has a get trap to invoke that trap or just do the normal access. If this were this way, the introduction of proxies in ES6 engines would have had performance implications for any code (whether it uses proxies or not), so obviously there has to be much more to it.

Saturday, 1 July 2017

Generics and Dynamic

Time ago I saw somewhere a fragment of code that rather confused me. It was a line where a generic method was receiving dynamic as type parameter. "dynamic" is something at the C# compiler level, but at the MSIL level it does not exist, it just gets translated into callsite magic for accessing to its methods and properties. So, if you run this code:

  public static void DoSomething<T>()
  {
   Console.WriteLine("type: " + typeof(T).FullName);
  }
  
  static void Main(string[] args)
  {
    DoSomething<String>();
   
    DoSomething<Object>(); 
   
    DoSomething<dynamic>();
   
   //Output:
   // type: System.String
   // type: System.Object
   // type: System.Object
 }

You can see the passing dynamic or Object is just the same, in the method you are receiving T as an Object. If you decompile it and check the MSIL you'll see indeed the compiler is just replacing dynamic by Object in the call.

So from the point of view of the invoked method "dynamic" has no particular meaning, it's from the caller side where it can add semantic value to write DoSomething<dynamic> rather than DoSomething<Object>.

Another related case is when a method returns generic type of dynamic. For example, in Dapper we have this method:

public static IEnumerable<dynamic> Query (this IDbConnection cnn, string sql, object param = null, SqlTransaction transaction ...

Again, at the MSIL that dynamic is just an Object, you can check the MSIL:

Here the semantic value of using dynamic is particularly powerful. It's telling you that the method is returning an object that whatever its type, contains all the fields that you expect and so you can access them without a problem. You can consume it like this:

var rows = connection.Query(...); 

If they had just signed the method as returning IEnumerable<Object> you could write something like this:

IEnumerable<dynamic> rows = connection.Query(...); 

But obviously it's pretty much more semantic to have expressed this "duck typing" in the signature.

By the way, this Query method indeed returns DapperRow objects (a class implementing IDynamicMetaObjectProvider). One could think why they don't just return an anonymous type (then it's going to be accessed via dynamic anyway). One reason is performance, with an anonymous type the dynamic access would be using a Reflection (with a cache mechanism, so it works faster than normal Reflection, but it's Reflection anyway), a type implementing IDyamicMetaObjectProvider implements itmself the access and dispatch rules, so it should be faster. But there is another more fundamental reason, based on what you can read here

Anonymous objects are internal, which means their members are very restricted outside of the assembly that declares them. dynamic respects accessibility, so pretends not to be able to see those members.

Notice that when you use Reflection you can decide to ask access to non public items, but dynamic uses Reflection "as a good guy", trying to access only public items, so for this case where Dapper is in an assembly and your code in another, Anonymous Types + dynamic is not an option. On the contrary, for code living in the same assembly, like the one below, it's a good option.

static dynamic CreateObject()
{
 
 return new {Name="Francois"};
 
}
  
static void Main(string[] args)
{
 dynamic o1 = CreateObject();
 Console.WriteLine(o1.Name);
}

Monday, 26 June 2017

Local Functions and Async

Following my previous post on Local Functions and Iterator methods, it's time now to see what I've learnt about async thanks to local functions

When reading the last part of this article, this paragraph was a great finding.

This exhibits the same issue as the iterator method. This method doesn’t synchronously throw exceptions, because it is marked with the ‘async’ modifier. Instead, it will return a faulted task. That Task object contains the exception that caused the fault. Calling code will not observe the exception until the Task returned from this method is awaited (or its result is examined).

So, let's compare these 2 methods:

public static Task<string> DoProcess1(string st)
  {
   Console.WriteLine("DoProcess1method started in: " +  Thread.CurrentThread.ManagedThreadId);
   if (st.StartsWith("throw")){
    throw new Exception("Can not process invalid string");
   }
   Console.WriteLine("string is valid to be processed");
   
   
   var t1 = Task.Run(() => {
                      Console.WriteLine("Time consuming operation running in: " +  Thread.CurrentThread.ManagedThreadId);
                      Thread.Sleep(1000);
                      return st.ToUpper();
                     });
   return t1;
  
  }
  
  public static async Task<string> DoProcess2(string st)
  {
   Console.WriteLine("DoProcess2 method started in: " +  Thread.CurrentThread.ManagedThreadId);
   if (st.StartsWith("throw")){
    throw new Exception("Can not process invalid string");
   }
   Console.WriteLine("string is valid to be processed");
   
   
   var res = await Task.Run(() => {
                      Console.WriteLine("Time consuming operation running in: " +  Thread.CurrentThread.ManagedThreadId);
                      Thread.Sleep(1000);
                      return st.ToUpper();
                     });
   
   return res;
  
  }

Both methods are basically the same, but the second is using await and hence we've had to decorate it with the async keyword. In both cases the first lines of the method (until Task.Run) are executed by the thread from the caller method, but while in Process1, the exception will be thrown immediatelly, in Process2 the compiler generated code will capture the exception and won't throw it until the Result of the returned Task is accessed. Let's see:

try{
    t1 = ProcessingService3.DoProcess1("throw aaaa");
    
    //we DON't reach this line
    Console.WriteLine("after Process1 call");
    Console.WriteLine(t1.Result);
   }
   catch (Exception ex){
    Console.WriteLine("Exception: " + ex);
   }
   
   Console.WriteLine("----------------");
   
   try{
    t1 = ProcessingService3.DoProcess2("throw bbbb");
    
    //we reach this line
    Console.WriteLine("after Process1 call");
    //exception occurs in the line below, when accessing Result
    Console.WriteLine(t1.Result);
   }
   catch (Exception ex){
    Console.WriteLine("Exception: " + ex);
   }

This is interesting, in DoProcess1, 0 or 1 Task object is created (depending on whether we throw or not before reaching Task.Run). In a method marked as async, like DoProcess2 the compiler creates at least 1 "main" Task object. This main Task object is always created, it does not involve a particular thread, but is used to hold the Result/Exceptions of the different Tasks created inside the method, and is returned to the invoker of the method. If DoProcess2 does not throw an Exception then it reaches Task.Run, and the compiler will add an invocation to ContinueWith on that second task, and so on for as many "await" calls we could have inside the async method. For the last one the ContinueWith will set a Result in that main Task created at the beginning of the call. If at some point an exeption happens the exception will be added to the Main Task and no more code of the asynchronous method will be run.

OK, the explanation above is confusing and not really accurate, but it helps me to get an approximate idea of what is going on in a method marked as async. If you really want to understand it, check this amazing post.

As you can see in that glorious article, for each method marked as async the compiler will create a class that is basically a State Machine. This State Machine has a MoveNext method that orchestrates the sequence of asynchronous calls. Each await call that exists in the async method is like a step, and where each awaited Task is asked (via ContinueWith) to call into MoveNext once it's done.

It's amazing to see how similar this compiler generated State Machine is to the one generated for an Iterator method. In the end, both things are basically the same, the big difference is that for Iterators is the consumer of the iterator who will be calling to MoveNext, while for the async-await thing it's the asynchronous method who once finished will call to MoveNext. Just as a reminder, in ES6 (while waiting for the await keyword to be introduced in ES7) people have been leveraging generators to sort of simulate await and avoid the callback hell.

Saturday, 24 June 2017

Local Functions and Iterators

In the end I find the introduction of Local Functions in C# 6 really useful, as I have learnt a couple of things reading about their uses. Thanks to them I've got a refresh and improvement of my understanding of how the compiler manages Iterators and Async methods. As explained here, in both cases local functions provide a useful pattern to manage validation in Iterators and Async Methods. This post will focus on Iterators.

I've had since its inception a more or less clear a view of the magic used by the compiler when it comes across an Iterator method. It creates a class implementing both IEnumerable and IEnumerator and moves the code of the iterator method into the MoveNext method in that class (that acts as a sort of State Machine), replacing the "yield" keyword (that does not exist at MSIL level) with assignments to the "Current" support field. No need to enter into more details when you have a perfect explanation here.

The thing is that maybe one could think that when we have code like this:


public IEnumerablet<string> GetMainDocuments(string url)
{
 if (url == null){
  throw new Exception("Empty!");
 }
 for (var i=0; i<5; i++){
  yield return downloader.download(url + "/item/i");
 }
}

The compiler could put the code that goes before the loop in a separate method, not in the MoveNext. A method that would be executed before we started the iteration with MoveNext, just invoked from the constructor of the state machine for example. This would make sense for certain cases of validation code, where we could want an immediate crash rather than a delayed crash when we decided to iterate. But we have to be aware that one of the basis of iterators is "lazy evaluation/deferred execution". In a way they are a bit like "promises", we have a "promise of a sequence" but each of its items does not materialize until the moment when we ask for it in the next iteration step. Have this in mind, and think that there are many cases where the "pre-iteration" code should be done just at the moment of starting to iterate: maybe it returns values used during the iteration, maybe it's still a validation, but one that depends on dynamic factors and has to be done just at the moment of starting to iterate, not before. So as the compiler can not differenciate those cases, absolutely all the code in your iterator method is put inside the MoveNext of the state machine.

For example think about this case:

public IEnumerable<string> GetMainDocuments(string pattern, DBHelper dbHelper)
{
 if (pattern == null){
  throw new Exception("Empty!");
 }
 
 if (!dbHelper.IsDbUp()){
  throw new Exception("DB down!!!");
 }
 
 IEnumerable<string> documents = dbHelper.GetDocuments("whatever query " + pattern + " whatever");
 
 for (var i=0; i<5; i++){
  yield return downloader.download(url + "/item/i");
 }
}

For the first contitional it would be useful to do it in a no deferred way, but for the second conditional, it makes more sense to do it just at the moment when data start to be needed. And for the third, again it makes more sense to do it not deferred (or maybe you wanted a snapshot at the moment when GetMainDocuments was called?) Yes, there are many possibilities, and the compiler can not just guess. So just have to adapt to his rules. Anything, inside the iterator will be deferred, so if there is code that you don't want lazy you'll have to separate it yourself

public IEnumerable<string> GetMainDocuments(string url)
{
 IEnumerable<string> getDocuments(string url){
  for (var i=0; i<5; i++){
   yield return downloader.download(url + "/item/i");
  }
 }
 
 if (url == null){
  throw new Exception("Empty!");
 }
 return getDocuments(url);
}
 

I have no idea of how the different JavaScript engines manage generators, but the issue with deferred execution is the same. Nothing of the code in your generator function will be executed until you start to iterate the created generator object.


function* nameGenerator(){
 //this console.log won't be executed until the first call to "next"
 console.log("inside iteration");
 yield "xuan";
 yield "xana";
 yield "iyan";
};

let gen = nameGenerator();

console.log("generator created\n");

console.log("starting loop");
for(let name of gen){
 console.log(name);
} 

//output:

// generator created

// starting loop
// inside iteration
// xuan
// xana
// iyan

Wednesday, 21 June 2017

A Distaste For Expression Bodied Members

I really like to see programming languages evolve, getting new features version after version. Even if some of those features are not particularly useful or revolutionary, but just a way to make code less verbose, I appreciate them, as I like to see evolution and dynamism, being forced to learn something new. Microsoft has put quite a bit of effort in saving us to type repetitive code and do our code more concise. The introduction of Automatic properties was a huge step, and the improvements done in the last versions (default values for Automatic properties, read only auto properties, and both features combined) are pretty sweet.


//Automatic property with default value
public string FavoriteCity {get; set;} = "Paris";

//Read Only Automatic property (I'll be setting its value in the constructor
public string BirthCity {get;}
 
//Read Only Automatic property with default value
public DateTime BirthDate {get;} = DateTime.Now;
  

Another "big feature" in terms of type strokes saving has been Expression Bodied Mmembers. Honestly, this feature feels like a bit irritating to me.

So now you can write a method like this:

public int FormatString(string st) => $"---{st}---";

//rather than like this:
public int FormatString(string st){
return $"---{st}---";
}

I don't see it particularly helpful, but guess others will find it cool. The problem for me comes when we use it with Automatic Properties. Let's see:



public string FrenchFullName => $"{this.LastName.ToUpper()}, {this.Name}";


//is equivalent to:
public string FrenchFullName 
{
 get{ return $"{this.LastName.ToUpper()}, {this.Name}";}

}

//And here an identical method

public string GetFrenchFullName() => $"{this.LastName.ToUpper()}, {this.Name}";

So in the first line I have a property and in the last line a method. At first sight it's not easy to see if it's a method or a property, the only difference is the extra "()". If you follow the correct nomenclature, properties should be nouns and methods should be verbs, there's no room for confusion, but anyway, I feel a bit uneasy with this similarity.

Furthermore, this syntax can be applied to create local functions (added in C# 7). So you can write alocal function in any of these 2 ways:

public static void InternalFunctionTest()
        {
   string applyFormat(int num){
    return $"[{num.ToString()}]";
   }
   
   string applyFormat1(int num) =>  $"[{num.ToString()}]";
  }

If you use the second way, it looks a bit similar to what we would have done in the past if we wanted to avoid polluting the class with a function that is only used from a certain method, write it as a lambda:

Func<int, string> applyFormat2 = (int num) => $"[{num.ToString()}]";

Under the covers, this is quite different from the expression bodied local function. In this case we are creating a new object, a delegate, while with the local function such object creation is not needed. There are more differences in terms on where the compiler creates the underlying method and the use of closures, but I've read some contradictory informations about this, so I'll have to spend some time to get the complete idea.

The "=>" syntax has been a pretty bad choice in my opinion and probably the source for my distaste for Expression Bodied Members . It's linked in my mind to delegates, so it takes my brain some extra cycles to break that association each time I see this symbol.

Saturday, 17 June 2017

Current Directory

The Current Directory (or Working Directory or Current Working Directoy) of a process is one of those topics that I seem to clearly understand, but after a while I partially forget and have to strive to rebuild it in my head, so I'll do a dump it here after that last build. I'm talking only about Windows, I guess the behaviour in Linux and the different UI Shells will be similar.

Every windows process has an associated Current Directory. This is the folder that the process will use as root for any relative path that it tries to access. You can see it with Process Explorer. So, how does a process get its working directory assigned?

The CreateProcess Win32 API function has a parameter to set the Current Directory. If NULL, the Current Directory of the parent process will be inherited. Higher level mechanisms used to create processes (and that ultimately make use of CreateProcess) will usually give you the option to provide the Current dDrectory (like in .Net with ProcessStartInfo).

OK, the above is good to know when we are writing a program that will launch another process, but how have the writers of common OS components handle this? meaning, how does this work when a program is started from the command line or Windows Explorer?

The Windows command line. cmd.exe has as Current Directory the folder where it is currently located. If you do a cd, the Current Directory will get updated accordingly. When you start a process from the command line, the process will inherit the Current Directory from cmd. This means that if you do this:

c:\Temp>cd c:\myAppFolder
c:\myAppFolder> myApp.exe

myApp.exe will have as working directory c:\myAppFolder

But if you do this:

c:\Temp>c:\myAppFolder\myApp.exe

myApp.exe will have as working directory c:\Temp

Windows Explorer. There are several ways in which you can launch a process from Windows Explorer (explorer.exe). Browsing to the folder containing the .exe file and double clicking on it, or typing the path to the exe in the Windows->Run window, is the same, Explorer will set up as Current Directory the folder where the .exe file is located.

An additional way to launch process from Windows Explorer is through a shortcut. If you check the properties of a shortcut you'll see a Start in field. If filled, that value will be used as Current Directory. If emtpy, Explorer will set the program's Current Directory to the folder where the shortcut is located (quite commonly the Desktop).

A process can easily change its working directory while running, by using the SetCurrentDirectory Windows API function. From .Net you can use the Directory.SetCurrentDirectory method.

As far as I know there is not a public API for changing the Current Directory of another process. The option that I've seen is pretty hackerish, injecting a dll in that process and invoking SetCurrentDirectory from its entry point.

As far as I know, by default Windows Services get as Current Directory C:\Windows\System32\, I guess because they inherit it from services.exe

Finally, there's an interesting quirk to bear in mind with .NET applications. The standard configuration system (the old school System.Configuration.ConfigurationManager, previous to the introduction in .Net Core of the much more advanced Microsoft.Extensions.Configuration) is pretty smart and does not use the Current Directory to locate the myApp.config file that has to be located next to myApp.exe, it will get the folder containing our binary and search the app.config in it, so in terms of configuration you don't care about the Current Directory and whether your app is started via explorer or from the command line or from another process. If you move to using a different kind of config file placed also next to your binary, you'll have to start to consider that the Current Directory could have different values and is not a reliable way to get the file.