Friday, 25 February 2011

JVM Threading made Elegant

When I started to learn C# and .Net a long while ago I had a basic Java background, and among many other things (delegates, events, PInvoke, pointers, using...) one of the features that made me feel much more comfortable in the (at that time) new ecosystem was threading. I'm not talking about advanced threading primitives here (neither then or now do I feel qualified to talk properly about sync issues), but about the very basics, how you create a Thread.

In .Net it feels pretty natural to me. It just adapts the mechanism used in "lower lever systems", passing a function pointer to Win32 CreateThread or Posix pthread_create. In .Net the perfect match for function pointers are delegates, so for whatever method you want to launch in its own thread (well, provided that it's parameterless or expects just one Object parameter), you just create a delegate about that method, then create a Thread object taking that delegate as parameter and that's all:

Thread th = new Thread(new ThreadStart(p1.SayHi));

simple, right?

Unfortunately in Java it's a different story. There is not equivalent to function pointers, so how do you feed the Thread object with the code it's going to launch? Two options here, either you inherit from Thread or implement the Runnable interface, putting in both cases your code in the run method. I clearly prefer the second option, as inheriting from Thread is rather limiting. If you program a method with the idea of being run in a separate thread implementing Runnable does not sound bad, but if what you want to run in a separate thread is a normal method programmed by others without threading in mind (which I think is perfectly valid, I don't see why "I'm going to run in a separate thread" should be part of the design of a method, I see it as running asynchronous, it can be the client who decides how to run such method), things get rather ugly. You would need to create an extra class implementing Runnable and invoke that method from inside the run method:

class RunnableSayHiWrapper implements Runnable
Person p1;
public RunnableSayHiWrapper(Person p)
this.p1 = p;

public void run()

Person p2 = new Person("xuan");
Runnable runnable2 = new RunnableSayHiWrapper(p2);
Thread th2 = new Thread(runnable2);

In many cases we can avoid typing the extra class by using an anonymous inner class, but given that as of today Java does not have closures yet... we have to resort to some unclean artifacts:

static Person p1 = new Person("xana");
public static void main(String[] args)
Runnable runnable = new Runnable() {
public void run() {
Thread th1 = new Thread(runnable);

What I'm referring to as unclean is that to have a reference to p1, we've had to add it to the class containing the calling method (in this case it's a static). The elegant thing would be that we just could pass that instance to the calling method and we had a closure that would trap that parameter as part of the lexical scope, but the lack of closures is a real pain...

Hopefully, if you're a JVM programmer you still can have fun and enjoy programming by using a beautiful, modern, powerful and fast evolving language like Groovy, and put aside all these Java limitations. Groovy has closures, and futher on, you can as if by magic implement interfaces with closures or maps, so creating a Runnable that you can feed to your Thread object is that simple and beautiful like this:

def p1 = new Person("xana");

def runnable1 = { p1.sayHi(); } as Runnable

th1 = new Thread(runnable1);

Furthermore, as we're using a closure, we get rid of one of the annoying limitations of .Net ThreadStart delegate, the number of parameters (0 for ThreadStart delegate or 1 for ParameterizedThreadStart delegate) of the method. There's no problem if our method (formatName in the example below) expects x parameters, as long as we have those variables available in the code where the closure is defined, they will get trapped:

s1 = "{";
s2 = "}";
def p3 = new Person("xabel");
def runnable3 = { p3.formatName(s1, s2); } as Runnable
th3 = new Thread(runnable3);

Of course, doing the same in C#, thanks to closures, is equally simple and elegant. By the way, the job done by the C# and Groovy compilers when dealing with anonymous methods-closures and Groovy closures is rather similar. In both cases a new class with fields for the trapped variables and a method for the code inside the closure are created under the covers.

string st1 = "{";
string st2 = "}";
Person p1 = new Person(){Name = "Xuan"};
//we need a closure here for the st1, st2 parameters
Thread th1 = new Thread( () => p1.FormatName(st1, st2));

Sometimes code can be astonishing beautiful, don't you think?

You can download source code here.

Saturday, 19 February 2011

Last month Interesting Stuff (January 2011)

Well, I'm starting off here with something I intend to do regularly, post an entry with links and maybe some very short descriptions/reminders of things that I've been reading lately and that I'd like to keep track of. It's mainly something for my own reference/future use, but anyway could be of some use to someone else out there.

I should have posted this some weeks ago, so these entries correspond to what I was reading/working with in last January:

  • Groovy:
    Implementing Interfaces with Closures, Maps and coercion

    Why we don't need anonymous inner classes in Groovy

  • .Net:
    Good points on using implicit typing (var) vs explicit typing in C#. It adds meaning, it's not just a matter of "wrist friendliness".

    Internal calls in .Net assemblies (yes, those methods for which Reflector is no longer your friend cause they are implemented in native code:
    MethodImplOptions.InternalCall attribute, which indicates (MSDN link):
    Specifies an internal call. An internal call is a call to a method that is implemented within the common language runtime itself.

    Benefits of the ExpandoObject, and a thorough article on them.
    Something a bit weird about the ExpandoObject is that to get access to a property in a ExpandoObject using a string (indexed access) you need to cast it to IDictionary. It's a bit weird, the reason is a because it does an explicit implementation of IDictionary, instead of
    a implicit implementation. The explanation for this decision is not clear to me:
    Asp.Net MVC 3 (and much more) is out!!!. IIS Express 7.5 works great and is an independent download now, not need to download the whole WebMatrix thing as previously needed.

  • Technology:
    The recent announcement by Microsoft explaining that Windows 8 will run on ARM, SOC architectures has caused quite a stir and speculation.

    The fact that I've known of Arduino since its first steps (one friend of mine was an early adopter), made me unaware of how terribly popular it has turned lately, taking it away from the MegaFreak land and bringing it to the "Fun for the masses" land. This is a good documentary. Diving into this a bit more I've ended up with some slightly different and pretty interesting stuff: .Net Microframework, Netduino and similar boards, some first impressions and crazy ideas.

A good sentence by Eric Lippert to finish this post:
The art of coding is figuring out what the right level of abstraction is for the audience.

Wednesday, 16 February 2011

Autumn - Sonbahar

I think Autumn-Sonbahar is the first genuinely Turkish film that I've watched (all the other ones were the German Turkish films by Fatih Akin).
I have to acknowledge that it didn't live up to my initial expectations. I'd read an interesting review that made me think of a more social, political story. There's a political component, but a soft one, as the whole film itself. What I mean with a "soft film" is that it's pretty slow, with very few dialogs, and with an a linear plot.

Yes, all this said, it could seem like I would not recommend this film, but in fact I do, at least if you're into finding out how this world works, breaths and stinks.
It opens a small window (sure the 90 minutes of filming could have been leveraged better to show a more complete picture) to an area of this planet that may be pretty unknown to many of us.

  • Did you know that there are left wing political prisoners in Turkey?

  • Did you know that after the fall of the Eastern Block, the economical situation of most of the former Soviet Republics is such that for some women (I know this is common in rural Ukraine, Moldava...) the best option is working as prostitutes in a country that our little Western brains would tend to consider less wealthy (Turkey)

  • Did you know thatNorth East Turkey, on the Black Sea coast, is an stunning area with very heavy rainfall and mountains rising above 3000 meters rather close to the sea? Yes, if you live in Green "Spain", sure it rings a bell. In fact, in this area you can find Hórreos that are terribly similar to Asturian ones.

Just those things aforementioned are enough to make this film an interesting way to spend your time before going to sleep. Moreover, there are some additional points that sure I'll remember from this film:

  • The absence of sex. I mean, a story about a prostitute... well, in a Western film you would expect all sort of nudity and sex on display. Maybe this is the main cultural difference that we can perceive throughout the film between Western Europe and Turkey, and I'm not judging here, indeed I'm not much sure of which approach to "public sexuality" I prefer. In this "distant" society, a prostitute wears rather more modest than the average teenager here... so the only sex-nudity in the film is a beutiful scene with the two main characters sharing their bed in fetal position.

  • The sequence when the beautiful Georgian female character leaves the library after having purchased a book and the owner says "even their hookers are well educated"...

  • When the beautiful Georgian woman says to the Turkish idealist "you spent the best years of your life in jail because you wanted socialism, are you crazy?"

Definitely, a rather appropiate film for those that like me have completely lost all their former political ideals... Time devours beauty, idealism and ourselves...

Saturday, 12 February 2011

A new Love Story

Today we've seen the announcement of what probably will be the biggest tech news of the year, the deep partnership between Nokia and Microsoft.
There have been rumors all over the last months, rumors that have been growing so loud in the last days that made today's announcement rather predictable.

I think both parts have done a really wise decision, Nokia needs Microsoft to avoid becoming just a middle class manufacturer (it's pretty clear that the Symbian + Smartphone equation does not work), and Microsoft needs Nokia to become a player again in the Mobile market. Nokia has great hardware, Microsoft has great software, users and reviewers unanimously praise Windows Phone, but it's lacking support from manufacturers that have gone mad about Android (and I don't blame them).

The net is buzzing with articles and mixed opinions on this matter. What is pretty funny to me is reading the brainwashed opinions of open source fundamentalists (I love sharing knowledge with others, but if selling hardware or selling "administration services" is OK, what's the problem with selling Software?). Many of these zealots seem terribly upset and angry, and swear not to buy a new Nokia device again. They wonder why Nokia has chosen Windows Phone over Android, and well, I think it's such a stupid question that it can only be prompted by the mind dullness induced by fanaticism. Nokia business is based on differentiation, and Android means just the contrary, because becoming one more among the army of Android sellers would be an obvious dead end for Nokia.
Sure they are not going to be the only manufacturer using Windows Phone, but they'll be the main one, so for good part of the public Windows Phone and Nokia will be almost synonymous, same as it's been with Symbian (though until recently other manufacturers like Ericsson, Samsung, Motorola, Siemens... also used Symbian).

I've also read something that increases my contempt for Google. Yes, I know, this blog is hosted by Blogger... but my perception of Google has dramatically changed in the last year, from the "gmail heroes, cool employers" to the "Chrome OS, I own your life, I'm cool cause I'm opensource but in fact I'm fucking CLOSE, fight Microsoft by all means..." arrogant zealots category, and this sentence from Vic Gundrota, VP of Engineering at Google, is disgusting and clearly shows how arrogant these Google idiots are:
Two turkeys do not make an Eagle

Such a stupid comment deservers one reply like:
Arrogance, narcisism and "coolness" well could make vultures fly over Mountain View

As a developer with a special love for C# and .Net, these are incredibly good news. Windows Phone + Nokia mean millions of devices running Silverlight, and this will be an incredible boost to the .Net platform, a boost that will spread to other enviroments (desktop, server...)
Now that I intended to walk my first developer steps on the Android path, seems like a different path begins to shine :-)

There's a good interview on CNN with Ballmer and Elop.

On my side, all I can add is well done guys!!!

Saturday, 5 February 2011

Groovy, MetaClasses and MOPs

In a recent post I said that I'm learning a bit of Groovy. In the last days I've continued to play around with the language and I have to say I'm pretty delighted by this beautiful piece of wisdom.

Thought I've never done any sort of compiler-interpreter programming, we could say that I'm a language freak, in the sense that I have great interest in any kind of advanced, new, unusual... features, new/different paradigms... that a language/framework provide. Expression Trees, lambdas, runtime code generation, Prototype-based programing, runtime weaving, MetaClasses... well, the list could go on for a while.

So, while diving into Groovy going through the User Guide, I found plenty of interesting stuff. "Closures" (I quote them cause as I already explained here they are not always real closures) are implemented in a very similar way to C# delegates, that is, compiler magic creates classes the inherit from Closure/MulticastDelegate. With this in mind, I soon realized that most of the beauties in the "Compile-time Metaprogramming - AST Transformations" are just compiler magic. The compile time metaprogramming capabilites provided by AST Transformations remind me of Boo's syntactic macros. So thought very useful and elegant, don't seem like "revolutionary" to me.

However, when getting to the Dynamic Groovy - ExpandoMetaClass section, things turned really exciting.
We have two pretty important concepts here:

  • Expando. I think I heard this concept for first time applied to javascript objects. An expando (or maybe open object) is an object that can be expanded, by adding to it new methods and properties. Of course, expando objects are the basis for Prototype-based languages like javascript. Python objects are also expandable objects (well, in both languages normal objects are mainly dictionaries of properties and functions), and .Net has a cool ExpandoObject.

  • Metaclass. I first heard of MetaClasses years ago, in my Pythonist times. Unfortunately it's been a long while since my last adventures in Python (thought it's full of nice features I think the the non-C syntax is a major drawback, at least for me), but I remember MetaClasses were an obscure concept that stayed on the Guru (or wannabe Guru) side. My understanding of Python MetaClasses was basically classes that allow you to create new classes. When a new class gets created (Python interpreter gets to a class sentence, its MetaClass is used to intefere in the creation process, adding, modifying behaviour, and thus determining the behaviour of instances of that Class.
    It fits quite well with the definition in wikipedia:
    In object-oriented programming, a metaclass is a class whose instances are classes. Just as an ordinary class defines the behavior of certain objects, a metaclass defines the behavior of certain classes and their instances. Not all object-oriented programming languages support metaclasses. Among those that do, the extent to which metaclasses can override any given aspect of class behavior varies. Each language has its own metaobject protocol, a set of rules that govern how objects, classes, and metaclasses interact.
    By the way, seems like in the last years MetaClass have been used in some real life scenarios, being its usage in Django broadly praised.

Groovy MetaClasses work at the class and instance level cause both instance objects and Class objects have a metaClass property. One note here:
In Groovy classes are first-class objects, that's why they have properties like metaClass. When we write Person.metaClass, it's not trying get access to a metaClass static property in Person class, but it's doing sort of a Person.getClass() and accessing the metaClass property in the returned Class object. OK, Java Class objects do not have a metaClass property, well, in Groovy (I'd like to have some additional information, cause it's a bit confusing to me), they have, check the Groovy JDK (don't confuse it with the Groovy API).

The main application of MetaClasses in Groovy is the ExpandoMetaClass (EMC). We can add methods (Closures) and properties to the ExpandoMetaClass of a Class object or an instance object and all instances of that class or that particular instance will get expanded with that method/property:

Person.metaClass.sayHi = {-> return + " says Hi"};

p1 = new Person("xana");
p1.metaClass.sayBye = {-> return + " says Bye"};

Simple, right?. So we have the same power that we have in javascript and Python. As I said above, in these languages objects are basically dictionaries and method-property access boils down to a dictionary lookup thing, but in Groovy things can't be that simple. When compiled to JVM bytecode each Groovy class gets compiled to a Java class (besides that, some extra classes are generated for Closures, categories, the main program...) Java classes/objects are closed, they can't be expanded with new properties or methods, and method invocation and property access is fixed at compile time (well, of course we have a small degree of dynamism in method invocation through method overloading and vTables, but vTables can't be modified at runtime, so no way to add/delete/update methods), so, how does this work?
how does a method invocation end up reaching a method added to an EMC?

well, all the magic comes from the fact that when type a simple method call like myInstance.myMethod(); the Groovy compiler does a nice job under the covers,
transforming the call into a call to ScriptByteCodeAdapter.invokeMethod, that in turn triggers a chain of method calls that get the MetaClasses involved:

ScriptBytecodeAdapter.invokeMethod(...) (static method)
InvokerHelper.invokeMethod(...) (static method)
Invoker.invokeMethod(...) (instance method called on InvokerHelper's single instance)
Invoker calls invokeMethod(...) on the MetaClass of our class (with exceptions, see below). It finds this MetaClass by looking it up in the MetaClassRegistry. The Invoker holds a single instance of this registry.

All this is well depicted here.
From the above, one new mistery, the MetaClassRegistry and how it relates to my per instance MetaClass and per class MetaClass. Well, I think the reference to MetaClassRegistry is misleading, with Groovy objects it's the metaClass property of the involved object what is used to find the MetaClass.
Well, I've gone through much googling, this nice pdf (but bear in mind that it's an old document and new versions of Groovy have brought some changes) and finally some debugging and source code exploration (eclipse rocks here!) and these are my findings:

All user defined Groovy classes implement the GroovyObject interface, which means that they have a metaClass property.
As I previously said, Classes also get a metaClass property.
At first, this Class.metaClass points to an instance of MetaClassImpl. (Person.metaClass)
When an instance (p1) of a Class (Person) is created, its metaClass property (p1.metaClass) will point to the same object pointed by the metaClass property of its Class (Person.metaClass).
When a method is added to a MetaClass (either the MetaClass of an instance or the MetaClass of a Class) (p1.metaClass = {...}; or Person.metaClass = {};) the affected object no longer points to the old MetaClassImpl object, but to a new ExpandoMetaClass object.
If it's been the Class (Person) who got the new EMC, all new
instances of that Class (p2, p3) will point to that EMC, and will be affected by additions to that EMC done from the Class. However, the instances already existing before the EMC got assigned to the Class, will still point to the old MetaClassImpl object, not being affected by the changes done to the EMC.
If it's been an instance who has got an EMC assigned (per instance metaClass: p1.metaClass.sayHi = {...};) it only affects to that instance, not to the Class metaClass or any other new instances.

You can check this code that adds some other marginal cases and should certify my conclusions.

All this usage of the metaClass property and the ScriptBytecodeAdapter is ingenious but should not impress any Python or javascript lover. What is pretty astonishing is that it works for normal Java classes!!!

String.metaClass.wrapInBrackets({->"{" + delegate + "}"});
println "xana".wrapInBrackets();

Normal Java classes (String for example) are closed and out of your control, you can't make them implement the GroovyObject Interface, so what?
This is where the MetaClassRegistry comes into play. As a normal class defined in Java does not have a metaClass property, when the method invocation gets to the point where it needs to check the MetaClass to obtain the method, it looks it up in the MetaClass Registry. I guess this is a Dictionary where Class objects are keys and MetaClass objects values. In old versions MetaClasses for Java classes only worked at the class level, but in modern Groovy it also works for instances, so I guess also instances are added to that Dictionary.
Another question here is how does the access to the (non existing) metaClass property work, I guess methodMissing will have something to do here.

This is the feature that really blew me away. Different languages provide different mechanisms for intercepting method calls and property access (I'm talking about the typical AOP thing, intercept method call, write a log and continue the call). In Java and C# interception is just confined to some compile time weaving or creating Dynamic Proxies at runtime. In Python we can use decorators or MetaClasses to "decorate" functions at creation time. Also in Python and in JavaScript we can go one step ahead, and given an existing object, patch it by traversing its list of methods and making each method reference point to a new function, that will contain the extra code and then invoke the original method. It's powerful, but it has a "hack" feeling, as something the language was not intended for.
That's where Groovy excels thanks to method invocations going through the invokeMethod method.
one class can implement the GroovyInterceptable interface and all method calls go through the implemented invokeMethod. This is a compile time decision, so not too especial.

class MyClass implements GroovyInterceptable
def hello(){ 'Hi' }

def invokeMethod(String name, Object args)
def result;
def metaMethod = this.metaClass.getMetaMethod(name,args);
result = "success, result: " + metaMethod.invoke(this,args);
result = "invocation failed";
return "intercepted $name(${args.join(', ')}) " + result;

more code here

The cool part is that we can do our interception decisions at runtime, adding the invokeMethod to the MetaClass of a class or the MetaClass of an instance. As we saw in the section above, method calls end up going to the invokeMethod of the corresponding MetaClass. Also bear in mind that what we saw above with respect to methods added to a Class MetaClass not being available in instances previously created also applies to this.

I've got a sample here.

I think something similar in terms of adding interception at runtime can be achieved in Python by means of __getattribute__. In this sample defines it inside the class declaration, but I think it should work adding it later dynamically.
One more feature that fairly amazed me is that in those rare cases where we're typing parameters, Groovy supports Multiple Dispatch!!!

Another cute feature that called my attention is Categories. They're similar to C#'s Extension methods, but more powerful cause it is limited to a block (use...) and it does not have the compile time type limitation of C#'s compiler. I mean, if we're typing a variable "p" to Parent, but at runtime it points to a Child object, and the Category was created to work with Child, it will work fine with our "p" variable (something that the C# compiler would not allow).

Metaclasses are also related to the broader concept of MetaObject Protocol.

The metaobject protocol approach ... is based on the idea that one can and should open languages up, allowing users to adjust the design and implementation to suit their particular needs. In other words, users are encouraged to participate in the language design process.

Well, the text above sounds impressive and I guess it's more ambicious than that we find in Groovy (for example, a MOP could allow us to modify how inheritance or exception handling work, how object comparison is done). Anyway Groovy MOP seems pretty powerful to me, cause it allows us to do all the nice things explained in the previous paragraphs.
Groovy's MOP mainly draws upon the existence of hooks/extension points:
taken from here

Groovy's MOP system includes some extension/hooks points that you can use to change how a class or an object behaves, mainly being:
getProperty/setProperty: control property access
invokeMethod: controls method invocation, you can use it to tweak parameters of existing methods or intercept not-yet existing ones
methodMissing: the preferred way to intercept non-existing methods
propertyMissing: also the preferred way to intercept non-existing properties

I've found a book about the DLR that seems to explain the MOP in our beloved "dynamic" C# 4. Implementing IDynamicMetaObjectProvider to return a DynamicMetaObject... seems like pretty interesting stuff.

Contrary to Java, Groovy is evolving very fast, and its designer have really great plans for next versions. There are 2 that specially caught my eye:

  • New Meta-Object Protocol

  • ability to pass expression trees / AST nodes as parameters (see C# 4's own expression tree)

Final musings
After this long post praising Groovy it's easy to figure out that I'd love to see an implementation of Groovy for .Net (IronGroovy?) but the question is, would it be worth the effort?
Why my doubts? Well, dynamic languages like Boo, the Iron ones (IronPython, IronRuby) and many incomplete implementations of other languages... don't seem to have caught in the .Net ecosystem with the same strength that in the Java Platform, and I think there's a good reason for this. Java (the language) seems like a stagnated language lacking many of the expressive features that many coders have discovered thanks to javascript, Python, Ruby... so many people programming for the Java Platform are more than receptive to alternatives. I think there's little doubt about the fact that C# is rather more expressive than Java. It's not just my taste, Jim Hugunin, that Java, .Net and Python genius also thinks so)
I will suffer some pain when I have to write code in Java now that I've learned to love the elegance of C#.

So what I think happens is that many .Net programmers do not feel the urge to embrace a new language cause C# serves them well (and given its fast paced evolution in these years, many goodies can be expected for next versions :-)

For those concerned about the performance of dynamic languages, and leaving benchmarks aside, I'll say: Who cares about performance when you can write such stylish code :-D