Some musings about Roslyn

There's been quite a lot of excitement in the last weeks around the first preview of the "Compiler as a Service" technology to be added to .Net 5 (aka Project Roslyn).

I have mixed feelings about this. On one side, I can't share all that much excitement, cause for the most part I don't see it as a revolutionary feature (as I see the DLR or even Linq), but as a rather needed feature that should not have taken so long to make it into the Framework. Furthermore, something similar has been present in Mono for a long time (one of the main reasons I always hurry up to install Mono on any new machine is the wonderful C# REPL console that comes with it). On the other side, having full access to the compiler pipeline means that I think very interesting things can be built on top of it, this made my imagination fly, but it suddenly hit the wall of reality. One of the fist and more obvious things that I could think of was adding your own new custom keywords and constructs to the language (turn it into an open language...), but unfortunately, seems like that's out of the scope for now

Actually, it isn't true that you can use Roslyn to extend C# with additional keywords. – Dustin Campbell Oct 21 at 19:35
thanks... corrected... although not in the first release I am pretty that this will be possible... – Yahia Oct 21 at 19:36

@DustinCampbell, What if you handled whatever compiler error that the pseudo keyword caused by generating code? – Rodrick Chapman Oct 21 at 20:17

You'd need to do a rewrite before passing it to the compiler. First, parse code with your special keywords. The code will parse and, unless the parser couldn't make heads or tails of it, the invalid keywords will show up as SkippedTokenTrivia in the resulting tree. Then, detect the skipped keywords and rewrite the tree with valid code (e.g. AOP weaving). Finally, pass the new tree to the compiler. This is definitely a hack though, and it's not guaranteed to work with future versions of Roslyn. E.g. the parser might produce not produce the same tree for broken code in future releases. – Dustin Campbell Oct 21 at 21:59

Anyway, I guess (even over Anders Hejlsberge dead body) compile time AOP will turn much more prevalent in the .Net world thanks to this.

Reflection is something that has always deeply appealed to me, from basic introspection to the full beauty of runtime code generation. The way to generate code at runtime in .Net has evolved over the years. Already in the first version we had 2 ways to do this:

• write some C# code, write it to a file, invoke the compiler (launch the csc.exe process) to create an Assembly and load that assembly
• use CodeDom to generate C# code and compile it (under the cover this also invokes csc.exe)
• use System.Reflection.Emit

By the way, we've got a good discussion of CodeDom vs Reflection.Emit here.

These options above have the drawback of having to create a new assembly for each new code that we want to run, which adds some overhead. Later on, (.Net 2.0) things got improved, allowing us to create new code by using LCG (Lightweight Code Generation) aka Dynamic Methods. No new assemblies are created/loaded, and the new code can be referenced from a delegate (using DynamicMethod.CreateDelegate). The main drawback is that we're not writing C# code here, but IL... and thought going low level in these times of higher and higher abstractions can have much appeal for the Geek inside us, you get a feeling of being brought back to the times of C and inline assembler :-)

.Net 3.5 came with new Code Generation candy in the form of Expression Trees, candy that got extra sugar in .Net 4, where Expression Trees can be used for creating statements, not just expressions.

In .NET Framework 4, the expression trees API also supports assignments and control flow expressions such as loops, conditional blocks, and try-catch blocks.

This gives us full power, but again, at the cost of a non trivial, not much natural syntax.

All this said, I think a common wish for many of us would be something that cute as JavaScript's almighty eval function. Finally, Roslyn seems to bring something similar to the table, but well, will have to see how it evolves, cause right now there seem to be some limitations

Can we now take a piece of code in a string, and compile that to a DynamicMethod?
We don't have this feature implemented yet, but it's definitely something on our radar.

One interesting topic here is, why Mono seems to have been some steps ahead in the "Compiler as a Service" area? Well, Mono has had a managed compiler since its early steps, while C# has been burdened by a native compiler so far. I think that's a big difference that we tend to overlook. Even contrary to what we could think based on how the native JavaScript interpreter (jscript.dll), Xml Parser (msxml.dll)... work (COM component in a dll that this way can be reused by IE, WSH, HTAs...), the C# compiler logic is not implemented in a separate dll-COM, but just in the csc.exe executable. This means that any C# to IL compilation involves spawning the csc.exe process. Even now I'm still amazed when I see the w3wp.exe process launch a csc.exe instance to compile some Razor view or .aspx page...