Self Hosting and http.sys

In the last week I've gone through a little nice "journey of discovery" where my current explorations with self hosted Web API and self hosted SignalR have led me to fully understand some stuff I've been partially aware of since 10 years ago (I've been putting together different small pieces until conforming a whole, very interesting, picture).

Back in 2006, while doing a short dive into Python's Web Development ecosystem (Django, Turbogears...), I felt delighted by the presence of basic http server modules that allowed you to test your applications sparing you from having to install and configure a "real" server and furthermore affording you to write applications with an embedded Web Server. Later on I found this idea again in other platforms like node.js or .Net WCF (that has featured self-hosting since its first version).

As of late, many other .Net libraries/frameworks provide self hosting functionality (Web API, SignalR via Owin, Service Stack...) this led me to think that most likely the .Net framework includes the core functionality for creating your own web server. Some searching took me to this StackOverflow question. So it seems like all the magic is provided by System.Net.HttpListener (it appears like SignalR directly uses HttpListener while WCF and Web API use an extra layer on top of it). Notice also that this has nothing to do with Hosted Web Core Applications that rely on IIS.

Well, HttpListener brings this embeddable Web Server to your application, but it does not implement the core http logic, this seems to be in the http.sys driver. I've been vaguely familiar with http.sys since the Window 2003 times. IIS6 meant an important evolution of our beloved Web Server, and the fundamental part came given by the introduction of http.sys, a kernel driver that, to put it simple, takes care of the http (and https) protocol. When you create a HttpListener, it registers itself with http.sys, so that http.sys routes traffic to your listener based on the url-prefix that you've registered. It's nicely explained in this article:

Basically, http.sys is a kernel-mode listener that has intrinsic knowledge of HTTP. Different parties can register with this listener to have requests forwarded to them. Typical examples nowadays include IIS 6 and SQL Server 2005

and in this other excellent one. There I've learnt that HttpListener is a managed wrapper around the HTTP Server API.

Notice that http.sys is a core component of the Windows networking subsystem, so it'll be present in any Windows PC from Windowx XP SP2 on. You can read more about it here, and you can verify that http.sys is installed in your system by typing: driverquery on your command line.

Opening listeners on http.sys is protected by ACL's, so unless the application creating the listener is running as Admin, you'll get an AddressAccessDeniedException. So, you'll have to use the netsh tool (running it as Admin) to add permissions for the user (that will later on run the application) to register a listener on the intended url-prefix. For example:
netsh http add urlacl url=http://+:80/MyUri user=DOMAIN\user

One of the coolest things with http.sys is that it allows port sharing between http applications, that is, different applications can be registered for the same port (but different url-prefix, I mean: app1 registers for localhost:80/app1 and app2 for localhost:80/app2). Obviously this is possible cause the applications do not listen directly on that port, it's http.sys who does it and then forwards accordingly to one or another application. This port sharing feature has been extended to other protocols by means of Net.TCP Port Sharing.

At the end of this excellent reading you'll find that http.sys is at the core of one of today's most confusing and painful topics on the modern web development arena, why on earth doesn't .Net 4.5 (and therefore WCF, SignalR...) support WebSocket in OS's other than Windows8 and Windows Server 2012?

The problem is with the way that HTTP.sys works in Windows 7/Server 2008/R2 and earlier OS versions. Rather than issue a patch that could result in large regressions for existing customers that rely on the current behavior of HTTP.sys, the newest versions of Windows include a modification to HTTP.sys that enables WebSockets.

This other post delves deeper into what "the way that http.sys works in Windows 7" means:

The current technical issue for our stack is that the low-level Windows HTTP driver that handles incoming HTTP request (http.sys) does not recognize the current websockets format as having a valid entity body. As you have noted, the lack of a content length header means that http.sys does not make the nonce bytes in the entity available to any upstack callers. That’s part of the work we will need to do to build websockets support into http.sys. Basically we need to tweak http.sys to recognize what is really a non-HTTP request, as an HTTP request.

I've read somewhere the WebSocket support in Windows 8/2012 is built in websocket.dll, so my guessing is that the "websockets enabled" http.sys in these OS's will take care of the "almost http like" websocket handshake, and then the normal communication will be handled by websocket.dll.