Parallel Programming

An fast write up mainly to share a couple of links (and a reminder of things I'd like to go more in depth in the near future)

In these times when multi-core processors are ubiquitous (even for smartphones in the near future) it's clear that parallel programming is more and more important. .Net framework has tried to address this doing Parallel Extensions mainstream by including them in .Net 4 framework.

Parallel Extensions look exciting and promising to me, so finding this free book (Patterns for Parallel Programming: Understanding and Applying Parallel Patterns with the .NET Framework 4) has been really good news. I've only gone through the first pages and I've already learnt some interesting things about dynamic partitioning, oversubscription...

This reminded me about another excellent free book (Threading in C# by Joseph Albahari) that I had been reading some months ago. It's great news to see that it's been updated for .Net 4, including a full new chapter on Parallel Extensions (and many updates to the previous chapters to reflect other MultiThreading improvements to the Framework).

We tend to think of Parallelism just in terms of Threads or Processes, but there's more to it, for example:

• SIMD instructions (that's SSE for x86 architecture). These instructions leverage Data Level Parallelism, performing the same instruction on multiple data, for example if we want to add different pairs of points, an only Add instruction could run in parallel over the different pairs. There's a good explanation here
  Mono has added support for SIMD instructions, which can be really useful for Multimedia applications doing heavy use of matrices and vectors. It's contained in an independent assembly that can be used both from Mono and from Microsoft's implementation.



• GPGPUs. The programmability of GPUs has gone a long way since the initial vertex, pixel shaders. These shaders are used for graphic tasks, so while advanced and interesting, do not seem so revolutionary as being able to run "normal calculations" on the GPU and use the results from the code running on the CPU.
  So far, there are main technologies in this field:
  
  
  
  • CUDA. This is the first one I was aware of. I remember how surprised I was when I saw some code a couple of years ago showing how dynamically compile and load into the GPU some code embedded in a Python script
  
  
  • OpenCL (Update, 2010/09/27): I've just read a very interesting entry on InfoQ about using OpenCL via pure Java (code written in a Java subset gets translated to OpenCL code at runtime). It's pretty interesting what the author says on current usages of GPUs in common programming:
    
    Looking at the OpenCL side, I have seen everything from financial services to energy exploration (I used to work in that field). I also know folk that are looking at GPUs for seismic data analysis. Applications that have the potential to see major benefits of using the GPU for computation include the big-data analytics domain.
    
  
  
  • Direct Compute. Thought it's Microsoft Centric there don't seem to be direct bindings from .Net, but there are some articles out there on how to make them work together.
  
  
  
  

By the way, back to the shaders, it's really amazing being able to run Shaders in your browser via Silverlight!!!