New IT Jargon

IT is a complex, constantly evolving world where it's rather difficoult for someone with an average brain to keep up with just a small part of the many things one can find interesting. What is quite simple (and sometimes can be pretty helpful) is keeping up with the new vocabulary. When I say helpful, I mean that knowing to name a few of the last cool IT things can make you look smarter than you are (but notice that some managers can consider that knowing what an Acronym stands for is right the same as being an expert on it, so it can end up causing you some trouble).

In the last weeks I've come across with some names/acronyms that can help you look cool :-D

• NoSQL is one of the most misleading terms of the last years. OK, NoSQL stores do not use SQL, but that's not the real point, the real point is that those Data Stores are not based on the Relational Model we find in RDBMS's. So when I read somewhere that some people had started to use the term NoRel I immediately loved it.
• Polyglot Persistence. When I first knew about the term polyglot programming I really liked it, it was a fast way to define what the programming world had evolved into, it's no longer enough to be proficient in one language/platform and blindly choose it for all your projects. I guess with Polyglot Persistence it's going to be right the same.
• BASE. The counterpart to ACID in a NoRel world.
• MEAN. I think it's been some years since the last time I heard about LAMP, and suddenly a modern, geek incarnation of that web stack is here: MongoDB, Express, AngularJS and Node. I'm almost certain I'll never use this combination. I love Node (but mainly as a scripting platform or just as a way to test JavaScript "proofs of concept") and I'm looking forward to have some time to play with Mongo and AngularJS, but I don't see myself doing any personal Web Development project, and in most companies the LOB (Line of Business) web applications are and will continue to be Java or .Net.
• MVW. Probably you're a bit fed up of having to stop to think about the distinction among MVC, MVP, MVVM... to explain someone which one (or variation of which one) you're using. Saying "I'm using Model View Whatever" can come really handy to spare you time in fruitless debates.
• Hoisting. In JavaScript variables declarations (but not initializations) are hoisted (that is, moved to the top). Function declaration (not function expressions) are also hoisted.
• IIEF. If you've been into JavaScript development in the last years is pretty likely that you've done some use of them, but chances are that you don't know their name: IIFE (Immediately Invoked Function Expression), I mean:

  (function () {  // open IIFE
                  var tmp = 100 - x;
                  ...
              }()); 
  

File Locking

I've gone through some problems with File Locking at work lately, so I think I'll write some notes about it here. We had an application appending lines (results) to a csv file in a shared folder. If that file is opened with Notepad++, results will continue to be appended successfully (and notepad++ will ask you to reload the file as it's been modified), however, if it's opened with Excel by someone with write access permissions on that file, new lines will fail to be appended to the file (we won't lose them as in that case we write to a secondary file, with means that in the end we'll have to consolidate both files). So Excel is locking the file while notepad++ isn't.

We requested Write access to be removed for our users, and just in case we also asked them to always copy the file to a local folder and open their local copy. Anyway, we've found lately that errors continue to happen (as we can see that new lines get added to the secondary file). After some bewilderment we finally found the reason, the errors were happening when a user was copying the file to their local folder (the file has grown enough to take some seconds to get copied, time enough for experiencing collisions with the main process trying to append new results to the file).

I have to admit that I had not much clear how "file locking" works, and I was quite confused on some points. The only kind of locking that I had in mind was that 2 processes (threads indeed) can not write to the same file at the same time, but from the above it seems like there are more cases where locking happens (I guess copying a file would mean just opening it in Read mode), so first I've done some basic tests with a .Net program opening a file with a StreamReader, StreamWriter or FileStrem with differen parameters, and trying to read/write from/to that file with some basic commands: echo hi ≶≶ File.txt / type File.txt.
Notice that I've performed all these tests on Windows, I'll test this on Linux when I have a chance.

1)One process opens a file for writing to it (either with a StreamWriter or a FileStream):

using (StreamWriter sw = new StreamWriter(filePath)) //careful, the StreamWriter empties the file!

//or:

using (FileStream fs = new FileStream(filePath, FileMode.Append, FileAccess.Write))

As expected, if another process tries to write to the same file (just a simple echo time >> file.txt), it'll fail with a The process cannot access the file because it is being used by another process error.

Notice however, that there will be no problem to read from this file from another process (a simple type file.txt)

2)Now, let's go for the interesting case, let's open a file for reading(with a StreamReader or a FileStream):

using (StreamReader sr = new StreamReader(filePath))

//or

using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))

and let's try to write to it (again with just a simple echo time >> file.txt), It'll fail So it seems like by default opening a file for reading prevents us from writing to it. This quite fits with the unexpected behaviour that we'd come across when trying to write while copying.
Again, notice that there will be no problem to read from the file from another process.

These results disconcerted me a bit, I guess there has to be a way to open a file for reading while allowing other processes to write to it. Trying to find some more information about File Locking in Windows it happened again that the Wikipedia seems to have just the perfect amount of information.

Microsoft Windows uses three distinct mechanisms to manage access to shared files:

• using share-access controls that allow applications to specify whole-file access-sharing for read, write, or delete
• using byte-range locks to arbitrate read and write access to regions within a single file
• by Windows file systems disallowing executing files from being opened for write or delete access

...

The sharing mode parameter in the CreateFile function used to open files determines file-sharing. Files can be opened to allow sharing the file for read, write, or delete access. Subsequent attempts to open the file must be compatible with all previously granted sharing-access to the file. When the file is closed, sharing-access restrictions are adjusted to remove the restrictions imposed by that specific file open.
Byte-range locking type is determined by the dwFlags parameter in the LockFileEx function used to lock a region of a file. The Windows API function LockFile can also be used and acquires an exclusive lock on the region of the file.

If you look at the different overloads for the FileStream constructor you'll find a few ones with a FileShare parameter. This is what the article refers to as "file access sharing". In my examples, I've not been using any of those overloads, so I guess I'm in this case:

Windows inherits the semantics of share-access controls from the MS-DOS system, where sharing was introduced in MS–DOS 3.3. Thus, an application must explicitly allow sharing; otherwise an application has exclusive read, write, and delete access to the file (other types of access, such as those to retrieve the attributes of a file are allowed.)

I found then this excellent answer, where a guy opens a file with FileShare.ReadWrite and then copies it by blocks. I've put it into my own FileCopyTaker class and done some tests, and it works like a charm. I can copy a large file and at the same time append additional information to it from another process.

I've uploaded it here.

It's clear from this that doing a file copy from Windows Explorer (or via the copy command) is not using this file-sharing feature at all, and one could wonder why. Well, I think that's the correct default behaviour. In most cases, modifying a file while it's being copied would leave the file in an unusable state (when the copy finishes the other process could still be writing information to the file, so it would be incomplete, and in most cases (save for my particular case of appending a line to a text file) it would mean broken.

You'll find people wondering why there's not a Windows function to check whether a file is Locked. The answer is clear, it would not be useful, as you could run into synchronization issues, I mean, given this code

if (!isFileLocked(filePath)){
	//Do stuff with the file
}

between the moment when isFileLocked returns false, and the moment you start to do things with it, another process could be locking the file and your code would crash... so in anyway, you should always put your file access code into a try-catch to deal with the possibility of the file being locked

Thread Suspend, Resume and Abort

At first sight aborting, suspending or resuming threads (in the .Net arena) seems pretty simple, the Thread class sports methods for each of these 3 tasks. However, if you check the documentation, you'll see that both Suspend and Resume are marked as obsolete and highly discouraged:

Do not use the Suspend and Resume methods to synchronize the activities of threads. You have no way of knowing what code a thread is executing when you suspend it. If you suspend a thread while it holds locks during a security permission evaluation, other threads in the AppDomain might be blocked. If you suspend a thread while it is executing a class constructor, other threads in the AppDomain that attempt to use that class are blocked. Deadlocks can occur very easily.

As for the native Win32 funcion SuspendThread you'll find equally discouraging statements:

This function is primarily designed for use by debuggers. It is not intended to be used for thread synchronization. Calling SuspendThread on a thread that owns a synchronization object, such as a mutex or critical section, can lead to a deadlock if the calling thread tries to obtain a synchronization object owned by a suspended thread. To avoid this situation, a thread within an application that is not a debugger should signal the other thread to suspend itself. The target thread must be designed to watch for this signal and respond appropriately.

Thread.Abort is not marked as obsolete and no advice against its use is explicitly given in MSDN, but you'll find multiple articles and discussions (like this pointing against its use, just for the same reasons as Suspend.

The basic idea is that these methods are too invasive, they'll just Suspend or Abort a thread irrespective of what it's doing at that moment, which can be pretty risky. We need then a more collaborative way to do this, basically a Thread should be periodically stopping to do its main task and checking for Suspend or Abort requests at moments when it can really proceed with such order, aborting/suspending itself accordingly. Obviously the "periodically stop to do its main task" thing can be complicated to accomplish depending on what that main task is... but that's apart from the suspend/abort logic.

For Abort it seems pretty simple. A given thread would just check for an Abort flag, and when true, it would just end.

while(!this.abort){ //check for Abort requests
     DoWork();
}

For Suspend/Resume it's a bit more complicated, we can check for a Suspend flag, but once suspended, how do we check for the Resume order? Well, signals are the main communication mechanism among threads, so we should just use that, a ManuaResetEvent initially set as signaled, we would indicate a Suspend request by resetting it, and then send a Resume command by setting it again.

while(!this.abort){ //check for Abort requests
     DoWork();
     this.resumeSuspend.WaitOne(); //check for Resume/Suspend requests
    }

To really understand it you'll have to see this full sample

Parallel Linq

Parallel Linq has been around for a long while, but I've hardly had the need nor the chance to play with it. Some discussions at work as to whether some parts of our current Perl application could get noticeable performance improvements by running them in parallel have whet my appetite for running operations like sorting, filtering... in parallel, and this has drawn me to take a look into Parallel Linq.

Apart from all the parallel magic, I find interesting the way how these parallel operations have been made available to Collections, so I'll write here some notes about it.

Given that Parallel Linq is a parallel implementation of Linq To Objects, the design is very similar. While Linq to Objects is implemented as a set of Extension Methods that apply to IEnumerable objects (I'm using IEnumerable to refer both to IEnumerable<TSource> and IEnumerable), Parallel Linq is implemented as a set of Extension Methods that apply to ParallelQuery objects. The former are located in the System.Linq.Enumerable class, and the latter in the System.Linq.ParallelEnumerable class.

If you have an IEnumerable object and want to switch from applying normal (sequential) Linq methods on it to its parallel counterpart, all you need is calling the AsParallel method on it. How does this work? AsParallel will return a ParallelQuery objet, so now the different Where, Find, First... extension methods will be taken from System.Linq.ParallelEnumerable rather than from System.Linq.Enumerable. Notice that ParallelQuery implements IEnumerable, but the resolution rules for Extension Methods guarantee that "the closest" to the given object are taken, meaning that Where(ParallelQuery... is chosen over Where(IEnumerable...

Let's say you have one collection and for whatever the reason (depending on different factors the parallel version could be slower than the sequential one) you want to apply some parallel methods and some sequential methods (and you're chaining all those calls), you can then use the AsSequential method to convert your ParallelQuery back into a plain and simple Enumerable. Thinking a bit about this, it seems clear that all what AsParallel and AsSequential classes do is wrap/unwrap your normal Enumerable Object in a ParallelQuery object. We can launch ILSpy to confirm this:

Indeed, internally the framework is using a ParallelEnumerableWrapper class, that inherits from ParallelQuery. Not sure about the rationale for this extra class.

You'll see that we also have an AsEnumerable method, that seems to do just the same as AsParallel. An yes, if you check its code, it just calls AsSequential.

You could be wondering why I'm using ILSpy to peek into the Framework code when the Framework's source code is publically available (both online and for download). Well, first I'm a creature of habit, and second, using a decompiler still gives me sort of a slight sense of "hacker" that the source code doesn't :-D

Order Preserving Object

Some weeks ago one coleague showed me a pretty cool perl library, Tie, that among other things allows to preserve the order in which items are added to a Hash (meaning that you can iterate the Hash in that order)

use Tie::IxHash;
my $items = {};
tie (%{$items}, 'Tie::IxHash');

Given the similarities between Perl and JavaScript I began to think of a way to accomplish the same in JasvaScript. Notice that JavaScript does not guarantee that the order in which a "for in" loop traverses an object (or Object.keys returns its keys) is the same in which those properties were inserted, the specification says it quite clearly. So far, it seems like many JavaScript engines preserve that order, but I guess this is just an effect of how they lay out objects in memory, it's an implementation detail on which we can not rely.

Achieving this feature in ES6 does not seem particularly complicated thanks to proxies. Notice that I'm using Direct Proxies, and at the time of this writing only Firefox seems to implement them, so I'm trying to make this code work just in Firefox (no node.js fun this time). Basically, we could create an OrderPreservingObject, by creating a proxy object around a plain JavaScript object and trapping the set and delete actions, so that it adds the key to an array that holds the order of that object.

function OrderPreservingObject(initObj){
	var obj = {
		orderedKeys: [],
	};
	//to prevent orderedKeys from showing up when iterating with "for in" we should declare it as non enumerable, 
	//but anyway, we should not use "for in" for iterating this object
	//ideally we would have managed to trap the "for in" and we would exclude it directly there
	var proxy = Proxy(obj, {
		
		//receiver is the proxy itself
		set: function(target,name,val,receiver){    
			//console.log("set invoked");
			target[name] = val;
			if (target.orderedKeys.findIndex(function(it){
				return it == name;
			}) == -1){
				target.orderedKeys.push(name);
			}
			return true;
		},
		deleteProperty: function(target,name){
			//console.log("delete invoked");
			delete target[name];
			var index = target.orderedKeys.findIndex(function(it){
				return it == name;
			});
			if (index != -1){
				target.orderedKeys.splice(index, 1);
			}
			return true;
		}, 
				
		keys: function(target){
			//console.log("keys invoked");
			return target.orderedKeys;
		}
	});
	//now that the proxy is set up, we can add the items in the initObj and they'll be added in order
	if (initObj){
		for (var key in initObj){
			proxy[key] = initObj[key];
		}
	}	
	return proxy;
}

Once we have the pieces to keep the order, the main problem is, how should we do to iterate the object following that order? The ideal solution would be iterating the object with "for in". "for in" iteration logic does not depend on our objects implementing a certain iteration method, so it can not be altered or hinted by users. Well, that's true for normal objects, but looking into the ES6 documentation, we see that in principle proxies are also able modify the "for in" behavior. So, I came up with this method/trap:

enumerate: function* (target){
	console.log("enumerate invoked");
	for (var i=0; i<target.orderedKeys.length; i++){
		yield target.orderedKeys[i];
	}
}

But I guess I'm missing something, cause the above does not trap the "for in". The "for in" runs as normal, iterating the object without taking into account our trap function (it iterates the object in order, but not based on our ordering extra logic, just cause for the moment Firefox's internal implementation of objects and "for in" happens to preserve the order). After a fast search I haven't found any information about how this "enumerate" trap is supposed to work, so I guess it'll have to wait.

In C# and Java iterating an object with foreach works just by obtaining an Enumerator/Iterator for that object and working with it. ES6 implements iterators, and while the "for in" loop will continue to work right the same as it does now, the new "for of" loop has been added to work with iterators and their next method. So well, maybe the best way to iterate my OrderPreservingObject would be to make it implement the iterator protocol and use it with a "for of" loop.

		__iterator__: function* (){
			for (var i=0; i<this.orderedKeys.length; i++){
				yield this.orderedKeys[i];
			}
		}

Unfortunately, the above does not work either. Trying to use it will fail in Firefox:
TypeError: dic1['@@iterator'] is not a function

So it seems like I'm left with no other option but iterating my object with a normal for loop, either using Object.keys

for (var i=0; i<Object.keys(dic1).length; i++){
	var key = Object.keys(dic1)[i];
	console.info(key + " - " + dic1[key]);
}

or directly using my orderedKeys array (that otherwise I would prefer to keep out of the Object interface)

for (var i = 0; i< dic1.orderedKeys.length; i++){
	var key = dic1.orderedKeys[i];
	console.info(key + " - " + dic1[key]);
}

Both solutions are quite verbose, so in the end, I came up with the idea of adding a forEach method (similar to the one in Array.prototype) to my Object:

forEach: function(actionFunc){
	//where actionFunc receives: item, key, tiedDic
	for(var i=0; i<this.orderedKeys.length; i++){
		var key = this.orderedKeys[i];
		actionFunc(this[key], key, this);
	}
}

so that it can be iterated like this:

dic1.forEach(function(item, key){
	console.info(key + " - " + item);
});

You can find the whole code here.

Definitely, I should revisit this in a few months as the adoption of ES6 evolves, to see if I finally make the "for in" trap or the "for of"/__iterator__ work.

Unfamous Resistenza

Over the last months I've come across in multiple occasions (quite normal, as they are all over Toulouse) with some nice stickers with a cool drawing and reading Unfamous Resistenza, something like this:

Given that many times these stickers can be found next to left wing stickers, which fortunately are very common here (NPA, Front de Gauche, all sort of Antifa designs...) I was expecting there could be something interesting behind them. Today I've finally done a search to see what this all was about, coming across with something pretty good. Unfamous Resistenza is a French collective of cultural activists, focusing on ethical and conscious works.

Just having a fast look around their site I've found 2 really interesting interviews, this one to Gutter, a Toulouse based Street Artist with a very dark and gloomy style, and this one to Dark Taz, an excellent photographer focused on Urban Exploration and Beauty in Dereliction.

I think I'll be spending some hours in their site in the next days and become a regular visitor. The fact of being in French is not putting me off, on one side, my French reading skills are slowly improving (reading Metro, 20 Minutes and Direct Matin every morning in the tram helps a bit :-), and on the other side, I want to force me to continue to read French materials. I joked some weeks ago with one friend that one of my aspirations in life is having enough French skills to be able to sit on a Parisian or Toulousain terrace enjoying a Café Noisette (coffee in France is occasionally pretty good, even slightly reminding me the unbeatable Portuguese coffee) while reading "Le Monde Diplomatique".

Dictionary throwing Exception

I've always found it quite bothering that trying to access a missing key in a Dictionary in .Net throws an Exception rather than returning null. Being so used to JavaScript dictionaries (and to Perl hashes now), where it will just return undefined/undef, I sometimes forget about the difference in .Net and miss using ContainsKey or TryGetValue, ending up with an Exception.

If we think about why the .Net designers took this decision, I think the explanation comes from the Reference Type/Value Type dichotomy. In a Dictionary<K,V>, V can be a Reference type or a Value type. For Reference types, if the key is not present we could return null without a problem, but this is not the case for Value types, as Value types can't be null. One solution would be having applied the same logic as in TryGetValue, and return the default for that type (0 if it's a number). I don't think that would be a good idea, cause you would not know then if the item was in the Dictionary or not. I think this behaviour is fine for TryGetValue, as it's not the standard way to access an element, but not for the the most common accessor to a Dictionary elements.

This has led me to think about how Java behaves for similar data structures. Looking at the Hashmap and HashTable documentation for the get method (yeah, not even in its almighty Java 8 does Java feature indexes... ) we can see that:

Returns the value to which the specified key is mapped, or null if this map contains no mapping for the key.

OK, sure that's cool but, how do they handle Value types (the 8 primitive types supported by Java)? After writing a few lines of code to check this and failing to compile it I realised that Java Generics do not support Primitive Types. Sure it's a serious limitation (caused I guess by the same "don't break previous code" aim that led them to choose their non reified flavour of generics), but for this case it gives you a small feature.

So this code in Java won't crash:

Integer numObj;
int numPrimitive;
Hashtable<String, Integer> myHashTable = new Hashtable<String, Integer>();
  myHashTable.put("Haute-Garonne", 31);
  
  numObj = myHashTable.get("Haute-Garonne");
  System.out.println("Haute-Garonne: " + numObj);
  
  numObj = myHashTable.get("Gironde"); //returns null
  System.out.println("Gironde: " + numObj); //works fine

but notice that obiously, trying to unbox will throw a NullPointerException:

try{
   numPrimitive = numObj; //NullPointerException when trying to unbox a null object into an int
  }
  catch(Exception ex){
   System.out.println("Exception: " + ex); 
  }

There's something rather curious here, not the fact of throwing an exception, that is pretty logical, but the name of the Exception. NullPointerException. The usage of the term "Pointer" in Java seems pretty weird to me. It looks to me that calling it NullReferenceException as .Net does would be quite more appropriate. Seems like I'm not the only one puzzled by such naming decision. Someone justifies it by the fact that pointers and references are indeed the same, the memory address of the object. I would say such a justification is quite wrong. In .Net and Java (and I guess almost all platforms) the fact that a reference holds the address of the referenced object is an implementation detail. Rather than that, a reference could just hold an ID for an entry in a table that would in turn point to the objects (something similar to Object Handles in Windows OS), and such implementation should be transparent to the programmer.

One could wonder why Reference Types are not nullable. I don't fully agree with the accepted answer here. I see it more in terms of how would you differentiate null from a valid value. I mean, I think both Java and .Net internally represent null with a 0. So, how would you distinguish an int or a float which value is 0 from one with no value (null)?

Related to this, the question of how JavaScript implementations store null and undefined comes to mind. After a fast search I haven't found an answer, I think I'll look deeper once I have a chance.