Windows Power States

I've recently received a new laptop from my employer, a pretty nice Lenovo machine. After the first days using it I realised there was something odd. When waking it up in the morning after having put it to sleep in the late evening, the battery level had quite dropped, one day the laptop was hot as if it had not really entered into sleep. I suspended it several times and it seemed to get into sleep OK, so maybe it was more a thing of waking up from the suspended state for some reason. The windows event log seemed to show it entering into suspension and going mute, but after several hours there would be entries in the log, so yes, something was waking it up.

Doing some searches for Windows 10 Sleep/Stand by issues I thought I've found what was happening. Windows 10 comes with a new Sleep mode, Modern Standby (S0 Low-power Idle) that, if your hardware supports it, is used rather than the previous S3 mode (my previous Lenovo work laptop, dating back to 2017 did not support Modern standby). This Modern Standby thing comes with 2 modes: Connected and Disconnected.

Devices that support Modern Standby can connect or disconnect from Wi-Fi or a wireless local area network while in standby.

- Connected Modern Standby will allow the device to remain connected to Wi-Fi while in standby. It will be able to receive and process notifications about new email messages, incoming calls. This is convenient, but makes the device drain the battery power faster.
- Disconnected Modern Standby allows longer battery life, but the device won't notify you about new events.

Resuming from "traditional" sleep mode has always been almost immediate, so for most users I don't see what's the use of Connected Modern Standby mode in a laptop. Receiving notifications while it's in standby and hence waking up?! I guess after that it will go back to sleep following the inactivity rules on your system, but if you're receiving notifications pretty often this will be a battery draining mess. I was thinking it could be something like that what was happening to my laptop, so I wanted to turn off this behaviour.

First I confirmed with powercfg -a that my new laptop is using this Connected Modern Standby thing.

As you can see, the traditional S3 mode is not supported by my Hardware. It seems Hardware can not support both Sleep states (read here

Modern Standby is only available on some SoC systems. When it's supported, the system does not support S1-S3.

So the solution is switching from Connected Modern Standby to Disonnected Modern Standby. For that you just have to go to Settings -> System -> Power & Sleep and in the Network connection settions select Always

Unfortunately I've faced once again the problem of finding my laptop hot and with a very low battery level in the morning, even when using this Disconnected mode... Checking the event log it seems the laptop got into sleep and woke up immediatelly, don't know why, so I've had to resort so far to the Hibernate mode to avoid problems. Anyway, it's been interesting to learn about the current types of Power States in modern Windows/modern Hardware.

Linux Memory Values

I've been trying to understand lately memory consumption information in Linux and I'll share my findings here.

My personal laptop has 8 GB of RAM (for sure, I can see it in the Bios), but the integrated Graphics card is taking 2.1 GB (and my Bios does not allow me to configure and reduce it!!!). The different programs that I've used in Linux to view my memory status only show the physical memory available to the system discounting those reserved 2.1 GB, hence 5.9 GB, and I have not found a way in Linux to see that the real physical amount of memory in my system is 8 GB.

I have to admit that things are a bit more clear in Windows. Task manager will show 6GBs in the main view, but at least it also has a "Hardware reserved: 2.1 GBs" reading. Furthermore, the "Windows About" window shows: Installed RAM 8.00 GB (5.92 GB usable).

The three main programs that I use to check my memory are the Ubuntu System Monitor and the top and free -m commands.

The information in the System Monitor is pretty straight forward. A 2 colors representation (as when checking a hard disk usage), and a "used" percentage. There is also the "Cache" value... that can seem quite more misterious, we'll understand what it represents in the next paragraphs.

The information displayed by top can be particularly confusing if we just read the first (total) and second (free) values. We should not care much about that "free", we should read just to the end to find the "avail Mem" value, that's what really matters in terms of how much memory we still have available (many of us would call it "free", but "free" has a different meaning in memory terms).

If we run free -m we really get the full picture, but we need some extra knowledge to interpret it.

              
		total        used        free      shared  buff/cache   available
Mem:           5920        1908        1704         112        2307        3610
Swap:          8092           0        8092

used is what we expect, the memory currently in use by our system. available is, as I've said for "top", what we really care in terms of how much memory my programs can still use without having to resort to swapping. And then we mainly have "free" and "buff/cache" (and "shared", but this is a small value, it represents the space used for tmpfs)

The main point to understand buff/cache and free is to be aware that Linux tries to take advantage of our RAM as much as possible (this is the same for Windows, but as Task Manager does not show any information about it, there's not much way for confusion). For that, it will store in memory pages read from disk, so that if we need them again we can take them from memory rather than reading them again from the slow disk. This is called Page Cache-Disk Cache. This is roughly what we see in free -m as buff/cache, and "free" is basically: total - used by programs - used by buff/cache - shared.

In computing, a page cache, sometimes also called disk cache,[2] is a transparent cache for the pages originating from a secondary storage device such as a hard disk drive (HDD) or a solid-state drive (SSD). The operating system keeps a page cache in otherwise unused portions of the main memory (RAM), resulting in quicker access to the contents of cached pages and overall performance improvements. A page cache is implemented in kernels with the paging memory management, and is mostly transparent to applications. Usually, all physical memory not directly allocated to applications is used by the operating system for the page cache. Since the memory would otherwise be idle and is easily reclaimed when applications request it, there is generally no associated performance penalty and the operating system might even report such memory as "free" or "available".

Another interesting point is the swap used value. In my case it's 0, which looks quite natural as I have plenty of available RAM, but sometimes values could be slightly more confusing, as explained here:

Sometimes the system will page something out (for whatever reason). If later that page is moved back to memory for a read operation, the copy in swap space is not deleted. If the same page is later paged out again, without being changed, it can do so without writing to the dis

Fix Broken Folder

I'll share this experience here in case it can help anyone. This morning I was copying a bunch of films on Ubuntu from my laptop to and external NTFS disk. The copying got stuck at 50% and after a while I cancelled it. The usb disk seemed to be still in use but anyway I disconnected it (as so many times before). When connecting it again it looked fine, save for the folder "Films" on which I was copying the files. Opening the folder from the UI it showed up empty. From the command line I could do a cd into it, but doing a ls would return reading directory '.': Input/output error

I did not enter into panic as all the other folders with important stuff (and already backed up in other disks) like personal pictures, documents... seemed intact, so I was only losing like 400 films... (many of them scattered over other old, smaller, external disks). Anyway, managing to recover the folder would be nice...

The first thing I found about was running the badblocks command to verify that the disk was physically OK (I honestly thought so, as all the other folders looked good I was more leaning to think of a file system error). badblocks was taking so long, so I did a fast search and found that it could take up to 70 hours!!! and just to verify something, not to fix the disk...

I read by passing something that I already knew but was missing, that even if NTFS disks are perfectly supported in Linux since a very long while... it's Microsoft who has the full knowledge of the system, so I thought I could check if Windows was able to recognize the folder.

Rebooting my laptop on Windows and trying to navigate the folder I got this error:
The File or Directory is Corrupted and Unreadable

A fast search brought up this article, that recommended doing a chkdsk /f on the problematic disk.

After a few seconds the first messages stating that some error had been found and had been fixed showed up, so it was quite reassuring. After less than 10 minutes (for a 2TBs disk with a 30% use disk), the procedure was finished and the contents of my "Films" folder were back again! I'm putting below some of the fix messages that chkdsk returned:

UEFI and the GPT

As I said in my previus post, I've not kept myself much up to date with the evolution of hardware for the last decade, so I had missed another huge change (not so noticeable as the SSD-NVMe thing, but highly important anyway). I'm talking about the modern UEFI and GPT partitions vs the old BIOS and MBR. You can read a short and clarifying article here.

In the past, we had the BIOS and in order to boot our OS's it needed a MBR in the first sector of the disk. This MBR could store information of up to 4 primary partitions as long with the boot loader. In order to have more than 4 partitions you needed to set one of those partitions as an extended partition.

Modern computers (since 2011 I think) come with UEFI rather than the old BIOS. UEFI can work with a MBR, but it's intended to work with a GPT. This new partitioning scheme is much more powerful, allowing for up to 128 primary partitions. There's still a MBR in the first sector of the disk, the Protective MBR, in case you want to use this disk with a BIOS based computer. After that you have the GPT lied over the next disk sectors (and with a backup copy right at the end of the disk), like this:

UEFI systems use an EFI System Partition that is normally the first disk partition and contains:

An ESP contains the boot loaders or kernel images for all installed operating systems (which are contained in other partitions), device driver files for hardware devices present in a computer and used by the firmware at boot time, system utility programs that are intended to be run before an operating system is booted, and data files such as error logs.

Usually USB drives and external disks come with a MBR partitioning scheme, but if you intend use it as a bootable USB device with multiple partitions you could find useful to change it to use a GPT scheme.

My Laptop Flies

I've recently bought a new laptop, nothing out of the ordinary, 8GBs of RAM, 512 GBs SSD, 4 physical cores - 8 logical ones... in principle it's just a bit better than the laptop I got at work 2 years ago, so I got shocked by how fast it boots, the Windows 10 logon screen shows up in no time! I then installed Ubuntu, and it's the same, it hardly takes 10 seconds for the laptop to be fully operational. I assume the Windows 10 in my work laptop is bloated with some corporate stuff... but anyway, that can not account for the difference, my personal laptop boots like 10 times faster! When I got my current work laptop I thought the SSD would make a huge difference (all my personal laptops so far had magnetic hard drives), and well, it works obviously faster, but I did not get the "Quantum Leap" feeling that I'm getting now with my new one.

I did not think more about it, just enjoyed the situation :-), but a few days ago when visiting the local shop (yes, I'm an old school idiot that whenever possible favors buying stuff in a physical store rather than online, Fuck you Amazon and alikes!) for another reason, I mentioned it to the owner and he told me "yes, it's really amazing, it's all thanks to the new NVMe thing, and yes, your laptop comes with that for sure".

Far away in the past I used to be quite into hardware (northbridge-southbridge chipsets, the different buses, the internals of processors and GPUs...) but that's not been the case in at least the last decade. Before buying this laptop I did some "research" to make sure I was buying something "decent". The 8 GBs RAM and 512 GBs SSD had been clear to me since a long while, but I learned that I should discard i3 processors and go for an i5 (or in my case an AMD Ryzen 5), and that I should have a USB C port (ideally it should be USB C charging enabled, but in the end few laptops support that so far). I was surprised to note that some processors do not implement SMT (HyperThreading in Intel's parlor...), so you don't have the physical vs logical/virtual cores thing. You'll still find for sale laptopts with a 4 physical/virtual cores, and even laptops with 2 physical/4 logical cores... So the 4 physical/8 logical combo became also a must-have for me. The other must-haves were: 14" screen, weight below 1.5KGs and an ethernet port. As for this last requirement, I honestly can not understand how someone can buy a laptop without an ethernet connector (yes, you can use an usb-ethernet adapter, but anyway...). Both in Asturies and in France, with the same laptop, my WiFi speed would never go above 50 Mbps, while the ethernet speed would be around 160-190 Mbps, very close to the official limit of 200 Mbps for both internet providers. On the other side, a touch screen was not in my list. My last laptops both have touch screen and I hardly ever use it.

So I was totally unaware of this NVMe thing, and it seems to me as an improvement as big (if not bigger) as when the multicore processors where introduced. This article is a pretty nice introduction. As a summary, we all know that SSDs are much faster than magnetic HDDs (with their moving parts), but until recently we were accessing them through a slow SATA bus, that was acting as a bottleneck. Now we're using the NVMe protocol over the PCI express bus. This takes advantage of the parallelism provided by SSD's, and we end up with this (from the article):

NVMe SSD reads and writes data literally four times faster than the SATA SSDs found in previous generations. Not only that, but it locates them 10 times as fast (seek). That’s on top of the four- to five-fold improvement in throughput and ten-fold improvement in seek times that was already provided by SATA SSDs when compared to hard drives.

The Odd Case of the Dual-core Netbook

I purchased a second hand laptop a few weeks ago (yes, it may sound like a bit retro/vintage thing to buy a netbook now when they've almost gone away... but let's say it took me a while to find a situation where a netbook could be useful to me).
It's a Vaio VPCW12J1E, and first of all, in case anyone is contemplating to get one, I want to state here that Lubuntu (light Ubuntu) works like a charm on this machine. WiFi will work nicely out of the box, sound card, everything, even the key shortcuts for brightness and sound! and it's absolutely astonishing to see how little memory it needs, really amazing, much more when we've got used to such resource hungry crappy software like Android...

This netbook already came with Windows 7 starter, which works pretty horribly performace wise, but it's not a Windows thing itself, it's something with the tons of (unnecessary) software that retailers add to it...
Well, the thing is that to my surprise Windows 7 tells me that I have a dual-core processor, which was quite unexpected cause it seemed to me that most netbooks (and more from that time, 2009) are single-core machines.

Windows Device Manager:

Windows Task Manager:

However, the Lubuntu Task Manager (lxtaskmanager) tells me it's a Single Core machine, pretty odd:

As you can see from one of the screenshots above, this is an Atom N280 processor, and as I suspected, Wikipedia tells us it's a Single-core Processor, so WTF? After a few searches the answer revealed itself in some forum, it's a Hyper-threading (HTT) thing!

Hyper-threading, wow, that brings up some old memories. A long while ago I was pretty much into how hardware works, and I quite well remember having printed and read some material about this "Virtual MultiProcessor" kind of thing, back in early summer 2002 (at that time the Itanium also seemed like the new king of the party) In a few years Multi-core processors became common place, and the Hyper-threading thing just was no more (or at least that's what I wrongly thought).
In short (not just to save space, but because I have a minimum knowledge about this topic), in an Hyper-threading processor some units in the processor core are duplicated (but not all of them, so it's very far from being a Multi-core/Multi-processsor), so that under some conditions it offers some sort of multiprocessing. The Operating System will see it as 2 cores, and will schedule 2 threads simultaneously, and then depending on the instructions/state of the processor, sometimes some sort of parallelism will happen.

This has made me muse over concepts I had almost forgotten, like the Processor Pipeline and Superscalar processors. Well Superscalar and Hyper-threading seem quite related, as in both cases some functional units in the processor are duplicated in order to provide some form of parallelism. This wikipedia entry is quite elightening. My understanding then is that with superscalar techniques (the attempt of) parallelism happens with instructions belonging to the same thread/process, and without any knowledge on the OS side, while with Hyper-threading, this (attempt of) parallelism happens between different threads/processes, and needs of the OS taking part on it.

This thing of Lubuntu's Task Manager showing only 1 processor made me wonder whether my OS would have the Hyper-threading support enabled, so I did a couple of checks to verify that HTT support was enabled:

• sudo dmidecode
  
  
  
  where the status: Populated, Enabled means that HTT is enabled
• run top and press the "1" key, so that processors are listed (Cpu0 and Cpu1):

This last check was particularly odd. As HTT is a very limited form of parallelism, it would appear fine to me if Linux decided to show me a single processor, but the fact of top displaying 2 processors and the Task Manager displaing 1, is confusing to say the least

32 vs 64 bits again

This post is a bit of complementary of this one I wrote some months ago. This week, someone at work said it was possible to install Windows 7 64 bits on a 32 bits processor. Well, I was pretty certain that was absolutely impossible (same as you can't run a 64 bits application on a 32 bits OS). Obviously the 64 bits OS will make use of the 64 bits registers (otherwise it would not be a 64 bits OS), and those registers are not available in your 32 bits processor (what I've heard is that it's possible to install it on a VM, but I think the performace is terrible, imagine what the VM will have to do to emulate the 64 bits registers...). Anyway, I don't like entering into a discussion before doing some previous extra research.

After checking a few articles I confirmed that I was correct and my colleage confused. A 64 bits OS can't run on a 32 bits processor, same as a 64 bits application can't run on a 32 bits OS. This is one of the best articles that I found, with some rather interesting tables:

Another interesting point they mention there and that should also seem obvious, but I'd never thought of it before, that you can't run 32 bit drivers on a 64 bits OS. The driver's kernel mode code can not run inside the WOW64 emulation system

One more thing that came to my mind is, given than as I explained in my post from October, 64 bits windows applications can't load 32 bits components (dll's, COM components...) what happens for example with Internet Explorer 64 bits and all those intranet applications that use some client side COM component? (we have several of them at work, some of them close to critical)
The answer is here. Your 64 bits Windows system includes 2 Internet Explorer versions, one 64 bits (the default one), and one 32 bits. So for those cases all you need is just launching the 32 bits version.

I think another interesting topic is how much memory is now available to the OS and to an application.

• In 32 bits systems (processor and OS), we had a maximum of 4 Physical GBs of memory accessible to the OS. That's the maximum address you can reference to from those 32 bits registers. However, you end up having quite less than 4 GBs of physical space to you, normally around 3 GBs. Why? because of Memory Mapped IO. The memory and registers of the I/O devices are mapped to (associated with) the upper physical addresses in the main memory. Bear in mind that then the (let's say) 512 MBs of your Graphic Card are reducing the amount of Physical Memory available to the OS, not because the Card is storing data there, but because those addresses are reserved to communicate with the GC physical memory... That's why usually a Windows 32 bits OS can't see more than 3 GBs of physical RAM...

  With a 64 bits system, as much more memory is addressable the memory addresses reserved for IO mapping will be further up from the top of the Physical memory available in your system.

  In the Virtual Memory arena, in a 32 bits system a 32 bits process has 4 GBs of Virtual Memory available to him, from which the 2 first GBs are reserved for the kernel, so indeed it's only 2 GBs of Virtual Memory what is available to the processor. Well, I can think of many applications for which only 2 GBs of memory seems pretty scarce...

• In 64 bits systems, the theoretical limit of Physical Memory that can be addressed is 2 to the power of 64 bytes (that's 16 TBs I think), but in practice most OSs (well, at least Windows) impose restrictions (in Windows this goes from 8 GBs to 2 TBs, you can check the specific figures here)

  For the Virtual Memory, the limitations imposed to the Physical Memory are not applied, and then we have that for each 64 bits process there are 8 TBs of User Virtual Memory available to it (the other 8 TBs are for the Kernel).

This is a good read regarding all this.

I guess this write up has helped me understand how important 64 bits Systems are now not just for big servers, but for a normal user (I'm limited to 3GBs of Physical memory in both my laptops, and well, with Visual Studio, Eclipse, FireFox and Lotus Notes running, that´s far from optimus.

Extend NTFS Volume

I've had and odd issue with a NTFS partition that I think it's worth to share, if not for others at least for a future me confronted to a similiar issue in the future.

I've been lately ejecting some Hard Disks from an old PC and putting them in external HD cases to have easy access to the tons of dubiously useful old stuff stored there.
One of them had some small System partitions (Windows, Linux) and some large (well, not by today's standards) Data partitions. As the System partitions are no longer needed I thought I would merge them with the Data ones. So, launch GParted from Ubuntu to start off the process.
So far GParted had been a well behaved piece of software that had helped me on many ocasions, but this time something went wrong and crashed in the middle of the process. To my horror, the 127 GBs NTFS partition that I was trying to merge with the 10 GBs System partition was unusable (and the 10 GBs one was gone).

I launched Windows 7, ran a chkdsk, and after some minutes my NTFS data partition was alive and well, but with the original 127GBs size (based on the Explorer readings), and the 10GBs one was missing.

Opening the Computer Management -> Disk Management MMC I was presented with this:



So now my partition was showed on the section below as taking up 136 GBs, but above it was showed as occupying the 127 GBs that Explorer was displaying. So, what the fuck???

After some googling I found out somewhere (sorry, I can't locate the link now), that probably this was a mismatch between the Partition and the NTFS Volume. The partition, as showed on the lower section, is 136 GBs, but the NTFS Volume (that is what Explorer and the upper section show), is 127 GBs.

Hopefully, this can be fixed fast and easy with a Windows utility (it's part of the OS normal installation) that I was unaware of, Diskpart, as explained here.

In my case, contrary to what is stated in the article, chkdsk was not showing the the same info as Explorer, but 133 GBs, but Diskpart was showing the expected 136 GBs.





So, just type:

DISKPART> select volume 5 (in my case)

DISKPART> extend filesystem


and as you can see in the below screenshot, we're done and well!