We have some Python scripts that at some point count the number of lines in a file. We do so by invoking wc -l, like this:
def count_lines_wc(file_path: str) -> int:
result = subprocess.run(
["wc", "-l", file_path],
check=True,
capture_output=True,
text=True,
)
return int(result.stdout.split()[0])
Well, counting lines does not seem like a big deal, so why not just do it in pure Python? The immediate approach would be something like:
def count_lines_1(file_path: str, encoding: str | None = None) -> int:
""" counts the number of lines in a file, without loading it all in memory (so it can be used for big files)"""
with open(file_path, "r", encoding=encoding) as fr:
return sum(1 for _ in fr)
The problem with that function when compared to wc -l is that we have to provide the encoding or use the default one (that depends on your system, it's what locale.getpreferredencoding() returns, that on Linux systems is utf-8, but on windows it's cp1252).
When we call "wc -l" we are not passing any encoding, so how does it do it? It just reads the file in binary mode and counts the bytes corresponding to a newline character, that is the 0X0A byte (or b"\n"). So we can do that in Python like this:
def count_lines_2(path: str) -> int:
"""
behaves like 'wc -l' command, as it's byte based it does not have to take into account encoding,
but it can be used only for files that use \n as line separator (so Linux, Windows, 'modern' MacOS)
"""
with open(path, "rb") as f:
return sum(chunk.count(b"\n") # or b"\x0A"
for chunk in iter(lambda: f.read(8192), b""))
So I'm reading chunks of n bytes (8192 bytes in this case) and counting the occurrences of the "\n" byte in it. Checking this with a GPT it gave me a more pythonic version:
def count_lines_3(file_path: str) -> int:
"""Counts newline bytes, like wc -l"""
with open(file_path, "rb") as f:
return sum(1 for _ in f) # reads binary lines split by b'\n'
This version is using something I was not aware of. We know that the file object that we obtain after opening a file in text mode is an iterator that yields lines. But the file object that we obtain when opening a file in binary mode (in this cases it's a binary file object) is also an iterable/iterator that iterates bytes splitting by the b"\n" byte (and that b"\n" byte is also included in the bytes yielded in each iteration, save for the last 'line' if it lacks it).
We have an interesting difference depending on whether the last line of a file contains a newline character or not. The standard defines that text files should end with a newline character, but not everybody follows the standard. If we have a text file which last line does not end with a newline character, using wc -l (or my count_lines_2), that work by counting "\n"'s will not count the last line, while using my count_lines_3 will count it.
We can see that difference by executing the above functions with 2 versions of a same text file, one version with a newline character in its last line and another version without it, we get:
for file_name in files:
print(f"-- Counting lines in {file_name}...")
for fn in [count_lines_wc, count_lines_1, count_lines_2, count_lines_3]:
print(f"{fn(file_name)} lines counted by {fn.__name__}")
# -- Counting lines in withEOL.txt...
# 4 lines counted by count_lines_wc
# 4 lines counted by count_lines_1
# 4 lines counted by count_lines_2
# 4 lines counted by count_lines_3
# -- Counting lines in noEOL.txt...
# 3 lines counted by count_lines_wc
# 4 lines counted by count_lines_1
# 3 lines counted by count_lines_2
# 4 lines counted by count_lines_3
Related to this, what if we want to count the lines of one file inside a zip? Obviously we can extract the file and use one of the above functions, but Python zipref.ZipFile allows us reading the file without extracting it. ZipFile.open opens a file in binary mode, and then we can either read it by chunks or iterate it, just as we did in the previous examples. So if we put the 2 previous files into a zip we have:
def count_lines_in_zip_file_1(zip_ref: zipfile.ZipFile, file_name: str) -> int:
# works like wc -l: count newline bytes without extracting to disk.
# "r" mode for zip_ref.open is equivalent to "rb" in the standard open, so we get bytes and can count b"\n".
num_lines = 0
with zip_ref.open(file_name, "r") as zipped_file:
for chunk in iter(lambda: zipped_file.read(1024 * 1024), b""):
num_lines += chunk.count(b"\n")
return num_lines
def count_lines_in_zip_file_2(zip_ref: zipfile.ZipFile, file_name: str) -> int:
# Count lines without extracting to disk. If the last line lacks a newline character, it won't be counted
# "r" mode for zip_ref.open is equivalent to "rb" in the standard open, so we get bytes and can count b"\n".
num_lines = 0
with zip_ref.open(file_name, "r") as zipped_file:
return sum(1 for _ in zipped_file) # reads binary lines split by b'\n' (b'0x0A') and counts them, so it behaves like wc -l
zip_ref = zipfile.ZipFile("files.zip", "r")
files = ["withEOL.txt", "noEOL.txt"]
for file_name in files:
print(f"-- Counting lines in {file_name} inside zip file...")
for fn in [count_lines_in_zip_file_1, count_lines_in_zip_file_2]:
print(f"{fn(zip_ref, file_name)} lines counted by {fn.__name__}")
# -- Counting lines in withEOL.txt inside zip file...
# 4 lines counted by count_lines_in_zip_file_1
# 4 lines counted by count_lines_in_zip_file_2
# -- Counting lines in noEOL.txt inside zip file...
# 3 lines counted by count_lines_in_zip_file_1
# 4 lines counted by count_lines_in_zip_file_2
