Python venvs

In the past I used to install Python modules globally, but since quite a while ago I'm careful to use separate virtual environments (venv) for all my projects. I guess anyone doing any non basic Python stuff will be familiar with venvs, so I'm not going to explain here what a venv is, but to provide some information that though pretty simple feels useful and interesting to me.

We create a new virtual environment with: python -m venv .venv. Notice that .venv is just the name of the folder where the venv will be created, we can use any name, but .venv (or env or venv) is a sort of convention. Then we activate the venv with: source .venv/bin/activate (or .venv\Scripts\activate in Windows).

The python version that we use when creating the virtual environment is the one the venv will use. That means that if we have several python versions (the system one, let's say 3.12, and several altinstalls, let's say 3.11, 3.13), if we create the venv using 3.13 (python3.13 -m venv .venv), when we activate the venv it will use the python3.13 altinstall, regardless of whether we type python, python3 or python3.13

That's so because inside the venv (.venv/bin) we have these symlinks:
python -> python3.13
python3 -> python3.13
python3.13 -> /usr/local/bin/python3.13

If we want to launch a python script in certain venv (I mean in one go, not the typical thing of opening a terminal, activating the venv in that terminal and then launching the python script), we can just put this in a launcher.sh script:
source /path/to/.venv/bin/activate && python /path/to/script.py
This will activate the venv in the bash process that runs the script and hence the python invocation will be done with the python pointed from the venv.

There's a more direct approach that I was not aware of until recently. We don't need to activate the venv, we can just type this:
/path/to/.venv/bin/python /path/to/script.py

All this works because the venv mechanism is implemented by python itself, it's not a third party addition. When we activate a venv with source .venv/bin/activate what is happening mainly is that it's prepending the path to .venv/bin to our PATH variable, that's all. That way we'll reach those symlinks that we've seen that point to the python installation used during the venv creation. So if in the end we're just running that global python installation, how is it that it will find the packages locally installed in: .venv/lib/python3.13/site-packages?

Well, that's so because when started, python checks if a pyenv.cfg file exists in a path relative to the path used for launching python (so in this case the path to that symlink). I guess it gets the path used for launching it by checking argv[0]. If that file (.venv/pyenv.cfg) exists, it will use it for:

• It adjusts sys.path to point sto the venv’s lib/python3.13/site-packages
• It sets sys.prefix and sys.exec_prefix to the venv directory
• It avoids loading global site-packages (unless configured to do so)

With regards to installing packages with pip in a venv, we have to notice that pip is a bootstrap python script and a python module. When we create a venv, 3 pip scripts are created in .venv/bin:

pip
pip3
pip3.13

Each of them is a python script with a shebang pointing to the python version used during the venv creation. They look like this:

$ cd .venv/bin 
$ more pip
#!/myProjects/my_app/.venv/bin/python3.13
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

And a pip module is installed inside the site-packages of that venv, (e.g: .venv/lib/python3.13/site-packages/pip). So when we run any of those pip scripts in the venv, they load the python version that was used when creating the venv, and that python version will see the pyenv.cfg file, prepend the .venv/site-packages to sys.path, and that way load the pip module in the .venv site-packages.