Extending Python

Last week I wrote about how Python gives us access to the bytecodes of one function and how several projects leverage this to create new code from those bytecodes. Python also gives us access to the Python source code of functions, which is also leveraged in interesting ways. To obtain the source code of one function (it also works for classes) we use the inspect.getsource() function. This cute function works good with "advanced" cases, like factory functions, but it won't work for example for code generated with exec/eval/compile (probably I'll write a short, separate post about that).

Looking into different ways to pipe/compose functions I've found a very interesting project that makes use of inspect.getsource(). The general idea behind it is to write a function using valid Python syntax but with the intent of giving that syntax a different meaning (in this particular case, composing/piping functions). As the syntax is correct the function is compiled by Python without a problem, but we indeed want to rewrite it at runtime giving that syntax a different meaning. So this smart guy wrote a decorator for that. The decorator receives the original function, gets its Python source code (via inspect.getsource()), parses it into an AST (using the ast standard module), transforms that AST (converting that fake-pipes syntax into real function piping/composition), and compiles the transformed AST into a new code-object that then is passed to the exec() function to obtain the new function. You should check the source code

I've found 2 projects [1] and [2] that I think are doing something similar to allow for concise/improved lambdas, but honestly I have not looked enough into their source code.

So what we've seen in the previous post and in this post so far is the ability to modify at runtime an existing function. For that existing function to exist the Python interpreter has had to get its source and convert it into an object, and for that the syntax of that function has to be valid. We can use tricks like marker functions or use a given syntax with a different meaning in mind, but we can not use a new, expanded (and invalid) syntax. Well, the amazing news is that indeed we can, but for that, such syntax has to be processed and transformed into standard Python code before the Python interpreter fails to process it, and that can be done at module loading time, by means of import hooks.

Import hooks allow us to modify a file being imported as a module (that can be any file, not a realPpython source code file), before Python tries to compile it and finds that its syntax is not supported. So we can write a hook that takes a pseudo-module containing some extra Python syntax and transforms it to standard Python code. It's the amazing Coconut language who made me aware of this.

Coconut is a variant of Python built for simple, elegant, Pythonic functional programming. Coconut syntax is a strict superset of the latest Python 3 syntax.

So the Coconut language is a superset of Python, which allows us writing code in a much more functional style (and yes, it brings (Multi)-statement lambdas into Python!!!) Coconut transpiles the coconut code to standard Python. You can do this ahead of time, but you can just let it happen at runtime (automatic-compilation), which feels much more natural to me. You can have a project with some modules written in standard Python and others in coconut (with its .coco extension) and run it, without having to remember to preprocess the .coco files first. It's import hooks who allow us doing that.

I have no knowledge at all about implementing import hooks, so I've taken a fast look into the source code out of curiosity. Importing the coconut.api module in our code will enable automatic-compilation by adding a finder object (an instance of CoconutImporter) to the list of meta path finders:

if coconut_importer not in sys.meta_path:
	sys.meta_path.insert(0, coconut_importer)
	...
	
class CoconutImporter(object):
    """Finder and loader for compiling Coconut files at import time."""
	...
	

I've found a similar project, the Hy language, a Lisp dialect embedded in Python. Same as coconut it comes with an import hook, so that you can import the lisp-like modules with no need of compiling ahead of time.

Finally, this project that makes writing import hooks almost trivial looks well worth a mention.