Python Adaptive Specializing Interpreter and Local Inline Cache

In this previous post I talked about how Python optimizes the complex attribute lookup process by means of a Global Attribute Cache/MRO cache. We'll see in this post the recent (since Python 3.11) complementary technique used to further optimize attribute lookup.

Time ago I talked about how the CPython bytecode interpreter is since version 3.11 an Adaptive Specializing Interpreter. This means that as it runs it monitors the code to find patterns that allow optimization by replacing generic bytecode instructions by specialized ones, this is also known as Quickening (this is similar to what we are used to with interpreter - layered JIT's combinations, only that working purely at a bytecode level). This specialization is not only a matter of changing one instruction by another, but also of storing (in an inline cache just next to the instruction, a per-site cache) additional information needed by the adaptative instruction.

Since 3.11, CPython rewrites hot instructions in place into **specialized** forms. Each specializable instruction is followed in the bytecode by **inline cache entries** that hold the facts the specialized form depends on.

For looking up an attribute we have a generic LOAD_ATTR instruction, that gets specialized like this:

1. The generic (adaptive) `LOAD_ATTR` keeps a warm-up counter.
2. When the counter runs out, the specializer looks at the current type and attribute (calling `_PyType_Lookup` once) and rewrites the instruction, for example to:
- `LOAD_ATTR_METHOD_WITH_VALUES`: method on a class whose instances use key-sharing dicts
- `LOAD_ATTR_METHOD_NO_DICT`: method on instances without a `__dict__` (for example `__slots__`)
- `LOAD_ATTR_INSTANCE_VALUE`, `LOAD_ATTR_SLOT`, `LOAD_ATTR_PROPERTY`, `LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN`, …

As you can see there are different specialized instructions for an attribute lookup that has ended up returning a method from a class and for one that ends up returning a value from an instance. We can read here:

Load methods for call: o.meth()
The actual address of the method is cached. Method loading now has no namespace lookups – even for classes with long inheritance chains.

Load attribute: o.attr
The attribute’s index inside the class/object’s namespace is cached. In most cases, attribute loading will require zero namespace lookups.

So for method lookup once the specialization is in place we no longer do a lookup in the Global Attribute Cache/MRO cache (that we can consider as a Global Type Cache), but directly have the address of the method in this Local Inline Cache next to the instruction.

For instance attributes it feels even more impressive. Without this optimization a normal lookup for person1.age would search the age attribute in the MRO (after the first attempt it would store in the MRO cache a "NOT_FOUND_IN_MRO" entry) and as not found there it would retrieve it from the dictionary in person1.__dict__. With this specialization, the Local Inline Cache next to the bytecode instruction stores an index to retrieve the value from the the internal structure that contains the values stored in person1.__dict__ (along with guards). This per-site cache works very nicely when we have loops traversing an iterable of instances of the same type, as the __dict__ structure looks the same for all of them and hence the index is valid for all of them.

A successful specialized fast path may therefore avoid both an MRO walk and a lookup in the more general type cache

Python Global Attribute Cache

In my previous post we saw how Python optimizes bound method invokation, It's time to see other particularly important optimizations, the one(s) used for attribute lookup. Because of the dynamic nature of Python, looking up an attribute involves a complex algorithm (I've talked about it here and optimizing it is key, not just to make Python fast, but almost to make it usable. Notice that this optimization is also very important when invoking a method (obj.do_something()), as method invokation involves first that complex look up process to find the callable object, then the creation of a bound method (which optimization/omission we saw in our previous post), and then calling/invoking that callable. So in modern Python there are 2 complementary optimizations that are used for attribute lookup. We'll see in this post the first of them:

- Global Attribute cache / MRO cache
We know that looking up an attribute (either through the LOAD_ATTR bytecode instruction or the dynamic getattr() function), involves following the MRO for that object. Of course Python allows us to patch an existing class, replacing one method by another, adding methods, changing base classes, but this is not particularly common. Most times classes remain static, which means that normally looking up one method/attribute through the MRO will repeatidly come up with the same result. That should prompt an idea, "let's use a cache!", and yes, that's what CPython does:

CPython therefore maintains an internal cache for type-level attribute resolution, historically called the method cache, although it is useful for class attributes in general.

That's the Global Attribute cache / MRO cache. It's useful both when looking up an attribute in an instance or directly in the class. Simplifying things a bit, we could see this cache as a sort of dictionary of [type, attribute-name]: value (it's indeed a direct-mapped cache). As explained in the linked post above when accessing an attribute in an instance the lookup mechanism searches the MRO first (class and base classes) and depending on what it finds (or not finds there) it will then search the instance. Because of this, not finding an attribute in the MRO cache is also valuable information, so the cache contains also sort of "NOT_FOUND_IN_MRO" entries. The MRO cache definitively helps in all cases.

PyObject_GenericGetAttr must call _PyType_Lookup first, even for ob.x where x lives in the instance dict, because a data descriptor on the type (such as a property) takes priority over the instance dict. For plain instance attributes that type lookup almost always finds nothing, and caching that“nothing” keeps every ob.x from walking the whole MRO.

Given the dynamic operations allowed by Python, different actions (that I mentioned above: monkey-patching, changing class hierarhcy) will invalidate cache entries, actions things like:

• type.__setattr__ and type.__delattr__: any Cls.attr = ... or del Cls.attr, dunders included (which also refresh the matching C slots).
• Assigning to __bases__ (which recomputes the MRO).
• A few other type mutations.

For this to work there's a key detail. The C structure used to represent Python classes has a tp_version_tag field that changes (after being invalidated) if the class mutates.

Key properties:
0 means “no valid version”. New classes start at 0.
Tags are assigned lazily from one interpreter-wide counter (next_version_tag). A tag is only handed out when something needs one: the specializer, or the global type attribute cache. Classes that are never looked at never get one.
Tags are unique across all types, not per-class sequences, and never reused. So a cached version == N can only ever match one exact state of one exact class.

Invalidation:
Any change that could affect attribute lookup calls PyType_Modified(type), which:
Sets tp_version_tag = 0 on the type.
Walks tp_subclasses recursively and clears theirs too, because a subclass’s MRO lookup depends on its bases' dicts. The next time a tag is needed, a new one is taken from the global counter.

Another key detail, entries in the Global Attribute Cache contain this tp_version_tag. So if a class gets a new tp_version_tag, both tags will no longer match and the cache value won't be used. A normal MRO lookup will be done and the cache entry will be overwritten. Better explained by a GPT:

The lookup (attribute name in type)
Compute the slot in the cache and check entry->version == type->tp_version_tag && entry->name == name (two comparisons). If both match, return entry->value.
On a miss, do the real MRO walk (find_name_in_mro). Then, if the type has or can be given a version tag and the name can be cached, overwrite that slot.

Python Bound Methods Optimization

We know Python is slow when compared to environments with massively optimized JIT's like the JVM or node.js (using more than one JIT, PGO, etc), but for an interpreted language with no JIT one could say that Python feels fast, particularly if we are aware of how, due to its enormous dynamism, basic actions involve rather complex steps. We saw here and here that calling a function (that at a semantic level means searching for a __call__ method in its type) is done in a more straightforward way.

We're going to look into some optimizations in this post and the next one. A key element is that these optimizations do not compromise language semantics. Under certain common circumstances the optimization will shortcut the way an action works based on the language semantics (like invoking a descriptor to create a bound method or searching the MRO to retrieve an attribute), but if we perform any changes that will render different the output of the optimization from the expected output based on language semantics, the optimization is reverted. Better explained by a GPT:

CPython may combine descriptor handling, instance storage, MRO lookup, a type-level lookup cache, per bytecode inline caches, version guards, and call paths that avoid temporary bound-method objects.
The central principle is:
The runtime must preserve every observable consequence of Python’s semantics, but it need not perform unobservable intermediate work.

Method invokation, bound-method elision
Invoking a method means looking up an attribute. That look up will find a function in a class, and as functions are descriptors the descriptor protocol will run returning a bound method (with __self__ and __func__ attributes). Then the bound method is called (semantically by looking up its __call__ attribute, but we know, from previous linked posts, that that gets optimized). The thing is that in most cases that bound method creation can be skipped. Let me explain:

class Person:
    def __init__(self, name):
        self.name = name

    def greet(self):
        return f"Hello, my name is {self.name}."    

p1 = Person("Iyan")
b_m = p1.greet
print(type(b_m ))  
print(b_m.__func__)
print(b_m.__self__)
# 
# 
# <__main__.Person object at 0x74ef724286e0>

print(b_m("Francois"))
# Hello Francois my name is Iyan.

In this case, the result of attribute access becomes visible to Python code (as we store it in the b_m variable for later use). For an ordinary Python function found in the class, the descriptor protocol produces a bound method containing __self__ and __func__, and calling that bound method is conceptually equivalent to: b_m.__func__(b_m.__self__, "Francois")

But most times we just invoke a method directly, like this:

p1.greet("Francois")

Creating a temporary method object merely to unpack those two references would be wasteful. CPython could instead retain the function and receiver separately and invoke: Person.say_hi(p1, "Xuan"). And basically that's what modern CPython does.

If we look at the bytecodes generated for this second case we'll see:

 23           LOAD_FAST_BORROW         0 (p)
              LOAD_ATTR                3 (greet + NULL|self)
              LOAD_CONST               1 ('Francois')
              CALL                     1
              POP_TOP
              LOAD_CONST               2 (None)
              RETURN_VALUE

That NULL|self tells the lookup algorithm to look up the function and put it in the stack (CPython is a stack based VM) along with the self value (the receiver, the object on which we've done the look up). After that, other parameters to be passed to the function are pushed in the stack and the function is called.

In the next post we'll see the optimizations used for attribute lookup

Counting Lines in Python

We have some Python scripts that at some point count the number of lines in a file. We do so by invoking wc -l, like this:

def count_lines_wc(file_path: str) -> int:
    result = subprocess.run(
        ["wc", "-l", file_path],
        check=True,
        capture_output=True,
        text=True,
    )
    return int(result.stdout.split()[0])

Well, counting lines does not seem like a big deal, so why not just do it in pure Python? The immediate approach would be something like:

    def count_lines_1(file_path: str, encoding: str | None = None) -> int:
        """ counts the number of lines in a file, without loading it all in memory (so it can be used for big files)"""
        with open(file_path, "r", encoding=encoding) as fr:
            return sum(1 for _ in fr)   

The problem with that function when compared to wc -l is that we have to provide the encoding or use the default one (that depends on your system, it's what locale.getpreferredencoding() returns, that on Linux systems is utf-8, but on windows it's cp1252).

When we call "wc -l" we are not passing any encoding, so how does it do it? It just reads the file in binary mode and counts the bytes corresponding to a newline character, that is the 0X0A byte (or b"\n"). So we can do that in Python like this:

def count_lines_2(path: str) -> int:
    """ 
    behaves like 'wc -l' command, as it's byte based it does not have to take into account encoding, 
    but it can be used only for files that use \n as line separator (so Linux, Windows, 'modern' MacOS)
    """
    with open(path, "rb") as f:
        return sum(chunk.count(b"\n") # or b"\x0A"
            for chunk in iter(lambda: f.read(8192), b""))

So I'm reading chunks of n bytes (8192 bytes in this case) and counting the occurrences of the "\n" byte in it. Checking this with a GPT it gave me a more pythonic version:

def count_lines_3(file_path: str) -> int:
    """Counts newline bytes, like wc -l"""
    with open(file_path, "rb") as f:
        return sum(1 for _ in f)  # reads binary lines split by b'\n'

This version is using something I was not aware of. We know that the file object that we obtain after opening a file in text mode is an iterator that yields lines. But the file object that we obtain when opening a file in binary mode (in this cases it's a binary file object) is also an iterable/iterator that iterates bytes splitting by the b"\n" byte (and that b"\n" byte is also included in the bytes yielded in each iteration, save for the last 'line' if it lacks it).

We have an interesting difference depending on whether the last line of a file contains a newline character or not. The standard defines that text files should end with a newline character, but not everybody follows the standard. If we have a text file which last line does not end with a newline character, using wc -l (or my count_lines_2), that work by counting "\n"'s will not count the last line, while using my count_lines_3 will count it.

We can see that difference by executing the above functions with 2 versions of a same text file, one version with a newline character in its last line and another version without it, we get:

for file_name in files:
    print(f"-- Counting lines in {file_name}...")
    for fn in [count_lines_wc, count_lines_1, count_lines_2, count_lines_3]:
        print(f"{fn(file_name)} lines counted by {fn.__name__}")

# -- Counting lines in withEOL.txt...
# 4 lines counted by count_lines_wc
# 4 lines counted by count_lines_1
# 4 lines counted by count_lines_2
# 4 lines counted by count_lines_3

# -- Counting lines in noEOL.txt...
# 3 lines counted by count_lines_wc
# 4 lines counted by count_lines_1
# 3 lines counted by count_lines_2
# 4 lines counted by count_lines_3

Related to this, what if we want to count the lines of one file inside a zip? Obviously we can extract the file and use one of the above functions, but Python zipref.ZipFile allows us reading the file without extracting it. ZipFile.open opens a file in binary mode, and then we can either read it by chunks or iterate it, just as we did in the previous examples. So if we put the 2 previous files into a zip we have:

def count_lines_in_zip_file_1(zip_ref: zipfile.ZipFile, file_name: str) -> int:
    # works like wc -l: count newline bytes without extracting to disk.
    # "r" mode for zip_ref.open is equivalent to "rb" in the standard open, so we get bytes and can count b"\n".
    num_lines = 0
    with zip_ref.open(file_name, "r") as zipped_file:
        for chunk in iter(lambda: zipped_file.read(1024 * 1024), b""):
            num_lines += chunk.count(b"\n")
    return num_lines

def count_lines_in_zip_file_2(zip_ref: zipfile.ZipFile, file_name: str) -> int:
    # Count lines without extracting to disk. If the last line lacks a newline character, it won't be counted
    # "r" mode for zip_ref.open is equivalent to "rb" in the standard open, so we get bytes and can count b"\n".
    num_lines = 0
    with zip_ref.open(file_name, "r") as zipped_file:
        return sum(1 for _ in zipped_file)  # reads binary lines split by b'\n' (b'0x0A') and counts them, so it behaves like wc -l



zip_ref = zipfile.ZipFile("files.zip", "r")

files = ["withEOL.txt", "noEOL.txt"]

for file_name in files:
    print(f"-- Counting lines in {file_name} inside zip file...")
    for fn in [count_lines_in_zip_file_1, count_lines_in_zip_file_2]:
        print(f"{fn(zip_ref, file_name)} lines counted by {fn.__name__}")
        

# -- Counting lines in withEOL.txt inside zip file...
# 4 lines counted by count_lines_in_zip_file_1
# 4 lines counted by count_lines_in_zip_file_2

# -- Counting lines in noEOL.txt inside zip file...
# 3 lines counted by count_lines_in_zip_file_1
# 4 lines counted by count_lines_in_zip_file_2

Python Enums missing_key

In a recent post about Python enums I mentioned the _missing_ method, that gets executed when accessing an enum member by value if that value is missing.

from enum import Enum
class Color(Enum):
    RED = "red"
    BLUE = "blue"
    @classmethod
    def _missing_(cls, value):
        value = value.lower()
        print(f"missing value: {value}")
        for member in cls:
            if member.value == value:
                return member
        return None

print(Color.RED is Color("red"))
# True

print(Color.RED is Color("rEd"))
# missing value: rEd
# True

We know that we can also get an enum member by name, using the [] syntax (subscripted access or key/name access), like color = Color["RED"], and well, it feels to me that it would be nice to have this kind missing functionality for that kind of access (let's call it 'missing_key'). I mean, defining a sort of missing_key method in our enum classes that would get invoked if the key/name used does not exist. And how can we hook this behaviour into the subscripted access to the enum? At the end of this post I talked about those __dunder_methods__ that work as behavioral hooks. We have hooks for invokation, attribute getting, attibute setting... and also one, __getitem__ for subscripted access. For this case, we'll be using the subscripted access in the class, so we have to implement it in a metaclass. The Enum class has EnumMeta as its metaclass, so we'll need our new metaclass to inherit from EnumMeta. When creating an Enum empowered with this "missing_key" functionality, it'll have to use our new metaclass, and inherit from Enum (or related child classes like IntEnum, StrEnum...).

`EnumMeta` implements the class-creation machinery (turning attributes into members, building `__members__` and `_value2member_map_`), but it relies on the base class (`Enum`) to provide the instance API and the initialization/constructor expectations used when creating each member. Without inheriting from `Enum`, those assumptions break and class creation fails or yields an object that isn't a proper enum.

So we define this metaclass, that when accessing a missing key will invoke a _missing_key method (if it exists) in the enum class.

from enum import EnumMeta, Enum

class MissingKeyEnum(EnumMeta):
    def __getitem__(cls, name):
        try:
            return super().__getitem__(name)
        except KeyError:
            print(f"Name '{name}' is not a valid member of {cls.__name__}")
            # Call a dedicated missing-name handler if present
            if hasattr(cls, "_missing_key"):
                return cls._missing_key(name)
            return None

An now we define a class that leverages the metaclass to allow case insensitive subscripted access:

class Query(Enum, metaclass=MissingKeyEnum):
    SELECT = "select"
    INSERT = "insert"
    UPDATE = "update"
    DELETE = "delete"

    @classmethod
    def _missing_key(cls, name):
        # This method is called when a name is not found in the enum
        # You can customize the behavior here, for example, allowing case insensitive access
        return getattr(cls, name.upper(), None)  # Return the enum member if it exists, otherwise return None



query1 = Query["INSERT"]  # This will return the enum member Query.INSERT
query2 = Query["Insert"]  # This will go through _missing_key and return Query.INSERT
print(query1 is query2)  # This will print True, as both refer to the same enum member

query3 = Query["insssseeerrtt"]  # This will go through _missing_key and return None
print(f"query3: {query3}")  # This will print None, as the name is not valid

Just to close this post I'll mention an interesting article that discusses inheriting from Enum vs using a new metaclass that inherits from EnumMeta.

Python Dynamic Class Creation part 2

In a recent post I mentioned the rough steps that Python executes to create a class when it comes across a class statement (we can call it the "class creation pipeline")

• Step 1 — Determine the metaclass. Python calls __build_class__ (a builtin), which inspects the bases and the explicit metaclass= kwarg to resolve which metaclass to use (with MRO-based metaclass conflict resolution).
• Step 2 — Prepare the namespace. The metaclass's __prepare__ classmethod is called: namespace = Meta.__prepare__('Foo', (Base,), **kwargs). This returns the dict (or dict-like object) that will serve as the class namespace. For type, this is just a plain dict. For enum.EnumMeta, for example, it returns a special _EnumDict.
• Step 3 — Execute the class body. The compiled code object for the class body is executed as a function, with the namespace from step 2 as its locals(). This is the key insight: it's essentially exec(body_code, globals(), namespace). After this, namespace contains {'x': 1, '__module__': ..., '__qualname__': ...}.
• Step 4 — Call the metaclass. Meta('Foo', (Base,), namespace) is called — which, for the default type, invokes type.__call__ → type.__new__ → type.__init__. This is where the actual class object is constructed.

And explained how when we create a class dynamically using type() (or any other metaclass), we are directly at step 4. Obviously we have to take care of deciding what metaclass to use and prepare the namespace ourselves, which means that we should invoke the metaclass __prepare__ method (hook) ourselves, something that I was not aware of. Deciding what metaclass to use may seem pretty straightforward, and it is if we intend to use a particular metaclass, but if we just want to use the one that corresponds based on the inheritance chain it's not that evident. Most times, when all base classes just use type, it's just type, but we really would have to take care of analysing the metaclasses of the classes in that inheritance chain.

So indeed, correctly building a class dynamically is more complex that it seemed to me... Well, not really, cause I had missed that since python 3.3 the types module comes with a nice function just for that, types.new_class(name, bases=(), kwds=None, exec_body=None). This function follows the 4 aforementioned steps in the class creation process. It will select the metaclass to use based on whether we provide a metaclass in kwds or the parent classes in bases. It will invoke __prepare__ to create and prepopulate the class namespace, and pass it over to the exec_body function for it to continue populating it (that exec_body function corresponds to the code that we will place in the class body of a normal class statement). Finally it will invoke the selected metaclass to create the class.

def populate(namespace):
	for field in fields:
		namespace[field.name] = create_descriptor(field)

Generated = types.new_class(
	"Generated",
	(Base,),
	{"metaclass": Meta},
	populate,
)

There's another function in types that should call your attention, types.prepare_class(name, bases=(), kwds=None). It returns the metaclass to be used and the namespace created by metaclass.__prepare__, so that then you can further populate that namespace (with the code that you would put in the exec_body callback if you were using new_class), and invoke that metaclass.

meta, namespace, remaining_keywords = types.prepare_class(
	"Generated",
	(Base,),
	{"metaclass": Meta},
)

for field in fields:
	namespace[field.name] = create_descriptor(field)

Generated = meta(
	"Generated",
	(Base,),
	namespace,
	**remaining_keywords,
)

So while new_class performs the 4 steps in the class creation pipeline, prepare_class performs the 2 first steps and leaves under your responsibility performing steps 3 and 4. We could say that create_class follows a callback-based style while prepare_class moves us into an imperative, phase-oriented style.

Sometimes it can be not possible to decide what metaclass should be used based on the base classes combination (and the explicit metaclass if we happen to have provided one). If the base classes drive us to different, uncompatible (not in the same inheritance chain), metaclasses, we get an error:

>>> class MetaA(type): pass
... 
>>> class MetaB(type): pass
... 
>>> class A(metaclass=MetaA): pass
... 
>>> class B(metaclass=MetaB): pass
... 
>>> class C(A, B): pass
... 
Traceback (most recent call last):
  File "python-input-4", line 1, in module
    class C(A, B): pass
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

If the metaclass obtained from the base classes and the one we provide explicitly do no belong to the same inheritance chain,

>>> class C(B, metaclass=A): pass
... 
Traceback (most recent call last):
  File "python-input-6", line 1, in module
    class C(B, metaclass=A): pass
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

Python Object Oriented Enums

Years ago, when I wrote this post about Python enums I mentioned this:

if you want something more advanced like the Java/Kotlin enums, where we can have multiple attributes as values and have instance methods (the archetypical Java Planets example), you can use the powerful aenum module (advanced enums), that just implements the Planets example.

Well, I've just found out that the standard Python enums support "enum members with associated data". You can add methods and properties to your enum classes, so you have a functionality pretty similar to the powerful Java enums. I've felt a bit like an idiot, cause after learning this from a code discussion with a GPT I realised that it's not any "python hidden feature", the standard enums documentation comes with a Planet example.

class Planet(Enum):
    MERCURY = (3.303e+23, 2.4397e6)
    VENUS   = (4.869e+24, 6.0518e6)
    EARTH   = (5.976e+24, 6.37814e6)
    MARS    = (6.421e+23, 3.3972e6)
    JUPITER = (1.9e+27,   7.1492e7)
    SATURN  = (5.688e+26, 6.0268e7)
    URANUS  = (8.686e+25, 2.5559e7)
    NEPTUNE = (1.024e+26, 2.4746e7)
    
    def __init__(self, mass, radius):
        self.mass = mass       # in kilograms
        self.radius = radius   # in meters
        
    @property
    def surface_gravity(self):
        # universal gravitational constant  (m3 kg-1 s-2)
        G = 6.67300E-11
        return G * self.mass / (self.radius * self.radius)

Planet.EARTH.value

Planet.EARTH.surface_gravity

That's the equivalent to the Java Planet one that years ago showed the world that enums could be much more than what we were used to.

public enum Planet {
    MERCURY (3.303e+23, 2.4397e6),
    VENUS   (4.869e+24, 6.0518e6),
    EARTH   (5.976e+24, 6.37814e6),
    MARS    (6.421e+23, 3.3972e6),
    JUPITER (1.9e+27,   7.1492e7),
    SATURN  (5.688e+26, 6.0268e7),
    URANUS  (8.686e+25, 2.5559e7),
    NEPTUNE (1.024e+26, 2.4746e7);

    private final double mass;   // in kilograms
    private final double radius; // in meters
    Planet(double mass, double radius) {
        this.mass = mass;
        this.radius = radius;
    }
    private double mass() { return mass; }
    private double radius() { return radius; }

    // universal gravitational constant  (m3 kg-1 s-2)
    public static final double G = 6.67300E-11;

    double surfaceGravity() {
        return G * mass / (radius * radius);
    }
    double surfaceWeight(double otherMass) {
        return otherMass * surfaceGravity();
    }

"Traditional" enums, like those in C or C# are just "a group of named constant values with type safety", they answered this question "How do I give names to a set of integer constants?". Java enums are "classes with a fixed set of instances defined at compile time", and they answer this question "How do I represent a fixed set of domain objects?"

Before Java added enums to the language, people were using the "Typesafe Enum Pattern" pattern to simulate "object-oriented enums". By the way, years ago I wrote about emulating Java enums in C#.

Java enums have an extra feature that is not directly implemented by Python enums, each member of the enum can have its own implementation of some methods (it's a per-instance override). Let's see an example:

enum Operation {
	PLUS { public int apply(int a, int b) { return a + b; } },
	
	TIMES { public int apply(int a, int b) { return a * b; } };
	
	public abstract int apply(int a, int b);
}

Python does not have a direct equivalent, a member can’t override a method. But there are idiomatic workarounds: carry the behaviour as data (a callable in the tuple) and dispatch on self inside one method:

class Operation(enum.Enum):
	PLUS = (operator.add,)
	TIMES = (operator.mul,)

	def __init__(self, fn: Callable[[int, int], int]) -> None:
		self.fn = fn

	def apply(self, a: int, b: int) -> int:
		return self.fn(a, b)

In Python we can dynamically look up an enum member both by name Color["RED"] and by value (Color("red")). The latter is not supported in Java. Related to this, we can customize behaviour when trying to access an element using a non existing value, by means of defining a _missing_ method:

from enum import Enum
class Color(Enum):
    RED = "red"
    BLUE = "blue"
    @classmethod
    def _missing_(cls, value):
        value = value.lower()
        for member in cls:
            if member.value == value:
                return member
        return None

Color.RED is Color("rEd")
# True