Secondary indexes

Catalogue and extraction

The value is opaque to the engine, so an index needs a function that extracts the keys to index. It's registered when the database opens:

db.define_index(IndexDef {
    id: 1,
    name: "by_endpoint",
    version: 3,
    unique: false,
    extract: |key, value| -> Vec<Vec<u8>> { /* ... */ },
})?;

The engine calls extract on every put and delete, and maintains the index inside the same transaction as the data. That atomicity is exactly what's bought by putting indexes in the engine instead of maintaining them in the caller's code.

Storage, one keyspace per index:

• Non-unique: [0x80 | id][indexed_key][primary_key] → empty
• Unique: [0x80 | id][indexed_key] → primary_key, with uniqueness guaranteed by key collision — no extra check needed.

Extractor versioning. The version field is stored in the catalogue. If a database reopens with a different extraction function and the version isn't bumped, the index would lie undetectably. With the version stored, the engine detects the mismatch and, depending on configuration, refuses to open or rebuilds the index. It's the only defence possible when the indexing logic lives in a function the caller supplies.

Range deletion

delete_range over a contiguous key range unhooks whole subtrees. Leaf page numbers live in their parent nodes, so leaves are never read: internal nodes are walked, their pointers are dumped to the free list in bulk, and the subtree is unhooked. The cost is proportional to the number of internal pages, not the number of deleted entries.

Pruning 833,000 records with 4 KB pages means reading about 25 internal pages instead of the 8,000 leaves holding the data — two orders of magnitude less I/O. Without this operation, daily retention pruning on every node would be a full tree walk, exactly the CPU cost the project exists to avoid.

Derived rule. Any index defined over a keyspace with retention must put the time component first in its index key. With [bucket][endpoint][pk], deleting the matching index entries is another contiguous range. Without the time prefix, index entries end up scattered across the whole index and pruning goes back to a full scan.