Search index¶
Experimental
The index layout carries a format marker (akms-records-index/v3) and is
rebuilt automatically when it changes. v2 adds review_cleared, set when
a later amend left a previously reviewed record unreviewed.
recall and wake read a derived SQLite database, one per store root, at
<store root>/.index/records.sqlite. It is never committed (the store's
.gitignore lists .index/), it can always be rebuilt from the event files,
and it belongs to one working tree: two worktrees of a repository each have
their own.
It needs the FTS5 module of the Python sqlite3 library, which the
python.org, Homebrew, python-build-standalone and common Linux builds
include. Without it, recall, wake and index fail with
fts5_unavailable; writes still succeed and the event files are unaffected.
Contents¶
| Table | Holds |
|---|---|
records |
One row per record that replays: record ID, head revision and event digest, scope, schema ref, name and digest, the digest of the extraction rules it was indexed with, review state, lifecycle, sensitivity, recorded_at, update time, the schema's time field (normalized to UTC) and the time used by --since/--until, origin, tags, source refs, superseded_by, the record IDs it names as supersedes, label, summary and review_cleared |
record_tags |
One row per tag, for exact tag filters |
records_fts |
FTS5 (unicode61 tokenizer) over the summary, the tags and the search text the schema's search_text rules select |
files, months, rules, meta |
Bookkeeping for the staleness check |
Records that do not replay (a gap, a conflict, a broken seal), records named in the erasure log, and summaries whose current revision cites a record named there are never indexed.
Keeping it current¶
Writes through the API update the index right after the event file is
linked: add, amend, supersede, retract, erase, import, and
schema register or schema rules set. An update is accepted only when it
carries a higher revision than the one indexed, so two writers that race can
never move the index backwards.
A writer first drops a pending marker in .index/pending/ and removes it
once the index row is written. If the writer dies in between, the marker stays
and the next read replays that record. Markers whose process has exited are
then removed.
Before every read, a cheap staleness check looks at:
- one
statper month directory; - the pending markers;
- the worktree's git
indexandHEADfiles (checkout, merge, reset, commit and pull all rewrite them); - the erasure log;
- the rules file of every schema version that has indexed records.
Only what changed is reconciled. A changed month directory is rescanned with
one stat per record directory, and only records whose directory changed are
replayed. A changed git signature rescans every month directory, which costs
about one stat per record. A changed rules digest re-extracts the records
of that schema version. When nothing changed, nothing is replayed.
Directory times are coarse on some filesystems (two seconds on FAT, a few milliseconds on ext4). A signature taken less than two seconds after its directory changed is stored as unknown and checked again on the next read, so a change made within the same clock tick is not missed.
Changes the cheap check does not see. Files copied in, restored or edited
by tools other than git and akms-records, without a new month directory.
After such a change run akms-records index reconcile (or doctor --fix).
You can also add akms-records index reconcile --scope ... to git
post-checkout and post-merge hooks, though the git signature already
covers those.
Commands¶
| Command | Effect |
|---|---|
index rebuild --scope S |
Drop every table and index every record again. |
index check --scope S |
Replay every record and compare with the index. Changes nothing. status is ok or drift (exit 1), with missing, extra and stale record IDs and the unindexable ones that do not replay. |
index reconcile --scope S |
Replay every record and fix every difference. Reports the IDs added, updated and removed. |
The scope only selects the store root: each command covers every scope in it.
doctor reports a differing index under index_drift (one item per record,
with problem missing, extra or stale, or format when the index has
another format), and an index that cannot be opened under
index_unavailable. doctor --fix reconciles it and lists the changed IDs
under repairs.index_reconciled. A store without an index yet is not a
problem: the first read builds it.
Rebuild parity is tested. A rebuild produces exactly the rows that incremental indexing produced, compared row by row (records, tags, FTS content and rules digests) after a mix of creates, amends, supersessions, retractions, erasures and several scopes.
Erasure¶
erase removes the record's rows from this worktree's index so that none of
its bytes remain in the database files:
- the connection runs with
PRAGMA secure_delete=ON, so SQLite overwrites deleted content; - the FTS5 table has its
secure-deleteoption on, so deleted tokens are removed from the full-text index instead of being marked deleted; - the erasure ends with
PRAGMA wal_checkpoint(TRUNCATE), which empties the write-ahead log.
With an SQLite older than 3.42, which lacks FTS5 secure-delete, deletions
also run the FTS5 optimize command and VACUUM. The test erases a record
and scans records.sqlite, -wal and -shm for its text.
The result's index field says what happened: purged (with
rows_removed and checkpoint), absent (no index yet), or deleted (the
index could not be opened, so its files were removed; it is derived). If
another process held the write-ahead log open, checkpoint is busy and the
erasure report's not_reached says so: run doctor --fix when no other
process uses the store.
Each worktree has its own index. Erasing in one worktree does not clean
another's index; the erasure log reaches it through git, and its next read
(or doctor --fix) removes the record there. Records named in the erasure
log are never indexed, even when a merge brings their files back.
Performance¶
Measured on an Apple M2 Ultra (macOS, APFS, warm cache, plain fsync) with
packages/akms_records/tests/bench_index.py: a synthetic corpus of
Zipf-weighted words in eight project scopes of one store root, spread over 24
months, 10 % of records amended; 200 queries (frequent words, rare words, two
words, word plus tag), 50 wakes with a 2,000-token budget, 300 timed creates
and amends with fsync and the index update.
| Metric | 10,000 records | 100,000 records | Target |
|---|---|---|---|
| Recall p95, query plus reading the top 10 records | 4.2 ms | 27.7 ms | 10 ms / 50 ms |
| Recall p95, query only | 2.6 ms | 27.2 ms | |
| Wake p95 | 43.9 ms | 472 ms | |
| Full rebuild | 2.2 s | 28.0 s | 60 s at 100,000 |
Create p95 (file, fsync, index) |
1.4 ms | 2.1 ms | 5 ms |
Amend p95 (file, fsync, index) |
2.0 ms | 3.1 ms | 15 ms / 40 ms |
| Index size | 24 MB | 195 MB | |
| Rebuild parity | equal | equal | equal |
wake reads every record row of the store roots it opens (to report what it
left out and why), so its cost grows with the root, not with the budget.
CI machines are slower; compare on similar hardware. Run it yourself: