Tek9

• Tek9DB
  • Install
    • Nix flakes
  • Open a database
  • Durability profiles
  • Documents
    • Bulk ingest
    • Indexed initial load
    • Bulk reads
  • Secondary indexes
  • Ordered range scans
  • Graph storage
  • Views
  • Performance rules
  • Benchmark regression gate
  • CI/CD
  • Close

Tek9DB

Tek9 is a local embedded document and graph database implemented in Common Lisp on top of LMDB.

Version 0.2 keeps the small Lisp-facing API while moving the hot paths toward LMDB-native storage structures:

• one LMDB environment per Tek9 database;
• cached named database handles;
• batched write transactions;
• ordered primary-key and secondary-index range scans;
• durable DUPSORT secondary indexes;
• grouped posting-list index builders for sequential rebuilds;
• graph-facing string IDs mapped once to compact internal uint64 row IDs;
• fixed-width graph adjacency records and predicate-specific adjacency indexes;
• full durability by default, with weaker durability profiles only when explicitly selected;
• comparative CI benchmarks that gate performance regressions without mutating the source branch.

Tek9 is still alpha software. Treat the on-disk format and public API as evolving until a stable release is cut.

Install

Clone into Quicklisp local projects:

git clone https://github.com/lost-rob0t/tek9.git ~/quicklisp/local-projects/tek9

Then:

(ql:quickload :tek9)
(use-package :tek9)

Nix flakes

Tek9 exposes packages.<system>.tek9 and packages.<system>.default for x86_64-linux and aarch64-linux. A downstream flake can add Tek9 to an SBCL closure without a developer-local checkout:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    tek9.url = "github:lost-rob0t/tek9";
    tek9.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { nixpkgs, tek9, ... }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };
    in
    {
      packages.${system}.default = pkgs.sbcl.withPackages (_: [
        tek9.packages.${system}.tek9
      ]);
    };
}

The package propagates its Lisp dependencies and LMDB runtime library. Database paths remain application-owned mutable paths; they are never placed in the Nix store.

Open a database

The default LMDB map reserves 16 GiB of virtual address space and supports 64 named databases. The map is a capacity ceiling, not a preallocated 16 GiB data file.

(defparameter *db*
  (open-database
   (new-database "quasar"
                 :path #P"/tmp/quasar-tek9/"
                 :max-size (* 64 1024 1024 1024))))

Keep the environment open for the lifetime of the application instead of reopening it for every operation.

Durability profiles

:full is the default and should be used for canonical user data.

(new-database "canonical" :path #P"/tmp/canonical/" :durability :full)

:metadata-lazy may lose the newest committed transaction after a machine crash while retaining LMDB's transactional structure. :nosync skips normal commit flushing and is intended only for disposable or reproducible stores where the caller explicitly accepts that tradeoff.

Do not turn sync off just to make a benchmark look good.

Documents

(put *db* (new-document :id "person:1"
                        :value '(:dtype "person" :name "Ada")))

(fetch* *db* "person:1")

Bulk ingest

Bulk writes share one LMDB write transaction instead of committing one transaction per document.

(put-bulk *db*
          (loop for i below 100000
                collect (new-document
                         :id (format nil "~12,'0d" i)
                         :value (list :n i))))

If IDs are already in LMDB key order, pass :sorted t. Tek9 checks the database's current final key and uses LMDB's append fast path only when the append precondition is valid; otherwise it falls back to normal insertion.

(put-bulk *db* sorted-documents :sorted t)

Large imports do not populate the legacy materialized-view change vector by default. Use :track-changes t only when that incremental view workflow needs it.

Indexed initial load

When a source database is empty and its secondary indexes are already registered, bulk-load can build source rows and indexes atomically in one write transaction.

(register-index
 *db*
 "dtype"
 (lambda (document)
   (getf (doc-value document) :dtype)))

(bulk-load *db* documents)

The source database must be empty. Each index groups postings by key, sorts the distinct keys and local posting lists, then feeds LMDB in ordered form. If a unique constraint fails, the complete source+index transaction rolls back.

This is an explicit RAM-for-sequential-I/O path, not an unconditional replacement for put-bulk. The CI benchmark keeps both strategies measurable.

Bulk reads

(fetch-bulk *db* '("000000000001" "000000000002" "000000000003"))

Arbitrary point IDs are resolved inside one read snapshot with direct LMDB MDB_GET operations. Exact-key cursor seeks benchmarked worse and were removed from this path.

Secondary indexes

A secondary index is another LMDB named database, not a Lisp hash-table cache. Non-unique indexes use DUPSORT so one indexed value maps directly to multiple sorted document IDs.

(register-index
 *db*
 "dtype"
 (lambda (document)
   (getf (doc-value document) :dtype)))

(rebuild-index *db* "dtype")

(index-document-ids *db* "dtype" "person")
(select-index *db* "dtype" "person")

Indexes are maintained in the same write transaction as document replacement and deletion.

rebuild-index is the bounded-memory streaming path. rebuild-index-fast groups postings in memory by index key and then append-loads those groups in sorted order. Whether the grouped path wins depends on corpus size and cardinality, so CI measures it instead of assuming it is faster.

(rebuild-index-fast *db* "dtype")

Integer indexes can use LMDB's native unsigned-64 ordering:

(register-index
 *db*
 "created"
 (lambda (document)
   (getf (doc-value document) :created))
 :key-type :uint64)

Ordered range scans

Primary keys are already ordered by LMDB. Tek9 can seek once and walk sequentially:

(select-primary-range *db* "person:1000" :end "person:1999")

Secondary indexes support the same pattern:

(select-index-range *db* "created" 1700000000 :end 1800000000)

Use an index or ordered range when the predicate is indexable. select remains the full-scan fallback for arbitrary Lisp predicates.

Graph storage

The graph API still exposes normal string IDs and logical graph names, but the physical graph/v2 hot path resolves them to compact internal IDs. Logical graphs share a constant set of LMDB databases; graph namespaces are isolated by the external-ID mapping.

graph/v2/meta
  counter-name -> uint64 next-row-id

graph/v2/node-map
  (graph-name, external-node-id) -> uint64 node-row

graph/v2/edge-map
  (graph-name, external-edge-id) -> uint64 edge-row

graph/v2/predicate-map
  predicate -> uint64 predicate-row

graph/v2/nodes
  uint64 node-row -> encoded node

graph/v2/edges
  uint64 edge-row -> encoded edge

graph/v2/out
  uint64 source-row -> DUPSORT fixed 16-byte (neighbor-row, edge-row)

graph/v2/in
  uint64 target-row -> DUPSORT fixed 16-byte (neighbor-row, edge-row)

graph/v2/out-predicate
  fixed 16-byte (source-row, predicate-row)
    -> DUPSORT fixed 16-byte (neighbor-row, edge-row)

graph/v2/in-predicate
  fixed 16-byte (target-row, predicate-row)
    -> DUPSORT fixed 16-byte (neighbor-row, edge-row)

The external strings are resolved at the API boundary. Neighbor traversal then stays on integer/fixed-width LMDB keyspaces and does not decode edge objects unless the caller actually asks for edges.

(defparameter *a* (make-instance 'node :id "a" :props '(:name "A")))
(defparameter *b* (make-instance 'node :id "b" :props '(:name "B")))

(put-nodes *db* (list *a* *b*) :database-name "investigation")

(put-edge *db*
          (make-instance 'edge
                         :source "a"
                         :predicate "knows"
                         :target "b")
          :database-name "investigation")

(fetch-node-neighbors *db*
                      "a"
                      :database-name "investigation"
                      :predicate "knows")

Edges have independent IDs, so parallel edges are preserved. Replacing an existing edge ID removes its old generic and predicate-specific adjacency entries before the new edge is inserted. Every edge mutation and all adjacency updates happen atomically in one LMDB write transaction.

Views

Views remain available for derived materialized results. Version 0.2 fixes their transaction boundaries and rebuild path. Secondary indexes should be preferred for normal equality and range lookup because they avoid arbitrary mapper execution across the full corpus.

Performance rules

1. Keep one LMDB environment open per application database.
2. Batch logically atomic writes into one transaction.
3. Use :sorted t only as a hint; Tek9 verifies whether LMDB append is actually safe.
4. Put selective predicates behind secondary indexes instead of decoding the whole corpus.
5. Use range cursors for ordered predicates.
6. Resolve external graph IDs once, then stay on internal uint64/fixed-width adjacency structures.
7. Use predicate-specific adjacency whenever the relationship type is known.
8. Keep graph DBI count constant instead of creating named databases per logical graph.
9. Choose a generous LMDB map size up front; it reserves address space rather than eagerly allocating the whole file.
10. Benchmark with the same durability profile production uses.
11. Do not keep an optimization merely because it sounds database-like; the CI benchmark decides whether it stays fast.

Tek9 intentionally does not add its own buffer pool, WAL, or page cache above LMDB. LMDB already owns those storage-engine responsibilities. Tek9 concentrates on key layout, indexing, batching, query planning, graph semantics, and the Common Lisp API.

Benchmark regression gate

The comparative benchmark runs with :full durability and currently covers:

• per-document durable commits versus batched writes;
• repeated point reads versus a shared read snapshot;
• decoded scans versus DUPSORT equality indexes;
• streaming versus grouped/sequential index rebuild;
• row-maintained versus bulk-built indexed initial ingest;
• edge-materializing graph traversal versus direct adjacency traversal.

Run it locally with:

(tek9-bench-compare:run)

CI emits structured benchmark JSON for the current commit and its parent and uploads both files as a GitHub Actions artifact. The benchmark job never commits generated data back onto the source branch, so a successful check always belongs to the actual PR head being reviewed.

The older files under bench/results/ are retained as historical benchmark snapshots from the previous write-back workflow. They are not updated automatically by CI.

CI/CD

Pull requests and pushes run independent correctness and performance gates:

• the full FiveAM suite on Ubuntu 22.04 and Ubuntu 24.04;
• Python syntax validation for the benchmark aggregation/regression helpers;
• a source-package smoke test;
• same-runner parent/current performance regression benchmarks;
• a final merge-gate job that fails unless every required lane succeeds.

CI uses read-only repository permissions and cancels stale runs when a branch advances. Official GitHub actions are pinned to commit SHAs rather than floating tags.

Pushing a tag such as v0.2.0 runs the release workflow. The tag must match the ASDF version in src/tek9.asd. The workflow reruns the full test suite, creates tek9-<version>.tar.gz plus a SHA-256 checksum, uploads the package as a workflow artifact, and publishes the same files in a GitHub release.

Close

(close-database *db*)