architecture

Technical deep-dive into @zabaca/lattice - a human-initiated, AI-powered knowledge graph CLI for markdown documentation.

Overview

Lattice extracts entities and relationships from markdown documents and syncs them directly to a DuckDB database, enabling semantic search across documentation. No frontmatter required - your markdown files stay clean.

┌─────────────────────────────────────────────────────────────────┐
│                     Lattice CLI (@zabaca/lattice)               │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────┐     ┌──────────────┐     ┌────────────────┐  │
│  │  Markdown    │────▶│   Lattice    │────▶│    DuckDB      │  │
│  │  Docs        │     │   Sync       │     │   (embedded)   │  │
│  │  ~/.lattice/ │     │              │     │                │  │
│  └──────────────┘     │  - Parse     │     │  - Nodes table │  │
│                       │  - Extract   │     │  - Rels table  │  │
│                       │  - Dedupe    │     │  - HNSW index  │  │
│                       └──────────────┘     └───────┬────────┘  │
│                              │                     │           │
│                              ▼                     │           │
│                       ┌──────────────┐             │           │
│                       │  Voyage AI   │◀────────────┘           │
│                       │  Embeddings  │   Semantic Search       │
│                       └──────────────┘                         │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Technology Stack

Layer	Technology	Purpose
CLI Framework	nest-commander	Command parsing and execution
Application	NestJS	Dependency injection, modules
Validation	Zod	Schema validation
Database	DuckDB	Embedded graph + vector storage
Vector Search	DuckDB VSS	HNSW index with cosine similarity
Embeddings	Voyage AI	Semantic search vectors (voyage-3-lite)
Runtime	Bun	Fast JavaScript runtime

Key Architecture Decisions

Embedded database: DuckDB requires zero external dependencies (no Docker, no Redis)
No frontmatter: Entities extracted directly to database, markdown files stay clean
Single file storage: All data in ~/.lattice/lattice.duckdb
HNSW vector indexing: Fast approximate nearest neighbor search

Project Structure

lattice/
├── src/
│   ├── commands/           # CLI command implementations
│   │   ├── sync.command.ts      # lattice sync
│   │   ├── status.command.ts    # lattice status
│   │   ├── query.command.ts     # lattice search, rels, sql
│   │   ├── ontology.command.ts  # lattice ontology
│   │   ├── init.command.ts      # lattice init
│   │   ├── extract.command.ts   # lattice extract (debug)
│   │   └── migrate.command.ts   # lattice migrate (v1→v2)
│   │
│   ├── sync/               # Document synchronization
│   │   ├── sync.service.ts      # Main sync orchestration
│   │   ├── manifest.service.ts  # Change detection (hashes)
│   │   ├── document-parser.service.ts
│   │   ├── entity-extractor.service.ts  # AI entity extraction
│   │   ├── cascade.service.ts   # Relationship impact analysis
│   │   ├── ontology.service.ts  # Schema introspection
│   │   └── database-change-detector.service.ts
│   │
│   ├── graph/              # DuckDB operations
│   │   ├── graph.service.ts     # SQL query execution
│   │   └── graph.types.ts       # Type definitions
│   │
│   ├── embedding/          # Vector embeddings
│   │   ├── embedding.service.ts
│   │   └── providers/
│   │       ├── voyage.provider.ts   # Production
│   │       ├── openai.provider.ts   # Alternative
│   │       └── mock.provider.ts     # Testing
│   │
│   ├── schemas/            # Zod schemas
│   │   ├── entity.schemas.ts
│   │   ├── graph.schemas.ts
│   │   └── config.schemas.ts
│   │
│   ├── pure/               # Pure functions (testable)
│   │   ├── hashing.ts
│   │   ├── embedding-text.ts
│   │   └── validation.ts
│   │
│   ├── utils/
│   │   ├── paths.ts        # ~/.lattice/ path utilities
│   │   └── frontmatter.ts  # Legacy frontmatter parsing
│   │
│   ├── app.module.ts       # NestJS root module
│   └── main.ts             # Application entry
│
├── commands/               # Claude Code slash commands
│   ├── research.md
│   └── graph-sync.md
│
└── dist/                   # Build output

Database Schema

Nodes Table

CREATE TABLE nodes (
    label VARCHAR NOT NULL,           -- Entity type: Document, Technology, etc.
    name VARCHAR NOT NULL,            -- Unique identifier
    properties JSON,                  -- Additional metadata (description, title, etc.)
    embedding FLOAT[512],             -- Vector for semantic search
    PRIMARY KEY(label, name)
);

-- HNSW index for vector similarity search
CREATE INDEX nodes_hnsw_idx ON nodes
USING HNSW (embedding)
WITH (metric = 'cosine');

Relationships Table

CREATE TABLE relationships (
    source_label VARCHAR NOT NULL,
    source_name VARCHAR NOT NULL,
    relation_type VARCHAR NOT NULL,   -- REFERENCES or APPEARS_IN
    target_label VARCHAR NOT NULL,
    target_name VARCHAR NOT NULL,
    properties JSON,
    PRIMARY KEY(source_label, source_name, relation_type, target_label, target_name)
);

Entity Labels

Label	Description	Example
`Document`	Markdown files (auto-created)	`/home/user/.lattice/docs/topic/file.md`
`Topic`	Research subjects	”Machine Learning”
`Technology`	Tools, frameworks, languages	”TypeScript”, “React”
`Concept`	Abstract ideas, patterns	”GraphRAG”, “Microservices”
`Tool`	Specific software tools	”Lattice”, “Docker”
`Process`	Workflows, procedures	”Entity Extraction”
`Person`	People	”Tim Davis”
`Organization`	Companies, teams	”Anthropic”
`Location`	Places, paths	”~/.lattice/docs/“

Relationship Types

Type	Created By	Description
`REFERENCES`	AI extraction	Semantic connection between entities
`APPEARS_IN`	Auto-generated	Links entity to source document

Sync Workflow

Change Detection

Lattice tracks document changes using content hashes stored in the database:

-- Hash tracking in nodes table properties
SELECT name, properties->>'contentHash' as hash
FROM nodes
WHERE label = 'Document';

Sync Phases

Phase 1: Detect changes (compare file hashes vs stored hashes)
Phase 2: Parse changed documents
Phase 3: Extract entities using Claude Code (via /graph-sync)
Phase 4: Deduplicate entities across documents
Phase 5: Upsert nodes with embeddings
Phase 6: Create APPEARS_IN relationships
Phase 7: Create REFERENCES relationships

Entity Deduplication

When the same entity appears in multiple documents, Lattice merges them:

Keeps the longest description
Tracks all source document paths
Single node with multiple APPEARS_IN relationships

CLI Commands

Command	Description
`lattice init`	Install Claude Code slash commands
`lattice sync`	Sync documents to graph
`lattice status`	Show pending changes
`lattice search <query>`	Semantic search
`lattice sql <query>`	Raw SQL queries
`lattice rels <name>`	Show relationships for entity
`lattice ontology`	Show derived schema
`lattice extract <file>`	Debug entity extraction
`lattice migrate`	Migrate from v1 to v2

Sync Options

lattice sync [paths...]      # Sync specific paths or all
lattice sync --force         # Force re-sync (rebuilds graph)
lattice sync --dry-run       # Preview changes

Vector Search

How It Works

User query is embedded using Voyage AI (voyage-3-lite)
DuckDB VSS extension performs HNSW approximate nearest neighbor search
Results ranked by cosine similarity

SQL Example

-- Find similar nodes using vector search
SELECT name, label,
       array_cosine_similarity(embedding, $query_embedding) as score
FROM nodes
WHERE embedding IS NOT NULL
ORDER BY score DESC
LIMIT 20;

Claude Code Integration

Lattice is designed as a Claude Code-first tool.

Slash Commands

Command	Description
`/research [topic]`	AI-assisted research workflow
`/graph-sync`	Batch entity extraction + sync

/research Workflow

1. Search existing docs via `lattice search`
2. Present findings to user
3. Ask if new research needed
4. If yes: WebSearch → Create docs
5. Remind user to run /graph-sync

/graph-sync Workflow

1. Run `lattice status` to find modified docs
2. For each modified doc:
   - Extract entities using Claude
   - Write directly to database
3. Run `lattice sync` to finalize
4. Report results

Design Philosophy

Human-initiated: Claude Code executes, but human approves
AI-powered extraction: LLM identifies entities from content
No frontmatter: Database-first, clean markdown files
Incremental sync: Only process changed documents
Embedded database: Zero external dependencies

Configuration

Environment Variables

Variable	Description	Default
`VOYAGE_API_KEY`	Voyage AI API key	required
`EMBEDDING_DIMENSIONS`	Vector dimensions	`512`

Storage Location

All data stored in ~/.lattice/:

~/.lattice/
├── docs/                  # Markdown documentation
├── lattice.duckdb         # Graph database
├── .sync-manifest.json    # Sync state (legacy, migrating to DB)
└── .env                   # API keys