lattice-knowledge-sharing

Comprehensive guide to sharing your ~/.lattice knowledge base with collaborators using P2P file synchronization.

Use Case

Share your Lattice knowledge base (~/.lattice/docs/ markdown files and lattice.duckdb database) with collaborators without relying on centralized cloud services like Dropbox or Google Drive.

Key Requirements for Lattice

1. Bidirectional sync - Both parties can add/edit documents
2. Database safety - DuckDB file (lattice.duckdb) needs special handling
3. Markdown-friendly - Small text files sync efficiently
4. Privacy - No third-party access to knowledge base content
5. Offline-first - Work independently, sync when connected

Recommended Solutions

1. Croc (Recommended for One-Time Sharing)

Best for: Quick, one-time file transfer with a simple share code

Croc is a command-line tool that makes sharing files as simple as sharing a code phrase. Perfect for sending research to a collaborator without setting up ongoing sync.

How Croc Works

┌─────────────┐        ┌─────────────┐        ┌─────────────┐
│   Sender    │◀──────▶│ Relay Server│◀──────▶│  Receiver   │
└─────────────┘        └─────────────┘        └─────────────┘
       │                                             │
       │  1. Generate code (e.g., 7-actress-plural)  │
       │  2. PAKE key exchange (password → key)      │
       │  3. Attempt direct P2P connection           │
       │  4. Fallback to relay if NAT blocks P2P     │
       │  5. End-to-end encrypted transfer           │
       └─────────────────────────────────────────────┘

Key features:

• Code phrase - Human-readable, one-time code (e.g., 7-actress-plural-pilgrim)
• End-to-end encrypted - Uses PAKE (Password Authenticated Key Exchange)
• NAT traversal - Works through firewalls via relay server
• Resume support - Interrupted transfers can be resumed
• Cross-platform - macOS, Linux, Windows, Android

Installation

# macOS
brew install croc

# Linux (Ubuntu/Debian)
curl https://getcroc.schollz.com | bash

# Or download binary from https://github.com/schollz/croc/releases

Sharing Lattice Research

Sender (you):

# Share a single topic directory
croc send ~/.lattice/docs/duckdb

# Output:
# Sending 'duckdb' (5 files, 142 KB)
# Code is: 7-actress-plural-pilgrim
# On the other computer, run:
#   croc 7-actress-plural-pilgrim

Receiver (collaborator):

# Receive the files
cd ~/.lattice/docs
croc 7-actress-plural-pilgrim

# Files are downloaded to current directory
# Now sync to their local knowledge graph
lattice sync

Sharing Multiple Topics

# Create a temporary archive of selected topics
cd ~/.lattice/docs
tar -czf /tmp/research-share.tar.gz duckdb/ agents/ bun-nestjs/

# Share the archive
croc send /tmp/research-share.tar.gz

# Receiver extracts and syncs
cd ~/.lattice/docs
tar -xzf research-share.tar.gz
lattice sync

Advanced: Self-Hosted Relay

For maximum privacy, run your own relay:

# On your server (needs ports 9009-9013)
croc relay

# Sender uses your relay
croc --relay "yourserver.com:9009" send ~/.lattice/docs/duckdb

# Receiver uses same relay
croc --relay "yourserver.com:9009" 7-actress-plural-pilgrim

Pros

• ✅ Simplest UX - just share a code
• ✅ No account or setup required
• ✅ Works through NAT/firewalls
• ✅ End-to-end encrypted
• ✅ Resume interrupted transfers

Cons

• ❌ One-time transfer (not ongoing sync)
• ❌ Both parties must be online simultaneously
• ❌ Requires manual lattice sync after receiving

Recommendation: Best for sharing research with someone once. For ongoing collaboration, use Syncthing.

2. Syncthing (Recommended for Ongoing Sync)

Best for: Continuous, bidirectional P2P sync with conflict handling

Why Syncthing for Lattice

• Open source - Free, no vendor lock-in
• True P2P - Direct device-to-device sync, no central server
• Conflict versioning - Preserves both versions when conflicts occur
• Selective sync - Can sync only docs/ folder, exclude database if needed
• Cross-platform - Works on macOS, Linux, Windows, Android

Setup for Lattice

# Install Syncthing
brew install syncthing  # macOS
# OR
apt install syncthing   # Ubuntu/Debian

# Start Syncthing
syncthing

# Access web UI at http://localhost:8384

Configuration:

1. Add remote device using Device ID exchange
2. Create folder sync for ~/.lattice/docs/
3. Enable versioning: “Simple File Versioning” or “Staggered File Versioning”
4. Important: Consider .stignore to exclude database:

lattice.duckdb
lattice.duckdb.wal
.sync-manifest.json

Database Sync Strategy

Option A: Exclude database, sync markdown only

• Safest approach - each user maintains their own lattice.duckdb
• Run lattice sync after receiving new markdown files
• Embeddings regenerated locally (requires Voyage API key)

Option B: Sync database with caution

• Only works if users never edit simultaneously
• Risk of database corruption if both write at once
• Better: Use LiteSync (see below) for proper database replication

Pros

• ✅ Simple setup and configuration
• ✅ Excellent conflict handling with versioning
• ✅ No ongoing costs or subscriptions
• ✅ Battle-tested and actively maintained
• ✅ Built-in file versioning prevents data loss

Cons

• ❌ Web UI less polished than commercial alternatives
• ❌ Requires both devices online simultaneously for sync (or relay server)
• ❌ No built-in database conflict resolution

Recommendation: Best choice for 90% of use cases. Exclude database, sync markdown only.

2. Resilio Sync

Best for: Users wanting polished UI and faster sync speeds

Why Resilio Sync for Lattice

• Fast - Uses BitTorrent protocol, faster than Syncthing in many cases
• Polished UI - More user-friendly than Syncthing
• Share links - Can share folders via link or QR code
• Selective sync - Fine-grained control over what syncs
• Encrypted - End-to-end encryption for shared folders

Setup for Lattice

# Download from https://www.resilio.com/individuals/
# Install and launch

# Generate link or add device manually
# Configure .syncignore for database files

Sync Strategy: Same as Syncthing - recommend excluding database, syncing markdown only.

Pros

• ✅ Beautiful, intuitive interface
• ✅ Very fast sync performance
• ✅ Easy sharing via links
• ✅ Mobile apps available

Cons

• ❌ Proprietary (free for personal use, paid for advanced features)
• ❌ Free version limited to 10 folders
• ❌ No built-in versioning (uses OS filesystem features)

Recommendation: Good alternative if you prefer polished UI over open source.

3. Git + SSH (Developer-Friendly Approach)

Best for: Developers comfortable with Git workflows

Why Git for Lattice

• Version control - Full history of all changes
• Conflict resolution - Excellent merge tools
• Free hosting - GitHub/GitLab private repos (or self-hosted)
• Gitignore - Easy to exclude database files
• Familiar workflow - If you already use Git daily

Setup for Lattice

cd ~/.lattice/docs/

# Initialize Git repo
git init
git add .
git commit -m "Initial commit"

# Add remote (GitHub, GitLab, or self-hosted)
git remote add origin git@github.com:username/lattice-kb.git
git push -u origin main

# .gitignore
echo "lattice.duckdb" >> ../.gitignore
echo "lattice.duckdb.wal" >> ../.gitignore
echo ".sync-manifest.json" >> ../.gitignore

Collaborator workflow:

# Clone repo
git clone git@github.com:username/lattice-kb.git ~/.lattice/docs/

# Pull updates
cd ~/.lattice/docs/
git pull

# Push changes
git add .
git commit -m "Add new research on topic X"
git push

Pros

• ✅ Full version control and history
• ✅ Excellent conflict resolution tools
• ✅ Works with existing GitHub/GitLab accounts
• ✅ Can use SSH for secure, P2P-like direct sync
• ✅ Familiar to developers

Cons

• ❌ Manual sync (need to commit/push/pull)
• ❌ Learning curve for non-developers
• ❌ Binary files (database) not handled well
• ❌ Not real-time sync

Recommendation: Best for technical teams already using Git. Can combine with Git hooks for semi-automatic sync.

4. LiteSync (Database-Specific Solution)

Best for: When you need to sync the DuckDB database itself

Why LiteSync for Lattice

• SQLite replication - Designed specifically for SQLite-compatible databases
• CRDT-based - Automatic conflict resolution
• Real-time sync - Changes propagate immediately
• P2P capable - Can sync peer-to-peer without central server

DuckDB Compatibility

Challenge: LiteSync is designed for SQLite, but DuckDB can attach/read SQLite databases.

Possible approach:

1. Export Lattice data to SQLite format
2. Use LiteSync to sync SQLite database
3. Attach SQLite to DuckDB for queries

Reality: This adds complexity. Better to sync markdown only and regenerate embeddings.

Pros

• ✅ Proper database conflict resolution
• ✅ Real-time synchronization
• ✅ Designed for database files

Cons

• ❌ Not natively compatible with DuckDB
• ❌ Requires conversion layer
• ❌ Additional complexity
• ❌ Commercial product (paid)

Recommendation: Only consider if you absolutely must sync the database. Otherwise, sync markdown only.

5. CRDT-Based Solutions (Future-Proof Approach)

Best for: Building true local-first collaborative features

CRDT Libraries

Automerge:

• JSON-based CRDT with Rust core
• Strong consistency guarantees
• Sync protocol included
• “PostgreSQL for local-first apps”

Yjs:

• Optimized for text editing
• Memory-efficient binary encoding
• Works with WebRTC for P2P sync
• Popular in collaborative editors

Application to Lattice

CRDTs excel at collaborative editing but require application-level integration. For Lattice, this would mean:

Potential architecture:

Markdown files → CRDT documents → Automerge/Yjs sync → Peers
                     ↓
                 DuckDB sync'd separately

Current state: Would require significant Lattice architecture changes.

Future possibility: Lattice could add CRDT support for true Google Docs-style collaboration on knowledge base.

Pros

• ✅ True offline-first collaboration
• ✅ Automatic conflict resolution
• ✅ No central server required
• ✅ Fine-grained sync (per-document, not per-file)

Cons

• ❌ Requires application integration (not just file sync)
• ❌ Complex to implement
• ❌ Overhead for metadata storage
• ❌ Not a drop-in solution

Recommendation: Not practical for current Lattice, but interesting future direction if real-time collaboration becomes a goal.

Comparison Matrix

Recommended Setup for Lattice

For Individual Collaboration (2-3 people)

Option 1: Syncthing (Recommended)

# 1. Install Syncthing
brew install syncthing

# 2. Start and configure
syncthing

# 3. Share ~/.lattice/docs/ folder
# 4. Add .stignore to exclude database
# 5. Each person runs `lattice sync` after changes

Why: Simple, reliable, free, works offline.

For Team/Organization

Option 2: Git + Private Repo

# 1. Create private GitHub/GitLab repo
# 2. Add ~/.lattice/docs/ to Git
# 3. Team clones and syncs via Git
# 4. Use pre-commit hooks to run `lattice sync`

Why: Audit trail, familiar workflow, scales well.

For Advanced Users

Option 3: Hybrid Approach

• Markdown: Syncthing or Git for file sync
• Database: Each user maintains own, runs lattice sync
• Embeddings: Generated locally (requires API key for each user)

Why: Best of both worlds - real-time file sync + local database ownership.

Database Considerations

Why NOT Sync DuckDB Directly

1. Corruption risk - Two users writing simultaneously corrupts database
2. No conflict resolution - DuckDB is single-writer by design
3. Large file size - Database file is larger than markdown files
4. Lock files - .wal files complicate sync

Recommended Database Strategy

Each user maintains their own database:

1. Sync markdown files only (using Syncthing, Resilio, or Git)
2. After receiving new/updated markdown:

   lattice sync

3. Lattice regenerates embeddings and updates local database
4. Result: Same knowledge base content, independent databases

Cost: Each user needs their own Voyage API key for embeddings.

Alternative (if embeddings are expensive):

Export/import embeddings separately:

# User A exports
lattice export-embeddings > embeddings.json

# User B imports
lattice import-embeddings < embeddings.json

(Note: These commands don’t exist yet, but could be added)

Security Considerations

End-to-End Encryption

Access Control

Syncthing: Device-level - only explicitly added devices sync Resilio: Link-based or device-based Git: Repository permissions (GitHub/GitLab)

Data Leakage Prevention

1. ✅ Never commit API keys to Git (use .env in .gitignore)
2. ✅ Exclude sensitive documents from sync if needed
3. ✅ Use encrypted file systems for local storage
4. ✅ Review .stignore / .gitignore carefully

Conflict Resolution

Markdown File Conflicts

Syncthing approach:

• Creates .sync-conflict versions
• Both files preserved
• User manually resolves

Git approach:

• Merge conflicts with <<<<<<< markers
• Use git mergetool for resolution
• Commit resolved version

Recommendation: Use descriptive commit messages / sync labels to help resolve conflicts.

Database Conflicts

Best practice: Avoid by not syncing database directly.

If you must sync database:

1. Coordinate edits (avoid simultaneous work)
2. Use “last write wins” strategy
3. Keep backups
4. Consider LiteSync for proper CRDT-based resolution

Future Improvements for Lattice

Potential Features

1. Export/import embeddings - Share computed embeddings without regenerating
2. Conflict-free entity merge - CRDT-based entity synchronization
3. Built-in sync command - lattice sync-remote <peer> using libp2p
4. Collaborative editing - Real-time multi-user document editing

Integration Ideas

• Syncthing plugin - Automatic lattice sync after file changes
• Git hooks - Pre-commit: validate markdown, Post-pull: run sync
• CRDT documents - Replace markdown files with Automerge documents

Quick Start Guide

30-Second Share (Croc)

Sender:

brew install croc                        # One-time install
croc send ~/.lattice/docs/duckdb         # Share a topic
# → Code is: 7-actress-plural-pilgrim

Receiver:

brew install croc                        # One-time install
cd ~/.lattice/docs
croc 7-actress-plural-pilgrim            # Receive files
lattice sync                             # Add to knowledge graph

1-Minute Setup (Syncthing)

# Install
brew install syncthing

# Start
syncthing &

# Open UI
open http://localhost:8384

# Add folder: ~/.lattice/docs/
# Share with collaborator's Device ID
# Create .stignore with:
#   lattice.duckdb
#   *.wal

First Sync

# After receiving files from collaborator
cd ~/.lattice
lattice sync  # Regenerate your local database

# Make changes
vim docs/my-topic/new-research.md

# Syncthing auto-syncs in background
# Collaborator receives file and runs `lattice sync`

Troubleshooting

Syncthing not syncing

• Check both devices are online
• Verify Device IDs are correct
• Check .stignore isn’t blocking files
• Review sync status in web UI

Database corruption

• Stop all Lattice processes
• Restore from backup (Syncthing versioning)
• Rebuild: rm lattice.duckdb && lattice sync

Merge conflicts in markdown

Git approach:

git status  # See conflicted files
# Edit files, remove <<<<<<< markers
git add .
git commit -m "Resolved merge conflict"

Syncthing approach:

# Find .sync-conflict files
find ~/.lattice -name "*.sync-conflict*"
# Manually merge changes
# Delete conflict files when done

Sources

1. Croc - GitHub
2. Syncthing Documentation
3. Resilio Sync vs Syncthing Comparison (2025)
4. Syncthing or Resilio Sync - Self-Hosted File Syncing
5. 5 Self-Hosted Markdown Editors That Sync Without the Cloud
6. HedgeDoc - Collaborative Markdown Editor
7. LiteSync - SQLite Replication
8. Automerge - CRDT Library
9. Yjs - Shared Data Types
10. Using CRDTs + Sync as a Database
11. CRDT Dictionary - Field Guide
12. DuckDB Concurrency Documentation

Last Updated: 2025-12-26 Recommended: Syncthing for markdown sync + local database regeneration