This guide explains how to export YouTube video transcripts from the k3s-deployed YouTube Transcript project into Lattice for knowledge graph indexing and semantic search.

Documents

DocumentDescription
Data AccessHow to access transcript data from k3s/MinIO
Export WorkflowSteps to export and sync to Lattice

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                        k3s Cluster                              │
├─────────────────────────────────────────────────────────────────┤
│  youtube-transcript-worker  →  SQLite DB  →  Litestream  →  MinIO
│  (GPU transcription)           (Prisma)      (replication)      │
└─────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼ litestream restore
┌─────────────────────────────────────────────────────────────────┐
│                        Local Machine                            │
├─────────────────────────────────────────────────────────────────┤
│  SQLite DB  →  export_to_lattice.py  →  Markdown Files          │
│                                             │                   │
│                                             ▼                   │
│                                      lattice sync               │
│                                             │                   │
│                                             ▼                   │
│                                      DuckDB Graph               │
│                                      (entities + embeddings)    │
└─────────────────────────────────────────────────────────────────┘

Quick Start

Terminal window
# 1. Restore database from MinIO
cd ~/Projects/uptownhr/agents/packages/youtube-transcript
AWS_ACCESS_KEY_ID=minio AWS_SECRET_ACCESS_KEY=minio123 \
  litestream restore -config litestream-restore.yml /tmp/youtube-transcripts.db


# 2. Export to markdown (test with 10 first)
python export_to_lattice.py --db /tmp/youtube-transcripts.db \
  --output ~/.lattice/docs/youtube-transcripts/ --limit 10


# 3. Sync to Lattice
lattice sync


# 4. Search transcripts
lattice search "housing market crash"

Data Stats

As of Dec 2025:

  • 658 videos tracked
  • 646 transcripts with full content
  • 646 AI-generated takeaways with analysis