export-workflow

This document describes the complete workflow for exporting YouTube transcripts to Lattice for knowledge graph indexing.

Overview

The workflow converts SQLite transcript data into Lattice-compatible markdown files, which are then synced to create a searchable knowledge graph.

SQLite DB → export_to_lattice.py → Markdown Files → lattice sync → DuckDB Graph

Prerequisites

1. Database restored locally (see Data Access)
2. Lattice installed globally: bun add -g @zabaca/lattice
3. Python 3.x with sqlite3 module (standard library)

Export Script

Location: ~/Projects/uptownhr/agents/packages/youtube-transcript/export_to_lattice.py

Usage

# Basic usage (exports all transcripts)
python export_to_lattice.py \
  --db /tmp/youtube-transcripts.db \
  --output ~/.lattice/docs/youtube-transcripts/

# Test with limited transcripts
python export_to_lattice.py \
  --db /tmp/youtube-transcripts.db \
  --output /tmp/lattice-test-export/ \
  --limit 10

# Skip existing files
python export_to_lattice.py \
  --db /tmp/youtube-transcripts.db \
  --output ~/.lattice/docs/youtube-transcripts/ \
  --skip-existing

Options

Output Format

Each transcript is exported as a markdown file with the following structure:

# Video Title Here

**Channel:** Channel Name
**YouTube ID:** [ABC123](https://youtube.com/watch?v=ABC123)
**Duration:** 15 minutes

## Takeaways

AI-generated summary and key points from the video...

## Full Transcript

Complete corrected transcript text...

Filename Convention

Files are named: {slugified-title}-{youtube-id}.md

Example: fastest-microcontroller-mcu-vs-cheapest-microproce-YnfpbCGyl-E.md

Complete Workflow

1. Restore the database

cd ~/Projects/uptownhr/agents/packages/youtube-transcript
AWS_ACCESS_KEY_ID=minio AWS_SECRET_ACCESS_KEY=minio123 \
  ~/bin/litestream restore -config litestream-restore.yml /tmp/youtube-transcripts.db

2. Export to markdown

# Test first with a small batch
python export_to_lattice.py \
  --db /tmp/youtube-transcripts.db \
  --output ~/.lattice/docs/youtube-transcripts/ \
  --limit 10

# Then export all
python export_to_lattice.py \
  --db /tmp/youtube-transcripts.db \
  --output ~/.lattice/docs/youtube-transcripts/

3. Sync to Lattice

# Check what will be synced
lattice status

# Sync documents to knowledge graph
lattice sync

# Verify with a search
lattice search "housing market crash"

Incremental Updates

For ongoing updates when new transcripts are added:

# 1. Restore latest database
AWS_ACCESS_KEY_ID=minio AWS_SECRET_ACCESS_KEY=minio123 \
  ~/bin/litestream restore -config litestream-restore.yml /tmp/youtube-transcripts.db

# 2. Export only new files
python export_to_lattice.py \
  --db /tmp/youtube-transcripts.db \
  --output ~/.lattice/docs/youtube-transcripts/ \
  --skip-existing

# 3. Sync changes
lattice sync

Search Examples

After syncing, you can search across all transcripts:

# Topic search
lattice search "interest rates mortgage"

# Channel-specific content
lattice search "car market prices wholesale"

# Technical topics
lattice search "microcontroller vs microprocessor"

Integration with Temporal Bridge

For conversations that reference YouTube content, the transcripts become searchable via Temporal Bridge:

// Search for relevant video content
mcp__temporal-bridge__search_project("housing market analysis", 5)

This enables AI assistants to reference specific video insights during conversations.

Troubleshooting

Export fails with “database locked”

The litestream restore creates a read-only snapshot. Ensure no other process is accessing the database.

Missing takeaways in output

Some older videos may not have AI-generated takeaways. The script handles this gracefully with “No takeaways generated.”

Lattice sync errors

Run lattice status to check for issues. Use lattice sync --force to rebuild the graph from scratch.

Related Documentation

• Data Access - Database restore and schema details
• Lattice Architecture - Knowledge graph setup
• Local Knowledge Graph - System overview