Content Pipeline: Sources to Production Diagram

Download fills .data/ with the raw files and a manifest; re-running is safe
Normalize and enrich turn them into problems, documents, topics and an importance score
Lessons come first, cards and audio are built from lessons, and chunks index lessons plus documents
Everything between download and publish stays in staging.duckdb on your machine

Free steps: download, importance, chunking, rendering mp3s, publish
Paid steps all go through llm.LLM, which refuses a call once logged spend reaches PIPELINE_MAX_USD (default $20)
Lessons raise NoSource and are not written when no downloaded material exists for a topic, so the model cannot answer from memory
Audio runs on OpenCode Go at a flat rate logged as $0, so the cap does not bind; it waits for off-peak hours and stops at 350 calls per 5 hours

Red flows write production: embed to Upstash, audio-upload to R2, publish to Supabase
Publish is one transaction, shows a diff first, and has a dry-run that rolls back
Cards arrive as draft: an admin sample review or pipeline swap makes them live (see the content status diagram)
Reading is the web app only; it never writes content back