Building Table Canon: TTRPG Campaign Memory Engine (Whisper + Pyannote + Structured Outputs + Music AI)

Community
, , , ,
1

Hey everyone!

Like a few others here who have built tools for tabletop gaming, I was tired of hand-written notes and generic meeting note-takers missing the mark on D&D/TTRPG sessions. Generic transcribers turn fantasy proper nouns like “Mordenkainen” into nonsense, don’t necessarily understand that the “cloaked figure” is the same as “Rodrigo the Hunter”, and treat every 4-hour recording as an isolated island without campaign context.

I’ve been building Table Canon—an end-to-end “campaign memory engine” that transforms multi-hour raw game audio into persistent lore, searchable campaign history, audio recaps, and prep hooks.

I wanted to share our tech stack, pipeline architecture, and some of the hard lessons we hit when scaling processing for long-form, noisy audio.

The Tech Stack

  • Transcription: whisper-large-v3-turbo (handles 3–4 hour long audio files fast with great multi-speaker accuracy)

    • We chunk the audio into 20-minute sections (with 15 second overlap) - processed in a custom pipeline in Modal
    • Chunks are then re-formed by finding overlapping text to stitch things back together

  • Diarization: pyannote (for speaker separation & voice profile calibration)

  • LLM Pipeline & Structured Extraction: OpenAI API with Structured Outputs (JSON Schema enforcement for entity [npc, locations, items, factions, players] extraction, campaign dossiers, and prep ideas [quests, mysteries, unfulfilled promises])

  • Audio Recaps & Narration: Kokoro & Chatterbox Turbo (fast TTS rendering for dramatic session overviews)

  • Tavern Songs / Anthems: ACE-Step-v1.5-XL-Turbo (renders custom session lyrics into shareable MP3 tavern ballads)

Key Features & Engineering Challenges

1. Speaker Calibration & Diarization

  • The Challenge: Raw Whisper doesn’t know who is talking. At a TTRPG table, players jump constantly between out-of-character jokes, in-character dialogue, and DM world-building.

  • The Solution: We run pyannote to generate speaker embeddings. By having players do a quick voice calibration, we match diarization tracks directly to player/character profiles. This stops out-of-character banter from polluting canonical world lore. There’s an added “after-the-fact” step that allows GMs to match voice clips to Players, in cases where the initial match didn’t hit.

2. Custom “World Term” Pre-Lexicon

  • The Challenge: Speech-to-text models hallucinate standard English words over homebrew campaign names,.

  • The Solution: DMs can pre-load a custom term list into the pipeline. Prompting Whisper and guiding downstream LLMs with a pre-pass fantasy dictionary vastly improves first-pass spelling accuracy.

3. Structured Outputs for Compounding Memory & Entity Dossiers

  • The Challenge: Summarizing single sessions is easy; maintaining a living campaign database across 30+ sessions without prompt bloat is hard.

  • The Solution: We rely heavily on Structured Outputs / JSON Schema to extract atomic campaign updates—new entities, updated NPC statuses, aliases, open quest hooks, and promises made by players. This feeds an entity dossier database, allowing the AI to retain long-term state across months of real-time play.

4. The “Ask” Query Engine

  • DMs can issue RAG queries across single sessions or the entire campaign history (“List every promise the paladin made to the Crimson Hand cult”), returning precise answers backed by line-item timestamps and transcript quotes.

  • A “backup” FTS on key words from the user prompt is there in cases where RAG embeddings don’t provide a good match

5. Audio Recaps & Session Songs

  • Structured recap JSON is fed into Kokoro / Chatterbox Turbo for atmospheric narration, or passed to ACE-Step-v1.5-XL-Turbo to turn session recaps into folk ballads you can play for your table before the next session.

Hard Lessons Learned / What Was Hard to Get Right

  1. Audio File Processing & VAD: Processing 4 hours of continuous audio in a single pass is a recipe for memory leaks and timeouts. We had to build robust chunking and Voice Activity Detection (VAD) pre-preprocessing steps before passing audio to Pyannote and Whisper.

  2. Deterministic Extraction via JSON Schema: Before OpenAI introduced strict Structured Outputs, LLMs would frequently alter field names between sessions, breaking database schema constraints for campaign entities. Strict schema enforcement was a game-changer for reliability.

  3. Context Window vs. Cost: Feeding full transcripts from 20 prior sessions into context is financially and technically impractical. Shifting to an Entity/Dossier delta extraction model (where each session only outputs changes to the world state) kept costs low while compounding campaign memory.

I’d love to hear how others in the community are tackling long-form audio diarization, multi-session memory/RAG, or structured state tracking for long-running projects!

Feel free to ask any technical questions about the architecture! See my profile for a link to the app - would love feedback from anyone who tries it.

2

Welcome to the community, @schilling.derek …

Any videos to show us how it works?

How long have you been working on it? Were you a dev before AI?

I’ve added a project tag. If you keep your updates to this thread only, it makes it easier on everyone to keep up to date on your project, which sounds sweet!

Thanks for stopping by.

3

hey @PaulBellow !
Great idea to post a video demo - I’ll get one created and share it here!

Been working on this project for about 4 months - I do have a background in data / product engineering, so that helps. Our company has been somewhat slow to adopt AI (though that’s changing quickly) - so I’ve been looking for outlets to test things out myself :slight_smile: