Voice Insight: turn call recordings into grounded, quote-checked insight

Purpose & scope

The app is a learning template for the voice-to-insight pattern. Speech-to-text acts as a grounding action: it turns audio into words with speakers and timings. Those words, the rubric and the rules are placed in the model's working memory (the prompt). The rubric and rules are the agent's procedural memory. The LLM performs a reasoning action and proposes claims, each with a quote. The verifier is the decision procedure: it evaluates every proposal against the transcript and keeps only the grounded ones. The vocabulary follows CoALA (after Sumers et al., 2023).

In scope

• Transcribe an uploaded call in any of the provider's languages, with language detection and two-speaker diarization.
• Compute talk share, silence, overtalk, monologue length, speaking rate and first-response time from word timings, with no model.
• Score the call against a nine-category rubric with structured JSON output, in the language it was spoken.
• Verify every quote for text, time (±5 s) and speaker; drop and count failures; withhold the score when nothing survives.
• Play the original audio in the browser and seek to any quote.
• Run as a private demo (access code), a public demo with a capped upload size, or fully open for local development.

Out of scope

Persistence of any kind (no database, history or accounts), files over 4.5 MB, more than two speakers, a human review loop that feeds corrections back into the rubric, streaming output and multi-tenant access control. Section 10 lists these as follow-ups.

Use cases

ID	Use case	Behaviour
UC1	Analyse a call	Checks the file size against the visitor's limit, relays the audio to AssemblyAI, polls until the transcript is ready, then returns a score, category scores, strengths and coaching points, each with a verified quote, plus measured metrics. About 50 s for a 10-minute call.
UC2	Inspect evidence	Clicking a quote, a timeline bar or a transcript line seeks the audio to that moment. The Provenance panel lists every dropped claim and its reason.
UC3	View sample output	Shows a built-in example result, including one discarded claim, under a banner. No API is called.
UC4	Unlock larger uploads	In public mode, a valid access code sets a signed cookie that lifts the upload cap from PUBLIC_MAX_MB to 4.5 MB. In private mode, the code is required to open /demo at all.

Architecture

There is one runtime part: a Next.js 16 app. Its browser side holds the audio file (played from an object URL), the job id and the result. Its server side is three stateless API routes plus an optional access gate in proxy.ts. Two third-party APIs do the heavy work: AssemblyAI for speech-to-text and xAI for the LLM. There is no database, storage bucket, queue or webhook.

Key flows

4.1 Transcription: upload → relay → poll

The browser posts the raw bytes. The route checks the size against the visitor's limit, relays the audio to AssemblyAI's upload endpoint, and starts a job with language_detection, speaker_labels and sentiment_analysis. The browser keeps the job id and polls every 3 s, so the server stays stateless.

4.2 Analysis: fetch → prompt → infer → verify

4.3 Verification: locate → time → speaker

Each quote is fuzzy-matched against the word stream (normalised Levenshtein similarity ≥ 0.85), then the nearest occurrence must be within TIMESTAMP_TOLERANCE_MS = 5000 of the claimed time, then the diarized speaker must match. A kept quote is snapped to the transcript's own timings. If claims were made and none survived, the overall and category scores are withheld.

4.4 Analyzer lifecycle

Backend design

File	Responsibility
src/app/api/transcribe/route.ts	Reads the body, enforces uploadLimitFor(cookie), uploads to AssemblyAI and submits the job. maxDuration = 60.
src/app/api/status/route.ts	Returns working, ready or failed for a job id.
src/app/api/insight/route.ts	Fetches the finished transcript, guesses the agent speaker, runs analyseCall. maxDuration = 300.
src/app/api/access/route.ts	POST exchanges the code for an HMAC cookie (7 days); DELETE clears it.
src/proxy.ts	Redirects or returns 401 for /demo and the three API routes when the gate is active.
src/lib/access.ts	Gate state, public mode (PUBLIC_MAX_MB), per-visitor upload limit, HMAC tokens, constant-time compare, open-redirect guard.
src/lib/providers/asr.ts · asr-assemblyai.ts	The AsrProvider interface and its AssemblyAI implementation, normalised to words, utterances, language and sentiment.
src/lib/providers/llm.ts	structuredCall(): OpenAI SDK against XAI_BASE_URL (default https://api.x.ai/v1), temperature: 0, response_format: json_schema with strict: true.
src/lib/quality-guide.ts	The nine rubric categories; drives the prompt, the schema enum and the chart.
src/lib/analysis/prompt.ts	SYSTEM_PROMPT, formatTranscript() ([start-end] A [agent]: text) and buildUserPrompt().
src/lib/analysis/schema.ts	zod schema for the model's answer, converted to JSON Schema; toInsight() maps it to app types.
src/lib/analysis/run.ts	Orchestrates prompt → model → parse → verify → metrics, and withholds scores when nothing is verified.
src/lib/verify.ts	Script-aware normalisation, quote location, time and speaker checks, drop reasons and stats.
src/lib/metrics.ts	Deterministic call metrics and the first-speaker heuristic.

System prompt

The user prompt adds the detected language, the duration, the rubric categories and the transcript in its original language, one [start_ms-end_ms] line per utterance.

API contract

Method	Path	Request	Response
POST	/api/transcribe	raw audio body, content-type audio/*	{"id"}
GET	/api/status?id=	n/a	{"state":"working"} · {"state":"ready"} · {"state":"failed","error"}
POST	/api/insight	{"id"}	{"transcript":{"utterances","words","sentimentSegments","languageCode","languageConfidence","durationMs","agentSpeakerLabel"},"model","insight","metrics","verification"}
POST	/api/access	{"code"}	{"ok":true} + Set-Cookie: voice_insight_access
DELETE	/api/access	n/a	{"ok":true}, cookie cleared

Errors: 400 for an empty body, a missing id or invalid JSON; 401 for a missing access cookie on a gated route or a wrong code; 409 when posting a code to a deployment with no ACCESS_CODE; 413 for a file over the visitor's limit; 422 when no speech was detected; 500 with a readable error for provider or schema-validation failures.

Data model

Nothing is persisted, so there is no schema to migrate. The shapes live in TypeScript (src/lib/analysis/types.ts, src/lib/providers/asr.ts), and the model's half is enforced by the zod schema in src/lib/analysis/schema.ts. One /api/insight response is the whole record, and it lives only in the browser tab.

Frontend design

• The root layout wraps every page in the shared demo look (newsroom tool, from the demo-doc skill): a masthead strip linking The Inference Times and this post, the pipeline as a stage strip, a status note with the public upload limit, a night-edition toggle, and a footer linking the design doc, source, fork and licence.
• demo/page.tsx is a server component: it reads the access cookie and passes maxBytes, gated and canUnlock to the client Analyzer, which owns all state.
• The Analyzer calls the three routes with plain fetch, polling /api/status every 3 s, and shows a stage bar with elapsed time.
• One seek(ms) function moves the <audio> element. Quotes, the call timeline and transcript lines all call it, which is what makes the evidence checkable by ear. Measured panels are labelled apart from the model's judgement, and Provenance shows how many quotes were checked and dropped.

Project layout

voice-insight-template/
├── README.md                       quick start, pipeline figure, deploy + spending limits
├── CONTRIBUTING.md · LICENSE       MIT
├── .env.example                    ASSEMBLYAI_API_KEY, XAI_API_KEY, ACCESS_CODE, PUBLIC_MAX_MB
├── .github/                        CI (lint · typecheck · test · build), issue + PR templates
├── docs/
│   ├── ARCHITECTURE.md             stage-by-stage walkthrough
│   ├── DESIGN.md                   this document
│   └── figures/                    fig-01 … fig-11 (hand-drawn SVG)
└── src/
    ├── proxy.ts                    optional access gate
    ├── app/                        layout · demo-look.css · page · demo/ · unlock/ · api/{transcribe, status, insight, access}
    ├── components/                 analyzer.tsx · charts/ · landing/ · demo-look/ · unlock-form · lock-button
    └── lib/                        access · limits · metrics · verify · quality-guide · analysis/ · providers/

Run & verify

No install needed to try it: the live demo runs in public mode with a capped upload size, and View sample output works without uploading anything. To run it yourself:

git clone https://github.com/nitingoswami-io/voice-insight-template.git && cd voice-insight-template
npm install
cp .env.example .env.local     # add ASSEMBLYAI_API_KEY and XAI_API_KEY
npm test                       # offline, no keys needed
npm run dev                    # http://localhost:3000

Check	Expected
npm test	110 tests pass with no network or keys.
GET / and GET /demo	200; with no ACCESS_CODE, no lock button is shown.
/demo → View sample output	A scored call with verified quotes, and one claim under Provenance discarded as quote_not_found ("Engineer Tuesday morning ko definitely aa jayega, I guarantee it.").
Upload a 10-minute support call (mono MP3)	About 50 s end to end; every coaching point carries a quote that plays the right moment when clicked.
PUBLIC_MAX_MB=0.001 and a 4 KB upload	413 with "The public demo accepts up to …". With a valid access cookie the same upload passes the cap.
ACCESS_CODE set: POST /api/transcribe with no cookie	401 {"error":"Access code required."}; /demo redirects to /unlock?next=%2Fdemo.

Next steps

Idea	What it teaches
Persist analyses and let a reviewer accept or reject each point	Episodic memory, and a human-gated learning action that tunes the rubric from feedback
Stereo (multichannel) recordings	Removes the speaker-role guess and turn-boundary errors
A "swap speakers" control	Small UI fix with a visible effect on every score
Direct-to-storage uploads with signed URLs	Lifts the 4.5 MB cap; how serverless body limits shape architecture
A second AsrProvider (self-hosted Whisper) and a vLLM endpoint	Provider seams; keeping transcripts on your own infrastructure
Aggregate the drop rate across many calls	Verification stats as a live hallucination metric

Contribute

Fork the repo, pick an idea from the table, and open a pull request. Bugs and unclear steps go in issues; the workflow is in CONTRIBUTING.md. Fork