Making voice bots debuggable, then making them improve themselves

Context

Voice bots fail in ways that are slow to find: latency, speech recognition, prompt drift, tool errors, platform fallbacks that look like the bot misbehaving. Engineers dug through recordings and logs one call at a time.

Argus: one command to debug any call

A FastAPI service I built from scratch and shipped behind a load balancer with a Jenkins rolling deploy and automatic rollback. It answers in Google Chat, from a CLI and over a REST API.

Give it any call identifier and it pulls metrics and logs from two OpenSearch clusters, the bot's configuration from the platform API, and the recording from S3, then explains what happened.

• Perceived latency measured from the recording itself (energy-based VAD + Gemini audio analysis): the gap the caller heard, not the gap the logs claim.
• Disconnect attribution that cross-checks the platform's “who hung up” against error logs and transport events.
• Recordings playable without VPN through an HMAC-signed proxy with per-account token derivation.
• A natural-language mode: Gemini Flash with 16 function-calling tools and per-thread memory, so follow-ups need no IDs.
• A daily Splunk On-Call alert digest, file-locked across workers.

AutoLoop, part 1: building a bot before go-live

The loop I run on every new bot, packaged so any FDE can run it from Cursor or Claude Code:

• Generate requirements.md for the bot from the customer's requirement document.
• Generate the scenarios: the eval suite the bot has to pass.
• Write a baseline prompt and four other prompting techniques (compact, decision table, chain-of-thought, persona-driven), run all five in chat simulations against the real bot runtime over WebSocket, and lock the technique that passes the most scenarios.
• Iterate: analyse failing transcripts (thinking-tag leaks, stalls, language mismatch, missing empathy), make a targeted prompt edit, re-run. Usually 2–4 rounds; one documented CCTV-support bot went from a 30% to an 85% behavioural score in 3.
• Validate with voice simulations on the in-house voice simulator, then real calls. Every run ends in a report and a PDF.

AutoLoop, part 2: improving bots already in production

Seven stages with a human approval gate in the middle. Nothing is applied without a person saying yes.

• Evaluate: benchmark sampled production calls on 8 deterministic and 6 LLM-as-judge dimensions (plus 12 use-case ones). Platform health is classified first, across 11 infrastructure failure patterns, so an outage never gets blamed on the prompt.
• Investigate: five data streams per call, a per-turn “was this actually LLM-generated?” check, one root-cause tag per call, and log lines traced back to the source code that emitted them.
• Recommend: six preflight gates (sample-size confidence, same-config evidence, grounding in a specific call and turn) before proposing an exact before/after diff to one prompt layer.
• Apply: after approval, a drift check, then a new assistant version with the actor recorded.
• Verify: re-benchmark only calls after the change (time-fenced), so old calls can't pollute the score.
• Compare: per-dimension deltas marked improved, stable or regressed; a regressed composite means roll back.
• Journey: a version-by-version audit of what changed, when, and by whom.

Packaging

AutoLoop ships as 20+ agent skills in all (stage skills, simulation skills and model-specific prompt skills for Gemini Flash, Gemma 4, GPT-4.1-mini and Qwen 3; 16 of them call the platform's REST APIs), an argus.sh CLI with about 30 commands, and an idempotent installer. Argus also hosts the CallEvaluator and ProdBotRecommender pipelines and a nightly evaluation harness.

What I'd do differently

• Define the eval metrics before building the tooling around them.
• Classify platform health from day one; I learned that one from the May incident.
• Invest in the voice simulator earlier, because chat simulations miss audio-level failures.