AI voice agents handle intake calls autonomously, but without systematic evaluation, law firms operate blind. An attorney whose AI agent fields 50 intake calls per week can't answer critical questions without listening to recordings: Did the agent collect required information from every caller? Which calls resulted in qualified leads versus time-wasters? Are callers having smooth experiences, or hanging up frustrated? When an outbound medical records series should auto-stop because records were received, how does the system know the call actually succeeded?

Manual call review doesn't scale. Listening to a 5-minute recording takes 5 minutes. Reviewing 50 calls weekly consumes 4+ hours of attorney or staff time that could be spent on billable work. Reading transcripts is faster but still time-intensive and requires subjective judgment about information collection adequacy. For lead qualification, attorneys need to know practice area, case type, injury severity, liability clarity, and viability, and extracting this from unstructured conversation transcripts means manual analysis for every inquiry.

We built Casegen's post-call processing system to eliminate this manual work entirely: two automated analysis pipelines that provide instant, objective assessment of every conversation while extracting structured case data for immediate lead qualification.

The Problem: Operating AI Agents Without Visibility

Law firms using AI voice agents face a critical gap between automation and understanding what's actually happening in conversations.

No systematic quality measurement. Without call-by-call evaluation, attorneys don't know whether their AI agents are performing well or poorly. Maybe the agent skips critical questions. Maybe callers become frustrated and hang up. Maybe the agent successfully completes calls but misses subtle case qualification signals. This blindness prevents optimization: you can't improve what you don't measure. The only way to know what's happening is manual review, which means either spending hours listening to recordings or operating on faith that the AI works correctly.

Lead qualification requires manual extraction. When a potential client calls describing their legal issue, attorneys need structured data: What practice area? What case type? How severe are injuries? Is liability clear? What's the viability score? This information exists in the conversation transcript, but extracting it manually means reading every call and making subjective categorization decisions. For a solo practitioner with 20 new leads weekly, this burns 2-3 hours that could be spent on case work. For firms with higher volume, manual extraction becomes impossible.

Outbound series need success detection. Recurring outbound calls, like bi-weekly medical records requests, should stop automatically when their goal is achieved. But determining "success" from raw call data requires understanding: Did the call complete? Was the duration sufficient to accomplish the objective? Did the conversation quality indicate goal achievement? Without automated evaluation, these series either run indefinitely (wasting attempts and annoying recipients) or require manual review to trigger cancellation.

Pattern identification stays invisible. Which calls consistently fail? Which agent configurations produce better outcomes? Do certain call types underperform? These patterns only emerge through systematic tracking and analysis. Individual call review misses the forest for the trees. Attorneys need aggregate metrics and trend analysis to identify systemic issues and optimization opportunities.

The result: law firms either waste hours on manual review or operate AI systems without understanding their performance, quality, or lead qualification.

Building Automated Post-Call Intelligence

We built two complementary automated analysis systems that together provide complete post-call intelligence without manual review.

Universal Call Quality Evaluation (All Calls)

Every conversation, whether inbound intake, outbound client check-in, medical provider coordination, or spam call, receives automated quality assessment through LLM analysis of the transcript and conversation metadata within seconds of call completion.

Three-dimensional scoring. The system analyzes conversations against the AI agent's instructions and scores performance across three dimensions:

• Script Following (0-10): Did the AI agent follow the prescribed conversation flow correctly? Evaluates question sequencing, adherence to critical rules (one question at a time, required empathy statements, fee disclosure verbatim), appropriate responses to user answers, and context awareness. The score reflects execution quality up to whatever point the call reached: a call that ends early due to interruption can still score 9/10 if the agent followed the script perfectly for what was covered.

• Caller Experience (0-10): How smooth and frustration-free did the call feel from the caller's perspective? Evaluates clarity of agent communication, appropriate pacing, respectful tone, minimal repetitions or interruptions, and effective recovery when misunderstandings occur. A score of 8-9 indicates a very good experience with only minor friction, while 0-3 signals the call was unusable or the caller became hostile.

• Call Success (Binary: 1 or 0): Did the call complete successfully with its intended purpose, or was it disrupted, dropped, or ended prematurely? This is not a 0-10 scale; it's a binary determination. A score of 1 means the call achieved its objective, while 0 indicates interruption before completion. Successful transfers count as success (1) if the agent correctly identified when to transfer per its instructions.

Intelligent edge case handling. The LLM evaluation model accounts for nuances that simple rule-based systems miss. If a caller volunteers information without being asked, the agent gets credit. If the call ends due to technical issues but the agent was performing well, Script Following and Caller Experience can still score high while Call Success correctly registers as failure. Transfer handling is sophisticated: if a transfer occurred and the caller had a poor experience beforehand (frustrated, confused, asked "are you a robot?"), Call Success = 0 despite the transfer; if the experience was good, Call Success = 1.

Real-time alerting. Scores trigger instant Slack notifications with visual indicators (green for ≥8, yellow for 5-7, red for <5), overall score, duration, breakdown by category, and links to full evaluation traces in Langfuse for detailed analysis. This enables immediate response when calls fail and pattern identification when performance degrades.

Outbound success determination. For outbound call series with auto-stop enabled, the evaluation determines whether the call met success criteria (call completed, duration ≥30 seconds, and for medical calls average score ≥6 or for intake calls information collected) to trigger series cancellation. This automation enables recurring series to stop precisely when their goal is achieved.

Complete observability. All scores and conversation metadata flow to Langfuse for long-term tracking, trend analysis, and debugging. Attorneys can identify patterns like "agent performance degraded after prompt change" or "calls shorter than 2 minutes consistently score poorly."

Unified Intake Processing (New Case & Existing Case Calls Only)

Intake calls require extracting structured case data that attorneys use for lead qualification and case triage. This processing happens exclusively for calls classified as NEW_CASE or EXISTING_CASE, executing a single comprehensive LLM analysis that delivers 67-75% cost reduction compared to the previous multi-call approach.

Complete case classification. The system analyzes conversation transcripts along with call metadata and the law firm's case classification schema to extract everything needed in one structured output:

• Practice area and case type: Legal category (Personal Injury, Family Law, Criminal, Estate Planning, Workers Compensation) and specific case type (Car Accident, Medical Malpractice, Divorce, DWI/DUI, Slip and Fall) based on facts described, not caller claims. Classification rules ensure accuracy: DUI/DWI cases route to the DWI practice area, traffic violations become Speeding Ticket, probate versus estate planning distinction hinges on whether the person is deceased.

• Call reason detection: Why the caller contacted the firm: new case inquiry, existing case status update, scheduling, document submission, payment inquiry, complaint, callback request, or general question.

• Comprehensive case summarization: Detailed incident information structured by case type: for auto accidents (what happened, location, witnesses, police report, vehicle details, airbag deployment), for dog bites (owner info, insurance, number of bites), for premises liability (location, hazard cause, warning signs). Includes featured items without category labels (injuries, defendant types, jurisdictions) and brief case summary for quick review.

• Injuries and treatments extraction: Medical detail including all mentioned injuries with specifics ("Concussion with ongoing headaches", "Fractured right wrist - dominant hand"), treatment history (ambulance transport, ER visit, diagnosis, medications, procedures), and pain scores by body part when mentioned.

• Case metadata: Date of loss (returned as structured object {year, month, day} handling relative dates like "yesterday" or "two months ago"), incident location, defendant type (Commercial/Private Party/Governmental), insurance limit if mentioned, and case quality score (1-5 rating based on injury severity, liability clarity, damages, evidence availability, statute of limitations compliance).

67-75% cost reduction. Previously, data collection required 3-4 separate LLM requests: one for classification, one for call reason detection, one for summarization, one for metadata extraction. Unifying these into one comprehensive prompt with structured output reduced cost by 67-75% while improving consistency (all extracted data comes from the same LLM analysis) and speed (parallel processing time eliminated).

Immediate lead qualification. Extracted data flows directly into case management tables. Attorneys opening a lead see instantly: "Personal Injury - Car Accident, Score: 4/5, Date of Loss: 2024-10-15, Commercial defendant, Injuries: Whiplash with ongoing neck pain, Broken right collarbone" without reading transcripts or listening to recordings. This enables immediate qualification decisions and intelligent routing (high-score PI cases to senior attorneys, low-score family law to paralegals).

Technical Implementation

The system architecture combines LLM-powered evaluation with structured data extraction, delivering comprehensive post-call intelligence at scale.

GPT-5 for universal quality evaluation. Every call triggers evaluation analysis using GPT-5, which reads the full transcript and conversation metadata to produce three-dimensional scoring. The model accounts for edge cases: voluntary information provision, technical interruptions, transfer scenarios, transcription errors. Prompts include the AI agent's specific instructions as context, enabling accurate assessment of whether the agent followed its prescribed conversation flow.

o3-mini with reasoning for intake processing. NEW_CASE and EXISTING_CASE calls receive structured data extraction using o3-mini with reasoning enabled. This model excels at complex extraction tasks where classification rules must be applied correctly (DUI → DWI practice area, deceased person → Probate) based on conversation facts. Structured output schemas with validation ensure consistent data format.

Real-time processing pipeline. Webhook handlers trigger immediately on call completion, launching parallel evaluation and post-processing pipelines. Quality evaluation runs for all calls; intake processing runs conditionally for case-related calls only. Both complete within seconds, enabling instant notifications and immediate data availability for attorney review.

Langfuse integration for observability. Full evaluation chain logging captures LLM inputs/outputs, scores with timestamps, conversation metadata (company, case ID, transfer status). This creates long-term trend analysis capability, debugging support when evaluations seem incorrect, and model performance improvement feedback loops.

Slack webhook notifications. Instant alerts deliver quality scores with visual indicators to designated channels. Red alerts (score <5) surface immediately for investigation. Green confirmations (score ≥8) provide confidence that calls succeeded. Yellow warnings (score 5-7) identify calls that warrant review but didn't fail critically.

Structured date parsing. Date of loss extraction returns {year, month, day} objects directly from LLM, handling relative expressions ("yesterday", "two months ago"), ambiguous formats ("back in March 2023"), and transcription errors ("July 20 eighth" → July 28th) without string parsing. This eliminates post-processing complexity and error-prone date interpretation logic.

Business Value

The core value centers on eliminating manual work that prevents law firms from systematically reviewing AI agent performance and qualifying leads at scale.

Systematic quality visibility. AI agents handling 50+ calls weekly generate massive transcript volume impossible to review manually. Automated evaluation provides instant visibility without listening to recordings. Attorneys see which calls succeeded (green scores), which had issues (yellow/red alerts), and drill into problems through Langfuse traces. This enables optimization: "script scores dropped after prompt update, reverting changes" or "criminal calls score low on information collection, needs revision." For a firm processing 700 calls monthly, this eliminates 8-10 hours of manual review, equivalent to $400-500 in staff time.

Outbound series intelligence. Recurring series for medical records or client check-ins need to know when to stop once the goal is achieved and records are received. Manual tracking doesn't work at scale. Automated success evaluation enables auto-stop logic: when a medical records call meets criteria (completed, ≥30s, average score ≥6), the series cancels future occurrences and notifies staff. This prevents unnecessary calls after objectives are met while ensuring persistence until success.

Instant lead qualification. Manual extraction means reading transcripts for every inquiry: a solo practitioner with 20 new leads weekly spends 2-3 hours on triage. Automated intake processing provides structured case data instantly: attorneys see "Personal Injury - Car Accident, Score: 4/5, Commercial defendant, Date of Loss: Oct 15 2024, Injuries: Whiplash, Broken collarbone" without reading anything. This enables intelligent routing and immediate decline decisions on unviable cases.

Cost-optimized post-processing. Unifying intake extraction into one comprehensive analysis reduced post-processing costs by 67-75% while improving consistency and speed. At 700 intake calls monthly, this saves ~$80 annually while delivering better data quality.

Complete performance audit trail. Every call generates detailed tracking: scores, metadata, timestamps, LLM traces. This creates an optimization record: "evaluated 700 calls this month with 94% success rate and 8.2/10 average caller experience" demonstrates system reliability, while historical trends identify when performance degrades.

Results

The Casegen post-call processing systems successfully eliminate manual call review and case data entry for law firms operating AI voice agents at scale. Processing ~700 calls monthly, the platform delivers instant quality assessment across all conversations and comprehensive intake data extraction for new cases, all automated within seconds of call completion.

Development delivered three-dimensional LLM quality scoring with intelligent edge case handling, real-time alerting with visual indicators and Langfuse trace links, outbound success criteria evaluation enabling series auto-stop logic, unified intake processing with 67-75% cost reduction, structured date parsing without post-processing, rule-based classification with LLM reasoning, case quality scoring for lead qualification, and comprehensive Langfuse observability.

The combination of universal quality evaluation and unified intake processing provides complete automation of tasks that previously required hours of attorney or staff time weekly. Law firms gain systematic quality visibility, instant lead qualification, and data-driven optimization capability, enabling them to compete with large practices on lead response speed and qualification efficiency through automated post-call intelligence that scales indefinitely.