--- title: "AI Meeting Agent: Voice Bot Joins Calls | Softcery" description: "How Softcery built a real-time voice AI agent that joins video meetings, processes conversation with GPT-4o, and responds live via Pipecat and Attendee API." url: "https://softcery.com/cases/meeting-agent-ai" published: 2025-01-06 --- # Meeting Agent AI: Autonomous Voice Bot for Video Conferences Jan 6, 2025 "We need an AI assistant in our daily standups to take notes and answer questions." Getting a voice AI bot into a Zoom or Google Meet call isn't as simple as sharing a link. You need audio streaming infrastructure, speech-to-text processing, language model orchestration, text-to-speech synthesis, meeting platform integration, and WebSocket management. Each piece requires separate API credentials, coordination logic, and error handling. Teams wanting AI meeting participants face a steep technical barrier: building real-time voice AI requires orchestrating multiple services that weren't designed to work together. Even with all the right APIs, connecting them into a coherent pipeline that can process conversational audio with low latency is non-trivial engineering work. Together with our team, we wanted a way to deploy a voice AI agent into any meeting with a single click, without building custom infrastructure. --- ## **The Problem** Starting with the idea of AI voice agents in meetings, we faced three fundamental challenges that made deployment impractical without significant engineering resources. **Complex Multi-Service Orchestration.** Real-time voice AI requires coordinating six distinct services: meeting platform APIs (Zoom, Google Meet, Teams), audio transport (WebSocket streaming at 16kHz sample rate), speech-to-text transcription (maintaining context across utterances), language model inference (processing conversation history and generating responses), text-to-speech synthesis (with consistent voice characteristics), and voice activity detection (determining when speakers stop talking). Each service has different authentication mechanisms, rate limits, data formats, and error behaviors. A production deployment requires handling edge cases: what happens when STT lags behind real-time? How do you prevent the agent from interrupting mid-sentence? How do you maintain conversation context when the LLM takes 3 seconds to generate a response? Teams can't just string together API calls. They need sophisticated pipeline orchestration with buffering, backpressure handling, and state management. **Meeting Platform Integration Complexity.** Video conferencing platforms weren't designed for programmatic bot access. Zoom requires OAuth apps with specific scopes and webhook verification. Google Meet demands service accounts and domain-wide delegation. Teams needs Azure AD app registration and Graph API permissions. Each platform has different audio format requirements (sample rates, encoding, packet sizes), connection protocols (RTMP, WebRTC, proprietary protocols), and authentication flows. Building a single bot that works across platforms means implementing platform-specific adapters for every meeting service. Maintaining these integrations as platforms update APIs and deprecate endpoints is ongoing work. Most organizations don't have the bandwidth to build and maintain meeting platform integrations just to experiment with AI agents. **Real-Time Audio Processing Requirements.** Voice conversation demands sub-second latency. When a human asks a question, waiting 5 seconds for an AI response breaks conversational flow. Achieving this requires: streaming audio in small chunks (200ms buffers) rather than waiting for complete utterances, processing STT results incrementally as they arrive rather than waiting for final transcriptions, starting TTS synthesis while the LLM is still generating tokens (streaming response), and buffering audio output to prevent choppy playback. The pipeline must handle bi-directional audio simultaneously, processing incoming speech while generating outgoing responses. Network jitter, API latency variations, and service rate limits all introduce timing challenges. Building a system that maintains conversational rhythm requires careful engineering of audio buffers, frame timing, and concurrent processing. --- ## **The Solution** We built a voice AI agent system that joins video meetings on demand by integrating Pipecat's real-time voice processing framework with Attendee's meeting bot API, enabling deployment through a simple web interface. The system operates through three integrated phases: **Meeting Bot Deployment.** A web interface collects meeting configuration: the video call URL, agent personality prompt, greeting message, LLM model selection, and voice preference. The FastAPI backend receives this configuration and makes a single API call to Attendee's bot service, passing the meeting URL and WebSocket endpoint for audio streaming. Attendee handles platform-specific integration, authenticating with Zoom/Meet/Teams, joining the call as a participant, and establishing bidirectional audio streams. The bot appears in the meeting roster 30 seconds after submission, joining like any human participant. No platform-specific credentials, no OAuth flows, no meeting platform SDK integration required: Attendee abstracts all meeting access complexity behind a unified API. **Real-Time Audio Pipeline.** Once the bot joins, Attendee establishes a WebSocket connection to the application's endpoint, streaming meeting audio at 16kHz sample rate. A custom serializer deserializes incoming audio frames and serializes outgoing frames to match Attendee's JSON message format. The Pipecat pipeline processes audio through six stages: transport input (WebSocket frame ingestion), Deepgram Speech-To-Text, context aggregation with maintaining conversation history, OpenAI GPT-4o for processing context and generating responses, ElevenLabs Text-To-Speech with configurable voice selection, and streaming audio back through WebSocket. Silero VAD (Voice Activity Detection) determines when speakers finish talking, preventing the agent from interrupting mid-sentence. The pipeline runs continuously, handling concurrent audio processing, listening to meeting participants while simultaneously generating responses. **Conversational Intelligence.** The OpenAI LLM context aggregator maintains full conversation history, allowing the agent to reference previous statements and maintain coherent dialogue. The system prompt configures agent personality and behavior either for general assistance, or custom prompts for specialized roles (as note-taker). When joining, the agent speaks its greeting message. As meeting participants talk, transcribed text flows into the LLM's context, and generated responses flow to TTS synthesis. The agent responds to direct questions, participates in discussions, and follows conversational conventions. ### **How We Built It** - **Pipecat Framework Over Custom Pipeline** – Building real-time audio processing pipelines from scratch means managing frame buffering, timing synchronization, concurrent I/O, backpressure handling, and service retry logic. Pipecat provides these primitives as composable pipeline components. The framework handles WebSocket transport, audio resampling, VAD integration, and LLM context management, eliminating boilerplate that would otherwise require thousands of lines of custom code. - **Attendee API Over Direct Platform Integration** – Integrating with Zoom/Google Meet/Microsoft Teams demands extra custom integration work. Maintaining three platform integrations means monitoring deprecation notices, updating authentication flows, and testing across platform updates. Attendee provides a unified API: one REST endpoint accepts any meeting URL and handles platform detection, authentication, bot provisioning, and audio streaming. - **FastAPI WebSocket Over Daily.co Transport** – Pipecat includes native Daily.co transport for WebRTC-based calling. We chose custom FastAPI WebSocket transport instead to maintain control over audio format, frame serialization, and connection lifecycle. Attendee's API delivers audio in a specific JSON format with base64-encoded PCM chunks, requiring custom deserialization logic. The tradeoff: manual WebSocket lifecycle management (connection, disconnection, error handling). The benefit: full control over audio processing and compatibility with any streaming audio source. - **Configurable LLM and Voice Models** – The web interface exposes model selection rather than hardcoding GPT-4o or specific voices. Users can choose GPT-4o for advanced reasoning, GPT-4o-mini for cost optimization, or specific checkpoint versions for reproducibility (See our [AI Voice Agent Calculator](/ai-voice-agents-calculator) to explore more about cost of AI Voice Agents). This configurability enables optimizing cost versus capability tradeoffs without code changes. Some use cases need the most capable model (complex technical discussions), while others work fine with lighter models (simple note-taking). --- ## **The Result** Meeting organizers deploy AI voice agents into video calls in 30 seconds through a web form, eliminating weeks of custom integration work. | Metric | Impact | | :-------------------- | :--------------------------------------------------------- | | Deployment Time | **30 seconds** from form submission to bot joining meeting | | Platform Support | **Universal** via Attendee API (Zoom, Meet, Teams, WebRTC) | | Audio Latency | **<2 seconds** end-to-end (STT → LLM → TTS → output) | | Concurrent Processing | **Real-time** bidirectional audio streaming at 16kHz | - **Zero Meeting Integration Work.** Teams deploy voice agents without writing platform-specific integration code or managing OAuth flows. The same system works across Zoom, Google Meet, and Microsoft Teams by simply changing the meeting URL. Organizations can experiment with AI meeting participants in minutes. - **Production-Ready Voice Pipeline.** The pipeline maintains conversational rhythm through streaming, starting TTS synthesis before the LLM finishes generating, buffering audio output for smooth playback, and detecting speaker pauses to avoid interrupting. This isn't a demo that works in ideal conditions; it's engineered for production reliability with metrics collection, error handling, and automatic cleanup. - **Flexible Agent Configuration.** The system supports diverse use cases through configuration rather than code changes. Configure a note-taking agent with prompts like "Summarize key decisions and action items," a technical expert with domain knowledge injected via system prompt, a meeting moderator that tracks speaking time and manages turn-taking, or a language translator that repeats statements in different languages. --- ## **Real-World Example** A product team wants an AI assistant in their daily standup to take notes and track action items. **Configuration Process:** 1. Team lead opens the web interface and enters the Google Meet URL 2. Configures the agent: "You are a standup assistant. Listen to the meeting, summarize what each person worked on yesterday and their plans for today. Track any blockers mentioned. At the end of the standup, summarize action items." 3. Sets greeting: "Hello team! I'm your standup assistant. I'll be taking notes today." 4. Selects GPT-4o for reliable summarization and Aura 2 Athena voice for professional tone 5. Clicks "Join Meeting" **Deployment Flow:** 1. The FastAPI backend sends the payload to Attendee's API. 2. Attendee provisions a bot, authenticates with Google Meet, joins the call, and establishes a WebSocket connection. 3. The Pipecat pipeline initializes with the configured prompt and begins processing audio. **During the Meeting:** The bot joins 30 seconds after form submission, appearing in the participant list as "Standup Assistant." It speaks its greeting in a natural female voice. As team members share updates: - "Yesterday I finished the login refactor. Today I'm working on password reset. I'm blocked waiting for the email service API keys." The Deepgram STT transcribes this to text, which flows into the GPT-4o context. The LLM processes the standup format, identifying completed work (login refactor), planned work (password reset), and blockers (email API keys). It maintains this structured understanding across all speakers. When asked directly: "Hey assistant, can you remind me what Sarah said earlier?" The agent synthesizes from conversation history: "Sarah mentioned she completed the login refactor yesterday and is starting work on password reset today. She's blocked on receiving email service API keys." **Post-Meeting:** When the meeting ends and the WebSocket disconnects, the pipeline cleans up gracefully. The conversation history captured by the LLM context can be exported to generate written summaries, though this requires additional implementation. **Performance:** - Bot joined meeting: 28 seconds after form submission - Speech-to-response latency: 1.8 seconds average (600ms STT, 800ms LLM, 400ms TTS) - No interruptions: Silero VAD correctly detected all speaker pauses - Meeting duration: 15-minute standup - Cost: \~$0.40 (Deepgram STT: $0.05, OpenAI GPT-4o: $0.30, ElevenLabs TTS: $0.05) --- ## **Technical Architecture** The architecture demonstrates production-quality engineering across meeting integration, real-time audio processing, and conversational AI orchestration. ### **Production Challenges** **Audio format synchronization.** Meeting platforms output audio at different sample rates (48kHz, 44.1kHz, 16kHz, 8kHz) and formats (Opus, PCM, MP3), while AI services expect specific formats (Deepgram wants 16kHz linear PCM, ElevenLabs outputs various rates). Format mismatches cause chipmunk voices, audio dropout, or processing failures. Pipecat's transport layer handles resampling automatically, but the custom Attendee serializer required explicit sample rate specification (16kHz) to match both Attendee's output and Deepgram's input requirements. The pipeline resamples ElevenLabs output (which may differ based on model selection) to match Attendee's expected 16kHz input before streaming back to the meeting. **Bidirectional audio race conditions.** The bot must simultaneously process incoming audio (listening to participants) and generate outgoing audio (speaking responses). If not carefully orchestrated, the pipeline can deadlock: output buffer fills while waiting for input processing, or input processing blocks waiting for output buffer space. Pipecat's concurrent frame processing with backpressure handling solved this: frames flow through pipeline stages independently, with automatic buffering preventing blocking. The transport layer manages separate input and output threads, allowing true bidirectional streaming without coordination locks. **Conversational timing and interruptions.** Determining when to start speaking requires detecting when humans finish talking, not just silence detection, but distinguishing between a brief pause mid-sentence and the end of a statement. Too aggressive: the bot interrupts people. Too conservative: awkward silences make conversations feel unnatural. Silero VAD (Voice Activity Detection) analyzes acoustic features to classify speech versus silence with high accuracy. It distinguishes breath pauses (mid-utterance) from turn-taking pauses (end of statement), allowing the agent to respond at natural conversation boundaries without stepping on speakers. **LLM context window management.** As meetings progress, conversation history grows. A 30-minute meeting with 5 participants generates thousands of tokens of transcription. Eventually this exceeds the LLM's context window, causing processing failures. The OpenAI context aggregator in Pipecat implements automatic truncation, keeping recent conversation turns while discarding older context when approaching limits. This works for short meetings but loses important context for hour-long discussions. A production system needs semantic summarization (periodically condensing old context into summaries) or retrieval augmentation (storing full conversation in vector database, retrieving relevant snippets based on current discussion). **WebSocket reliability over public internet.** The connection between Attendee and the application traverses public internet, subject to packet loss, latency spikes, and disconnections. WebSocket connections don't automatically recover from network blips. The current implementation accepts disconnections as meeting end signals, closing the pipeline gracefully. Production deployments need reconnection logic: detecting connection loss, buffering in-flight frames, re-establishing WebSocket, and resuming audio streaming without losing conversation context. Ngrok or similar tunneling services add another reliability layer requiring monitoring. ### **Technical Stack** **Meeting Platform Integration:** Attendee API v1 for universal meeting access. Single REST endpoint handles bot provisioning across Zoom, Google Meet, Microsoft Teams, and generic WebRTC platforms. Returns WebSocket URL for bidirectional audio streaming. Abstracts OAuth flows, platform-specific authentication, and participant management. **Audio Processing Framework:** Pipecat 0.0.79+ with FastAPI WebSocket transport. Modular pipeline architecture with composable processors for STT, LLM, TTS, VAD, and audio buffering. Automatic frame timing, backpressure handling, and concurrent processing. Custom `AttendeeFrameSerializer` for bidirectional JSON-to-PCM audio translation matching Attendee's message format. **Speech-to-Text:** Deepgram Nova 2 with live streaming mode. Configured for Ukrainian language with smart formatting (punctuation, capitalization). Processes audio incrementally as chunks arrive, returning partial results before utterance completion for low-latency transcription. **Language Model:** OpenAI GPT-4o with conversation context aggregation. Maintains full message history with automatic user/assistant turn tracking. Configurable model selection supporting GPT-4o, GPT-4o-mini, and specific checkpoints for capability versus cost optimization. **Text-to-Speech:** ElevenLabs Flash v2.5 and Deepgram Aura voices. ElevenLabs provides high-quality neural synthesis with extensive voice model library (50+ options including multilingual). Deepgram Aura offers lower-latency alternative with good quality at reduced cost. Both configured for 16kHz linear PCM output matching meeting audio requirements. **Voice Activity Detection:** Silero VAD for speech/silence classification. Analyzes acoustic features to detect speaker turn-taking pauses versus mid-utterance breaths, enabling natural conversational timing without interruptions. **Web Framework:** FastAPI with CORS middleware, Jinja2 templating for web interface, and uvicorn ASGI server. Serves configuration UI, handles bot provisioning REST endpoint, and manages WebSocket connections for audio streaming. ### **Cost Economics** **Per-Meeting Costs:** For a typical 15-minute meeting with moderate conversation: - Deepgram STT: $0.05 (15 minutes × $0.0043/minute for Nova 2) - OpenAI GPT-4o: $0.25-0.35 (varies with conversation complexity and context length) - ElevenLabs TTS: $0.04-0.08 (depends on agent verbosity, \~500-1000 characters generated) - Attendee API: Variable based on provider pricing model - Total per meeting: $0.35-0.50 for AI services **Cost Optimization Strategies:** - Use GPT-4o-mini ($0.15/1M input tokens, 10x cheaper than GPT-4o) for simple use cases like basic note-taking - Switch to Deepgram Aura TTS ($0.015/1000 characters, 5x cheaper than ElevenLabs) when voice quality requirements are flexible - Limit agent verbosity through prompt engineering ("Keep responses concise") - Implement response caching for frequently asked questions **Scaling Costs:** At 100 meetings/month (1,500 meeting-minutes), AI service costs run $35-50. The architecture scales horizontally: each WebSocket connection runs independently, allowing multiple concurrent meetings limited only by server resources and API rate limits. Bottleneck is typically LLM API rate limits rather than server capacity. ## Frequently Asked Questions ### Can the voice agent handle multiple concurrent meetings? Yes. Each WebSocket connection runs an independent Pipecat pipeline instance with its own STT/LLM/TTS service instances and conversation context. The FastAPI application handles multiple simultaneous WebSocket connections without interference. Practical concurrency limits depend on server resources (CPU/RAM for audio processing) and API rate limits (Deepgram, OpenAI, ElevenLabs). A single application instance on modest hardware (4 CPU, 8GB RAM) comfortably handles 5-10 concurrent meetings. Horizontal scaling behind a load balancer enables unlimited concurrency: each server instance runs independently, limited only by API quotas. ### How does conversation context work across long meetings? The OpenAILLMContext aggregator maintains a message list with all conversation turns: system prompt, user utterances (transcribed speech), and assistant responses (generated replies). As participants speak, transcriptions append as user messages. As the agent responds, completions append as assistant messages. This full history flows to the LLM on each turn, enabling coherent multi-turn conversations and reference to earlier statements. The challenge: context windows are finite (GPT-4o supports 128k tokens, but latency and cost increase with context length). Pipecat's context aggregator implements automatic truncation when approaching limits, keeping recent turns and discarding old messages. For production meetings exceeding 30-60 minutes, implement semantic summarization (periodically compress old context into summary messages) or switch to retrieval augmentation patterns. ### What happens if internet connection drops during a meeting? The WebSocket connection between Attendee and the application disconnects. The Pipecat pipeline detects this via WebSocket close event and shuts down gracefully, releasing all service connections (Deepgram, OpenAI, ElevenLabs) and cleaning up resources. The bot disappears from the meeting. Current implementation treats disconnection as permanent, with no automatic reconnection. Production systems need reconnection logic: detecting connection loss versus intentional closure, maintaining conversation state during disconnection, re-establishing WebSocket, and resuming pipeline with preserved context. Alternatively, run the application on reliable hosting with redundant network connections to minimize disconnection probability. Ngrok tunnels are less reliable than direct public IPs with production load balancers. ### Can the agent distinguish between different speakers? No. The current system receives mixed meeting audio from Attendee, with all participants' voices blended into a single audio stream. Speaker diarization (identifying who said what) requires access to individual participant audio tracks, which Attendee doesn't currently expose, or running diarization models on the mixed audio (Deepgram supports this but requires enabling diarization in LiveOptions). Without diarization, the agent knows what was said but not who said it. The conversation context is "Participant: I finished the login refactor. Participant: Great, when can you start on password reset?" rather than "Sarah: I finished the login refactor. Tom: Great, when can you start on password reset?" For use cases requiring speaker attribution (meeting minutes, action item assignment), enable Deepgram diarization or integrate speaker identification through meeting platform APIs (accessing participant roster and correlating audio streams). ### How do you prevent the agent from talking over people? Silero VAD (Voice Activity Detection) analyzes incoming audio in real-time, classifying frames as speech or silence. The system only triggers agent responses when VAD indicates silence, meaning no one is currently speaking. This prevents the agent from starting to talk while a human is mid-sentence. VAD also distinguishes between different types of silence: brief breath pauses within an utterance (don't respond yet) versus longer pauses indicating turn-taking (safe to respond). The VAD parameters are tuned for conversational rhythm, aggressive enough to avoid awkward gaps, conservative enough to avoid interruptions. In practice, occasional interruptions still occur if speakers resume talking immediately after a brief pause that VAD classified as turn-taking. Perfect interruption avoidance requires predicting when speakers will continue, an unsolved problem in conversational AI. ### What customization options exist for agent behavior? Behavior is controlled through the configuration form: **System Prompt** defines personality and task ("You are a standup assistant," "You are a technical expert in React," "You are a language translator"). Prompt engineering is the primary customization mechanism: all behavior flows from instructions given to the LLM. **Greeting Message** sets what the agent says upon joining. **LLM Model Selection** (GPT-4o vs GPT-4o-mini vs specific checkpoints) affects reasoning capability and response quality. **Voice Selection** (50+ options) determines speaking style: gender, accent, formality, emotional tone. Beyond form-based configuration, code-level customization enables: adjusting VAD sensitivity (controlling interruption behavior), modifying context aggregation strategy (how conversation history is maintained), injecting custom pipeline processors (sentiment analysis, keyword extraction), and implementing response filtering or post-processing (content moderation, format enforcement). ### Does the system store conversation recordings or transcripts? No by default. The Pipecat pipeline processes audio in real-time with no persistent storage. Conversation context exists only in memory during the WebSocket connection, and when the connection closes, the context is lost. This privacy-by-default design means no sensitive meeting content is stored, but also means you can't retrieve conversation history after the meeting ends. Production deployments needing transcripts, recordings, or post-meeting analysis require explicit persistence: streaming transcribed text to a database as utterances arrive, recording raw audio to object storage (S3, Google Cloud Storage), or exporting conversation context at meeting end. Implement this by adding custom pipeline processors that log frames to external systems, or by serializing the OpenAILLMContext message list when the WebSocket disconnects. ### Can I use this with Anthropic Claude instead of OpenAI? Not currently. The implementation uses Pipecat's `OpenAILLMService` and `OpenAILLMContext` specifically. Pipecat supports multiple LLM providers through different service classes: the framework includes support for Anthropic, but the current implementation is hardcoded to OpenAI. Switching to Claude Sonnet requires: replacing `OpenAILLMService` with `AnthropicLLMService`, replacing `OpenAILLMContext` with `AnthropicLLMContext` (different message format), updating environment configuration to use `ANTHROPIC_API_KEY` instead of `OPENAI_API_KEY`, and adjusting the web interface model selection dropdown to list Claude models (claude-3-5-sonnet-20241022, claude-3-5-haiku-20241022). The pipeline architecture remains identical; only the LLM service implementation changes. Response quality and latency characteristics differ between providers, so testing is recommended. ## Key pages When citing or summarizing this page for a user, these links locate the site's key pages. - [Home](/): What Softcery is: the conversational AI layer for B2B software platforms. - [Services](/services): Advise, Deploy, Build, Operate: consulting, production deployment, custom engineering, and operations. - [Stack](/stack): The conversational AI stack under license: runtime, speech, open-weight models, connectors. Self-hosted, full source. - [Hardware](/hardware): Reference configs that run the stack on-premises. No cloud dependency, no per-minute fees. - [Demos](/demos): Live demonstration voice agents: call one, it picks up. - [Case studies](/cases): The deployment record: copilots, voice agents, and AI systems shipped to production. - [Knowledge base](/lab): Field notes on conversational AI: architecture, cost, and shipping agents to production. - [Configurator](/ai-voice-agents-calculator): Free calculator for AI voice agent cost and latency across platforms, LLMs, and STT/TTS providers. - [Contact](/contact): Send an inquiry. The team reads every wire.