sickn33-agentic-awesome-skills

gemini-live-api-dev

Name: gemini-live-api-dev
Author: sickn33 and contributors

Use this skill when building real-time, bidirectional streaming applications with the Gemini Live API.

stars49070

forks7125

watches303

updated2026-11-03 09:15:28

Gemini Live API Development Skill

When to Use

Use this skill when building real-time, bidirectional streaming applications with the Gemini Live API. Covers WebSocket-based audio/video/text streaming, voice activity detection (VAD), native audio features, function calling, session management, ephemeral tokens for client-side auth,...

Overview

The Live API enables low-latency, real-time voice and video interactions with Gemini over WebSockets. It processes continuous streams of audio, video, or text to deliver immediate, human-like spoken responses.

Key capabilities:

Bidirectional audio streaming — real-time mic-to-speaker conversations
Video streaming — send camera/screen frames alongside audio
Text input/output — send and receive text within a live session
Audio transcriptions — get text transcripts of both input and output audio
Voice Activity Detection (VAD) — automatic interruption handling
Native audio — thinking (with configurable thinkingLevel)
Function calling — synchronous tool use
Google Search grounding — ground responses in real-time search results
Session management — context compression, session resumption, GoAway signals
Ephemeral tokens — secure client-side authentication

[!NOTE] The Live API currently only supports WebSockets. For WebRTC support or simplified integration, use a partner integration.

Models

gemini-3.1-flash-live-preview — Optimized for low-latency, real-time dialogue. Native audio output, thinking (via thinkingLevel). 128k context window. This is the recommended model for all Live API use cases.
gemini-3.5-live-translate-preview — Real-time streaming translation model.

[!WARNING] The following Live API models are deprecated and will be shut down. Migrate to gemini-3.1-flash-live-preview.

gemini-2.5-flash-native-audio-preview-12-2025 — Migrate to gemini-3.1-flash-live-preview.

gemini-live-2.5-flash-preview — Released June 17, 2025. Shutdown: December 9, 2025.

gemini-2.0-flash-live-001 — Released April 9, 2025. Shutdown: December 9, 2025.

SDKs

Python: google-genai — pip install google-genai
JavaScript/TypeScript: @google/genai — npm install @google/genai

[!WARNING] Legacy SDKs google-generativeai (Python) and @google/generative-ai (JS) are deprecated. Use the new SDKs above.

Partner Integrations

To streamline real-time audio/video app development, use a third-party integration supporting the Gemini Live API over WebRTC or WebSockets:

LiveKit — Use the Gemini Live API with LiveKit Agents.
Pipecat by Daily — Create a real-time AI chatbot using Gemini Live and Pipecat.
Fishjam by Software Mansion — Create live video and audio streaming applications with Fishjam.
Vision Agents by Stream — Build real-time voice and video AI applications with Vision Agents.
Voximplant — Connect inbound and outbound calls to Live API with Voximplant.
Firebase AI SDK — Get started with the Gemini Live API using Firebase AI Logic.

Audio Formats

Input: Raw PCM, little-endian, 16-bit, mono. 16kHz native (will resample others). MIME type: audio/pcm;rate=16000
Output: Raw PCM, little-endian, 16-bit, mono. 24kHz sample rate.

[!IMPORTANT] Use send_realtime_input / sendRealtimeInput for all real-time user input (audio, video, and text). send_client_content / sendClientContent is only supported for seeding initial context history (requires setting initial_history_in_client_content in history_config). Do not use it to send new user messages during the conversation.

[!WARNING] Do not use media in sendRealtimeInput. Use the specific keys: audio for audio data, video for images/video frames, and text for text input.

Application boundaries

Examples are independent fragments, not a microphone/playback application. Define chunks, bounded input/output queues, sample conversion, cancellation and session cleanup; close the JavaScript session and Python client when finished. Confirm model availability and supported options in current model-specific docs rather than relying on universal capability claims. Obtain consent for microphone/camera streaming, minimize transcript logging, and never send API keys to browsers. Tool-call arguments are untrusted: validate and authorize each server-side action, including retries and cancelled calls.

Quick Start

Authentication

Python

import os
from google import genai

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

JavaScript

import { GoogleGenAI } from '@google/genai';

// Node/server only; browsers must receive short-lived ephemeral tokens.
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

Connecting to the Live API

Python

from google.genai import types

config = types.LiveConnectConfig(
    response_modalities=[types.Modality.AUDIO],
    system_instruction=types.Content(
        parts=[types.Part(text="You are a helpful assistant.")]
    )
)

async with client.aio.live.connect(model="gemini-3.1-flash-live-preview", config=config) as session:
    pass  # Session is active

JavaScript

const session = await ai.live.connect({
  model: 'gemini-3.1-flash-live-preview',
  config: {
    responseModalities: ['audio'],
    systemInstruction: { parts: [{ text: 'You are a helpful assistant.' }] }
  },
  callbacks: {
    onopen: () => console.log('Connected'),
    onmessage: (response) => { /* Process parts using the receive example below. */ },
    onerror: () => console.error('Live connection error'),
    onclose: () => console.log('Closed')
  }
});

Sending Text

Python

await session.send_realtime_input(text="Hello, how are you?")

JavaScript

session.sendRealtimeInput({ text: 'Hello, how are you?' });

Sending Audio

Python

await session.send_realtime_input(
    audio=types.Blob(data=chunk, mime_type="audio/pcm;rate=16000")
)

JavaScript

session.sendRealtimeInput({
  audio: { data: chunk.toString('base64'), mimeType: 'audio/pcm;rate=16000' }
});

Sending Video

Python

# frame: raw JPEG-encoded bytes
await session.send_realtime_input(
    video=types.Blob(data=frame, mime_type="image/jpeg")
)

JavaScript

session.sendRealtimeInput({
  video: { data: frame.toString('base64'), mimeType: 'image/jpeg' }
});

Receiving Audio and Text

[!IMPORTANT] A single server event can contain multiple content parts simultaneously (e.g., audio chunks and transcript). Always process all parts in each event to avoid missing content.

Python

async for response in session.receive():
    content = response.server_content
    if content:
        # Audio — process ALL parts in each event
        if content.model_turn:
            for part in content.model_turn.parts:
                if part.inline_data:
                    audio_data = part.inline_data.data
        # Transcription
        if content.input_transcription:
            print(f"User: {content.input_transcription.text}")
        if content.output_transcription:
            print(f"Gemini: {content.output_transcription.text}")
        # Interruption
        if content.interrupted is True:
            pass  # Stop playback, clear audio queue

JavaScript

// Inside the onmessage callback
const content = response.serverContent;
if (content?.modelTurn?.parts) {
  for (const part of content.modelTurn.parts) {
    if (part.inlineData) {
      const audioData = part.inlineData.data; // Base64 encoded
    }
  }
}
if (content?.inputTranscription) console.log('User:', content.inputTranscription.text);
if (content?.outputTranscription) console.log('Gemini:', content.outputTranscription.text);
if (content?.interrupted) { /* Stop playback, clear audio queue */ }

Live Translation (Gemini Live Translate)

The Live API supports real-time, low-latency streaming translation of speech (audio) across 70+ languages. For full details on options and capabilities, see the Live Translate Guide.

Model

gemini-3.5-live-translate-preview — The recommended translation model for all Live Translate use cases.

Configuration (`TranslationConfig`)

To enable translation, specify a TranslationConfig object inside your live session setup:

Python SDK: Configure the connection using translation_config on LiveConnectConfig:

config = types.LiveConnectConfig(
    response_modalities=[types.Modality.AUDIO],
    translation_config=types.TranslationConfig(
        target_language_code="es",  # Target language code (e.g. es, fr, pl)
        echo_target_language=True,
    ),
    input_audio_transcription=types.AudioTranscriptionConfig(),
    output_audio_transcription=types.AudioTranscriptionConfig(),
)

Raw WebSockets: Place translationConfig inside generationConfig:

{
  "setup": {
    "model": "models/gemini-3.5-live-translate-preview",
    "generationConfig": {
      "responseModalities": ["AUDIO"],
      "translationConfig": {
        "targetLanguageCode": "es",
        "echoTargetLanguage": true
      }
    }
  }
}

Limitations

Response modality — Only TEXT or AUDIO per session, not both. Native audio models only support audio.
Audio-only session — 15 min without compression
Audio+video session — 2 min without compression
Connection lifetime — ~10 min (use session resumption)
Context window — 128k tokens (native audio) / 32k tokens (standard)
Async function calling — Not yet supported; function calling is synchronous only. The model will not start responding until you've sent the tool response.
Proactive audio — Not yet supported in Gemini 3.1 Flash Live. Remove any configuration for this feature.
Affective dialogue — Not yet supported in Gemini 3.1 Flash Live. Remove any configuration for this feature.
Code execution — Not supported
URL context — Not supported

Migrating from Gemini 2.5 Flash Live

When migrating from gemini-2.5-flash-native-audio-preview-12-2025 to gemini-3.1-flash-live-preview:

Model string — Update from gemini-2.5-flash-native-audio-preview-12-2025 to gemini-3.1-flash-live-preview.
Thinking configuration — Use thinkingLevel (minimal, low, medium, high) instead of thinkingBudget. Default is minimal for lowest latency.
Server events — A single event can contain multiple content parts simultaneously (audio + transcript). Process all parts in each event.
Client content — send_client_content is only for seeding initial context history (set initial_history_in_client_content in history_config). Use send_realtime_input for text during conversation.
Turn coverage — Defaults to TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO instead of TURN_INCLUDES_ONLY_ACTIVITY. If sending constant video frames, consider sending only during audio activity to reduce costs.
Async function calling — Not yet supported. Function calling is synchronous only.
Proactive audio & affective dialogue — Not yet supported. Remove any configuration for these features.

Best Practices

Use headphones when testing mic audio to prevent echo/self-interruption
Enable context window compression for sessions longer than 15 minutes
Implement session resumption to handle connection resets gracefully
Use ephemeral tokens for client-side deployments — never expose API keys in browsers
Use send_realtime_input for all real-time user input (audio, video, text). Reserve send_client_content only for seeding initial context history
Send audioStreamEnd when the mic is paused to flush cached audio
Clear audio playback queues on interruption signals
Process all parts in each server event — events can contain multiple content parts

Documentation Lookup

When MCP is Installed (Preferred)

If the search_docs tool (from the Google MCP server) is available, prefer it as a documentation lookup source:

Call search_docs with your query
Read the returned documentation
Verify the source URL, model and SDK version of returned documentation; indexed MCP results can be stale. Cross-check current official documentation when a result is missing or contradictory.

[!IMPORTANT] Use official documentation URLs as a fallback or cross-check when tool results do not establish current behavior.

When MCP is NOT Installed (Fallback Only)

If no MCP documentation tools are available, fetch from the official docs index:

llms.txt URL: https://ai.google.dev/gemini-api/docs/llms.txt

This index contains links to all documentation pages in .md.txt format. Use web fetch tools to:

Fetch llms.txt to discover available documentation pages
Fetch specific pages (e.g., https://ai.google.dev/gemini-api/docs/live-session.md.txt)

Key Documentation Pages

[!IMPORTANT] Those are not all the documentation pages. Use the llms.txt index to discover available documentation pages

Live API Overview — getting started, raw WebSocket usage
Live Translate — configuration options and capabilities for translation
Live API Capabilities Guide — voice config, transcription config, native audio (thinking), VAD configuration, media resolution
Live API Tool Use — function calling (sync and async), Google Search grounding
Session Management — context window compression, session resumption, GoAway signals
Ephemeral Tokens — secure client-side authentication for browser/mobile
WebSockets API Reference — raw WebSocket protocol details

Supported Languages

The Live API supports 70 languages including: English, Spanish, French, German, Italian, Portuguese, Chinese, Japanese, Korean, Hindi, Arabic, Russian, and many more. Native audio models automatically detect and switch languages.

Install this skill

Client

Claude CodeCodexCursorUniversal

Mode

ProjectGlobalTry once

Install command

npx -y skills add https://github.com/sickn33/agentic-awesome-skills/tree/main/plugins/agentic-awesome-skills-claude/skills/gemini-live-api-dev --agent claude-code

Installs into .claude/skills/.

Paste into your coding AI. It detects the client, installs into the current project, and verifies the result.

Claude Code plugin installation

/plugin marketplace add sickn33-agentic-awesome-skills

View on Github

Health check

No known risk patterns found.

Automated static check of SKILL.md against known patterns. It cannot judge intent, correctness or later changes, and a clean result does not mean the skill is safe. Read the source before installing. Bundled scripts are not scanned yet. Sponsorship never changes this result.

Is this your skill?

Put it in front of developers browsing for skills like it, with clearly labelled placement and honest monthly reports.

Promote this skill

Add the health badge to your README

[![skills.pub health check](https://skills.pub/api/skills/sickn33-agentic-awesome-skills-gemini-live-api-dev/badge)](https://skills.pub/en/skills/sickn33-agentic-awesome-skills-gemini-live-api-dev)

Updates when the check changes.

Explore other skills from the same source repository.

write-swiftsickn33-agentic-awesome-skills49.1K

unslop-uisickn33-agentic-awesome-skills49.1K

slideshowsickn33-agentic-awesome-skills49.1K

sfxmintsickn33-agentic-awesome-skills49.1K

Put your skill hereReach developers while they are looking for a skill like yours. Clearly labelled placement, honest monthly exposure reports.See how it works

sickn33 and contributors

How to use this skill?

Select a client and mode, then copy and run the command in your project terminal.
Alternatively, give the AI installation prompt to your coding AI. For Try once, paste the command output into your AI session.

Review third-party skills before installing them.

Have any questions? report an issue

Gemini Live API Development Skill

When to Use

Overview

Models

SDKs

Partner Integrations

Audio Formats

Application boundaries

Quick Start

Authentication

Python

JavaScript

Connecting to the Live API

Python

JavaScript

Sending Text

Python

JavaScript

Sending Audio

Python

JavaScript

Sending Video

Python

JavaScript

Receiving Audio and Text

Python

JavaScript

Live Translation (Gemini Live Translate)

Model

Configuration (TranslationConfig)

Limitations

Migrating from Gemini 2.5 Flash Live

Best Practices

Documentation Lookup

When MCP is Installed (Preferred)

When MCP is NOT Installed (Fallback Only)

Key Documentation Pages

Supported Languages

Configuration (`TranslationConfig`)