Evaluation for live agents

Supported in ADKPython v2.6.0

You can evaluate a live agent the way it actually gets used: a simulated user speaks its turns as audio, your agent answers over a real bidirectional session, and you score what it said back. The eval sets, criteria, and adk eval loop are the same ones you already use for text agents, covered in Evaluate agents.

Drive the agent with speech

The llm_audio user simulator synthesizes each simulated user turn with a text-to-speech model and streams it to your agent as audio. That runs the path your users take end to end: speech in, voice activity detection, turn-taking, speech out, transcription. Feeding text to a voice agent skips all of it.

{
  "user_simulator_config": {
    "type": "llm_audio",
    "model": "gemini-3.7-flash",
    "max_allowed_invocations": 10,
    "audio_model": "gemini-3.1-flash-tts-preview",
    "audio_model_configuration": {
      "response_modalities": ["AUDIO"],
      "speech_config": {
        "voice_config": {
          "prebuilt_voice_config": { "voice_name": "Kore" }
        },
        "language_code": "en-US"
      }
    }
  }
}

Two models do different jobs here. model decides what the simulated user says next, and audio_model turns that into speech. Changing voice_name and language_code is how you test the agent against different voices and accents, which is the kind of regression a text eval cannot catch.

Your eval cases stay as they are. The same conversation scenario or fixed conversation drives a text run or a voice run, so you can reuse a suite you already have. For the full schema, personas, and how to write scenarios, see Audio user simulation.

Score with rubrics

A spoken reply is right in dozens of different phrasings, so criteria that compare against a reference string will fail correct answers. Rubric-based judges let you write the intent in natural language once and apply it across every conversation in the suite:

Criterion	Scores
rubric_based_final_response_quality_v1	A single turn's reply
rubric_based_tool_use_quality_v1	Whether tools were called correctly
rubric_based_multi_turn_trajectory_quality_v1	The conversation end to end

{
  "criteria": {
    "rubric_based_multi_turn_trajectory_quality_v1": {
      "threshold": 0.7,
      "judge_model_options": { "judge_model": "gemini-3.7-flash" },
      "rubrics": [
        {
          "rubric_id": "verifies_identity_first",
          "rubric_content": {
            "text_property": "Across the call, the agent confirms the caller's name and validates their date of birth before disclosing any appointment details."
          }
        }
      ]
    }
  }
}

Reach for the trajectory criterion with workflows, where what matters is that the agents ran in order and handed off cleanly rather than what any single turn said. Where the answer is genuinely fixed, tool_trajectory_avg_score still checks the exact sequence of tool calls and ignores phrasing entirely.

Run an eval

Add a live_model_config block to test_config.json. It is what puts the eval in live mode, and it is required for Live models, which are not served over the unary generateContent endpoint that text evals use:

{
  "live_model_config": {
    "timeout_seconds": 300
  }
}

timeout_seconds (default 300) caps how long ADK waits for a turn to finish. Raise it if your agent narrates long tool calls, lower it to fail a stuck session faster.

adk eval path/to/your_agent \
  path/to/your_agent/live.evalset.json \
  --config_file_path path/to/your_agent/test_config.json

This needs the eval extras (pip install "google-adk[eval]") and credentials for the Live API and the TTS model. The same run is available through AgentEvaluator, which is how you put voice evals in CI.

In adk web, the eval dialog has a Standard | Live toggle that exposes the input modality and the simulated user's voice and language. When the run finishes, ADK reassembles the audio into a transcript with a playable clip on every turn, so you can hear how the agent sounded instead of only reading what it said.

Sample

The live_workflow sample is a complete voice eval you can run: three live agents in a graph workflow, a tool call in the middle, and an eval set and test_config.json wired up with all three rubric criteria.