Turning server VAD off does not reset the internal timers

[tmmvn]

Please read this first

• Have you read the docs? Agents SDK docs
• Have you searched for related issues? Others may have faced similar issues.

Describe the bug

If you disable VAD server turn detection, the internal timers do not get reset. Thus, if you re-enable VAD later, it will immediately trigger an idle message if the VAD was off longer than the idle timer.

Debug information

• Agents SDK version: v0.8.2
• Runtime environment (e.g. Node.js 22.16.0)

Repro steps

Update the examples/realtime-demo main.ts with this:

// @ts-expect-error Typescript doesn't know about the css module
import './style.css';
import {
  connectButton,
  disconnectButton,
  log,
  muteButton,
  setButtonStates,
} from './utils';

import { z } from 'zod';
import { RealtimeAgent, RealtimeSession, tool } from '@openai/agents-realtime';

const getTemperature = tool({
  name: 'getTemperature',
  description: 'Get the temperature for a given city',
  parameters: z.object({
    city: z.string(),
  }),
  execute: async ({ city }) => {
    await preventInterruption();
    setTimeout(() => allowInterruption(), 20000);
    return `The temperature in ${city} is warm and sunny.`;
  },
});

const getWind = tool({
  name: 'getWind',
  description: 'Get the wind chill for a given city',
  parameters: z.object({
    city: z.string(),
  }),
  execute: async ({ city }) => {
    await preventInterruption();
    setTimeout(() => allowInterruption(), 20000);
    return `The wind in ${city} is strong coming from the West with a chill`;
  },
});

const weatherAgent = new RealtimeAgent({
  name: 'Weather Agent',
  instructions: 'You are a weather expert. You speak in a very long winded shakesperian play way.',
  handoffDescription: 'You can handoff to the weather agent if you need to.',
  tools: [getTemperature, getWind],
});

const agent = new RealtimeAgent({
  name: 'Greeter',
  instructions:
    'You are a greeter. Always greet the user with a "top of the morning, then hand off as the user will want to know the weather."',
  handoffs: [weatherAgent],
});

//weatherAgent.handoffs.push(agent);

const session = new RealtimeSession(agent, {
  config: {
    audio: {
      input: {
        turnDetection: {
          type: 'server_vad',
          threshold: 0.31,
          prefixPaddingMs: 600,
          silenceDurationMs: 200,
          idleTimeoutMs: 5000,
        }
      }
    }
  }
});

const preventInterruption = async () => {
  console.log("Preventing interruptions by disabling turn detection.")
  session.transport.sendEvent({
    type: "session.update",
    session: {
      type: "realtime",
      audio: {
        input: {
          turn_detection: null,
        },
      },
    },
  });
}

const allowInterruption = async () => {
  console.log("Allowing interruptions by re-enabling turn detection.")
  session.transport.sendEvent({
    type: "session.update",
    session: {
      type: "realtime",
      audio: {
        input: {
          turn_detection: {
            type: 'server_vad',
            threshold: 0.31,
            prefix_padding_ms: 600,
            silence_duration_ms: 200,
            idle_timeout_ms: 5000,
          }
        },
      },
    },
  });
}

session.on('transport_event', (event) => {
  // this logs the events coming directly from the Realtime API server
  log(event);
});

connectButton.addEventListener('click', async () => {
  const apiKey = prompt(
    'Enter ephemeral API key. Run `pnpm -F realtime-demo generate-token` to get a token.',
  );
  if (!apiKey) {
    return;
  }
  await session.connect({
    apiKey,
  });
  setButtonStates('unmuted');
});

disconnectButton.addEventListener('click', () => {
  session.close();
  setButtonStates('disconnected');
});

muteButton.addEventListener('click', () => {
  const newMutedState = !session.muted;
  session.mute(newMutedState);
  setButtonStates(newMutedState ? 'muted' : 'unmuted');
});

Get your token and run the demo. Say yes to switch to the weather agent. Wait for a bit and the agent will do the idle prompt. Now ask for weather in Boston. Observe that right after the agent response ends, it triggers another idle prompt from the agent without giving the user a chance to talk.

It is even more obvious when doing something like a legal disclaimer where you don't want the user to be able to interrupt, and you turn the server VAD back on on the speech end event.

Expected behavior

Turning the VAD off and then back on should reset all the VAD related timers or not start until speech has ended.

[seratch]

Thanks for sharing this. If I understand the issue correctly, this sounds like a thing that could be enhanced on the API server side. We'll explore the possibility to take actions for it.