Live context notes

A live context note tells the agent something it can't hear on the call: a payment that went through, an order that shipped, the page the caller is looking at, a supervisor's steer. The agent takes it into account from its next reply. A note never counts as something the caller said. It isn't added to the transcript, it doesn't move a flow, and it doesn't make the agent answer it.

A note can also ask the agent to tell the caller (respond: "when_idle"). The agent then speaks up as soon as the line is free, in its own words and in the caller's language.

Notes work on every live call: phone calls on every carrier, SIP and web calls. A note can come from:

How the agent sees notes

Before each reply, the agent's prompt gets one block after the conversation. The block lists the call's notes oldest first, each with who sent it and how old it is. Notes from the caller's app are marked unverified: the agent is told they never prove a payment, an identity or eligibility, and to describe them as what the caller's app shows.

• Keys replace. A note with a key replaces the earlier note with the same key, so state that changes (a payment status, a cart, the current page) is always one current note. Without a key, notes add up.
• Keys are per sender. A note from the caller's app never replaces a note from your systems, and yours never replaces one of the app's. Your backend, your dashboard users and the apps you installed share one set of keys.
• Your notes come first. When the prompt can't fit every note, it takes your systems' notes first (keyed ones before un-keyed, newest first), then the caller's app's. A busy or tampered page can never crowd your notes out.
• Lines the agent says on its own — a silence check-in, a "take your time" nudge, a goodbye, a welcome back after a hold, a flow step's opener — also have the notes in view, as background.

Telling the caller: respond: "when_idle"

Send respond: "when_idle" with the note when the caller should hear the news.

• As soon as the line is free. The agent speaks up once nobody else has the floor: it isn't talking, the caller isn't, and the caller doesn't owe it an answer to a question it just asked. It never talks over the caller.
• If the caller speaks first, the agent's reply tells them instead.
• In a wait the caller asked for ("one moment, I'm paying"), the news ends the wait: the agent tells them and the call carries on.
• Told once. The note counts as told once a reply has carried it, or once the agent's line about it has actually been spoken. A line written but never played tells nobody, and neither does a check-in or a nudge that happened to have the note in view.
• A retry is not told twice. If you send the same note again (same key, text and respond) after the agent already had it in view, it isn't announced again.
• One at a time. A newer when_idle note takes over from an older one that hasn't been spoken yet. The older note is still told in the next reply.
• Your agent's own words. The line is written by the agent's own model, in the caller's language. There is no fixed text.
• Not in a text chat. In a text chat (a web call in chat mode) the agent does not write to the caller about a note on its own: when_idle there works like none. The note answers next_turn, and the agent's next reply tells the caller.

When the agent sees a note

• Usually from its next reply. With when_idle, it also speaks up once the line is free, or tells the caller in its next reply.
• Straight away, for a when_idle note that arrives while the line is free: the agent speaks up about it now.
• When it gets the call back, if the agent is away from the call: the caller put it on hold, or a transfer is under way that may hand the call back.

The answer to each note says which of these applies (its delivery). A call where a person has the call for good — a manual (softphone) call, a takeover by your team, or a transfer that went through — has no agent to tell, so notes there are refused.

Size: how much fits

A note has to fit in the room the agent's prompt leaves beside the conversation, and beside the notes the call already holds. Your notes count first, then the caller's app's. A note that doesn't fit is refused, and the answer says how many of its characters would fit right now.

• The room shrinks as the call grows. "Fits right now" is the honest answer. Keep notes short and current: one keyed note per piece of state.
• When the call's notes outgrow what the model could ever hold, the oldest are dropped to make way: the caller's app's notes first, then your un-keyed notes. Your keyed notes are never dropped. The call page marks what was dropped.
• On a speech-to-speech agent, the room is the realtime model's own.

"Still here, just busy": activity

When the caller goes quiet because they are busy — paying, reading a message, typing in a form — tell the agent with an activity ping, so it doesn't ask "are you still there?".

• It restarts the agent's silence check-in. In a wait the caller asked for, it restarts that wait's own clock instead: its nudges and its close count from now, and the wait goes on. It never makes the agent speak.
• The answer says how long until the agent would speak up unprompted: its check-in or, with check-ins off, its silence hang-up. Ping again before it runs out while the caller is still busy.
• Sometimes nothing is armed: a hold, a wait the agent is still acknowledging, check-ins and the silence hang-up both off, or a speech-to-speech agent. Try again once the conversation moves on.
• In the browser, the web SDK can ping for you while the user types or clicks (autoActivity).

Speech-to-speech agents

On OpenAI Realtime, Gemini 3.8 Live and Gemini 3.1 Flash Live, a note goes silently into the model's own conversation. A later note with the same key replaces it there too. when_idle works the same way: the model speaks up once the line is free. It never interrupts the caller, never while it is still answering them, and never while a tool call is running.

• A note sent before the realtime model has connected waits, and is delivered as soon as it connects.
• Other Gemini Live models refuse notes. So does a Custom API agent.

The API

Send a note with POST /api/sessions/{id}/context, ping with POST /api/sessions/{id}/activity, and list a call's notes with GET /api/sessions/{id}/context-notes. The requests, every answer and error, the WebSocket replies and SDK examples are in Sessions & calls API → Tell the agent something mid‑call. The JSON shapes are in API responses.

Notes from the caller's app

Notes from the caller's own app are off by default. Turn them on per agent:

• In the builder: Call handling → "Notes from the caller's app".
• In the API: sessionConfig.liveContext.acceptClientNotes: true (see the agent field reference).

The caller controls their own app, so these notes are always marked unverified to the agent, never replace your notes, and get only the room your notes leave. Each call's app shares the browser keypad's allowance: a burst of 16, then 5 a second, and at most 300 per call. Anything more is refused with rate_limited.

A reseller's client can turn the setting on or off for their own agents when their provider grants it. When a client creates an agent, they can leave it as their provider's starter set it.

After the call

• The call page shows a "Notes during the call" card. It lists every note with its sender, its key, and what became of it: replaced by a later note, dropped to make room, spoken up about by the agent, or held while the agent was away. While the call is still on, the card refreshes by itself. The debug timeline has a row for each note (context.note), for each when_idle outcome (context.react), and for each activity ping that restarted something (activity), with who sent it.
• Reseller clients see their own calls' notes in the client portal, without who sent them.
• Post-call analysis is grounded on the notes your systems sent: a statement the agent made after a note that supports it is not a hallucination. The judge only sees notes up to the last transcript line it reads. Notes from the caller's app are never grounds.
• Privacy. A note is stored with the call's PII policy applied to its text and key, exactly like a transcript turn. It is deleted with the transcript when content retention runs. The agent itself received the note as you sent it.

When not to use a note

A note reaches the agent's next reply. The agent starts answering as soon as the caller stops talking, so a note sent in reaction to what the caller just said arrives too late for that answer. To answer the caller's current question with your data, give the agent a tool: it calls your API and waits for the result.