Realtime ASR (WebSocket)

Realtime ASR streams audio over a WebSocket and returns transcripts as you speak — provisional partial results while audio is still flowing, then authoritative final results per utterance. Use it for live captioning, meeting transcription, and voice interfaces.

For one-shot transcription of a complete audio file, use the batch ASR endpoint instead.

Endpoint

wss://api.shisa.ai/ws/asr/realtime

Authenticate during the WebSocket handshake with your ASR bearer token:

Authorization: Bearer YOUR_API_KEY

Requires realtime access

Your API key needs explicit access to the shisa/asr-realtime service. Access to batch ASR, chat, translation, or TTS does not grant realtime ASR access. If your key uses a service allowlist, add shisa/asr-realtime to it — otherwise the handshake is rejected.

Server-side only

Realtime ASR is designed for backend use. Keep your API key server-side: relay audio from the browser to your own server, and connect to Shisa from there.

How it works

Open a WebSocket to the endpoint with the Authorization header.
Send one session.update message to set the audio format and language.
Wait for session.created.
Stream raw audio as base64 input_audio.append messages (50–250 ms per chunk is a good starting point).
Show asr.partial_result as live provisional text; replace it when asr.final_result arrives.
Send session.close when finished.

Audio requirements

Send raw PCM — no container, no compression:

Property	Value
Encoding	`pcm_s16le` (signed 16-bit little-endian)
Sample rate	`16000` Hz
Channels	`1` (mono)
Chunk cadence	50–250 ms recommended (examples use 100 ms)

Resample and downmix in your client before sending. Do not send WAV headers, Ogg, MP3, data URLs, float PCM, or stereo audio.

Client messages

`session.update`

Send exactly one session.update before any audio. A fixed-language session is the simplest option:

{
  "type": "session.update",
  "session": {
    "input_audio_format": "pcm_s16le",
    "sample_rate": 16000,
    "channels": 1,
    "language": "ja"
  }
}

Field	Required	Values	Notes
`input_audio_format`	Yes	`pcm_s16le`	Raw signed 16-bit little-endian PCM.
`sample_rate`	Yes	`16000`	Resample to 16 kHz before sending.
`channels`	Yes	`1`	Mono only. Downmix stereo first.
`language`	Yes	`ja`, `en`, `zh`, `auto`	Use a fixed language when you know it. Use `auto` for automatic language identification (see below).
`default_language`	Auto only	`ja`, `en`, `zh`	Fallback when detection is not confident.
`language_detection_mode`	Auto only	`session`, `utterance`	`session` detects once; `utterance` detects per segment (advanced — see below).

`input_audio.append`

Stream audio chunks as base64-encoded PCM:

{
  "type": "input_audio.append",
  "audio": "<base64 pcm_s16le bytes>"
}

`session.close`

Close the session gracefully when done:

{ "type": "session.close" }

Language identification (optional)

If you don't know the spoken language ahead of time, set language: "auto". Two modes are available.

Session detection identifies the language once, near the start, and uses it for the whole session:

{
  "type": "session.update",
  "session": {
    "input_audio_format": "pcm_s16le",
    "sample_rate": 16000,
    "channels": 1,
    "language": "auto",
    "default_language": "en",
    "language_detection_mode": "session"
  }
}

The session emits session.language_detecting, then one session.language_detected carrying the chosen language.

Per-utterance detection (language_detection_mode: "utterance") detects the language of each utterance separately, which is useful for mixed-language audio. It emits utterance.language_detected per utterance. This mode must be enabled on the backend; if it isn't, the session returns an error with code: "invalid_config" — fall back to a fixed language or session detection in that case.

When detection can't choose confidently, it falls back to default_language, and the event carries source: "default" with a reason (for example no_speech, short_utterance, inconclusive, or timeout). Always treat asr.final_result.text as authoritative — don't rewrite a transcript based on a language event alone.

Service events

All events are JSON text messages. After the session is created, every event carries a session_id and a monotonically increasing seq.

`session.created`

The session is accepted; you may start sending audio:

{
  "type": "session.created",
  "session_id": "asr_sess_...",
  "seq": 1,
  "format": { "encoding": "pcm_s16le", "sample_rate": 16000, "channels": 1 }
}

Speech boundaries

speech_started and speech_stopped mark voice-activity endpoints for each utterance:

{
  "type": "speech_started",
  "session_id": "asr_sess_...",
  "seq": 4,
  "utterance_id": "utt_0001",
  "audio_start_ms": 420
}

{
  "type": "speech_stopped",
  "session_id": "asr_sess_...",
  "seq": 9,
  "utterance_id": "utt_0001",
  "audio_start_ms": 420,
  "audio_end_ms": 2860,
  "reason": "vad_endpoint"
}

Common stop reasons are vad_endpoint, max_buffer, and session_close.

`asr.partial_result`

Provisional text for an in-progress utterance. Display the latest partial per utterance_id, replacing the previous one — don't append every partial to history:

{
  "type": "asr.partial_result",
  "session_id": "asr_sess_...",
  "seq": 7,
  "utterance_id": "utt_0001",
  "result_id": "utt_0001:p3",
  "text": "今日の会議は3時から",
  "is_final": false,
  "audio_start_ms": 420,
  "audio_end_ms": 2260
}

`asr.final_result`

Authoritative text for the utterance. Use replaces to drop the provisional partials you were displaying:

{
  "type": "asr.final_result",
  "session_id": "asr_sess_...",
  "seq": 12,
  "utterance_id": "utt_0001",
  "result_id": "utt_0001:final",
  "replaces": ["utt_0001:p1", "utt_0001:p2", "utt_0001:p3"],
  "replaces_audio_range_ms": [420, 2860],
  "text": "今日の会議は3時からです。",
  "is_final": true,
  "audio_start_ms": 420,
  "audio_end_ms": 2860
}

Finals are asynchronous and can arrive after later speech events. Very long utterances may be split into continuation finals that carry extra continuation_of and overlap_mode metadata; display only the logical audio_start_ms–audio_end_ms range.

`session.usage`

Cumulative usage; billing uses the latest event. billable_duration is in whole seconds:

{
  "type": "session.usage",
  "session_id": "asr_sess_...",
  "seq": 20,
  "final": true,
  "usage": {
    "input_duration": 18.3036875,
    "billable_duration": 19,
    "utterance_count": 3,
    "final_result_count": 3,
    "status": "completed",
    "final": true
  }
}

`error`

{
  "type": "error",
  "code": "finalization_failed",
  "message": "Finalization failed",
  "fatal": false,
  "session_id": "asr_sess_...",
  "seq": 18,
  "utterance_id": "utt_0002"
}

Fatal errors are followed by the socket closing. Non-fatal utterance errors let the session continue.

Handling transcripts

Key provisional text by utterance_id.
Replace the active partial when a newer asr.partial_result arrives.
On asr.final_result, remove the replaces IDs and commit the final text.
Ignore empty final text for display (keep it for diagnostics).
Never render session.usage as transcript text.

Next steps

Transcribe complete files with the batch ASR endpoint.
Review supported formats and languages in Audio & languages.
See how usage is billed on Pricing.

Endpoint​

How it works​

Audio requirements​

Client messages​

session.update​

input_audio.append​

session.close​

Language identification (optional)​

Service events​

session.created​

Speech boundaries​

asr.partial_result​

asr.final_result​

session.usage​

error​

Handling transcripts​

Next steps​