Realtime ASR (WebSocket)

Realtime ASR streams audio over a WebSocket and returns transcripts as you speak — provisional partial results while audio is still flowing, then authoritative final results per utterance. Use it for live captioning, meeting transcription, and voice interfaces.

For one-shot transcription of a complete audio file, use the batch ASR endpoint instead.

Endpoint

wss://api.shisa.ai/ws/asr/realtime

Authenticate during the WebSocket handshake with your ASR bearer token:

Authorization: Bearer YOUR_API_KEY

Requires realtime access

Your API key needs explicit access to the shisa/asr-realtime service. Access to batch ASR, chat, translation, or TTS does not grant realtime ASR access. If your key uses a service allowlist, add shisa/asr-realtime to it — otherwise the handshake is rejected.

Server-side only

Realtime ASR is designed for backend use. Keep your API key server-side: relay audio from the browser to your own server, and connect to Shisa from there.

How it works

Open a WebSocket to the endpoint with the Authorization header.
Send one session.update message to set the audio format and language.
Wait for session.created.
Stream raw audio as base64 input_audio.append messages (50–250 ms per chunk is a good starting point).
Show asr.partial_result as live provisional text; replace it when asr.final_result arrives.
Send session.close when finished.

Audio requirements

Send raw PCM — no container, no compression:

Property	Value
Encoding	`pcm_s16le` (signed 16-bit little-endian)
Sample rate	`16000` Hz
Channels	`1` (mono)
Chunk cadence	50–250 ms recommended (examples use 100 ms)

Resample and downmix in your client before sending. Do not send WAV headers, Ogg, MP3, data URLs, float PCM, or stereo audio.

Client messages

`session.update`

Send exactly one session.update before any audio. A fixed-language session is the simplest option:

{
  "type": "session.update",
  "session": {
    "input_audio_format": "pcm_s16le",
    "sample_rate": 16000,
    "channels": 1,
    "language": "ja"
  }
}

Field	Required	Values	Notes
`input_audio_format`	Yes	`pcm_s16le`	Raw signed 16-bit little-endian PCM.
`sample_rate`	Yes	`16000`	Resample to 16 kHz before sending.
`channels`	Yes	`1`	Mono only. Downmix stereo first.
`language`	Yes	`ja`, `en`, `zh`, `auto`	Use a fixed language when you know it. Use `auto` for automatic language identification (see below).
`default_language`	Auto only	`ja`, `en`, `zh`	Fallback when detection is not confident.
`language_detection_mode`	Auto only	`session`, `utterance`	`session` detects once; `utterance` detects per segment (advanced — see below).

`input_audio.append`

Stream audio chunks as base64-encoded PCM:

{
  "type": "input_audio.append",
  "audio": "<base64 pcm_s16le bytes>"
}

`session.close`

Close the session gracefully when done:

{ "type": "session.close" }

Language identification (optional)

If you don't know the spoken language ahead of time, set language: "auto". Two modes are available.

Session detection identifies the language once, near the start, and uses it for the whole session:

{
  "type": "session.update",
  "session": {
    "input_audio_format": "pcm_s16le",
    "sample_rate": 16000,
    "channels": 1,
    "language": "auto",
    "default_language": "en",
    "language_detection_mode": "session"
  }
}

The session emits session.language_detecting, then one session.language_detected carrying the chosen language.

Per-utterance detection (language_detection_mode: "utterance") detects the language of each utterance separately, which is useful for mixed-language audio. It emits utterance.language_detected per utterance. This mode must be enabled on the backend; if it isn't, the session returns an error with code: "invalid_config" — fall back to a fixed language or session detection in that case.

When detection can't choose confidently, it falls back to default_language, and the event carries source: "default" with a reason (for example no_speech, short_utterance, inconclusive, or timeout). Always treat asr.final_result.text as authoritative — don't rewrite a transcript based on a language event alone.

Router-managed translation (optional)

The router can translate each non-empty asr.final_result asynchronously. Your key must have access to both shisa/asr-realtime and shisa/translate; translation is disabled until you opt in for the current WebSocket session.

After session.update, send:

{
  "type": "router.translation.update",
  "id": "translate-en",
  "enabled": true,
  "target_langs": ["en"]
}

Field	Required	Notes
`enabled`	Yes	Boolean. Set `false` to disable translation; target languages are then omitted.
`target_langs`	When enabling	One to three unique, valid language codes.
`id`	No	Client correlation ID, echoed in the control response; maximum 36 characters.
`context`	No	Context supplied to every translation; maximum 2,000 Unicode codepoints.
`keywords`	No	Glossary array; maximum 20 entries and 100 bytes per entry.

An accepted update returns:

{
  "type": "router.translation.updated",
  "id": "translate-en",
  "enabled": true,
  "target_langs": ["en"]
}

Invalid configuration returns router.translation.error, for example:

{
  "type": "router.translation.error",
  "id": "translate-en",
  "code": "too_many_target_langs",
  "message": "At most 3 target languages are allowed"
}

Configuration error codes include invalid_json, invalid_message_type, invalid_id, id_too_long, invalid_enabled, missing_target_langs, too_many_target_langs, duplicate_target_lang, invalid_target_lang, context_too_long, too_many_keywords, keyword_too_long, translation_access_denied, and translation_service_unavailable. Other reserved router.* messages return router.error with unknown_router_message or router_message_too_large.

Source-language resolution

For each final, the router resolves the source language from the first valid value in this order: (1) asr.final_result.language, (2) the latest session.language_detected, then (3) the fixed language from session.update.session.language. It does not directly consume a standalone utterance.language_detected event for translation. If no valid language is known when a final arrives, that target receives translation.error with code: "source_language_unknown".

Service events

All events are JSON text messages. After the session is created, ASR backend events carry a session_id and a monotonically increasing seq. Router-generated router.* and translation.* events do not carry those fields; correlate translation results and errors to ASR finals with source_result_id.

`session.created`

The session is accepted; you may start sending audio:

{
  "type": "session.created",
  "session_id": "asr_sess_...",
  "seq": 1,
  "format": { "encoding": "pcm_s16le", "sample_rate": 16000, "channels": 1 }
}

Speech boundaries

speech_started and speech_stopped mark voice-activity endpoints for each utterance:

{
  "type": "speech_started",
  "session_id": "asr_sess_...",
  "seq": 4,
  "utterance_id": "utt_0001",
  "audio_start_ms": 420
}

{
  "type": "speech_stopped",
  "session_id": "asr_sess_...",
  "seq": 9,
  "utterance_id": "utt_0001",
  "audio_start_ms": 420,
  "audio_end_ms": 2860,
  "reason": "vad_endpoint"
}

Common stop reasons are vad_endpoint, max_buffer, and session_close.

`asr.partial_result`

Provisional text for an in-progress utterance. Display the latest partial per utterance_id, replacing the previous one — don't append every partial to history:

{
  "type": "asr.partial_result",
  "session_id": "asr_sess_...",
  "seq": 7,
  "utterance_id": "utt_0001",
  "result_id": "utt_0001:p3",
  "text": "今日の会議は3時から",
  "is_final": false,
  "audio_start_ms": 420,
  "audio_end_ms": 2260
}

`asr.final_result`

Authoritative text for the utterance. Use replaces to drop the provisional partials you were displaying:

{
  "type": "asr.final_result",
  "session_id": "asr_sess_...",
  "seq": 12,
  "utterance_id": "utt_0001",
  "result_id": "utt_0001:final",
  "replaces": ["utt_0001:p1", "utt_0001:p2", "utt_0001:p3"],
  "replaces_audio_range_ms": [420, 2860],
  "text": "今日の会議は3時からです。",
  "language": "ja",
  "is_final": true,
  "audio_start_ms": 420,
  "audio_end_ms": 2860
}

Finals are asynchronous and can arrive after later speech events. Very long utterances may be split into continuation finals that carry extra continuation_of and overlap_mode metadata; display only the logical audio_start_ms–audio_end_ms range.

`translation.final_result`

When translation is enabled, the router emits one result per target language for each translatable ASR final:

{
  "type": "translation.final_result",
  "utterance_id": "utt_0001",
  "source_result_id": "utt_0001:final",
  "source_text": "今日の会議は3時からです。",
  "text": "Today's meeting starts at 3 o'clock.",
  "source_lang": "ja",
  "target_lang": "en",
  "model": "shisa-v2.1-unphi4-14b"
}

Attach the result to the exact ASR final identified by source_result_id. Translation is asynchronous, and results for multiple target languages may arrive in any order. model is the model ID reported by the backend response and may differ from the default shisa-ai/chotto request alias.

`translation.error`

A failed target emits a target-specific error rather than closing the ASR session:

{
  "type": "translation.error",
  "code": "source_language_unknown",
  "message": "Cannot translate ASR final result before source language is known",
  "utterance_id": "utt_0001",
  "source_result_id": "utt_0001:final",
  "target_lang": "en"
}

Other common codes include same_source_target_lang, translation_access_denied, translation_service_unavailable, translation_rate_limited, translation_rate_limiter_unavailable, translation_failed, and source_text_too_long. Empty final text is not translated and emits neither a translation result nor a translation error.

`session.usage`

Cumulative usage; billing uses the latest event. billable_duration is in whole seconds:

{
  "type": "session.usage",
  "session_id": "asr_sess_...",
  "seq": 20,
  "final": true,
  "usage": {
    "input_duration": 18.3036875,
    "billable_duration": 19,
    "utterance_count": 3,
    "final_result_count": 3,
    "status": "completed",
    "final": true
  }
}

`error`

{
  "type": "error",
  "code": "finalization_failed",
  "message": "Finalization failed",
  "fatal": false,
  "session_id": "asr_sess_...",
  "seq": 18,
  "utterance_id": "utt_0002"
}

Fatal errors are followed by the socket closing. Non-fatal utterance errors let the session continue.

Handling transcripts

Key provisional text by utterance_id.
Replace the active partial when a newer asr.partial_result arrives.
On asr.final_result, remove the replaces IDs and commit the final text.
Attach translated text by source_result_id; do not assume target-language result order.
Ignore empty final text for display (keep it for diagnostics).
Never render session.usage as transcript text.

Next steps

Transcribe complete files with the batch ASR endpoint.
Review supported formats and languages in Audio & languages.
See how usage is billed on Pricing.

Endpoint​

How it works​

Audio requirements​

Client messages​

session.update​

input_audio.append​

session.close​

Language identification (optional)​

Router-managed translation (optional)​

Service events​

session.created​

Speech boundaries​

asr.partial_result​

asr.final_result​

translation.final_result​

translation.error​

session.usage​

error​

Handling transcripts​

Next steps​