Skip to main content

Realtime ASR (WebSocket)

Realtime ASR streams audio over a WebSocket and returns transcripts as you speak — provisional partial results while audio is still flowing, then authoritative final results per utterance. Use it for live captioning, meeting transcription, and voice interfaces.

For one-shot transcription of a complete audio file, use the batch ASR endpoint instead.

Endpoint​

wss://api.shisa.ai/ws/asr/realtime

Authenticate during the WebSocket handshake with your ASR bearer token:

Authorization: Bearer YOUR_API_KEY
Access

Realtime ASR is public to normally authorized active API keys. If your key uses an allowedServices allowlist, include shisa/asr-realtime; deniedServices still takes precedence. The default limit is 3 concurrent WebSocket sessions per API key. Individual keys may be configured with a different limit.

Server-side only

Realtime ASR is designed for backend use. Keep your API key server-side: relay audio from the browser to your own server, and connect to Shisa from there.

How it works​

  1. Open a WebSocket to the endpoint with the Authorization header.
  2. Send one session.update message to set the audio format and language.
  3. Wait for session.created.
  4. Stream raw audio as base64 input_audio.append messages (50–250 ms per chunk is a good starting point).
  5. Show asr.partial_result as live provisional text; replace it when asr.final_result arrives.
  6. Send session.close when finished.

Audio requirements​

Send raw PCM — no container, no compression:

PropertyValue
Encodingpcm_s16le (signed 16-bit little-endian)
Sample rate16000 Hz
Channels1 (mono)
Chunk cadence50–250 ms recommended (examples use 100 ms)

Resample and downmix in your client before sending. Do not send WAV headers, Ogg, MP3, data URLs, float PCM, or stereo audio.

Client messages​

session.update​

Send exactly one session.update before any audio. A fixed-language session is the simplest option:

{
"type": "session.update",
"session": {
"input_audio_format": "pcm_s16le",
"sample_rate": 16000,
"channels": 1,
"language": "ja"
}
}
FieldRequiredValuesNotes
input_audio_formatYespcm_s16leRaw signed 16-bit little-endian PCM.
sample_rateYes16000Resample to 16 kHz before sending.
channelsYes1Mono only. Downmix stereo first.
languageYesja, en, zh, autoUse a fixed language when you know it. Use auto for automatic language identification (see below).
default_languageAuto onlyja, en, zhFallback when detection is not confident.
language_detection_modeAuto onlysession, utterancesession detects once; utterance detects per segment (advanced — see below).
hotwordsNoArray of stringsGlobal finalization vocabulary; up to 50 items (separate from the language-mapped limits).
hotwords_by_languageNoObject or nullFinalization vocabulary keyed by resolved language; up to 5 keys, 100 items per language, and 300 mapped items in total.

Hotwords​

Hotwords bias finalization toward your vocabulary — product names, jargon, and proper nouns. Configure them once, in the initial session.update; they apply to every utterance in that WebSocket session. To change them, close the session and open a new one with a new session.update. A typical call-center integration uses one WebSocket session per customer call, not one session per conversational turn.

A fixed-language session is the simplest setup:

{
"type": "session.update",
"session": {
"input_audio_format": "pcm_s16le",
"sample_rate": 16000,
"channels": 1,
"language": "ja",
"hotwords": ["シーサー", "Shisa AI"]
}
}

Hotwords affect only asr.final_result. Live asr.partial_result events may still contain an uncorrected spelling; replace the displayed partial text when the final result arrives. Service events never echo the configured values.

Limits​

The global hotwords list and the language-mapped lists have separate limits:

ConfigurationLimit
Global hotwords50 items
Language keys5
Items per language100
Total language-mapped items300
Length per item1–100 Unicode characters

Each item must be unique within its list (case-insensitively) and contain 1–100 Unicode characters (technically, Unicode scalar values), with no leading/trailing whitespace or control characters. The service rejects invalid configuration rather than trimming, normalizing, truncating, or deduplicating values.

Language-specific hotwords (advanced)​

A session may combine global fallback hotwords with language-specific lists in its single initial session.update:

{
"type": "session.update",
"session": {
"input_audio_format": "pcm_s16le",
"sample_rate": 16000,
"channels": 1,
"language": "auto",
"default_language": "en",
"language_detection_mode": "utterance",
"hotwords": ["global fallback"],
"hotwords_by_language": {
"ja": ["シーサー"],
"en": ["Shisa AI"],
"zh": ["狮萨"]
}
}
}

After fixed-language or LID resolution, a non-empty exact matching mapped list wins; an absent or empty matching list falls back to hotwords. Keys must be exact lowercase ASCII two-letter codes; use canonical ja, en, and zh for current deployments.

Availability of hotwords_by_language may depend on the selected deployment. If it is unavailable, the session returns an invalid_config error. In that case, use the global hotwords field or contact support.

Message-size policy​

The router accepts ordinary client messages up to 65,536 bytes. The initial session.update may be up to 524,288 bytes when its sole top-level type is exactly session.update; this accommodates bounded fields such as hotwords_by_language. Limits apply to the complete assembled WebSocket message in UTF-8 wire bytes.

Other client messages above 65,536 bytes, or any client message above 524,288 bytes, close with 1009 and are not forwarded. Backend-to-client messages remain limited to 65,536 bytes. Operators may configure stricter limits, so serialize compactly and treat 1009 as a terminal session error.

input_audio.append​

Stream audio chunks as base64-encoded PCM:

{
"type": "input_audio.append",
"audio": "<base64 pcm_s16le bytes>"
}

session.close​

Close the session gracefully when done:

{ "type": "session.close" }

Language identification (optional)​

If you don't know the spoken language ahead of time, set language: "auto". Two modes are available.

Session detection identifies the language once, near the start, and uses it for the whole session:

{
"type": "session.update",
"session": {
"input_audio_format": "pcm_s16le",
"sample_rate": 16000,
"channels": 1,
"language": "auto",
"default_language": "en",
"language_detection_mode": "session"
}
}

The session emits session.language_detecting, then one session.language_detected carrying the chosen language.

Per-utterance detection (language_detection_mode: "utterance") detects the language of each utterance separately, which is useful for mixed-language audio. It emits utterance.language_detected per utterance. This mode must be enabled on the backend; if it isn't, the session returns an error with code: "invalid_config" — fall back to a fixed language or session detection in that case.

When detection can't choose confidently, it falls back to default_language, and the event carries source: "default" with a reason (for example no_speech, short_utterance, inconclusive, timeout, unsupported, queue_full, buffer_limit, session_close, shutdown, vad_error, or lid_error). Always treat asr.final_result.text as authoritative — don't rewrite a transcript based on a language event alone.

Router-managed translation (optional)​

The router can translate each non-empty asr.final_result asynchronously. Your key must have access to both shisa/asr-realtime and shisa/translate; translation is disabled until you opt in for the current WebSocket session.

After session.update, send:

{
"type": "router.translation.update",
"id": "translate-en",
"enabled": true,
"target_langs": ["en"]
}
FieldRequiredNotes
enabledYesBoolean. Set false to disable translation; target languages are then omitted.
target_langsWhen enablingOne to three unique, valid language codes.
idNoClient correlation ID, echoed in the control response; maximum 36 characters.
contextNoContext supplied to every translation; maximum 2,000 Unicode codepoints.
keywordsNoGlossary array; maximum 20 entries and 100 bytes per entry.

An accepted update returns:

{
"type": "router.translation.updated",
"id": "translate-en",
"enabled": true,
"target_langs": ["en"]
}

Invalid configuration returns router.translation.error, for example:

{
"type": "router.translation.error",
"id": "translate-en",
"code": "too_many_target_langs",
"message": "At most 3 target languages are allowed"
}

Configuration error codes include invalid_json, invalid_message_type, invalid_id, id_too_long, invalid_enabled, missing_target_langs, too_many_target_langs, duplicate_target_lang, invalid_target_lang, context_too_long, too_many_keywords, keyword_too_long, translation_access_denied, and translation_service_unavailable. Other reserved router.* messages return router.error with unknown_router_message or router_message_too_large.

Source-language resolution

For each final, the router resolves the source language from the first valid value in this order: (1) asr.final_result.language, (2) the latest session.language_detected, then (3) the fixed language from session.update.session.language. It does not directly consume a standalone utterance.language_detected event for translation. If no valid language is known when a final arrives, that target receives translation.error with code: "source_language_unknown".

Service events​

All events are JSON text messages. After the session is created, ASR backend events carry a session_id and a monotonically increasing seq. Router-generated router.* and translation.* events do not carry those fields; correlate translation results and errors to ASR finals with source_result_id.

session.created​

The session is accepted; you may start sending audio:

{
"type": "session.created",
"session_id": "asr_sess_...",
"seq": 1,
"format": { "encoding": "pcm_s16le", "sample_rate": 16000, "channels": 1 }
}

Speech boundaries​

speech_started and speech_stopped mark voice-activity endpoints for each utterance:

{
"type": "speech_started",
"session_id": "asr_sess_...",
"seq": 4,
"utterance_id": "utt_0001",
"audio_start_ms": 420
}
{
"type": "speech_stopped",
"session_id": "asr_sess_...",
"seq": 9,
"utterance_id": "utt_0001",
"audio_start_ms": 420,
"audio_end_ms": 2860,
"reason": "vad_endpoint"
}

Common stop reasons are vad_endpoint, max_buffer, and session_close.

asr.partial_result​

Provisional text for an in-progress utterance. Display the latest partial per utterance_id, replacing the previous one — don't append every partial to history:

{
"type": "asr.partial_result",
"session_id": "asr_sess_...",
"seq": 7,
"utterance_id": "utt_0001",
"result_id": "utt_0001:p3",
"text": "今日の会議は3時から",
"is_final": false,
"audio_start_ms": 420,
"audio_end_ms": 2260
}

asr.final_result​

Authoritative text for the utterance. Use replaces to drop the provisional partials you were displaying:

{
"type": "asr.final_result",
"session_id": "asr_sess_...",
"seq": 12,
"utterance_id": "utt_0001",
"result_id": "utt_0001:final",
"replaces": ["utt_0001:p1", "utt_0001:p2", "utt_0001:p3"],
"replaces_audio_range_ms": [420, 2860],
"text": "今日の会議は3時からです。",
"language": "ja",
"is_final": true,
"audio_start_ms": 420,
"audio_end_ms": 2860
}

Finals are asynchronous and can arrive after later speech events. Very long utterances may be split into continuation finals that carry continuation_of, overlap_mode, leading_context_ms, and payload_audio_range_ms. audio_start_ms/audio_end_ms describe the logical utterance range; payload_audio_range_ms includes the overlapped context sent to final ASR, and leading_context_ms is the context duration before audio_start_ms. Display only the logical range.

translation.final_result​

When translation is enabled, the router emits one result per target language for each translatable ASR final:

{
"type": "translation.final_result",
"utterance_id": "utt_0001",
"source_result_id": "utt_0001:final",
"source_text": "今日の会議は3時からです。",
"text": "Today's meeting starts at 3 o'clock.",
"source_lang": "ja",
"target_lang": "en",
"model": "shisa-v2.1-unphi4-14b"
}

Attach the result to the exact ASR final identified by source_result_id. Translation is asynchronous, and results for multiple target languages may arrive in any order. model is the model ID reported by the backend response and may differ from the default shisa-ai/chotto request alias.

translation.error​

A failed target emits a target-specific error rather than closing the ASR session:

{
"type": "translation.error",
"code": "source_language_unknown",
"message": "Cannot translate ASR final result before source language is known",
"utterance_id": "utt_0001",
"source_result_id": "utt_0001:final",
"target_lang": "en"
}

Other common codes include same_source_target_lang, translation_access_denied, translation_service_unavailable, translation_rate_limited, translation_rate_limiter_unavailable, translation_failed, and source_text_too_long. Empty final text is not translated and emits neither a translation result nor a translation error.

session.usage​

Cumulative usage; billing uses the latest event. billable_duration is in whole seconds. The top-level model is the deployment-selected finalization model used for accounting/debugging; clients do not choose the realtime partial engine:

{
"type": "session.usage",
"session_id": "asr_sess_...",
"seq": 20,
"final": true,
"model": "shisa-ai/shisa-asr",
"usage": {
"input_duration": 18.3036875,
"billable_duration": 19,
"input_bytes": 585718,
"sample_rate": 16000,
"channels": 1,
"utterance_count": 3,
"final_result_count": 3,
"final_error_count": 0,
"final_no_speech_count": 0,
"partial_result_count": 17,
"error_count": 0,
"status": "completed",
"final": true
}
}

Handling errors​

A healthy session connects, streams, and closes without any error events. When something does need your attention, the API returns a clear code and, where possible, tells you what to do next.

Errors can appear at two points: while the connection is being established, as an HTTP response with an error object; or during the WebSocket session, as a JSON event with type: "error".

If the connection is refused​

If the upgrade is not accepted, no WebSocket is created. The response is a normal HTTP error:

{
"error": {
"code": "service_access_denied",
"message": "Service access denied"
}
}

The common causes are easy to fix:

HTTPCodeWhat to do
401auth_token_requiredSend an Authorization: Bearer <key> header.
401invalid_token_formatUse the full shsk:... key, including the prefix.
401invalid_tokenCheck that the key is current and active. See Authentication.
401legacy_api_key_rejectedReplace the legacy key with a current API key.
403service_access_deniedAdd shisa/asr-realtime to the key's allowedServices, or remove it from deniedServices.
403ws_origin_deniedConnect from an allowed origin, or omit the Origin header from server-side clients.
429rate_limited, service_rate_limitedSlow down and retry with backoff. See Rate limits.
429ws_connection_limit_exceededClose idle sessions or wait for one to finish. Individual keys may be configured with a different limit; the error body reports the applicable limit and current active count.

Occasionally the service is briefly unable to accept new sessions — for example during capacity changes or maintenance. These conditions are temporary and return 503; retry with backoff. They are not caused by your request.

Temporary service codes
HTTPCodeMeaning
503auth_unavailableAPI key verification is temporarily unavailable.
503rate_limiter_unavailable, service_rate_limiter_unavailableRate-limit checks are temporarily unavailable.
503ws_connection_quota_unavailableConnection-quota tracking is temporarily unavailable.
503backends_at_capacity, backend_unavailableNo realtime ASR capacity is available right now.
503websocket_drainingThe service is finishing a planned restart and is not accepting new sessions.
503service_rate_limit_policy_invalidThe key's rate-limit configuration needs a service-side update. Retrying unchanged will not help; contact support.

During a session​

After a successful upgrade, the backend forwards JSON error events. Most apply to a single utterance and let the session continue; only fatal: true ends the session. Handle unknown code values defensively, because the backend may add codes over time:

{
"type": "error",
"code": "finalization_failed",
"message": "Finalization failed",
"fatal": false,
"session_id": "asr_sess_...",
"seq": 18,
"utterance_id": "utt_0002"
}
CodeWhat to do
invalid_json, invalid_stateCheck message ordering: send exactly one session.update before audio, and use known message types.
invalid_audioSend raw pcm_s16le at 16 kHz mono; do not send headers or compressed audio.
invalid_configCheck the selected language/LID mode and validate hotwords against the documented count, length, uniqueness, and character rules. If hotwords_by_language is unavailable on the deployment, use global hotwords or contact support.
frame_too_largeSend a smaller individual message or audio chunk.
limit_exceededA session, buffer, recognizer backlog, or queue limit was reached. Stop or close the current session as appropriate, then retry with backoff. If it persists, contact support.
finalization_timeout, finalization_cancelled, finalization_failedRetry the utterance. If it keeps happening, contact support.
service_errorA temporary service-side issue. Retry; contact support if it persists.
fatal: trueThe session ends after this event. Reconnect and resume.

Connection close behavior​

  • A refused connection is a normal HTTP response; no WebSocket is created.
  • Invalid language-specific hotword configuration emits an invalid_config event followed by close 1008.
  • A client message above 65,536 bytes closes 1009; the initial top-level session.update may use up to 524,288 bytes.
  • Fatal in-session errors are followed by a close frame.
  • Log the close code and reason, and reconnect with backoff for temporary capacity, backend, or network conditions.

Handling transcripts​

  • Key provisional text by utterance_id.
  • Replace the active partial when a newer asr.partial_result arrives.
  • On asr.final_result, remove the replaces IDs and commit the final text.
  • Attach translated text by source_result_id; do not assume target-language result order.
  • Ignore empty final text for display (keep it for diagnostics).
  • Never render session.usage as transcript text.

Next steps​