Realtime ASR (WebSocket)
Realtime ASR streams audio over a WebSocket and returns transcripts as you speak — provisional partial results while audio is still flowing, then authoritative final results per utterance. Use it for live captioning, meeting transcription, and voice interfaces.
For one-shot transcription of a complete audio file, use the batch ASR endpoint instead.
Endpoint
wss://api.shisa.ai/ws/asr/realtime
Authenticate during the WebSocket handshake with your ASR bearer token:
Authorization: Bearer YOUR_API_KEY
Realtime ASR is public to normally authorized active API keys. If your key uses an allowedServices allowlist, include shisa/asr-realtime; deniedServices still takes precedence. The default limit is 3 concurrent WebSocket sessions per API key. Individual keys may be configured with a different limit.
Realtime ASR is designed for backend use. Keep your API key server-side: relay audio from the browser to your own server, and connect to Shisa from there.
How it works
- Open a WebSocket to the endpoint with the
Authorizationheader. - Send one
session.updatemessage to set the audio format and language. - Wait for
session.created. - Stream raw audio as base64
input_audio.appendmessages (50–250 ms per chunk is a good starting point). - Show
asr.partial_resultas live provisional text; replace it whenasr.final_resultarrives. - Send
session.closewhen finished.
Audio requirements
Send raw PCM — no container, no compression:
| Property | Value |
|---|---|
| Encoding | pcm_s16le (signed 16-bit little-endian) |
| Sample rate | 16000 Hz |
| Channels | 1 (mono) |
| Chunk cadence | 50–250 ms recommended (examples use 100 ms) |
Resample and downmix in your client before sending. Do not send WAV headers, Ogg, MP3, data URLs, float PCM, or stereo audio.
Client messages
session.update
Send exactly one session.update before any audio. A fixed-language session is the simplest option:
{
"type": "session.update",
"session": {
"input_audio_format": "pcm_s16le",
"sample_rate": 16000,
"channels": 1,
"language": "ja"
}
}
| Field | Required | Values | Notes |
|---|---|---|---|
input_audio_format | Yes | pcm_s16le | Raw signed 16-bit little-endian PCM. |
sample_rate | Yes | 16000 | Resample to 16 kHz before sending. |
channels | Yes | 1 | Mono only. Downmix stereo first. |
language | Yes | ja, en, zh, auto | Use a fixed language when you know it. Use auto for automatic language identification (see below). |
default_language | Auto only | ja, en, zh | Fallback when detection is not confident. |
language_detection_mode | Auto only | session, utterance | session detects once; utterance detects per segment (advanced — see below). |
hotwords | No | Array of strings | Global finalization vocabulary; up to 50 items (separate from the language-mapped limits). |
hotwords_by_language | No | Object or null | Finalization vocabulary keyed by resolved language; up to 5 keys, 100 items per language, and 300 mapped items in total. |
Hotwords
Hotwords bias finalization toward your vocabulary — product names, jargon, and proper nouns. Configure them once, in the initial session.update; they apply to every utterance in that WebSocket session. To change them, close the session and open a new one with a new session.update. A typical call-center integration uses one WebSocket session per customer call, not one session per conversational turn.
A fixed-language session is the simplest setup:
{
"type": "session.update",
"session": {
"input_audio_format": "pcm_s16le",
"sample_rate": 16000,
"channels": 1,
"language": "ja",
"hotwords": ["シーサー", "Shisa AI"]
}
}
Hotwords affect only asr.final_result. Live asr.partial_result events may still contain an uncorrected spelling; replace the displayed partial text when the final result arrives. Service events never echo the configured values.
Limits
The global hotwords list and the language-mapped lists have separate limits:
| Configuration | Limit |
|---|---|
Global hotwords | 50 items |
| Language keys | 5 |
| Items per language | 100 |
| Total language-mapped items | 300 |
| Length per item | 1–100 Unicode characters |
Each item must be unique within its list (case-insensitively) and contain 1–100 Unicode characters (technically, Unicode scalar values), with no leading/trailing whitespace or control characters. The service rejects invalid configuration rather than trimming, normalizing, truncating, or deduplicating values.
Language-specific hotwords (advanced)
A session may combine global fallback hotwords with language-specific lists in its single initial session.update:
{
"type": "session.update",
"session": {
"input_audio_format": "pcm_s16le",
"sample_rate": 16000,
"channels": 1,
"language": "auto",
"default_language": "en",
"language_detection_mode": "utterance",
"hotwords": ["global fallback"],
"hotwords_by_language": {
"ja": ["シーサー"],
"en": ["Shisa AI"],
"zh": ["狮萨"]
}
}
}
After fixed-language or LID resolution, a non-empty exact matching mapped list wins; an absent or empty matching list falls back to hotwords. Keys must be exact lowercase ASCII two-letter codes; use canonical ja, en, and zh for current deployments.
Availability of hotwords_by_language may depend on the selected deployment. If it is unavailable, the session returns an invalid_config error. In that case, use the global hotwords field or contact support.
Message-size policy
The router accepts ordinary client messages up to 65,536 bytes. The initial session.update may be up to 524,288 bytes when its sole top-level type is exactly session.update; this accommodates bounded fields such as hotwords_by_language. Limits apply to the complete assembled WebSocket message in UTF-8 wire bytes.
Other client messages above 65,536 bytes, or any client message above 524,288 bytes, close with 1009 and are not forwarded. Backend-to-client messages remain limited to 65,536 bytes. Operators may configure stricter limits, so serialize compactly and treat 1009 as a terminal session error.
input_audio.append
Stream audio chunks as base64-encoded PCM:
{
"type": "input_audio.append",
"audio": "<base64 pcm_s16le bytes>"
}
session.close
Close the session gracefully when done:
{ "type": "session.close" }
Language identification (optional)
If you don't know the spoken language ahead of time, set language: "auto". Two modes are available.
Session detection identifies the language once, near the start, and uses it for the whole session:
{
"type": "session.update",
"session": {
"input_audio_format": "pcm_s16le",
"sample_rate": 16000,
"channels": 1,
"language": "auto",
"default_language": "en",
"language_detection_mode": "session"
}
}
The session emits session.language_detecting, then one session.language_detected carrying the chosen language.
Per-utterance detection (language_detection_mode: "utterance") detects the language of each utterance separately, which is useful for mixed-language audio. It emits utterance.language_detected per utterance. This mode must be enabled on the backend; if it isn't, the session returns an error with code: "invalid_config" — fall back to a fixed language or session detection in that case.
When detection can't choose confidently, it falls back to default_language, and the event carries source: "default" with a reason (for example no_speech, short_utterance, inconclusive, timeout, unsupported, queue_full, buffer_limit, session_close, shutdown, vad_error, or lid_error). Always treat asr.final_result.text as authoritative — don't rewrite a transcript based on a language event alone.
Router-managed translation (optional)
The router can translate each non-empty asr.final_result asynchronously. Your key must have access to both shisa/asr-realtime and shisa/translate; translation is disabled until you opt in for the current WebSocket session.
After session.update, send:
{
"type": "router.translation.update",
"id": "translate-en",
"enabled": true,
"target_langs": ["en"]
}
| Field | Required | Notes |
|---|---|---|
enabled | Yes | Boolean. Set false to disable translation; target languages are then omitted. |
target_langs | When enabling | One to three unique, valid language codes. |
id | No | Client correlation ID, echoed in the control response; maximum 36 characters. |
context | No | Context supplied to every translation; maximum 2,000 Unicode codepoints. |
keywords | No | Glossary array; maximum 20 entries and 100 bytes per entry. |
An accepted update returns:
{
"type": "router.translation.updated",
"id": "translate-en",
"enabled": true,
"target_langs": ["en"]
}
Invalid configuration returns router.translation.error, for example:
{
"type": "router.translation.error",
"id": "translate-en",
"code": "too_many_target_langs",
"message": "At most 3 target languages are allowed"
}
Configuration error codes include invalid_json, invalid_message_type, invalid_id, id_too_long, invalid_enabled, missing_target_langs, too_many_target_langs, duplicate_target_lang, invalid_target_lang, context_too_long, too_many_keywords, keyword_too_long, translation_access_denied, and translation_service_unavailable. Other reserved router.* messages return router.error with unknown_router_message or router_message_too_large.
For each final, the router resolves the source language from the first valid value in this order: (1) asr.final_result.language, (2) the latest session.language_detected, then (3) the fixed language from session.update.session.language. It does not directly consume a standalone utterance.language_detected event for translation. If no valid language is known when a final arrives, that target receives translation.error with code: "source_language_unknown".
Service events
All events are JSON text messages. After the session is created, ASR backend events carry a session_id and a monotonically increasing seq. Router-generated router.* and translation.* events do not carry those fields; correlate translation results and errors to ASR finals with source_result_id.
session.created
The session is accepted; you may start sending audio:
{
"type": "session.created",
"session_id": "asr_sess_...",
"seq": 1,
"format": { "encoding": "pcm_s16le", "sample_rate": 16000, "channels": 1 }
}
Speech boundaries
speech_started and speech_stopped mark voice-activity endpoints for each utterance:
{
"type": "speech_started",
"session_id": "asr_sess_...",
"seq": 4,
"utterance_id": "utt_0001",
"audio_start_ms": 420
}
{
"type": "speech_stopped",
"session_id": "asr_sess_...",
"seq": 9,
"utterance_id": "utt_0001",
"audio_start_ms": 420,
"audio_end_ms": 2860,
"reason": "vad_endpoint"
}
Common stop reasons are vad_endpoint, max_buffer, and session_close.
asr.partial_result
Provisional text for an in-progress utterance. Display the latest partial per utterance_id, replacing the previous one — don't append every partial to history:
{
"type": "asr.partial_result",
"session_id": "asr_sess_...",
"seq": 7,
"utterance_id": "utt_0001",
"result_id": "utt_0001:p3",
"text": "今日の会議は3時から",
"is_final": false,
"audio_start_ms": 420,
"audio_end_ms": 2260
}
asr.final_result
Authoritative text for the utterance. Use replaces to drop the provisional partials you were displaying:
{
"type": "asr.final_result",
"session_id": "asr_sess_...",
"seq": 12,
"utterance_id": "utt_0001",
"result_id": "utt_0001:final",
"replaces": ["utt_0001:p1", "utt_0001:p2", "utt_0001:p3"],
"replaces_audio_range_ms": [420, 2860],
"text": "今日の会議は3時からです。",
"language": "ja",
"is_final": true,
"audio_start_ms": 420,
"audio_end_ms": 2860
}
Finals are asynchronous and can arrive after later speech events. Very long utterances may be split into continuation finals that carry continuation_of, overlap_mode, leading_context_ms, and payload_audio_range_ms. audio_start_ms/audio_end_ms describe the logical utterance range; payload_audio_range_ms includes the overlapped context sent to final ASR, and leading_context_ms is the context duration before audio_start_ms. Display only the logical range.
translation.final_result
When translation is enabled, the router emits one result per target language for each translatable ASR final:
{
"type": "translation.final_result",
"utterance_id": "utt_0001",
"source_result_id": "utt_0001:final",
"source_text": "今日の会議は3時からです。",
"text": "Today's meeting starts at 3 o'clock.",
"source_lang": "ja",
"target_lang": "en",
"model": "shisa-v2.1-unphi4-14b"
}
Attach the result to the exact ASR final identified by source_result_id. Translation is asynchronous, and results for multiple target languages may arrive in any order. model is the model ID reported by the backend response and may differ from the default shisa-ai/chotto request alias.
translation.error
A failed target emits a target-specific error rather than closing the ASR session:
{
"type": "translation.error",
"code": "source_language_unknown",
"message": "Cannot translate ASR final result before source language is known",
"utterance_id": "utt_0001",
"source_result_id": "utt_0001:final",
"target_lang": "en"
}
Other common codes include same_source_target_lang, translation_access_denied, translation_service_unavailable, translation_rate_limited, translation_rate_limiter_unavailable, translation_failed, and source_text_too_long. Empty final text is not translated and emits neither a translation result nor a translation error.
session.usage
Cumulative usage; billing uses the latest event. billable_duration is in whole seconds. The top-level model is the deployment-selected finalization model used for accounting/debugging; clients do not choose the realtime partial engine:
{
"type": "session.usage",
"session_id": "asr_sess_...",
"seq": 20,
"final": true,
"model": "shisa-ai/shisa-asr",
"usage": {
"input_duration": 18.3036875,
"billable_duration": 19,
"input_bytes": 585718,
"sample_rate": 16000,
"channels": 1,
"utterance_count": 3,
"final_result_count": 3,
"final_error_count": 0,
"final_no_speech_count": 0,
"partial_result_count": 17,
"error_count": 0,
"status": "completed",
"final": true
}
}
Handling errors
A healthy session connects, streams, and closes without any error events. When something does need your attention, the API returns a clear code and, where possible, tells you what to do next.
Errors can appear at two points: while the connection is being established, as an HTTP response with an error object; or during the WebSocket session, as a JSON event with type: "error".
If the connection is refused
If the upgrade is not accepted, no WebSocket is created. The response is a normal HTTP error:
{
"error": {
"code": "service_access_denied",
"message": "Service access denied"
}
}
The common causes are easy to fix:
| HTTP | Code | What to do |
|---|---|---|
401 | auth_token_required | Send an Authorization: Bearer <key> header. |
401 | invalid_token_format | Use the full shsk:... key, including the prefix. |
401 | invalid_token | Check that the key is current and active. See Authentication. |
401 | legacy_api_key_rejected | Replace the legacy key with a current API key. |
403 | service_access_denied | Add shisa/asr-realtime to the key's allowedServices, or remove it from deniedServices. |
403 | ws_origin_denied | Connect from an allowed origin, or omit the Origin header from server-side clients. |
429 | rate_limited, service_rate_limited | Slow down and retry with backoff. See Rate limits. |
429 | ws_connection_limit_exceeded | Close idle sessions or wait for one to finish. Individual keys may be configured with a different limit; the error body reports the applicable limit and current active count. |
Occasionally the service is briefly unable to accept new sessions — for example during capacity changes or maintenance. These conditions are temporary and return 503; retry with backoff. They are not caused by your request.
Temporary service codes
| HTTP | Code | Meaning |
|---|---|---|
503 | auth_unavailable | API key verification is temporarily unavailable. |
503 | rate_limiter_unavailable, service_rate_limiter_unavailable | Rate-limit checks are temporarily unavailable. |
503 | ws_connection_quota_unavailable | Connection-quota tracking is temporarily unavailable. |
503 | backends_at_capacity, backend_unavailable | No realtime ASR capacity is available right now. |
503 | websocket_draining | The service is finishing a planned restart and is not accepting new sessions. |
503 | service_rate_limit_policy_invalid | The key's rate-limit configuration needs a service-side update. Retrying unchanged will not help; contact support. |
During a session
After a successful upgrade, the backend forwards JSON error events. Most apply to a single utterance and let the session continue; only fatal: true ends the session. Handle unknown code values defensively, because the backend may add codes over time:
{
"type": "error",
"code": "finalization_failed",
"message": "Finalization failed",
"fatal": false,
"session_id": "asr_sess_...",
"seq": 18,
"utterance_id": "utt_0002"
}
| Code | What to do |
|---|---|
invalid_json, invalid_state | Check message ordering: send exactly one session.update before audio, and use known message types. |
invalid_audio | Send raw pcm_s16le at 16 kHz mono; do not send headers or compressed audio. |
invalid_config | Check the selected language/LID mode and validate hotwords against the documented count, length, uniqueness, and character rules. If hotwords_by_language is unavailable on the deployment, use global hotwords or contact support. |
frame_too_large | Send a smaller individual message or audio chunk. |
limit_exceeded | A session, buffer, recognizer backlog, or queue limit was reached. Stop or close the current session as appropriate, then retry with backoff. If it persists, contact support. |
finalization_timeout, finalization_cancelled, finalization_failed | Retry the utterance. If it keeps happening, contact support. |
service_error | A temporary service-side issue. Retry; contact support if it persists. |
fatal: true | The session ends after this event. Reconnect and resume. |
Connection close behavior
- A refused connection is a normal HTTP response; no WebSocket is created.
- Invalid language-specific hotword configuration emits an
invalid_configevent followed by close1008. - A client message above 65,536 bytes closes
1009; the initial top-levelsession.updatemay use up to 524,288 bytes. - Fatal in-session errors are followed by a close frame.
- Log the close code and reason, and reconnect with backoff for temporary capacity, backend, or network conditions.
Handling transcripts
- Key provisional text by
utterance_id. - Replace the active partial when a newer
asr.partial_resultarrives. - On
asr.final_result, remove thereplacesIDs and commit the final text. - Attach translated text by
source_result_id; do not assume target-language result order. - Ignore empty final text for display (keep it for diagnostics).
- Never render
session.usageas transcript text.
Next steps
- Transcribe complete files with the batch ASR endpoint.
- Review supported formats and languages in Audio & languages.
- See how usage is billed on Pricing.