Live Translation, Real-Time Multimodal Interpretation Across 60 Languages with 2.8s Latency

Real-time speech translation is one of the toughest challenges in applied AI. The model must begin translating before the speaker has even finished their sentence. Any added delay shatters the sense of seamless, live communication. Alibaba’s Qwen team has been steadily improving with each new release. Their newest model, Qwen3.5-LiveTranslate-Flash, cuts that latency to just 2.8 seconds and broadens language support to 60 input languages.

A Meaningful Jump From the Previous Release

The earlier Qwen3-LiveTranslate-Flash supported 18 input languages with roughly three seconds of delay. Qwen3.5-LiveTranslate-Flash trims that to 2.8 seconds, widens input coverage to 60 languages, and introduces speech output in 29 languages. That represents more than a threefold increase in input language support. For developers building multilingual applications, this largely eliminates the need to swap between different language-specific models in most global enterprise use cases.

The reduced latency stems from a method the team calls “reading unit” processing. Instead of holding off until a complete sentence arrives before generating a translation, the model determines when a segment has accumulated enough meaning to commit to an output. It then streams results continuously while the speaker is still mid-sentence. This follows the same principle as semantic unit prediction but with a more refined implementation that trims an additional 200 milliseconds.

Vision Is Now a First-Class Input

Most translation systems rely solely as their input signal. That works well in controlled studio environments. But it falls apart in a packed conference hall, a bustling trading floor, or any setting with overlapping voices and poor acoustics.

Qwen3.5-LiveTranslate-Flash takes a different path. It processes visual cues alongside audio — on-screen text, objects being shown, lip movements, and hand gestures. When a word sounds ambiguous or the audio quality drops, the visual information steps in and helps the model make a more accurate translation. This is far from a minor addition. In real-world deployments, clean audio can never be taken for granted. Having a visual channel means the model copes with the unpredictable nature of live interpretation far better than audio-only alternatives.

Voice Cloning Happens in Real Time

This is the standout feature of the Qwen3.5 release. Conventional translation systems swap the speaker’s voice with a generic synthetic one. Qwen3.5-LiveTranslate-Flash, by contrast, replicates the original speaker’s distinctive vocal characteristics during the translation process itself. A single sentence of spoken input is all the model needs to perform this acoustic adaptation.

For the audience on the other end, the translated speech sounds as though the same person is speaking in the target language — not a robotic stand-in. Whether it’s live conference interpretation, multilingual livestreams, or international customer support calls, this makes a real difference. The overall experience feels noticeably more natural and human compared to what existing systems offer.

Configure Domain-Specific Keywords

One recurring weakness of translation models in professional contexts is their handling of proper nouns and specialized terminology. A model interpreting a medical briefing might repeatedly mistranslate a drug name. A legal session could stumble over a technical statute reference.

Qwen3.5-LiveTranslate-Flash tackles this with dynamic keyword configuration at runtime. Developers can supply a glossary of brand names, medical terms, legal phrases, or technical jargon, and the model handles those terms with significantly greater accuracy. This capability is absent from most general-purpose translation tools.

Here is the paraphrased version of the HTML content, with the text rewritten for clarity and ease of reading while preserving the original structure and language:

APIs and it addresses a genuine need for specialized enterprise deployments.

Benchmark Performance

When evaluated on FLEURS and CoVoST2 — two widely recognized benchmarks for multilingual speech translation — Qwen3.5-LiveTranslate-Flash surpasses leading commercial alternatives. FLEURS assesses translation accuracy across numerous language pairs under realistic acoustic conditions. CoVoST2 includes 21 speech translation directions, serving as a reliable indicator of multilingual pipeline performance.

Marktechpost’s Visual Explainer

# Install dependencies
pip install websocket-client pyaudio

# Set your API key as an environment variable
export DASHSCOPE_API_KEY="your_key_here"

import json, websocket, os

API_KEY = os.getenv("DASHSCOPE_API_KEY")
API_URL = (
    "wss://dashscope-intl.aliyuncs.com"
    "/api-ws/v1/realtime"
    "?model=qwen3-livetranslate-flash-realtime"
)

def on_open(ws):
    print("Connected to Qwen3.5-LiveTranslate-Flash")

def on_message(ws, message):
    data = json.loads(message)
    print("Translation event:", data)

def on_error(ws, error):
    print("Error:", error)

ws = websocket.WebSocketApp(
    API_URL,

[
    "Authorization: Bearer " + API_KEY],
    on_open=on_open,
    on_message=on_message,
    on_error=on_error
)
ws.run_forever()

import base64, pyaudio

# Audio input settings: 16kHz, 16-bit PCM mono
INPUT_RATE    = 16000
INPUT_CHUNK   = 1600  # 100ms per chunk
INPUT_FORMAT  = pyaudio.paInt16
INPUT_CHANNELS = 1

def on_open(ws):
    # 1. Send session configuration first
    session_cfg = {
        "type": "session.update",
        "session": {
            "input_audio_transcription": {
                "language": "zh"  # source: Chinese
            },
            "translation": {
                "target_language": "en"  # target: English
            }
        }
    }
    ws.send(json.dumps(session_cfg))

    # 2. Stream audio from the microphone
    pa = pyaudio.PyAudio()
    stream = pa.open(
        rate=INPUT_RATE, channels=INPUT_CHANNELS,
        format=INPUT_FORMAT, input=True,
        frames_per_buffer=INPUT_CHUNK
    )
    while True:
        chunk = stream.read(INPUT_CHUNK)
        audio_b64 = base64.b64encode(chunk).decode()
        ws.send(json.dumps({
            "type": "input_audio_buffer.append",
            "audio": audio_b64
        }))

import cv2, base64, threading, time

def stream_video_frames(ws):
    cap = cv2.VideoCapture(0)  # 0 = default camera
    while True:
        ret, frame = cap.read()
        if not ret:
            break
        # Encode frame as JPEG → base64
        _, buf = cv2.imencode(".jpg", frame)
        img_b64 = base64.b64encode(buf).decode()
        ws.send(json.dumps({
            "type": "input_image_buffer.append",
            "image": img_b64
        }))
        time.sleep(0.5)  # ~2fps is sufficient

# Run video streaming in a separate thread
threading.Thread(
    target=stream_video_frames,
    args=(ws,), daemon=True
).start()

# Add to your session.update payload
session_cfg = {
    "type": "session.update",
    "session": {
        "input_audio_transcription": {
            "language": "zh"
        },
        "translation": {
            "target_language": "en"
        },
        # Inject domain keywords here
        "keywords": [
            {"source": "达芬奇机器人",  "target": "da Vinci Surgical System"},
            {"source": "腹腔镜",      "target": "laparoscope"},
            {"source": "实体瘤",      "target": "solid tumor"}
        ]
    }
}