← Examples

Streaming chat

A chat that streams tokens as they decode. Each turn reuses the cached prompt of the last one, so only the new message is prefilled.

Qwen2.5-0.5B-Instruct, about 500 MB on first use

The code

const llm = await load("Qwen/Qwen2.5-0.5B-Instruct");
const stream = llm.chat(history, { maxTokens: 400 });
for await (const text of stream) reply.append(text);
(await stream.stats).cachedTokens; // prompt tokens reused from the last turn

Full source: site/src/examples/chat.ts