Benchmarks
Kiln and transformers.js 4.3.2 in the same browser on the same machine, on the same inputs, both on WebGPU in fp32 unless noted (transformers.js runs the repos' ONNX exports, Kiln the original weights). Times are end to end, the median of three warm calls after one warm-up.
Speed
How many times faster Kiln is (transformers.js's time over Kiln's, or Kiln's tokens per second over transformers.js's). The dashed line is parity. Hover or focus a bar for both timings.
All measurements
Where Kiln is slower
- Loading from the cache: 1.8 s against 0.9 s for SmolLM2-135M when the two were measured together. Kiln now caches its converted weights, which about halves its cached loads (1.0–1.9 s against 2.0–3.4 s in a later, slower session); the pair has not been measured together since.
- Download size: transformers.js's ONNX repos ship pre-quantized files, so a q4 model downloads less; Kiln downloads the original weights once and quantizes them.
- Masked-LM logits through
AutoModelForMaskedLM: 0.15 s against 0.11 s (transformers.js on WASM) for nine tokens (0.24 s before the head was batched). Thefill-maskpipeline, which reads one position, is 2× faster. - The runtime itself is smaller: Kiln is 241 KB gzipped with its worker; transformers.js adds 6.7 MB of WASM.
Accuracy
Speed only counts if the outputs are right. Each model was checked against transformers.js on the same inputs, and many against Python's transformers or an independent NumPy forward pass:
| Model | Checked against | Result |
|---|---|---|
| LLMs (f32) | transformers.js; a NumPy reference (Gemma 3) | identical greedy tokens |
| BERT family | Python transformers | scores equal to six decimals |
| Tokenizers, 31 repos | transformers.js 4.3.2 | identical ids |
| Whisper | transformers.js | same text, segments and word timestamps; features within 1.2e-7 |
| ViT, DeiT, CLIP | transformers.js, same pixels | probabilities within 2.2e-6 |
| DETR, YOLOS | transformers.js, same pixels | same detections, boxes within 2.3e-6 |
| RMBG-1.4, SegFormer, Depth Anything | transformers.js, same pixels | a few output bytes one level apart (at most 20 of 307,200) |
| OWLv2 | Python's documented output | 0.665 / 0.613 against 0.665 / 0.614, same boxes |
| Donut DocVQA | transformers.js | same answers to 8 questions on 5 pages |
| SmolVLM | transformers.js, same tiles | same greedy replies in 17 runs |
| VITS | transformers.js, noise off | 33,024 samples within 1.4e-4 |
| LoRA adapters | NumPy, adapters merged | same greedy tokens |
| Image preprocessing | PIL + NumPy | identical pixel values |
Some transformers.js exports differ from their models (Florence-2's attention scale, Kokoro's loudness, OWL models that fail to load in 4.3.2); the technical notes explain each case.
Setup
- An Intel Mac with Iris Plus Graphics 650 (an integrated GPU), Chrome. A modest GPU: discrete GPUs and Apple silicon are faster for both libraries.
- On an Apple M3 (MacBook Air, Chromium), Kiln decodes SmolLM2-135M in q4 at about 307 tokens per second, against 59–74 on the Intel GPU; transformers.js was not measured there.
- GPU clocks on this machine drift 10–15% between runs, so each pair was measured in one session.
- The scripts and raw notes behind every row are in the repository's technical notes (docs/technical-notes.md), with the outputs compared.