ONNX Runtime Web Puts YOLO, Whisper, and BERT Directly in the Browser

Browser AI definitely can't do without onnxruntime, open-sourced by Microsoft

Browser/Node.js/Bun can run it,

Load ONNX model, inference, get results. Zero backend, zero Python.

onnxruntime-web, official from Microsoft, 23k stars on GitHub.

Quick Start

A YOLO object detection example:

import * as ort from "https://cdn.jsdelivr.net/npm/onnxruntime-web"
const session = await ort.InferenceSession.create("yolov5s.onnx")
const tensor = new ort.Tensor("float32", pixels, [1, 3, 640, 640])
const results = await session.run({ images: tensor })
console.log(results.boxes.data) // Output detection boxes [x1,y1,x2,y2]

More ONNX Models

ONNX is an open-source model format; PyTorch and TensorFlow can both export to it. Runtime Web brings it into the browser for local inference.

The onnx/models repository is packed with ready-made models, contributed by Microsoft, Tencent, Megvii, and others:

Model	Chinese Name	Use Case
MobileNet	Lightweight Classification	Mobile image classification, 14MB, fast enough
SqueezeNet	Compressed Classification	Even smaller than MobileNet, 5MB
ResNet	Deep Residual Network	Classic classification model, high accuracy
YOLOv5	Object Detection	Real-time object bounding boxes
SSD	Single-Stage Detection	Lightweight object detection, speed-first
DeepLabV3	Semantic Segmentation	Pixel-level segmentation, for cutouts
Whisper	Speech Recognition	Audio to text, multilingual
BERT	Text Understanding	Text classification, Q&A, semantic analysis
GPT-2	Text Generation	Dialogue, text continuation, copywriting

Check out the ONNX Runtime Web Demo.

Architecture

Load ONNX → ONNX Runtime Web → Browser Inference

No matter the original framework, exporting to ONNX makes it a universal language. The underlying layer compiles to WebAssembly via Emscripten; the browser becomes the inference engine.

Common Projects Based on onnxruntime-web

The ecosystem is mature; the following are all open-source projects that directly use it.

Whisper Browser Edition

OpenAI's Whisper speech recognition model, converted to ONNX and run in the browser:

const session = await ort.InferenceSession.create("whisper-tiny.onnx")

Whisper Web is official from Microsoft, with complete examples in the GitHub repository.

Real-ESRGAN Image Upscaling

4x super-resolution for images, upscaled directly in the browser:

const session = await ort.InferenceSession.create("real-esrgan.onnx")

Real-ESRGAN-Web is by the original author xinntao, 43k stars on GitHub.

TensorFlow.js Alternative

If you don't want to use TF.js, onnxruntime-web is a drop-in replacement.

ONNX Runtime Web is official from Microsoft, directly replacing TF.js for browser inference.

face-api.js Replacement

Face recognition, detection, expression analysis. ONNX model + runtime-web combo:

const session = await ort.InferenceSession.create("face_detection.onnx")

face-api.js has 53k stars on GitHub, and its underlying layer also uses ONNX inference.

Corresponding previous article: Web-side, 6.5MB face recognition model, Google framework, fast and accurate

Transformers.js

HuggingFace's browser-side Transformer inference library, using onnxruntime-web under the hood:

import { pipeline } from "https://cdn.jsdelivr.net/npm/@huggingface/transformers"
const classifier = await pipeline("text-classification")

Transformers.js has 12k stars on GitHub; BERT and GPT models run directly in the browser.

MediaPipe Web

Google's MediaPipe, for gesture, face, and pose recognition, uses an ONNX inference layer:

import { FilesetResolver, FaceLandmarker } from "@mediapipe/tasks-vision"
const vision = await FilesetResolver.forVisionTasks(CDN_URL)
const faceLandmarker = await FaceLandmarker.create(vision, options)

MediaPipe Web is official from Google, 23k stars on GitHub.

Corresponding previous article: Web-side, 5.5MB Google model, body landmark recognition, no lag even on video