Media, fonts and models

What imports

The engine reads the first sixteen bytes of a file to find its container. The file name does not matter. Decoding uses the browser's WebCodecs, so the codec must be one the browser decodes.

Kind	Containers	Codecs	Read by
Video	MP4, MOV, M4V	H.264, H.265, VP9, AV1	mp4box; mediabunny when mp4box fails
Video	WebM, MKV	VP8, VP9, AV1, H.264, where the browser decodes them	mediabunny
Sound of a video, or an M4A	MP4, MOV, M4A	AAC, MP3, FLAC, Opus, AC-3, where the browser decodes them	mp4box and WebCodecs
Audio files	MP3, WAV, Ogg, FLAC, WebM, MKV	what the browser decodes	mediabunny and WebCodecs, then the browser's decodeAudioData
Images	PNG, JPEG, WebP, GIF, BMP, SVG	none	the browser's image decoder

Things to know:

• H.264 decodes everywhere. H.265 depends on the browser and the hardware. Firefox refuses it, and the error names Chrome or Edge.
• A phone video with a rotation tag plays upright. The rotation is applied when drawing, never baked into the file.
• A fragmented MP4 (written with moof boxes) can import with fewer frames than it holds.
• A sound file is decoded whole into memory, about ten megabytes per minute of stereo.
• A GIF imports as a still picture, its first frame. An image clip lasts five seconds unless you pass durationSec.

The editor's file pickers accept fewer types than the engine does. The video picker lists MP4 and WebM. The audio picker lists any audio type. The image picker lists PNG, JPEG, WebP, SVG, GIF and BMP. The backend methods importVideo, importAudio and importImage accept any file the engine can read.

Proxies

A proxy is a lighter copy of a video, made for the preview. The engine makes one in the background after an import and when a project opens.

Rule	Value
Made when the video is wider than	1280 pixels
Or taller than	720 pixels
Or when the keyframes are further apart than	2 seconds on average
Never on	iPhone, iPad, Android

Clips read the original until the proxy is done, then switch to it. Export and transcription always read the original. Progress arrives as the proxy-progress event, from 0 to 1, then null when the proxy is finished, failed or stopped. When a proxy fails, the original keeps playing. Nothing else changes.

Media from a web address

importFromUrl downloads a file into file storage as a stream. It then adds the file to the project's media, like a picked file.

import type { EditorBackend } from "@vidstudio/engine";

export async function addFromServer(backend: EditorBackend): Promise<void> {
  const asset = await backend.importFromUrl("https://media.example.com/clips/intro.mp4");
  await backend.addTrackFromAsset(asset.id, "video");
  const sound = await backend.importFromUrl("https://media.example.com/clips/intro.mp4", { type: "audio" });
  await backend.addTrackFromAsset(sound.id, "audio");
}

type: "audio" keeps only the sound of a video. type: "video" on a file without video rejects with the code type-mismatch. A failed download rejects with import-failed. The file takes its name from the last part of the address. A name without an extension gets one from the content type. The download sends no cookies to another origin.

An address on your own origin needs nothing. An address on another origin needs headers from that server.

Your page	Headers the file server must send
Without cross-origin isolation	Access-Control-Allow-Origin: https://your-site.example (or *)
With cross-origin isolation (the two headers from Getting started)	The header above, plus Cross-Origin-Resource-Policy: cross-origin

The same rules apply to a web address returned by a resolver in Save and load.

Fonts

Text clips use twelve fonts, one regular weight each, as .woff2 files under node_modules/@vidstudio/engine/assets/fonts. All twelve are under the SIL Open Font License, included as OFL.txt.

Category	Fonts
Sans serif	Inter (the default), Roboto, Montserrat, Open Sans, Poppins
Serif	Playfair Display, Lora
Display	Bebas Neue, Oswald
Handwriting	Dancing Script, Pacifico
Monospace	JetBrains Mono

The editor loads a font the first time a clip uses it, from /fonts/ by default. Copy the folder there, or serve it elsewhere and set options.assets.fonts to that address, ending in a slash. A font that fails to load logs a warning, and the text draws in the browser's fallback font.

cp -r node_modules/@vidstudio/engine/assets/fonts public/fonts

Models

Captions come from Whisper speech models that run on the device. The editor picks the model by device. Both are English only: the .en models accept no other language.

Device	Model	Size
Desktop	onnx-community/whisper-base.en	294 MB
iPhone, iPad, Android	onnx-community/whisper-tiny.en	154 MB

vidstudio-fetch-models puts the files into a folder you serve. It downloads from Hugging Face at pinned revisions and skips files already present. It writes the Apache 2.0 licence and a notice file beside them.

npx vidstudio-fetch-models public/models

Flag	Effect
--models base.en	Only the models named, comma separated
--models all	Every model in the manifest, including three word-timestamped variants the editor does not use
--from <folder>	Copy from an existing models folder instead of downloading
--no-whisper	Skip the speech models
--no-mediapipe	Skip the background removal files

The engine loads the models from /models/ by default. Serve the folder elsewhere and set options.assets.models, ending in a slash. With a full web address, the files are fetched on first use and cached in the browser's cache storage. With features={{ captions: false }} the editor never downloads a model.

Transcription also loads the onnxruntime runtime files from cdn.jsdelivr.net on first use. A page that is offline, or whose content security policy blocks that address, cannot transcribe.

Background removal files

vidstudio-fetch-models also writes mediapipe/wasm/ and mediapipe/selfie_segmenter.tflite, about 19 MB together, under the models folder. The editor's effect list has no background removal today. The engine draws the effect only for a loaded document that already carries an effect of type background-removal. Such an export runs on the page thread instead of the worker. Pass --no-mediapipe if you never load such a document.