Guides * [Getting started](/developers) * [Projects and storage](/developers/projects-and-storage) * [Media, fonts and models](/developers/media) * [Save and load](/developers/save-and-load) * [Theme, features and events](/developers/theming-and-events) * [Driving the engine](/developers/engine) * [Limits and support](/developers/limits-and-support) * [Troubleshooting](/developers/troubleshooting) More * [Reference](/developers/reference/) * [Everything as one text file](/developers/llms-full.txt) # Media, fonts and models ## What imports The engine reads the first sixteen bytes of a file to find its container. The file name does not matter. Decoding uses the browser's WebCodecs, so the codec must be one the browser decodes. | Kind | Containers | Codecs | Read by | | --------------------------- | ------------------------------ | ---------------------------------------------------------- | ------------------------------------------------------------ | | Video | MP4, MOV, M4V | H.264, H.265, VP9, AV1 | mp4box; mediabunny when mp4box fails | | Video | WebM, MKV | VP8, VP9, AV1, H.264, where the browser decodes them | mediabunny | | Sound of a video, or an M4A | MP4, MOV, M4A | AAC, MP3, FLAC, Opus, AC-3, where the browser decodes them | mp4box and WebCodecs | | Audio files | MP3, WAV, Ogg, FLAC, WebM, MKV | what the browser decodes | mediabunny and WebCodecs, then the browser's decodeAudioData | | Images | PNG, JPEG, WebP, GIF, BMP, SVG | none | the browser's image decoder | Things to know: * H.264 decodes everywhere. H.265 depends on the browser and the hardware. Firefox refuses it, and the error names Chrome or Edge. * A phone video with a rotation tag plays upright. The rotation is applied when drawing, never baked into the file. * A fragmented MP4 (written with `moof` boxes) can import with fewer frames than it holds. * A sound file is decoded whole into memory, about ten megabytes per minute of stereo. * A GIF imports as a still picture, its first frame. An image clip lasts five seconds unless you pass `durationSec`. The editor's file pickers accept fewer types than the engine does. The video picker lists MP4 and WebM. The audio picker lists any audio type. The image picker lists PNG, JPEG, WebP, SVG, GIF and BMP. The backend methods `importVideo`, `importAudio` and `importImage` accept any file the engine can read. ## Proxies A proxy is a lighter copy of a video, made for the preview. The engine makes one in the background after an import and when a project opens. | Rule | Value | | -------------------------------------------- | --------------------- | | Made when the video is wider than | 1280 pixels | | Or taller than | 720 pixels | | Or when the keyframes are further apart than | 2 seconds on average | | Never on | iPhone, iPad, Android | Clips read the original until the proxy is done, then switch to it. Export and transcription always read the original. Progress arrives as the `proxy-progress` event, from 0 to 1, then `null` when the proxy is finished, failed or stopped. When a proxy fails, the original keeps playing. Nothing else changes. ## Media from a web address `importFromUrl` downloads a file into file storage as a stream. It then adds the file to the project's media, like a picked file. ```ts import type { EditorBackend } from "@vidstudio/engine"; export async function addFromServer(backend: EditorBackend): Promise { const asset = await backend.importFromUrl("https://media.example.com/clips/intro.mp4"); await backend.addTrackFromAsset(asset.id, "video"); const sound = await backend.importFromUrl("https://media.example.com/clips/intro.mp4", { type: "audio" }); await backend.addTrackFromAsset(sound.id, "audio"); } ``` `type: "audio"` keeps only the sound of a video. `type: "video"` on a file without video rejects with the code `type-mismatch`. A failed download rejects with `import-failed`. The file takes its name from the last part of the address. A name without an extension gets one from the content type. The download sends no cookies to another origin. An address on your own origin needs nothing. An address on another origin needs headers from that server. | Your page | Headers the file server must send | | --------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | Without cross-origin isolation | Access-Control-Allow-Origin: https://your-site.example (or \*) | | With cross-origin isolation (the two headers from [Getting started](/developers)) | The header above, plus Cross-Origin-Resource-Policy: cross-origin | The same rules apply to a web address returned by a resolver in [Save and load](/developers/save-and-load). ## Fonts Text clips use twelve fonts, one regular weight each, as `.woff2` files under `node_modules/@vidstudio/engine/assets/fonts`. All twelve are under the SIL Open Font License, included as `OFL.txt`. | Category | Fonts | | ----------- | ----------------------------------------------------------- | | Sans serif | Inter (the default), Roboto, Montserrat, Open Sans, Poppins | | Serif | Playfair Display, Lora | | Display | Bebas Neue, Oswald | | Handwriting | Dancing Script, Pacifico | | Monospace | JetBrains Mono | The editor loads a font the first time a clip uses it, from `/fonts/` by default. Copy the folder there, or serve it elsewhere and set `options.assets.fonts` to that address, ending in a slash. A font that fails to load logs a warning, and the text draws in the browser's fallback font. ``` cp -r node_modules/@vidstudio/engine/assets/fonts public/fonts ``` ## Models Captions come from Whisper speech models that run on the device. The editor picks the model by device. Both are English only: the `.en` models accept no other language. | Device | Model | Size | | --------------------- | ------------------------------ | ------ | | Desktop | onnx-community/whisper-base.en | 294 MB | | iPhone, iPad, Android | onnx-community/whisper-tiny.en | 154 MB | `vidstudio-fetch-models` puts the files into a folder you serve. It downloads from Hugging Face at pinned revisions and skips files already present. It writes the Apache 2.0 licence and a notice file beside them. ``` npx vidstudio-fetch-models public/models ``` | Flag | Effect | | ----------------- | ---------------------------------------------------------------------------------------------- | | \--models base.en | Only the models named, comma separated | | \--models all | Every model in the manifest, including three word-timestamped variants the editor does not use | | \--from | Copy from an existing models folder instead of downloading | | \--no-whisper | Skip the speech models | | \--no-mediapipe | Skip the background removal files | The engine loads the models from `/models/` by default. Serve the folder elsewhere and set `options.assets.models`, ending in a slash. With a full web address, the files are fetched on first use and cached in the browser's cache storage. With `features={{ captions: false }}` the editor never downloads a model. Transcription also loads the onnxruntime runtime files from `cdn.jsdelivr.net` on first use. A page that is offline, or whose content security policy blocks that address, cannot transcribe. ## Background removal files `vidstudio-fetch-models` also writes `mediapipe/wasm/` and `mediapipe/selfie_segmenter.tflite`, about 19 MB together, under the models folder. The editor's effect list has no background removal today. The engine draws the effect only for a loaded document that already carries an effect of type `background-removal`. Such an export runs on the page thread instead of the worker. Pass `--no-mediapipe` if you never load such a document. --- Source: [https://vidstudio.app/developers/media](https://vidstudio.app/developers/media)