I have been working on a browser-based video editor for a while now, and as a result I have run into plenty of video-related problems. While documenting as many of them as I could remember, I came back to one of my favourites: a problem for which I really struggled to find a satisfying solution.

Simply stated, I discovered that my implementation could require roughly twice the size of a video file in RAM just to optimize an MP4 for fast start.

Optimizing a video for fast start matters when serving video on the web because it allows playback to begin before the entire file has been downloaded. This can improve the user experience and, when the video is important to the initial page render, can also affect loading performance such as Largest Contentful Paint. Fast start itself is not a Google ranking factor, although Core Web Vitals are used by Google's ranking systems.

But what exactly does fast start mean?

In a normal MP4, the file contains a box called moov. You can think of it as the index of the video. It contains metadata that tells a player where the encoded samples are located, how large they are, when they should be decoded and presented, and which samples are keyframes.

For fast start, we want that index near the beginning of the file rather than at the end. FFmpeg's faststart option does exactly this by running a second pass and moving the moov box to the beginning.

Without this optimization, a player may need to fetch the end of the file before it has enough information to start playback. With moov near the front, the player can discover the structure of the video early and begin requesting and decoding the media it needs.

To understand why moving this box is more complicated than it sounds, we need a few MP4 fundamentals.

A few MP4 fundamentals

An MP4 is made up of boxes. A real file can contain many of them, but three are particularly useful for this explanation:

ftyp, which describes the file format and compatibility;
mdat, which contains the encoded media data;
moov, which contains the metadata describing that media.

There is another complication: encoded video is not necessarily processed in the same order in which we eventually watch it.

A video stream can have a decode order and a presentation order. Some frames depend on other frames, including frames that may appear later in playback. The decoder therefore sometimes needs to process frames in one order and display them in another.

The container metadata is what allows all of this to be reconstructed. Among other things, the tables inside moov tell the demuxer where samples live in the file and how they relate to time. This is why we cannot simply jump to an arbitrary byte offset, assume it contains H.264, and start decoding meaningful video from there.

A straightforward MP4 muxer can therefore do something very convenient: write the media data first, learn where everything ended up, and write the moov metadata afterwards. By then all of the offsets and timing information are known.

Unfortunately, that gives us exactly the layout we do not want for fast start.

So we move moov to the front.

Moving moov breaks the offsets

And this creates an amusing problem: moov itself contains offsets pointing to the media data. Once we insert moov before that media data, those offsets are no longer correct.

Suppose some media previously started at:

header + X

After inserting a moov box of size M before it, that same data is now roughly at:

header + M + X

The relevant chunk offsets inside the metadata therefore have to be rewritten to point to their new positions. This is an important detail of the format: chunk offsets are absolute file offsets, so moving metadata in front of the media changes them.

The algorithm itself is conceptually simple. Read the existing structure, move moov, adjust the affected offsets, and write the rearranged MP4.

Where the new file gets written

The interesting problem for me was where the new file gets written.

The usual fast-start implementation is a second pass that produces rearranged output. If the original file is kept while the new one is being written, peak storage approaches the size of the input plus the size of the output. In other words, roughly twice the video size.

On a normal machine, that usually means temporary disk usage.

I was doing it with an FFmpeg WASM build whose filesystem lived in WASM-backed memory.

That changed the problem completely.

By the time the last byte of the optimised video had been written, I could have both the original file and almost an entire second copy sitting in memory at the same time. In practice the process needs additional working memory as well, so the true peak can be even higher.

For a 100 MB video this is annoying.

For a multi-gigabyte video it becomes an architectural problem.

And because this editor runs entirely inside the browser, the machine doing the work belongs to the user. Memory is limited, browser environments have their own constraints, and the situation becomes even more restrictive on mobile devices, which I also want to support.

That was the part I originally underestimated. Moving one piece of metadata from the end of a file to the beginning sounds almost trivial. Doing it when the file might itself be larger than the memory available to your application is a very different problem.

I am glad I ran into it, because otherwise I probably would never have dug this far into how MP4 files are actually laid out.

What I did instead

I eventually reworked the pipeline around WebCodecs together with browser file APIs, which allowed me to rely on file-backed storage instead of keeping the entire transformation inside FFmpeg WASM's in-memory filesystem. WebCodecs itself handles encoding and decoding; browser file APIs are what provide the file-backed part.

That solution, though, deserves its own article.

If you just want the practical side of fast start, and how to check whether your own videos are optimised, that is covered in How to Optimize Video for Web.