IPFS as a first-class citizen in FFmpeg, who’s next?

FFmpeg (opens new window)is the leading media framework that allows you to watch videos in almost any format. This framework is at the very core of many applications (think, for example, of OBS Studio (opens new window), KODI (opens new window), VLC (opens new window), and even some game engines rely on it). Allowing FFmpeg to handle the IPFS protocol enables IPFS usage in many more applications than previously possible.

This post will explain how native IPFS support in FFmpeg came to be, how it can be used, how it might affect you and what the future could potentially hold!

#IPFS in FFmpeg. How did that happen?

In late 2021 I had an issue on my home server. A relatively simple setup with just one large-capacity HDD running on a ODROID-XU4 running KODI. Many people likely use this as their typical “home server” setup. I happily used it to consume my media until, one day, an OS update broke my network access to the server. Days of debugging couldn’t resolve the issue, which sparked a simple idea: “I already use IPFS; why can’t I use that to play my media?”

#Play media from a gateway

This idea seemed so logical and straightforward to me that I set out to figure out how to play my media via IPFS instead of KODI. It turns out KODI has so-called STRM files (opens new window). You can dump the URL to your media, and it works. While this worked, my URLs now looked like: http://10.0.3.3/ipfs/bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi. IPFS, being a distributed protocol, felt centralized in this case, as I’d have to ask for my video through 1 specific gateway (opens new window). Instead, I wanted the STRM file to look like this: ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi. In this ideal case, a gateway URL would be handled by “something” else internally. In any case, I should not have to specify a gateway. In my mind having this made it possible that a video – any resource really – only has to be found on the IPFS network for it to work. For me, my server could be online, but my data could also be served from anywhere else! A powerful concept; native to IPFS.

But this did not work, KODI is not aware of the IPFS protocol. So this again got me wandering through the KODI codebase to figure out how it handles STRM files and where I might need to make a change to add IPFS support?

#Enter FFmpeg

STRM files aside, it turns out that whatever is in there is eventually handled in FFmpeg (opens new window). Of course, there is more to it than this, but it became clear that getting my ideal way to play video in KODI was diving into the FFmpeg codebase.

In FFmpeg you also have the ffplay utility, which is used to play anything that FFmpeg supports. In this tool, my video would happily play when provided as an HTTP URL:
ffplay http://10.0.3.3/ipfs/bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi
but not as:
ffplay ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi

To me, the most straightforward approach was twofold.

1. Let FFmpeg detect which gateway you use.
2. Rewrite that ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi internally to a http URL for that detected gateway and play the media.

As a proof of concept, I mainly cared about rewriting the URL so that it looks like FFmpeg has native IPFS support from the users’ point of view. However, it turns out that FFmpeg already had quite a few protocols where stacking (or piping) was supported. For example, the crypto (opens new window)protocol (ffplay crypto+file://…) first gets your file and pulls it through the crypto protocol to decrypt it before playing. This piping mechanism verified to me that the IPFS solution was technically feasible. And as FFmpeg already had quite advanced support for HTTP, it seemed conceptually simple enough.

#Hacked together prototype leads to Open Grant

After a mere evening of hacking I already had a very rough prototype working ffplay ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi

While I made rapid progress here, it became clear that further work would potentially require a lot of time. Also, FFmpeg would only be the bottom layer in the media stack – I would still not have IPFS support in at least a couple of media players out there that use FFmpeg under the hood.

I eventually settled in wanting to support IPFS in VLC (opens new window), MPV (opens new window)and KODI (opens new window)and created this open grant (opens new window)for that.

After the Open Grant was approved, I set out to implement IPFS support in FFmpeg properly this time.

Note that anyone can apply for an Open Grant! If you have a great idea you’re passionate about you should consider applying for one. You can find more information on an Open Grants works here (opens new window).

#FFmpeg support leads to gateway detection spec

While implementing IPFS support in FFmpeg it became clear that detecting a potentially running gateway was not trivial. The idea here is that FFmpeg will detect a locally running gateway if it’s run in it’s default settings. This issue on it’s own ended up in drafting a spec for go-ipfs to expose a running gateway in a manner that can be detected by external applications. If you’re interested, this spec can be found here (opens new window).

A visual representation of this spec looks like this:

https://blog.ipfs.io/2022-08-01-ipfs-and-ffmpeg/