Creating a VOD (Video On Demand) Platform with Rails & FFMPEG

If you want the TL;DR version, here's the github link

Intro

Recently, I started working on a project where I wanted to allow users to upload videos, then allow them and other users to playback those videos. I was able to find numerous tutorials showing how to make Youtube clones in rails, but nearly every tutorial went along the lines of "upload video as .mp4, save the video to storage, play back video as mp4".

While this approach can work in limited instances, it has some serious drawbacks. For one, mp4 files have no way to truly be streamed. Sure, there's modern tricks to fragment mp4 files to give the illusion of streaming, but this solution hardly holds up in low bandwidth network settings, and mobile support can be hit or miss.

Instead, the HTTP Live Streaming (HLS), format developed by Apple, provides a way for video to sent to the client in an adaptive format that allows quality to be automatically adjusted depending on the client's internet connection. HLS files are also segmented by default, which provides the benefit of reducing the bandwidth required to start playing video.

The below tutorial for building a rails VOD platform is by no means production ready, but it should serve as a good jumping off point for anyone who's looking to build out a more fully-featured rails video service.

So with that, let's get started.

The version of Ruby & Rails I'm using:

Ruby Version: 3.0.2

Rails Version: 6.1.4.1

For a sample mp4 video, I'm using this 1080p first 30s version of the famous blender made video, big buck bunny

Project Setup

The instructions I'll be providing for this setup run the app inside a docker container.

Originally, I was developing this project on MacOS, but ran into permissions issues with the streamio-ffmpeg gem accessing the system installed version of FFMPEG. Instead of wasting time troubleshooting that, I decided this was a good opportunity to dockerize my app.

To start create a folder with your desired project name, ie:

mkdir rails-vod-example && cd rails-vod-example

Create a Dockerfile in your project root:

touch Dockerfile

Once the Dockerfile has been created, add the following code.

Dockerfile

FROM ruby:3.0.2

RUN apt-get update -qq \
    && apt-get install -y ffmpeg nodejs \
    npm

ADD . /app
WORKDIR /app
COPY Gemfile /app/Gemfile
COPY Gemfile.lock /app/Gemfile.lock
RUN bundle install
RUN npm install -g yarn

EXPOSE 3000
CMD ["bash"]

We'll then need to create a Gemfile with Rails in our project root so that we can run rails new inside our container.

touch Gemfile

And add the following to the newly created Gemfile (this will get overwritten once we run rails new in the container)
Gemfile

source 'https://rubygems.org'
gem 'rails', '~>6'

Create a Gemfile.lock as well

touch Gemfile.lock

Next create docker-compose.yml file in your project root and add the following configuration files to it.

touch docker-compose.yml

docker-compose.yml

version: '3.8'
services:
  web:
    build: .
    command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
    volumes:
      - .:/app
    ports:
      - "3000:3000"

Run this command to

docker-compose run --no-deps web rails new . --force

With the new project files generated run the following command which will change the ownership of the newly created rails files from root to your user. (Docker runs as root so the rails new command generates the files as the root user)

sudo chown -R $USER:$USER .

Then you'll need to reinstall the new Gemfile dependencies, run docker-compose build to re-run the bundle install

docker-compose build

Once that's complete, running docker-compose up should start the rails app and make it accessible on localhost:3000/

Building the bones of our app

For storage, we're going to use the Shrine gem, and for video transcoding we'll use ffmpeg via the streamio-ffmpeg gem.

Add these two lines to your Gemfile, then install them with docker-compose build.

gem 'shrine', '~> 3.0'
gem 'streamio-ffmpeg', '~> 3.0'

Next, we're going to use Rails scaffold to generate CRUD operations for our Video model. We add the name:string to create a new name field for our video and original_video_data:text to add the field Shrine needs for storing data about our uploaded file.

rails g scaffold Video name:string original_video_data:text

If your curious about what the orginal_video_data field is, and why it's suffixed with _data, I'd recommend reading the Shrine Getting Started docs as our code so far closely follows that.

In addition to creating the controllers, models, views, etc. The scaffold command will also create the neccessary routes for our Videos to be accessible at the /videos endpoint.

Let's add a default route so we don't have to type /videos everytime we want to test our application.

Update your config/routes.rb code to look like the below:

config/routes.rb

Rails.application.routes.draw do
  root to: "videos#index"
  resources :videos
  # For details on the DSL available within this file, see https://guides.rubyonrails.org/routing.html
end

Configuring Shrine:

To configure Shrine, we need to start by creating a Shrine initializer. I'm not going to go into too much detail about how Shrine works here, but the gist of things is that the below commands create two Shrine stores, a cache and permanent.

With the settings provided below Shrine is configured to use the local file system, and the cache and permanent storage are located in the Rails public/ directory and then placed into public/uploads & public/uploads/cache due to the prefix settings.

Create a new file config/initializers/shrine.rb and add the following code:

config/initializers/shrine.rb

require "shrine"

# File System Storage
require "shrine/storage/file_system"
Shrine.storages = { 
  cache: Shrine::Storage::FileSystem.new("public", prefix: "uploads/cache"), # temporary 
  store: Shrine::Storage::FileSystem.new("public", prefix: "uploads"),       # permanent 
}

Shrine.plugin :activerecord 
Shrine.plugin :cached_attachment_data # for retaining the cached file across form redisplays 
Shrine.plugin :restore_cached_data # re-extract metadata when attaching a cached file 

After configuring Shrine, let's test our rails app is running correctly by typing the command

docker-compose run web rails db:create db:migrate

Then start our rails server with:

docker-compose up

If all is well, you should see something similar to the below image:

With everything running smoothly, the next step is to create our Uploader. Start by creating a new folder in the /app directory called uploaders, and create a new .rb file for our uploader called video_uploader.rb

In a minute, this is where we will add the processing logic for how to handle video uploads, but for now, let's start with the bare minimum by adding the following code to video_uploader.rb

app/uploaders/video_uploader.rb

class VideoUploader < Shrine

end

Before we go any further with the processing logic for our uploader, let's make sure it's connected to our Video model. Shrine makes this incredibly easy. In app/models/video.rb add the following code so that your model looks like the below:

class Video < ApplicationRecord
  include VideoUploader::Attachment(:original_video)
end

If you'll recall, when we created our Video object earlier, we named the field in the database original_video_data, but in the model file we use just original_video instead. The _data suffix is a Shrine naming convention, hence the reason for it.

In app/views/videos/_form.html.erb, update the form field for the original_video file field to the below code. According to Shrine docs, the hidden_field ensures that if a user updates a video object without uploading a new orginal_video file, the cached file gets used instead of being overwritten to an empty file.

app/views/videos/_form.html.erb

<div class="field">
  <%= form.label :original_video, "Video File" %>
  <%= form.hidden_field :original_video, value: video.cached_original_video_data %>
  <%= form.file_field :original_video %>
</div>

The updated _form.html.erb should look like this:

<%= form_with(model: video) do |form| %>
  <% if video.errors.any? %>
    <div id="error_explanation">
      <h2><%= pluralize(video.errors.count, "error") %> prohibited this video from being saved:</h2>

      <ul>
        <% video.errors.each do |error| %>
          <li><%= error.full_message %></li>
        <% end %>
      </ul>
    </div>
  <% end %>
  <div class="field">
    <%= form.label :name %>
    <%= form.text_field :name %>
  </div>

  <div class="field">
    <%= form.label :original_video, "Video File" %>
    <%= form.hidden_field :original_video, value: video.cached_original_video_data %>
    <%= form.file_field :original_video %>
  </div>

  <div class="actions">
    <%= form.submit %>
  </div>
<% end %>

And finally, in our Videos Controller, we have to remember to permit the new :original_video parameter so that the file can actually be saved.

Update the video_params to this:

# Only allow a list of trusted parameters through.
def video_params
  params.require(:video).permit(:name, :original_video)
end

The full controller should now look like this:

app/controllers/videos_controller.rb

class VideosController < ApplicationController
  before_action :set_video, only: %i[ show edit update destroy ]

  # GET /videos or /videos.json
  def index
    @videos = Video.all
  end

  # GET /videos/1 or /videos/1.json
  def show
  end

  # GET /videos/new
  def new
    @video = Video.new
  end

  # GET /videos/1/edit
  def edit
  end

  # POST /videos or /videos.json
  def create
    @video = Video.new(video_params)

    respond_to do |format|
      if @video.save
        format.html { redirect_to @video, notice: "Video was successfully created." }
        format.json { render :show, status: :created, location: @video }
      else
        format.html { render :new, status: :unprocessable_entity }
        format.json { render json: @video.errors, status: :unprocessable_entity }
      end
    end
  end

  # PATCH/PUT /videos/1 or /videos/1.json
  def update
    respond_to do |format|
      if @video.update(video_params)
        format.html { redirect_to @video, notice: "Video was successfully updated." }
        format.json { render :show, status: :ok, location: @video }
      else
        format.html { render :edit, status: :unprocessable_entity }
        format.json { render json: @video.errors, status: :unprocessable_entity }
      end
    end
  end

  # DELETE /videos/1 or /videos/1.json
  def destroy
    @video.destroy
    respond_to do |format|
      format.html { redirect_to videos_url, notice: "Video was successfully destroyed." }
      format.json { head :no_content }
    end
  end

  private
    # Use callbacks to share common setup or constraints between actions.
    def set_video
      @video = Video.find(params[:id])
    end

    # Only allow a list of trusted parameters through.
    def video_params
      params.require(:video).permit(:name, :original_video)
    end
end

With all the above added, we can now test that Shrine is uploading and saving our .mp4 file. Make sure your rails server is running, then navigate to localhost:3000

NOTE: In rails, the initializers get called... well, on initialization. That means if you make changes to your shrine.rb file, make sure to stop and restart your server so that the changes will take affect.

At the videos index, click "New Video", give your video a name, and then select an mp4 file to upload.

Just remember, since we haven't done any frontend work (like adding a file upload progress bar) there won't be any feedback as the file gets uploaded. So just make sure to use a small .mp4 file for testing purposes.

Once the file uploads, you should get a green confirmation message that the video was created successfully. Let's update our show view, so we can see more information about our video.

app/views/videos/show.html.erb

<p id="notice"><%= notice %></p>

<p><strong>Name: </strong><%= @video.name %></p>

<p><strong>@video.original_video_url</strong> <%= @video.original_video_url %></p>

<%= link_to 'Edit', edit_video_path(@video) %> |
<%= link_to 'Back', videos_path %>

Updating the show view to the above code and navigating to localhost:3000/videos/1 you should see page similar to the following:

NOTE: Shrine automatically generates a UUID for file uploads but retains the original filename in the _data field. That's why the filename is different.

And if you navigate to your /public/uploads folder in your project, you should see that your .mp4 has been uploaded with it's Shrine ID as the filename.

For housekeeping andtesting purposes, go ahead and hit "Back" then use the "Destroy" button to delete your file. You'll notice it removes the item from the database, and deletes the associated video file from the permanent storage. (You might notice that the cache file & folder is still there, but handling that is outside the scope of what I'm aiming to cover in this tutorial)

Building our HLS VOD Service

NOTE: At the time of writing this, I had done some, albeit basic, refactoring to the code to make it a little more DRY. Instead of going step by step through the trial and error I went through to end up with this code, I've decided to just present the completed version as-is, then at the end explain some of the earlier roadblocks and why I made certain decisions

With the basic CRUD and upload functionality of our VOD service in place, it's time build out the actual HLS processing logic in our VideoUploader class.

Getting started:

We'll need to extend Shrine with two plugins, processing & versions. The processing plugin exposes a method process to our VideoUploader class that allows us to apply transforming processes to our file everytime a new file is uploaded.

The versions plugin further extends the processing plugin by giving us the ability to store arrays of files in our object. The plugin being named "versions" can be a bit misleading, but as you'll see from our use-case, we can use it beyond just versioning files.

At the top of the VideoUploader class, add the versions and processing plugins to Shrine:

class VideoUploader < Shrine
  plugin :versions
  plugin :processing

  ...

Next, we're going to override the initialize method in our uploader to generate a uuid which we will use for our transcoding ouput folder & filename. Add the following code to your VideoUploader class below the plugins.

def initialize(*args)
    super
    @@_uuid = SecureRandom.uuid
end

Now we're going to override Shrine's generate_location method in order to customize the file path our transcoded videos are saved at.

To briefly explain, generate_location gets called for each file that gets saved. Later in our code we'll be adding an array, called hls_playlist, to hold all the .ts and .m3u8 files that get generated by ffmpeg. So generate_location gets called for each one of those files that gets added.

The generate_location function then checks if the file being saved is of type .ts or .m3u8, and if it is saves it with the name of the file. (In this case the filename will be the uuid with some additional information about the encoding options appended as we'll see later)

Here's the code for generate_location to be added to the VideoUploader class.

def generate_location(io, record: nil, **)
    basename, extname = super.split(".")
    if extname == 'ts' || extname == 'm3u8'
        location = "#{@@_uuid}/#{File.basename(io.to_path)}"
    else
        location = "#{@@_uuid}/#{@@_uuid}.#{extname.to_s}"
    end
end

Next we're going to create a method to generate our HLS playlist, and add that newly created playlist to the master .m3u8 file. This method requires that an .m3u8 file already be open (which we pass in as the master_playlist param) and that a streamio-ffmpeg movie object already be created (which we pass in with the movie param)

The path and uuid params are both generated from the uuid that we created in the initialize method.

We set the options for ffmpeg in its own variable, and use the custom attribute to pass in cli options to ffmpeg for encoding the hls stream. Using %W instead of %w allows us to pass in variables to the array via Ruby string interpolation.

In the ffmpeg custom array, the flag -vf scale=#{width}:-2 allows us to specify a new video width while retaining the original videos aspect ratio. I'm not going to go over the rest of the ffmpeg settings in here, other than just to say these settings definitely need to be tweaked before actually being used.

In the movie.transcode call the first option is the output path of the file (you can see we join the path variable to gets passed to the function, with the uuid then the transcoding settings of the file)

After FFMPEG transcodes the movie, we use Ruby's Dir.glob to iterate over each of the generated .ts files and get the value of the highest bandwidth and calculate the average bandwidth before writing those and the rest of the .m3u8 information into the master.m3u8 playlist file.

def generate_hls_playlist(movie, path, uuid, width, bitrate, master_playlist)
    ffmpeg_options = {validate: false, custom: %W(-profile:v baseline -level 3.0 -vf scale=#{width}:-2 -b:v #{bitrate} -start_number 0 -hls_time 2 -hls_list_size 0 -f hls) }
    transcoded_movie = movie.transcode(File.join("#{path}", "#{uuid}-w#{width}-#{bitrate}.m3u8"), ffmpeg_options)

    bandwidth = 0
    avg_bandwidth = 0
    avg_bandwidth_counter = 0

    Dir.glob("#{path}/#{uuid}-w#{width}-#{bitrate}*.ts") do |ts|
        movie = FFMPEG::Movie.new(ts)
        if movie.bitrate > bandwidth
            bandwidth = movie.bitrate
        end
        avg_bandwidth += movie.bitrate
        avg_bandwidth_counter += 1
    end

    avg_bandwidth = (avg_bandwidth / avg_bandwidth_counter)

    master_playlist.write("#EXT-X-STREAM-INF:BANDWIDTH=#{bandwidth},AVERAGE-BANDWIDTH=#{avg_bandwidth},CODECS=\"avc1.640028,mp4a.40.5\",RESOLUTION=#{transcoded_movie.resolution},FRAME-RATE=#{transcoded_movie.frame_rate.to_f}\n")
    master_playlist.write(File.join("/uploads", uuid, "#{File.basename(transcoded_movie.path)}\n") )
end

Whew. still with me?

Finally, we use the process method that we called earlier to run the video transcoding on our uploaded file.

Using the versions plugin we create a variable versions which is a hash that contains both the original file, and an array we're naming hls_playlist that will hold all of the .ts and .m3u8 files that ffmpeg generates.

We use io.download to get the original file, then open up our new FFMPEG::Movie object as a variable movie, and create the master.m3u8 playlist file and write the required first lines.

Next we make a call to the method we just wrote to transcode our video, generate_hls_playlist

process(:store) do |io, **options|
    versions = { original: io, hls_playlist: [] }

    io.download do |original|
        path = File.join(Rails.root, "tmp", "#{@@_uuid}")
        FileUtils.mkdir_p(path) unless File.exist?(path)

        movie = FFMPEG::Movie.new(original.path)

        master_playlist = File.open(File.join("#{path}","#{@@_uuid}-master.m3u8"), "w")
        master_playlist.write("#EXTM3U\n")
        master_playlist.write("#EXT-X-VERSION:3\n")
        master_playlist.write("#EXT-X-INDEPENDENT-SEGMENTS\n")

        generate_hls_playlist(movie, path, @@_uuid, 720, "2000k", master_playlist)
        generate_hls_playlist(movie, path, @@_uuid, 1080, "4000k", master_playlist)

        versions[:hls_playlist] << File.open(master_playlist.path)

        Dir.glob("#{path}/#{@@_uuid}-w*.m3u8") do |m3u8|
            versions[:hls_playlist] << File.open(m3u8)
        end
        Dir.glob("#{path}/#{@@_uuid}*.ts").each do |ts|
            versions[:hls_playlist] << File.open(ts)
        end

        master_playlist.close
        FileUtils.rm_rf("#{path}")
    end
    versions
end

With all that code added, your video_uploader.rb file should look like the following:

class VideoUploader < Shrine
    plugin :versions
    plugin :processing

    def initialize(*args)
        super
        @@_uuid = SecureRandom.uuid
    end

    def generate_location(io, record: nil, **)
        basename, extname = super.split(".")
        if extname == 'ts' || extname == 'm3u8'
            location = "#{@@_uuid}/#{File.basename(io.to_path)}"
        else
            location = "#{@@_uuid}/#{@@_uuid}.#{extname.to_s}"
        end
    end

    def generate_hls_playlist(movie, path, uuid, width, bitrate, master_playlist)
        ffmpeg_options = {validate: false, custom: %W(-profile:v baseline -level 3.0 -vf scale=#{width}:-2 -b:v #{bitrate} -start_number 0 -hls_time 2 -hls_list_size 0 -f hls) }
        transcoded_movie = movie.transcode(File.join("#{path}", "#{uuid}-w#{width}-#{bitrate}.m3u8"), ffmpeg_options)

        bandwidth = 0
        avg_bandwidth = 0
        avg_bandwidth_counter = 0

        Dir.glob("#{path}/#{uuid}-w#{width}-#{bitrate}*.ts") do |ts|
            movie = FFMPEG::Movie.new(ts)
            if movie.bitrate > bandwidth
                bandwidth = movie.bitrate
            end
            avg_bandwidth += movie.bitrate
            avg_bandwidth_counter += 1
        end

        avg_bandwidth = (avg_bandwidth / avg_bandwidth_counter)

        master_playlist.write("#EXT-X-STREAM-INF:BANDWIDTH=#{bandwidth},AVERAGE-BANDWIDTH=#{avg_bandwidth},CODECS=\"avc1.640028,mp4a.40.5\",RESOLUTION=#{transcoded_movie.resolution},FRAME-RATE=#{transcoded_movie.frame_rate.to_f}\n")
        master_playlist.write(File.join("/uploads", uuid, "#{File.basename(transcoded_movie.path)}\n") )
    end

    process(:store) do |io, **options|
        versions = { original: io, hls_playlist: [] }

        io.download do |original|
            path = File.join(Rails.root, "tmp", "#{@@_uuid}")
            FileUtils.mkdir_p(path) unless File.exist?(path)

            movie = FFMPEG::Movie.new(original.path)

            master_playlist = File.open(File.join("#{path}","#{@@_uuid}-master.m3u8"), "w")
            master_playlist.write("#EXTM3U\n")
            master_playlist.write("#EXT-X-VERSION:3\n")
            master_playlist.write("#EXT-X-INDEPENDENT-SEGMENTS\n")

            generate_hls_playlist(movie, path, @@_uuid, 720, "2000k", master_playlist)
            generate_hls_playlist(movie, path, @@_uuid, 1080, "4000k", master_playlist)

            versions[:hls_playlist] << File.open(master_playlist.path)

            Dir.glob("#{path}/#{@@_uuid}-w*.m3u8") do |m3u8|
                versions[:hls_playlist] << File.open(m3u8)
            end
            Dir.glob("#{path}/#{@@_uuid}*.ts").each do |ts|
                versions[:hls_playlist] << File.open(ts)
            end

            master_playlist.close
            FileUtils.rm_rf("#{path}")
        end
        versions
    end
end

Restart your rails server, open up to the Videos index, create a new video, and let it upload.

After the video uploads you should see the following rails error screen:

Believe it or not, this is actually what we want to see. When we added the :versions plugin to our VideoUploader class, it changed the way that we need to access the file from our erb code.

Update the app/views/videos/show.html.erb to be the following. Note that since :hls_playlist is an array, we're calling .first.url, because when we created the :hls_playlist we set the master.m3u8 file to be the first element in the array.

<p id="notice"><%= notice %></p>

<p><strong>Name: </strong><%= @video.name %></p>

<p><strong>@video.original_video[:hls_playlist].first.url</strong> <%= @video.original_video[:hls_playlist].first.url %></p>

<%= link_to 'Edit', edit_video_path(@video) %> |
<%= link_to 'Back', videos_path %>

With that update made, navigating back to the show page for your newly uploaded video should echo back the url of the master .m3u8 playlist file. That's great, but now how do we actually play it?

In a real app, you'd want to pick a video player for your frontend of choice that supports HLS, but to quickly see what we're working with, I'm going to use a cdn version of HLS.js to get us up and running with a video player.

Add the following code to your show view, it's the example code from the hls.js github modified to use our :hls_playlist uploaded file as the video source.

<script src="https://cdnjs.cloudflare.com/ajax/libs/hls.js/0.5.14/hls.min.js" integrity="sha512-js37JxjD6gtmJ3N2Qzl9vQm4wcmTilFffk0nTSKzgr3p6aitg73LR205203wTzCCC/NZYO2TAxSa0Lr2VMLQvQ==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>

<video id="video" controls></video>
<script>
  var video = document.getElementById('video');
  var videoSrc = '<%= @video.original_video[:hls_playlist].first.url %>';
  if (Hls.isSupported()) {
    console.log("HLS Supported")
    var hls = new Hls();
    hls.loadSource(videoSrc);
    hls.attachMedia(video);
  }
  // HLS.js is not supported on platforms that do not have Media Source
  // Extensions (MSE) enabled.
  //
  // When the browser has built-in HLS support (check using `canPlayType`),
  // we can provide an HLS manifest (i.e. .m3u8 URL) directly to the video
  // element through the `src` property. This is using the built-in support
  // of the plain video element, without using HLS.js.
  //
  // Note: it would be more normal to wait on the 'canplay' event below however
  // on Safari (where you are most likely to find built-in HLS support) the
  // video.src URL must be on the user-driven white-list before a 'canplay'
  // event will be emitted; the last video event that can be reliably
  // listened-for when the URL is not on the white-list is 'loadedmetadata'.
  else if (video.canPlayType('application/vnd.apple.mpegurl')) {
    video.src = videoSrc;
  }
</script>

Voila, we have HLS video!

Closing Notes:

There's still a lot of work left to do for this to be anywhere near ready to go into a real application. Some of the biggest areas for improvements are adding frontend feedback on video upload progress & moving the video processing transcoding to a background job to free up the webserver resources.

Another major area for improvement is that I've implemented no error / exception handling. Definitely something that should be added in.

The actual m3u8 transcoding processes will also need to be setup as per your individual project's needs. In the example here, I transcoded the lowest quality version at a lower resolution to attempt to further cut down on bandwidth needs, but if you'd like to do something similar to youtube, where they have different resolutions ie 1080p, 720p, etc, it should be simple enough to create new arrays in the versions hash and name save the corresponding resolution to the variable.

If I do a pt. 2 on this tutorial the next thing I'd like to tackle is hosting the videos on AWS S3 and using Cloudfront as a CDN. This also opens up more possibilities in terms of restricting video access only to authenticated users via signed urls and signed cookies.