Build an Auto-Scaling Transcoding Platform Using Python, FFmpeg & AWS

In this post, I’ll describe the building blocks of a resilient self-hosted transcoding platform using open source tools and AWS.

For part two, I’ll share a sample python project that allows you to bootstrap this in minutes.

General principles

When building a system like this, you should never compromise on these:

  • Self-healing (AWS ASG)

Most of those can be attained effortlessly through SaaS solutions and/or your cloud provider services.

Infrastructure diagram

AWS Infra diagram

You can replace the compute layer with lambda, for example, just bear in mind that you have an execution time limit of 15 minutes.
The cheapest option will always depend on your workload (video length, output formats, amount of videos, schedules, etc), this architecture can be adapted to the execution layer that suits you better.

The cheapest option in AWS is to have a pool of workers using an auto-scaling group that provisions spot instances.

Using the proposed architecture, premature instance termination will have no effect on our service. This means we can safely use spot instances and enjoy up to a 90% discount. On average, my total discount has been about 70% versus On-Demand pricing.

This will be orders of magnitude cheaper than any SaaS video transcoder on the market.

Transcoder node diagram

Key points for the system (AWS version)

  • SQS: Send your jobs to a queue, here I’m using SQS, but any queue/stream service or even Redis should handle this job just fine.
    Remember to set your message visibility timeout longer than your longest-running transcode job or you’ll have the message go back to the queue.

FFmpeg command

This command will output using the HLS. You can easily change this to any other format. I chose this one in case you need HLS and the information for it can bit a bit scattered since it offers a lot of options.

FFmpeg command to transcode a video file to HLS

DOWNLOAD_PATH: this is the path to the original file you want to transcode
HLS_PATH: this should be the destination base folder
split: split input into multiple outputs
map: map the different outputs with the specified formats
-f hls: instructs FFmpeg to convert to HLS format
hls_playlist_type vod: this will make sure you have the full playlist inside the m3u8 file
strftime: special param that will format the filename based on the machine datetime and auto-create the needed file structure (very useful)
version_%v: this allows you to have a folder for each version (_0, _1, _3)

The following script serves to show a very basic example of how an AWS SQS worker could process the videos.

simple python script to illustrate the system

How does this all sound? Is there anything you’d like me to expand on? Let me know your thoughts in the comments section below (and hit the clap if this was useful)!

Stay tuned for the next post. Follow so you won’t miss it!

Principal Engineer @ Farfetch