Parallel ffmpeg stream manipulation

This post will illustrate how we can apply different operations (audio transcoding, subtitles selection, video, chapters and fonts passthrough) on multistream MKVs with ffmpeg, while optimizing for multicore systems.

More precisely, here is what we are trying to achieve:

• transcode the Japanese audio track from a entire anime season, from a superfluously lossless Flac to the more efficient Opus format¹
• shed the useless (to me) English audio track (also in Flac, and 5.1…), as well as the unneeded “Signs-only” subtitle tracks
• pass-through the video track
• retain all attachments (fonts, chapters…)
• preserve file modification times
• replace the original files with the new ones
• do all this as fast and efficiently as possible 😎

When starting, I knew how to accomplish this in two steps using two different tools:

1. Identify and remove the unwanted streams with mkvmerge^(source):

   mkvmerge -i *.mkv
   File '01.mkv': container: Matroska
   Track ID 0: video (HEVC/H.265/MPEG-H)
   Track ID 1: audio (Flac)
   Track ID 2: audio (Flac)
   Track ID 3: subtitles (SubStationAlpha)
   Track ID 4: subtitles (SubStationAlpha)
   Track ID 5: subtitles (HDMV PGS)
   Track ID 6: subtitles (HDMV PGS)
   Attachment ID 1: type 'application/vnd.ms-opentype', size 30712 bytes, file name 'font.ttf'
   (...)
   
   for f in *.mkv; do mkvmerge -o "remuxed/$f" --audio-tracks '!eng' --subtitle-tracks '!4,6' "$f"; done
   

2. Use ffmpeg to transcode the remaining audio track to Opus, using a generous 192 kbps (96k per audio channel) bitrate and then do the cleanup:

   for f in remuxed/*.mkv; do ffmpeg -i "$f" -map 0 -c copy -c:a libopus -b:a 192k "tmp_$f"; touch -r "$f" "tmp_$f"; mv "tmp_$f" "$f"; done
   

   Where -map 0 ports over all input streams, -c copy keeps them intact, and -c:a libopus -b:a 192k modulates this by transcoding all audio streams (in our case, the one we kept) to 192k Opus. Finally, touch -r updates our file modtime to reflect the original one, and mv overwrite the source with the modified file (warning: make sure to always test on a copy first, otherwise data loss will incur!).

From there it was rather easy to merge it all in one fell for loop:

for f in *.mkv; do \
    mkvmerge -o "tmp_$f" --audio-tracks '!eng' --subtitle-tracks '!4,6' "$f"; \
    ffmpeg -i "tmp_$f" -map 0 -c copy -c:a libopus -b:a 192k "tmp2_$f"; \
    touch -r "$f" "tmp2_$f"; \
    mv "tmp2_$f" "$f"; \
    rm "tmp_$f"; \
done

However, now the next step was to get rid of the superfluous mkvmerge and do it all with the Ever Almighty Multimedia Swiss-Army Knife®, ffmpeg. For that I had to dig quite a bit, but here is the result:

for f in *.mkv; do ffmpeg -i "$f" -map 0 -map -0:a:1 -map -0:s:1 -map -0:s:3 -c copy -c:a libopus -b:a 192k "tmp_$f"; touch -r "$f" "tmp_$f"; mv "tmp_$f" "$f"; done

As before we first select all streams via -map 0, then make use of negative mapping (notice the minus sign at the front of the subsequent -map values) to get rid of the unwanted tracks (second audio track, second and fourth sub tracks… Beware, this time we are using ffmpeg’s indexes/IDs which are relative to the category and 0-indexed). Then it is as before, we copy all remaining streams without changing them (-c copy) except audio ones which are transcoded to Opus.

Now this is all well and dandy, but this command runs sequentially, iterating over each file while queuing up the others, disregarding our multicore CPU architecture and thus slowing down the process. How can we run all this concurrently, dispatching a job to each CPU thread?

GNU Parallel² to the rescue!

parallel ffmpeg -i {} -map 0 -map -0:a:1 -map -0:s:1 -map -0:s:3 -c copy -c:a libopus -b:a 192k tmp_{}';' touch -r {} tmp_{}';' mv tmp_{} {} ::: *.mkv

Et voilà, that’s how you efficiently and stylishly reduce a bloated 10.9GB series to a lean and mean 4.8GB one. Great success! 👍👏🤗