One Pull Request per month: January

This year I'll be doing a challenge: submit at least one Pull Request to a FLOSS project. January's PR is a fix to No audio is recorded with videos #148 , a bug in Elementary OS 6's "rewrite" of the Camera app. The backend was changed to pure GStreamer and this introduced some regressions. However, it also allows for better device detection and possibly more control over video and audio settings. Hopefully this new version will solve many of the "Camera does not work on XXX" issues.

Having worked on Camera in the past (see my contributions in my previous post), I decided to tackle a new issue: no sound was being recorded on the app. After some investigation I realized that this was expected: the app never requested sound to be recorded!

GStreamer works with data processing pipelines with three basic elements:

1. sources, such as webcams, microphones or files, forward data from an external source to the pipeline
2. sinks, such as files or user interface elements, receive data from the pipeline and either save it or display it.
3. elements receive data from the pipeline, transform it and forward this result to other elements or a sink.

Thus, data flow from one or more sources, passes thru many elements that transform it and finally reaches a sink where it is stored or shown to the user. The following pipeline was being used:

"v4l2src device=%s name=v4l2src !"
"video/x-raw, width=640, height=480, framerate=30/1 ! " +
"videoflip method=horizontal-flip name=hflip ! " +
"videobalance name=balance ! " +
"tee name=tee ! " +
"queue leaky=downstream max-size-buffers=10 ! " +
"videoconvert ! " +
"videoscale ! " +
"gtksink name=gtksink"

We only have a single source, v4l2src, which corresponds to a Video for Linux 2 device (usually webcams or other external USB cameras). Data is passed thru many elements using a "!" character until reaching a gtksink that shows the video preview. To record videos a new branch was connected to the tee element that included a video encoder and a filesink that finally stored the video to a file. Thus, the current pipeline only has a single source that produces video frames.

To record audio we need a source that produces audio. This can be done by adding an alsasrc element that produces audio using the ALSA sound system. Gstreamer offers a very neat way of testing pipelines: the gst-launch-1.0 app.
The following sequence of elements obtains data from the microphone using ALSA, encodes it using the Vorbis codec and packages it into a Ogg container.

gst-launch-1.0 -e alsasrc ! audioconvert ! vorbisenc ! oggmux !  filesink location="aa.ogg"

Notice that we still can't produce a file with both audio and video! Both pipelines produce only one or the other. That's where the oggmux element comes into play: its role is to combine audio and video streams into a stream. Thus, to record audio as well we only need to add the elements above to the existing pipeline and link them to the existing oggmux element. The code below does the job and is the basis of my January PR: Add audio recording elements to record_bin. #153 .

var alsasrc = Gst.ElementFactory.make ("alsasrc", null);
if (alsasrc == null) {
    missing_messages += Gst.PbUtils.missing_element_installer_detail_new ("alsasrc");
}

var audio_queue = Gst.ElementFactory.make ("queue", null);

var audio_convert = Gst.ElementFactory.make ("audioconvert", null);
if (audio_convert == null) {
    missing_messages += Gst.PbUtils.missing_element_installer_detail_new ("audioconvert");
}

var audio_vorbis = Gst.ElementFactory.make ("vorbisenc", null);
if (audio_vorbis == null) {
    missing_messages += Gst.PbUtils.missing_element_installer_detail_new ("vorbisenc");
}

// ...

record_bin.add_many (alsasrc, audio_queue, audio_convert, audio_vorbis);
alsasrc.link_many (audio_queue, audio_convert, audio_vorbis, muxer);

The code is rather self explanatory and is basically a Vala version of the command line pipeline I showed earlier. And... that's it. I'm now waiting for the maintainers to review it and begin the improvements so it eventually gets merged. Now I only have 11 PRs left for 2021. Let's keep them coming!