Implementing SVE2 for Opus Codec Library Part 1: Package Installation

Introduction

Previously, we identified several packages that do not support SVE2 codes yet. We ultimately decided that Opus Audio Codec is the best candidate. In this post, we will explore the package in detail and see how we can implement SVE2 into it.

Before We Start...

When we clone the package, we can see the following files:

$ ls
AUTHORS          cmake           LICENSE_PLEASE_READ.txt  meson.build        opus_sources.mk         silk             update_version
autogen.sh       CMakeLists.txt  m4                       meson_options.txt  opus-uninstalled.pc.in  silk_headers.mk  win32
celt             configure.ac    Makefile.am              NEWS               README                  silk_sources.mk
celt_headers.mk  COPYING         Makefile.mips            opus_headers.mk    README.draft            src
celt_sources.mk  doc             Makefile.unix            opus.m4            releases.sha2           tests
ChangeLog        include         meson                    opus.pc.in         scripts                 training

As we can see, the packages contains several Makefile and configure template files. These gives us an idea that this package may use the GNU Autotools to generate Makefile and configure scripts. To have a clear understanding of how we can install this package, we can read the README file.

# README
# ...

1) Clone the repository:

    % git clone https://gitlab.xiph.org/xiph/opus.git
    % cd opus

2) Compiling the source

    % ./autogen.sh
    % ./configure
    % make

3) Install the codec libraries (optional)

    % sudo make install

Once you have compiled the codec, there will be a opus_demo executable
in the top directory.

Usage: opus_demo [-e] <application> <sampling rate (Hz)> <channels (1/2)>
         <bits per second> [options] <input> <output>
       opus_demo -d <sampling rate (Hz)> <channels (1/2)> [options]
         <input> <output>

# ...

Now, let's follow the instruction. Once we run the ./autogen.sh, we get the following list of files.

$ ./autogen.sh
$ ls
aclocal.m4       CMakeLists.txt  doc                      Makefile.mips      opus.pc.in              silk_headers.mk
AUTHORS          compile         include                  Makefile.unix      opus_sources.mk         silk_sources.mk
autogen.sh       config.guess    INSTALL                  meson              opus-uninstalled.pc.in  src
autom4te.cache   config.h.in     install-sh               meson.build        package_version         test-driver
celt             config.sub      LICENSE_PLEASE_READ.txt  meson_options.txt  README                  tests
celt_headers.mk  configure       ltmain.sh                missing            README.draft            training
celt_sources.mk  configure.ac    m4                       NEWS               releases.sha2           update_version
ChangeLog        COPYING         Makefile.am              opus_headers.mk    scripts                 win32
cmake            depcomp         Makefile.in              opus.m4            silk

We have more files now. The notable files are Makefile.in and configure script file. Makefile.in is generated from Makefile.am file but still is missing some values that are going to be filled with the configure script. Now, let's run the configure script.

$ ./configure
$ ls
aclocal.m4       config.guess   include                  Makefile.unix      opus-uninstalled.pc     stamp-h1
AUTHORS          config.h       INSTALL                  meson              opus-uninstalled.pc.in  test-driver
autogen.sh       config.h.in    install-sh               meson.build        package_version         tests
autom4te.cache   config.log     libtool                  meson_options.txt  README                  training
celt             config.status  LICENSE_PLEASE_READ.txt  missing            README.draft            update_version
celt_headers.mk  config.sub     ltmain.sh                NEWS               releases.sha2           win32
celt_sources.mk  configure      m4                       opus_headers.mk    scripts
ChangeLog        configure.ac   Makefile                 opus.m4            silk
cmake            COPYING        Makefile.am              opus.pc            silk_headers.mk
CMakeLists.txt   depcomp        Makefile.in              opus.pc.in         silk_sources.mk
compile          doc            Makefile.mips            opus_sources.mk    src

Not surprisingly, we get a Makefile amongst the newly generated files. If we inspect the Makefile, we can see what CFLAG it uses to compile the package.

# Makefile
# ...

CFLAGS = -g -O2 -fvisibility=hidden -D_FORTIFY_SOURCE=2 -W -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes

# ...

From these flags, we can know that the package does not utilize auto-vectorization. This means we can also implement SVE2 codes by utilizing auto-vectorization in this package.

Now, let's compile the package. For this, we can assign more jobs in parallel at a time when we execute the Makefile to increase the speed of compilation. In general, we can calculate the number by doubling the core number plus one. In our case, we can use a value of 24 (16 cores * 2 + 1) to keep every core busy with jobs.

$ make -j 24
$ ls
aclocal.m4       config.guess   include                  Makefile.mips      opus.pc                 silk
AUTHORS          config.h       INSTALL                  Makefile.unix      opus.pc.in              silk_headers.mk
autogen.sh       config.h.in    install-sh               meson              opus_sources.mk         silk_sources.mk
autom4te.cache   config.log     libopus.la               meson.build        opus-uninstalled.pc     src
celt             config.status  libtool                  meson_options.txt  opus-uninstalled.pc.in  stamp-h1
celt_headers.mk  config.sub     LICENSE_PLEASE_READ.txt  missing            package_version         test-driver
celt_sources.mk  configure      ltmain.sh                NEWS               README                  tests
ChangeLog        configure.ac   m4                       opus_compare       README.draft            training
cmake            COPYING        Makefile                 opus_demo          releases.sha2           trivial_example
CMakeLists.txt   depcomp        Makefile.am              opus_headers.mk    repacketizer_demo       update_version
compile          doc            Makefile.in              opus.m4            scripts                 win32

As the README file mentioned, we have a executable file called opus_demo. When we run it, we can see the package is successfully compiled.

$ ./opus_demo
Usage: /home/swji1/opus/.libs/opus_demo [-e] <application> <sampling rate (Hz)> <channels (1/2)> <bits per second>  [options] <input> <output>
       /home/swji1/opus/.libs/opus_demo -d <sampling rate (Hz)> <channels (1/2)> [options] <input> <output>

application: voip | audio | restricted-lowdelay
options:
-e                   : only runs the encoder (output the bit-stream)
-d                   : only runs the decoder (reads the bit-stream as input)
-cbr                 : enable constant bitrate; default: variable bitrate
-cvbr                : enable constrained variable bitrate; default: unconstrained
-delayed-decision    : use look-ahead for speech/music detection (experts only); default: disabled
-bandwidth <NB|MB|WB|SWB|FB> : audio bandwidth (from narrowband to fullband); default: sampling rate
-framesize <2.5|5|10|20|40|60|80|100|120> : frame size in ms; default: 20
-max_payload <bytes> : maximum payload size in bytes, default: 1024
-complexity <comp>   : complexity, 0 (lowest) ... 10 (highest); default: 10
-inbandfec           : enable SILK inband FEC
-forcemono           : force mono encoding, even for stereo input
-dtx                 : enable SILK DTX
-loss <perc>         : simulate packet loss, in percent (0-100); default: 0

Testing

But, how we validate if the binary works as intended? For this, we can refer to the README again.

# README
# ...

== Testing ==

This package includes a collection of automated unit and system tests
which SHOULD be run after compiling the package especially the first
time it is run on a new platform.

To run the integrated tests:

    % make check

# ...

Thankfully, the authors provide a set of unit tests to validate the integrity of the executable file. Using this, we can check if the package is compiled correctly.

$ make check
PASS: celt/tests/test_unit_cwrs32
PASS: celt/tests/test_unit_dft
PASS: celt/tests/test_unit_entropy
PASS: celt/tests/test_unit_laplace
PASS: celt/tests/test_unit_mathops
PASS: celt/tests/test_unit_mdct
PASS: celt/tests/test_unit_rotation
PASS: celt/tests/test_unit_types
PASS: silk/tests/test_unit_LPC_inv_pred_gain
PASS: tests/test_opus_api
PASS: tests/test_opus_decode
PASS: tests/test_opus_encode
PASS: tests/test_opus_padding
PASS: tests/test_opus_projection
============================================================================
Testsuite summary for opus 1.3.1-107-gccaaffa9
============================================================================
# TOTAL: 14
# PASS:  14
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

Conclusion

In this post, we explored the package and learned how to compile it to generate binary files to execute. We also confirmed that the package does not utilize the auto-vectorization. So, we may try implementing the vectorization in two ways: compiler intrinsics or auto-vectorization. In the next post, we will see how we can add SVE2 codes by using intrinsics.