Modern Software Development Tools and oneAPI Part 1

This will be the last blog post for this year (unless, I manage to get a second one in by the stroke of midnight tomorrow!).
I wanted to end 2022 with a departure from the last two blog posts. In this one, we're going to be looking at oneAPI toolchain from a different perspective.

I wanted to build a pure oneAPI environment that uses two things:

• A different build system than your usual CMake and
• an opportunity to use a Linux based IDE to write code.

Most HPC code runs on Linux in Data center. Linux is known for using arcane tools like VIM or Emacs and a build system. Sure, you can use VScode on Linux and that can be a viable option. However, I am an open-source person; I live and breathe the ideology and I am a true believer. oneAPI is an open platform, and we should use it on an open platform.

The two pieces of software I want to introduce you all is GNOME Builder and Meson.

GNOME Builder

GNOME Builder is an IDE that is primarily targeted at building GNOME-based applications. Unlike your typical IDE, GNOME Builder uses containers for building software internally. Containers in these case are flatpak SDKs - containers that contain everything you need to build an application. With GNOME Builder, you can get started writing code without the tedium of installing the entire software development toolchain you'd need to build said applications. That means, you aren't going to need to install a compiler, linker, libraries and anything else. Everything is all-inclusive - much like a vacation resort in Cancun, say! :-)

Back to Builder! oneAPI is not one of the options for building software inside GNOME Builder. GNOME Builder includes containers to build against GNOME software.

GNOME Builder is the brainchild of Christian Hergert, a long time Free Software programmer who was frustrated with the current state of tools to build software within the application community and started the Builder project about 7 years ago and was initially funded by a kickstarter. Through some great luck, Christian is now paid by Red Hat to help build GNOME Builder as well as improving the application building story on Linux and other platforms.

There is something wonderfully intriguing about Builder which is why I'm highlighting it here and why I picked it as the IDE of choice to write oneAPI-related code.

This one special trick

Builder will allow you to use a podman or docker container to run everything. So in this case, we're going to create that all-inclusive experience! Once you've done the work of building a container with all the oneAPI tools into it, others can then re-use the container as a fixed environment and other folx can clone the project you are working on and then easily collaborate.

Meson

The next piece of software I wanted to highlight is Meson. Meson is a build system that is written in python and tries to be an intuitive system that tries to do the right thing through easily understandable syntax. For those of you who have ever used autoconf - this was the application community's response to autoconf.

Autoconf during its heyday was a massive boon to those who were writing code that could work on many different UNIX and Linux distributions. However, autoconf was difficult to figure out and most folx simply copy from another project and then move on. Writing anything sophisticated required extensive, expended effort.

In my personal opinion, build systems are really hard to get right and there are so many oddities in how we build software that a build system has to get right, in order to be effective.

Meson is written by Finnish programmer, Jussi Pakkanen, who was frustrated with the current state of build systems and their arcane configuration syntax and sometimes rather unexpected behaviors!

Meson can be better described as a system that generates the configurations for build systems to use. It isn't a full-fledged build system like Make or CMake. In fact, you could easily re-use CMake configuration files in Meson. It has a concept of a backend and can generate config for Xcode on MacOS, VScode on Windows and Ninja on Linux systems. Meson easily integrates with profilers and debuggers and is designed not to build within source tree but in a designed build area.

For those not familiar with ninja, it is an extremely fast build system that has been shown to be effective in building software very quickly!!

Build an oneAPI Container

In this first part, we will focus on building an oneAPI container based on the Ubuntu 20.04 LTS release since that is what oneAPI works optimally on.

I will be using Fedora 37 SilverBlue edition. I like SilverBlue as it is built with containerized environments in mind. It allows you to build different container environments that you can enter and exit from on the command line and still easily integrate with the desktop.

Let's start then with building our oneAPI environment so that it will be able to run a simple oneAPI sample program.

Staying true to the spirit of open-source, I will build this environment from source and only use what's available on GitHub.

To start, find and install the 'distrobox' tool on your distro.

dnf install distrobox -y

Distrobox allows you to create containerized environments from the command line.

I use podman as I live in the Fedora world. Podman is a command line compatible version of Docker. It's reasonable that the following could be done through a Dockerfile but the oneAPI libraries changes often enough that this blog post would become stale in short order.

$ distrobox create oneapi -i docker.io/library/ubuntu:20.04

This will create a container called oneapi with an Ubuntu 20.04 setup.

$ distrobox enter oneapi

Will let you enter the container.

The beauty of distrobox is that you are in this container, it has mounted your home directory and you have essentially inherited your desktop system but the container is Ubuntu - and so you can use the Ubuntu distro tools to install software. Pretty neat, huh?

The first step is to build the DPC++ oneAPI compiler from source.

$ mkdir -p ~/src/sycl_workspace

You can use whatever area you want. I'm following this guide to build the compiler.

Prequisites

Let's first grab our pre-requisites for building.

$ apt install git python3 gcc c++ libstdc++ libstdc++-9-dev python3-pip python3-distutils python-distutils-extra python3-psutil -y
$ pip3 install meson
$ pip3 install ninja

Now that we have our build environment.

Build DPC++ Compiler

$ cd ~/src/sycl_workspace
$ export DPCPP_HOME=`pwd`
$ git clone https://github.com/intel/llvm -b sycl

We can start the actual build:

$ python $DPCPP_HOME/llvm/buildbot/configure.py
$ python $DPCPP_HOME/llvm/buildbot/compile.py

At the end of this exercise, you should have a working oneAPI DPC++ compiler.

But we aren't done yet - we still need to add some of the oneAPI libraries and runtimes to make our simple oneAPI example work.

We first need to install our low level runtimes: the things that recognizes accelerators. For now, we'll use the ones that recognize the x86 Intel processors as that is what is on my laptop right now.

We will first need to identify the latest versions of the runtimes we need to download. You need to look this up in the dependency.conf.

$ sudo mkdir -p /opt/intel 
$ sudo mkdir -p /etc/OpenCL/vendors/intel_fpgaemu.icd
$ cd /tmp
$ wget https://github.com/intel/llvm/releases/download/2022-WW50/oclcpuexp-2022.15.12.0.01_rel.tar.gz
$ wget https://github.com/intel/llvm/releases/download/2022-WW50/fpgaemu-2022.15.12.0.01_rel.tar.gz
$ sudo bash
# cd /opt/intel
# mkdir oclfpgaemu-<fpga_version>
# cd oclfpgaemu-<fpga_version>
# tar xvfpz /tmp/fpgaemu-2022.15.12.0.01_rel.tar.gz
# cd ..
# mkdir oclcpuexp_<cpu_version>
# cd oclcpuexp-<cpu_version>
# tar xvfpz /tmp/oclcpuexp-<cpu_version>
# cd ..

Now to create some configuration files.

# pwd
/opt/intel
# echo  /opt/intel/oclfpgaemu_<fpga_version>/x64/libintelocl_emu.so >
  /etc/OpenCL/vendors/intel_fpgaemu.icd
# echo /opt/intel/oclcpuexp_<cpu_version>/x64/libintelocl.so >
  /etc/OpenCL/vendors/intel_expcpu.icd

We'll need to grab a release of oneTBB from github

$ cd /tmp
$ wget https://github.com/oneapi-src/oneTBB/releases/download/v2021.7.0/oneapi-tbb-2021.7.0-lin.tgz

and now extract it.

$ cd /opt/intel
$ sudo bash
# tar xvfpz /tmp/oneapi-tbb-2021.7.0-lin.tgz

We'll need to reference some of the libraries in the oneTBB directory in our build.

# ln -s /opt/intel/oneapi-tbb-<tbb_version>/lib/intel64/gcc4.8/libtbb.so /opt/intel/oclfpgaemu_<fpga_version>/x64
# ln -s /opt/intel/oneapi-tbb-<tbb_version>/lib/intel64/gcc4.8/libtbbmalloc.so /opt/intel/oclfpgaemu_<fpga_version>/x64
# ln -s /opt/intel/oneapi-tbb-<tbb_version>/lib/intel64/gcc4.8/libtbb.so.12 /opt/intel/oclfpgaemu_<fpga_version>/x64
# ln -s /opt/intel/oneapi-tbb-<tbb_version>/lib/intel64/gcc4.8/libtbbmalloc.so.2 /opt/intel/oclfpgaemu_<fpga_version>/x64

# ln -s /opt/intel/oneapi-tbb-<tbb_version>/lib/intel64/gcc4.8/libtbb.so /opt/intel/oclcpuexp_<cpu_version>/x64
# ln -s /opt/intel/oneapi-tbb-<tbb_version>/lib/intel64/gcc4.8/libtbbmalloc.so /opt/intel/oclcpuexp_<cpu_version>/x64
# ln -s /opt/intel/oneapi-tbb-<tbb_version>/lib/intel64/gcc4.8/libtbb.so.12 /opt/intel/oclcpuexp_<cpu_version>/x64
# ln -s /opt/intel/oneapi-tbb-<tbb_version>/lib/intel64/gcc4.8/libtbbmalloc.so.2 /opt/intel/oclcpuexp_<cpu_version>/x64

Now we need to configure the library paths:

# echo /opt/intel/oclfpgaemu_<fpga_version>/x64 > /etc/ld.so.conf.d/libintelopenclexp.conf
# echo /opt/intel/oclcpuexp_<cpu_version>/x64 >> /etc/ld.so.conf.d/libintelopenclexp.conf
# ldconfig -f /etc/ld.so.conf.d/libintelopenclexp.conf

and we're done! Now we need to make sure that this toolchain actually works. So run this test.

Make sure you are not root.

$ python $DPCPP_HOME/llvm/buildbot/check.py

If you come back with no failure then, congratulations, you're in good shape!! Sometimes, there might be a few missing dependencies, especially when it comes to python.

We are now ready to create a simple SYCL application and test. I'm going to re-use the one that is located on Github.

Let's create our workspace and build this sample project.

$ mkdir -p ~/src/simple-oneapi/
$ cd ~/src/simple-oneapi
$ export PATH=$DPCPP_HOME/llvm/build/bin:$PATH
$ export LD_LIBRARY_PATH=$DPCPP_HOME/llvm/build/lib:$LD_LIBRARY_PATH
$ cat > simple-oneapi.cpp

#include <sycl/sycl.hpp>

int main() {
  // Creating buffer of 4 ints to be used inside the kernel code
  sycl::buffer<sycl::cl_int, 1> Buffer(4);

  // Creating SYCL queue
  sycl::queue Queue;

  // Size of index space for kernel
  sycl::range<1> NumOfWorkItems{Buffer.size()};

  // Submitting command group(work) to queue
  Queue.submit([&](sycl::handler &cgh) {
    // Getting write only access to the buffer on a device
    auto Accessor = Buffer.get_access<sycl::access::mode::write>(cgh);
    // Executing kernel
    cgh.parallel_for<class FillBuffer>(
        NumOfWorkItems, [=](sycl::id<1> WIid) {
          // Fill buffer with indexes
          Accessor[WIid] = (sycl::cl_int)WIid.get(0);
        });
  });

  // Getting read only access to the buffer on the host.
  // Implicit barrier waiting for queue to complete the work.
  const auto HostAccessor = Buffer.get_access<sycl::access::mode::read>();

  // Check the results
  bool MismatchFound = false;
  for (size_t I = 0; I < Buffer.size(); ++I) {
    if (HostAccessor[I] != I) {
      std::cout << "The result is incorrect for element: " << I
                << " , expected: " << I << " , got: " << HostAccessor[I]
                << std::endl;
      MismatchFound = true;
    }
  }

  if (!MismatchFound) {
    std::cout << "The results are correct!" << std::endl;
  }

  return MismatchFound;
}

Let's build our simple oneapi source code!

$ clang++ -fsycl simple-sycl-app.cpp -o simple-sycl-app

It should compile and run without any errors.

If all works as anticipated, you should have a working setup.

Setting up the container to code SYCL when you enter

Now, the next step is to make this container useful when you enter and have it always ready to build a sycl app.

Exit out of the container using the 'exit' command and you should be back on the host operating system.

Type:
$ uname -a

On my system, I get:

Linux fedora 6.0.13-300.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Dec 14 16:15:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Re-enter the container:

$ distrobox enter oneapi
$ uname -a
Linux oneapi.fedora 6.0.13-300.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Dec 14 16:15:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

You will notice that after "Linux" when you are in the container, there is a "oneapi" prefix to fedora.

We can take advantage of that. Let's make sure that when we enter the container that we can set things up from the shell perspective to be ready to write SYCL code.

Add this bit to your .bashrc:

oneapi=``uname -a | grep -c oneapi``

if [ $oneapi -gt 0 ]; then
   echo "Initializing oneAPI"
   export DPCPP_HOME="/var/home/sri/src/dpcplusplus"
   export PATH="$PATH:/var/home/sri/.local/bin:$DPCPP_HOME/llvm/build/bin"
   export LD_LIBRARY_PATH="$DPCPP_HOME/llvm/build/lib"
fi

Make sure you replace 'sri' with your your login details

Now when we enter the 'oneapi' container our environment will be properly initialized.

Let's stop here, and we'll pick it up in the next post. The next post will focus on creating a meson setup around this simple oneapi code. Part 3 will focus on taking our meson configured source code and using GNOME Builder to build it. Stay tuned!

Photo by iMattSmart on Unsplash