Deploying Designer products with multi-stage builds

Apama 10.3 introduces support for building and deploying Software AG Designer projects using multi-stage Docker builds and a new builder image. This article explains how it works and how to use it.

Issue 4, 2018	Download PDF

Adding Docker support to Designer projects

With 10.3 a new context menu item has been added to all Apama projects in Designer:

This will add a Dockerfile to the top level of your project. For the majority of EPL-only Apama projects the default Dockerfile requires no further changes to function. This will deploy Query files and integrate with Digital Event Services as well as correctly sequence any injections and event files at startup. It also includes your entire configuration, both custom and connectivity bundles in your project.

Building Designer projects

Once you have added Docker support you can use the Docker client to build an image from your project. This can be done on Windows® using the Docker client for Windows and either a local Linux® Docker install or a remote Docker server. It can also be done from Linux as part of a CI pipeline.

By default this will use the public Docker Store images for Apama:

docker build –t apama\_app ProjectDirectory

To use Docker Store you will have to first call docker login with your Docker credentials. Alternatively, if you have pushed your Apama base images into a custom repository you can override the images at build time:

docker build –t apama\_app \ --build-arg APAMA\_IMAGE=repository/apama- correlator \ --build-arg APAMA\_BUILDER=repository/apama builder \ ProjectDirectory

The resulting application image can then be run locally or pushed to a repository for cloud deployment with Docker or Kubernetes®. Simply running the container will automatically start Apama and deploy the configured application into it.

Multi-stage Dockerfile

A multi-stage Dockerfile which builds Apama looks like this:

ARG APAMA\_VERSION=10.3 ARG APAMA\_BUILDER=store/softwareag/apama-builder:${APAMA\_VERSION} ARG APAMA\_IMAGE=store/softwareag/apama-correlator:${APAMA\_VERSION} # Use the build environment FROM ${APAMA\_BUILDER} as builder # Copy project assets into build environment COPY --chown=sagadmin:sagadmin . /Project # Deploy the apama project RUN engine\_deploy --outputDeployDir ${APAMA\_WORK}/Project\_deployed \ /Project # Final image is based on the runtime base image FROM ${APAMA\_IMAGE} # Copy deployed project from the build environment COPY --from=builder ${APAMA\_WORK}/Project\_deployed \ ${APAMA\_WORK}/Project\_deployed WORKDIR ${APAMA\_WORK} # Run the deployed project CMD ["correlator", "--config", "Project\_deployed"]

With a few additions to handle Digital Event Services applications, this is exactly what you get when you add Docker support to your project through Designer. You can also implement this yourself if you want to tailor your image build process. In particular, if you have additional build steps, such as building plug-ins, you can edit the Dockerfile added by Designer to include those build steps in the builder image and then copy the results into the runtime image. Due to the multi-stage build you do not need to worry about any temporary files created during build steps being in the final image. Only things which were explicitly copied from the builder image will be included. This allows you to install compilers and other build tools during your build process. The builder image already provides a JDK and a Python® 3 interpreter.

Using multi-stage builds in CI pipelines

A common method of deploying Docker images is to use a CI pipeline direct from your source control to deployment. You can also do this with Docker-deployed Apama projects. Simply add Docker support to your project and commit it to your source repository. Since all of the Apama-specific build steps happen within the builder image your build server only needs to have the source control and Docker client tools available on it. Everything else will happen inside a properly configured environment inside Docker. The CI process will then produce an image pushed to a repository which can be directly deployed from any Docker or Kubernetes configuration.

In addition to building in your CI pipeline you can also run tests. The Apama builder image comes with a full Pysys and Python 3 environment which can be run as part of your project Dockerfile:

FROM ${APAMA\_BUILDER} as builder … RUN cd tests; pysys run … FROM ${APAMA\_IMAGE}

If the test run fails it will fail your Docker build. This means that your CI pipeline can validate the application with the test suite before deployment. Because the tests are being run in the builder environment none of the test output or even source directories need to be copied into the final deployed image.

Summary

The Apama builder image for Docker along with Software AG Designer integration provides a seamless development to deployment workflow including support for CI pipelines containing the full build and test process before producing images suitable for deployment in all of the major cloud providers.

To learn more, check the Software AG Apama documentation and the Docker samples available in your Apama installation and on https://github.com/SoftwareAG/apama-streaming-analytics-docker-samples