Cost-Efficient Integration Testing on vendor appliances running on EC2

We are building applications that are relying on monolithic commercial applications like SAP Business Objects or Tableau Software. Following our quality standard, we want to run reliable, efficient and cost-effective tests to validate our work. Here is how we are doing that using AWS EC2 instances.

The need

Integration Testing and Acceptance Testing are necessary steps to insure quality in our products. But this kind of tests needs to interact with existing vendor solutions, not available in an SaaS model. For example we are working a lot around SAP BusinessObjects and we need to run tests against this kind of software.

The first idea

An easy solution is to deploy an integration server, secure it to deny everyone the possibility to modify the content (it needs to stay stable since tests are relying on the content), and run it all time. Do that for example in your existing local infrastructure or drop that on a EC2 at AWS. This is a simple solution but it might be very costly (you only need it a couple of minutes a day ! ) and because content need to be the same you probably want to reset it at its original state before performing additional tests. You pay around 143 US$ for a t2.xlarge when it’s up all time.

The solution

Because our build process is managed by Maven we are searching for a Maven plugin solution for that. And lo and behold it exists !

We have decided to fork an existing project and adapt it to our specific needs. The plugin is able to create an EC2 instance on-demand based on an existing image when the integration test goal is launched and terminates it at the end of the goal. The customization allows us to wait until the vendor application is fully started on the EC2 (by listening to TCP ports).

It doesn’t rely on an existing EC2 instance but on a Amazon Machine Image (AMI) and this is good news :

• each integration test ran is done on the exact same AMI each time, we can rely on the server's content being absolutely consistent through time.
• multiple integration tests can be launched in parallel (2 developers, jenkins server…), each one will create a specific instance.

The project integration is very simple. You can call for the proper maven plugin in the pom.xml file :

<build>
<plugins>
    <plugin>
     <groupId>com.gbs.maven.plugins</groupId>
     <artifactId>ec2-maven-plugin</artifactId>
     <version>1.1.8</version>
     <configuration>
      <accessKey>XXX</accessKey>
      <secretKey>xxxx</secretKey>
      <ami>${bobj.ami.id}</ami>
      <key>gbandsmith</key>
      <type>t2.large</type>
      <securityGroups>
       <securityGroup>bobj</securityGroup>
      </securityGroups>
      <tags>
       <tag>
        <key>Name</key>
        <value>sapbi-integration-test</value>
       </tag>
      </tags>
      <waitPorts>
       <waitPort>6400</waitPort>
       <waitPort>6402</waitPort>
      </waitPorts>
     </configuration>
     <executions>
      <execution>
       <id>start-ec2-instance</id>
       <phase>pre-integration-test</phase>
       <goals>
        <goal>launchinstance</goal>
       </goals>
      </execution>
      <execution>
       <id>stop-ec2-instance</id>
       <phase>post-integration-test</phase>
       <goals>
        <goal>terminateinstance</goal>
       </goals>
       <configuration>
        <instanceId>${ec2.instance.id}</instanceId>
       </configuration>
      </execution>
     </executions>
    </plugin>
</build>

The execution and terminate goals are linked to pre-integration-test and post-integration-test phase of the classic Maven Failsafe plugin.

To use the AWS DNS entry we use the configuration of failsafe to set a System Property variable that we can use in our code:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <configuration>
    <systemPropertyVariables>
      <myvar>${ec2.instance.dns}</myvar>
    </systemPropertyVariables>
  </configuration>
</plugin>

We now have a robust integration test system that rely on AWS, with a very low cost (around 30US$ per month for a 10-person project).

Next

There is still some room for improvement. A few ideas that we have:

• Replace the failsafe plugin by something else. Currently when a test fail then the post-integration-test step is not trigged so the EC2 instance is not terminated.
• Optimize the build when we build a multi-module Maven project (currently each Maven module starts an independent instance and we‘re losing time when multiple modules are running IT tests).