Testcontainers for Hashicorp Consul and Vault

Testcontainers for Vault, Consul and LocalStack with DIND.

In my earlier post, I touched on some interesting architectural patterns for Configuration and Secret Management for your microservices on k8s. I highly recommend reading if you haven’t already. This article builds on the same idea and highlights the importance of integration tests of your service and how we could leverage Testcontainers to ease out some of these challenges such as divergent test environments and configuration or even mock components used in your tests that may not be enough to catch issues early on in your continuous integration pipelines and provide adequeate code coverage.

Testcontainers

Testcontainers is a JVM library that allows users to run and manage Docker images with Java code and frameworks such as JUnit. The most common use-cases for Testcontainers include integration tests against microservices with external dependencies such as Vault, Consul, AWS, Databases, Cache Frameworks, and more. With an API-based approach, it is easy to manage the lifecycle of a container along with the configuration that may be required for your services.

Integration tests with Testcontainers

Some obvious benefits for integration testing with Testcontainers:

• TestContainers can help your tests mirror managed environments and components as closely as possible.
  • For sourcing secrets and configuration from Vault (includes the various secret backends)and consul.
  • Integration with cloud providers such as AWS (most commonly used services if not all), GCloud (incubating), Azure (incubating).
  • Databases such as PostgreSQL or MySQL etc.
  • Kafka, RabbitMQ for distributed messaging
  • The list goes on. Check out the full list of Testcontainer modules.
• Testing compatibility and tech stack upgrades for client libraries and dependencies such as spring cloud (Vault, Consul, AWS), AWS SDK, etc.

Why Vault and Consul?

Secret Management in microservices needs to be high on your priority list to build secure and scalable microservices. Secrets can be sensitive , dynamic, and time-bound. They require proper access control models with audit logs and encryption. We also need to support unique life-cycle policies and rotations for microservices. While there are a few options that may work for your needs, Hashicorp Vault is certainly the most popular and comprehensive solution in this space.

Configuration Management also presents a set of challenges with microservices. Support for static and dynamic configuration, externalized configurations, watching for changes, and updating application configuration without any service disruption are some of the key features we would expect from microservices. In addition, the microservice ecosystem in mature organizations often leads to a web with many microservices deployed but also inter-connected where Service Discovery or even a Service Mesh becomes important in a cloud infrastructure to manage endpoint configuration, load balance, security, etc. Hashicorp Consulis a great fit for meeting these requirements.

Testcontainers with Vault

The Hashicorp Vault Testcontainer module aims to solve your app’s integration testing with Vault. You can use it to source static and dynamic credentials for your application as well as test how your application may behave with Vault by writing different test scenarios in Junit such as corner cases like lease rotations, lease expiry, exception handling, etc.

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>vault</artifactId>
    <version>${testcontainers.version}</version>
    <scope>test</scope>
</dependency>

Testcontainers for Consul

One of the challenges we hit was that there wasn’t a Testcontainer module for consul. Based on some discussions with the test container teams (#4680) we decided to fork off an existing project, polished it a bit, and published this artifact for the OSS community as part of Houghton Mifflin Harcourt. This also turns out to be a tiny milestone in some ways as it is our first (out of many more to come) OSS artifact on Sonatype!!!

<dependency>
 <groupId>com.hmhco</groupId>
 <artifactId>testcontainers-consul</artifactId>
 <version>0.0.4</version>
 <scope>test</scope>
</dependency>

The module supports legacy and newer versions of Consul, ACL, Clustering, and more. The project can be found on Github and we welcome contributions via pull request and/or the discussion forum for any issues or improvements! Eventually, the hope would be to get this module added to testcontainers-java and be supported alongside the rest of the modules.

Some useful pointers when working with Testcontainers

• There may be test scenarios that require a Testcontainer to be able to talk to another in the scope of a test. TestContainers networking support makes it easy for you to do that with a generic host address “ host.testcontainers.internal:{port} ” when looking up containers and ports that may be exposed.
• An example would be to write a test when integrating Vault and Consul KV via ACL or even Vault integrating with other secret backends such as AWS (localstack) or Database backends (PostgreSQL or similar.)
• Supports Junit 4 and Jupiter/Junit 5 and Spock. Choose what’s best for your test framework.
• Manage Testcontainer life-cyclesappropriately.
• Often you may want to re-use a Testcontainer across your tests. This may also help speed up your test phase.
• Use containers for your tests and pipelines. Various patterns including DIND are available.
• Running an image for every test method, image per class, or even running one image for all integration test executions. When sharing an image we need to pay close attention to test data and rollback to clean up the state after test execution.
• Configure testcontainers.properties when working with private docker registries.
• hub.image.name.prefix ={your_private_registry}
• Also see image dependencies for what you may need on your private registry for Testcontainers and getting around any Docker Rate Limiting.
• Configuring logback to see Testcontainer logs is useful at development time and troubleshooting any issues. You could also stream container logs if you choose to.
• Use container labels and image pull policy as appropriate to make sure you benefit from any caching for images that are not changing often. Please see advanced options for more details on this.
• Configure wait_timeouts for containers. Also not a bad idea to Assert the container is up before you kick off your tests.

Assert.assertTrue(yourContainer.isRunning()));

• Follow the FIRST principle for your tests as defined in the book Clean Code: A Handbook of Agile Software Craftsmanship written by Robert C. Martin. It’s a great read for coding best practices and I highly recommend it!

Getting Started

• A quick start for a spring-boot service with Spring Cloud Vault and Consul has been provided.
• This example includes integration tests for Consul KV, Vault Secret Backends (KV, Consul, AWS).
• It also includes a docker-compose recipe for running all of these integrations on your local development environment.

Summary

The addition of consul as an independent Testcontainer module allows you to integrate Vault and Consul for your test pipelines and add a lot more coverage for your code! I also hope the quickstart examples provided above serve as a good starting point for adding integration tests to your microservice and standardizing your development and test pipelines with best practices for configuration and secret management.

Also, tune in to HashiTalks 2022 on Feb 17/18, 2022 if you are interested in learning more about HashiCorp Vault, Consul, and many more services for your cloud infrastructure.

Speaker card for Hashitalks 2022.

If you would like to learn more about Spring Cloud and integration with Vault and Consul for your microservice please join me at the HashiTalks on Feb 17, 12:05–12:35 GMT. We have a great lineup for speakers and topics this year and looking forward to speaking at the event as well as learning a lot more from the Hashicorp user group!

See you there!