What is mutation testing?
Mutation testing is a type of testing that validates new and existing tests. More formal definition from Wikipedia:
Mutation testing (or mutation analysis or program mutation) is used to design new software tests and evaluate the quality of existing software tests.
Why do we need mutation testing?
The problem that solves mutation testing is checking the validity of the existing and new tests. I often have met invalid unit tests that have covered only the happy path and no other paths.
When does it make sense for mutation testing?
So, if overall coverage is below 80%, you can skip mutation testing. You can return to mutation testing once you reach a threshold of 80% coverage.
How is mutation testing working?
The mutation testing itself is a form of white box testing. More or less mutation testing, trying to mutate or change your code and check that unit tests will fail.
The process of changing your code is called mutation. The mutation can be killed or survived during the test phase. The killed mutation is a good sign, notifying us that the unit tests caught the mutation and failed the unit test; on the other hand, if the mutation survived, that is a bad sign that the unit test did not die with code changes.
Implementation of mutation
First, we should add the following dependency on TestContainer for PostgreSQL.
In your pom.xml add following dependency.
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>postgresql</artifactId>
<scope>test</scope>
</dependency>
The next step is to add a basic class test, which will be responsible for creating a PostgreSQL test container.
package io.vrnsky.mutationdemo;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.annotation.DirtiesContext;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
@Testcontainers
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@DirtiesContext
public class DatabaseIntegrationTest {
@Container
public static PostgreSQLContainer<?> postgreSQLContainer = new PostgreSQLContainer<>()
.withPassword("inmemory")
.withUsername("inmemory");
@DynamicPropertySource
static void postgresqlProperties(DynamicPropertyRegistry registry) {
registry.add("spring.datasource.url", postgreSQLContainer::getJdbcUrl);
registry.add("spring.datasource.password", postgreSQLContainer::getPassword);
registry.add("spring.datasource.username", postgreSQLContainer::getUsername);
registry.add("spring.datasource.platform", () -> PostgreSQLContainer.NAME);
}
}
Let’s implement some basic functionality — the basic CRUD application.
The project will be to manage the book entity.
package io.vrnsky.mutationdemo.entity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.time.LocalDate;
import java.util.UUID;
@Data
@Table(name = "BOOK")
@Entity
@AllArgsConstructor
@NoArgsConstructor
public class Book {
@Id
@GeneratedValue
@Column(name = "ID")
private UUID id;
@Column(name = "AUTHOR")
private String author;
@Column(name = "ISBN")
private String isbn;
@Column(name = "PUBLISHED_DATE")
private LocalDate publishedDate;
@Column(name = "DELETED")
private boolean deleted;
}
To manage an entity, you need to create an interface that extends the JPA Repository interface.
package io.vrnsky.mutationdemo.repository;
import io.vrnsky.mutationdemo.entity.Book;
import org.springframework.data.jpa.repository.JpaRepository;
import java.util.UUID;
public interface BookRepository extends JpaRepository<Book, UUID> {
}
According to the most used structure of spring boot services, we need two more layers — service and controller. Let’s implement that
package io.vrnsky.mutationdemo.service;
import io.vrnsky.mutationdemo.entity.Book;
import io.vrnsky.mutationdemo.repository.BookRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.util.UUID;
@Slf4j
@Service
@RequiredArgsConstructor
public class BookService {
private final BookRepository bookRepository;
public Book create(Book book) {
return bookRepository.save(book);
}
public Book getById(UUID bookId) {
var book = bookRepository.findById(bookId);
if (book.isEmpty()) {
throw new RuntimeException();
}
return book.get();
}
public Book update(Book book) {
var existingBook = bookRepository.findById(book.getId());
if (existingBook.isEmpty()) {
throw new RuntimeException();
}
return bookRepository.save(book);
}
public void delete(UUID bookId) {
var existingBook = bookRepository.findById(bookId);
if (existingBook.isEmpty()) {
throw new RuntimeException();
}
existingBook.get().setDeleted(true);
bookRepository.save(existingBook.get());
}
}
As you can see, for now, we are using RuntimeException. It is a terrible practice. Sonar Qube also agrees with me. We will fix it later with custom runtime exceptions and controller advice.
BookController.java
package io.vrnsky.mutationdemo.controller;
import io.vrnsky.mutationdemo.entity.Book;
import io.vrnsky.mutationdemo.service.BookService;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import java.util.UUID;
@RestController
@RequiredArgsConstructor
public class BookController {
private final BookService bookService;
@PostMapping("/book")
public Book create(@RequestBody Book book) {
return bookService.create(book);
}
@GetMapping("/book/{bookId}")
public Book getById(@PathVariable UUID bookId) {
return bookService.getById(bookId);
}
@PutMapping("/book/{bookId}")
public Book update(@RequestBody Book book) {
return bookService.update(book);
}
@DeleteMapping("/{bookId}")
public void delete(@PathVariable UUID bookId) {
bookService.delete(bookId);
}
}
There is one more thing that we also need to improve, which is added documentation to our controllers. We will do it later, and I will use SpringDoc for that.
package io.vrnsky.mutationdemo.controller;
import io.vrnsky.mutationdemo.entity.Book;
import io.vrnsky.mutationdemo.service.BookService;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import java.util.UUID;
@RestController
@RequiredArgsConstructor
public class BookController {
private final BookService bookService;
@PostMapping("/book")
public Book create(@RequestBody Book book) {
return bookService.create(book);
}
@GetMapping("/book/{bookId}")
public Book getById(@PathVariable UUID bookId) {
return bookService.getById(bookId);
}
@PutMapping("/book/{bookId}")
public Book update(@RequestBody Book book) {
return bookService.update(book);
}
@DeleteMapping("/{bookId}")
public void delete(@PathVariable UUID bookId) {
bookService.delete(bookId);
}
}
It’s time to move on to write the unit test.
package io.vrnsky.mutationdemo.service;
import io.vrnsky.mutationdemo.DatabaseIntegrationTest;
import io.vrnsky.mutationdemo.entity.Book;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import java.time.LocalDate;
import java.util.UUID;
class BookServiceTest extends DatabaseIntegrationTest {
@Autowired
private BookService bookService;
@Test
@DisplayName("Test case: Creation of book")
@Tag("Positive")
void testThatCreationOfBookWorksCorrectly() {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
var createdBook = bookService.create(book);
Assertions.assertNotNull(createdBook.getId());
Assertions.assertEquals(createdBook.getAuthor(), book.getAuthor());
Assertions.assertEquals(createdBook.getIsbn(), book.getIsbn());
Assertions.assertEquals(createdBook.getPublishedDate(), book.getPublishedDate());
Assertions.assertFalse(createdBook.isDeleted());
}
@Test
@DisplayName("Test case: Update book")
@Tag("positive")
void testThatUpdateOfBookWorksCorrectly() {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
var createdBook = bookService.create(book);
createdBook.setDeleted(true);
var updatedBook = bookService.update(createdBook);
Assertions.assertTrue(updatedBook.isDeleted());
}
@Test
@DisplayName("Test case: Get book by id")
@Tag("Positive")
void testThatGetBookByIdWorksCorrectly() {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
var createdBook = bookService.create(book);
var existingBook = bookService.getById(createdBook.getId());
Assertions.assertNotNull(existingBook);
}
@Test
@DisplayName("Test case: Delete book")
@Tag("Positive")
void testThatDeleteBookWorksCorrectly() {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
var createdBook = bookService.create(book);
bookService.delete(createdBook.getId());
var deletedBook = bookService.getById(createdBook.getId());
Assertions.assertTrue(deletedBook.isDeleted());
}
@Test
@DisplayName("Test case: Update not existing book")
@Tag("Negative")
void testThatUpdateOfNotExistingBookThrowsException() {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
Assertions.assertThrows(RuntimeException.class, () -> bookService.update(book));
}
@Test
@DisplayName("Test case: Get by id not existing book")
@Tag("Negative")
void testThatGetByIdOfNotExistingBookThrowException() {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
Assertions.assertThrows(RuntimeException.class, () -> bookService.update(book));
}
@Test
@DisplayName("Test case: Deleting not existing book")
void testThatDeletionOfNotExistingBookThrowsException() {
Assertions.assertThrows(RuntimeException.class, () -> bookService.delete(UUID.randomUUID()));
}
}
Now, let’s move to the controller layer.
package io.vrnsky.mutationdemo.controller;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.vrnsky.mutationdemo.DatabaseIntegrationTest;
import io.vrnsky.mutationdemo.entity.Book;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.http.MediaType;
import org.springframework.test.web.servlet.MockMvc;
import java.time.LocalDate;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.delete;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.put;
import static org.springframework.test.web.servlet.result.MockMvcResultHandlers.print;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
@AutoConfigureMockMvc
class BookControllerTest extends DatabaseIntegrationTest {
@Autowired
private MockMvc mockMvc;
@Autowired
private ObjectMapper objectMapper;
@Test
@DisplayName("Test case: Creating a book")
@Tag("Positive")
void testThatCreateBookCorrectlyCreateBook() throws Exception {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
mockMvc.perform(post("/book")
.content(objectMapper.writeValueAsString(book))
.contentType(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andExpect(jsonPath("$.id").isNotEmpty())
.andDo(print());
}
@Test
@DisplayName("Test case: Updating existing book")
@Tag("Positive")
void testThatUpdatingExistingBookWorksCorrectly() throws Exception {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
var mvcResponse = mockMvc.perform(post("/book")
.content(objectMapper.writeValueAsString(book))
.contentType(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andExpect(jsonPath("$.id").isNotEmpty())
.andReturn();
var createdBook = objectMapper.readValue(mvcResponse.getResponse().getContentAsString(), Book.class);
createdBook.setAuthor("Yegor");
mockMvc.perform(put("/book/" + createdBook.getId())
.content(objectMapper.writeValueAsString(createdBook))
.contentType(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andExpect(jsonPath("$.author").value("Yegor"));
}
@Test
@DisplayName("Test case: Get existing book by id")
@Tag("Positive")
void testThatGetExistingBookByIdWorksCorrectly() throws Exception {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
var mvcResponse = mockMvc.perform(post("/book")
.content(objectMapper.writeValueAsString(book))
.contentType(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andExpect(jsonPath("$.id").isNotEmpty())
.andReturn();
var createdBook = objectMapper.readValue(mvcResponse.getResponse().getContentAsString(), Book.class);
createdBook.setAuthor("Yegor");
mockMvc.perform(get("/book/" + createdBook.getId())
.content(objectMapper.writeValueAsString(createdBook))
.contentType(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andExpect(jsonPath("$.id").value(createdBook.getId().toString()))
.andExpect(jsonPath("$.author").value("John Doe"));
}
@Test
@DisplayName("Test case: Deletion of existing book works correctly")
@Tag("Positive")
void testThatDeletionOfWorkBookWorksCorrectly() throws Exception {
var book = new Book(null, "John Doe", "123", LocalDate.now(), false);
var mvcResponse = mockMvc.perform(post("/book")
.content(objectMapper.writeValueAsString(book))
.contentType(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andExpect(jsonPath("$.id").isNotEmpty())
.andReturn();
var createdBook = objectMapper.readValue(mvcResponse.getResponse().getContentAsString(), Book.class);
createdBook.setAuthor("Yegor");
mockMvc.perform(delete("/" + createdBook.getId())
.content(objectMapper.writeValueAsString(createdBook))
.contentType(MediaType.APPLICATION_JSON))
.andExpect(status().isOk());
mockMvc.perform(get("/book/" + createdBook.getId())
.content(objectMapper.writeValueAsString(createdBook))
.contentType(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andExpect(jsonPath("$.id").value(createdBook.getId().toString()))
.andExpect(jsonPath("$.deleted").value(true));
}
}
The next step is to add the Pit plugin to perform mutation testing. By default, the plugin will try to mutate all code, so be precise about what you want to test.
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.15.3</version>
<dependencies>
<dependency>
<groupId>org.pitest</groupId>
<artifactId>pitest-junit5-plugin</artifactId>
<version>1.2.1</version>
</dependency>
</dependencies>
<configuration>
<timeoutConstant>15000</timeoutConstant>
<verbose>true</verbose>
<targetClasses>
<param>io.vrnsky.mutationdemo.service.*</param>
<param>io.vrnsky.mutationdemo.controller.*</param>
</targetClasses>
<targetTests>
<param>io.vrnsky.mutationdemo.service.*</param>
<param>io.vrnsky.mutationdemo.controler.*</param>
</targetTests>
<features>
<feature>+auto_threads</feature>
</features>
</configuration>
</plugin>
There is one more thing: Pitest is running in single-thread mode until you explicitly configure the auto threads feature.. As you can see, I have set verbose to true. It helps get more detailed information on what is going on.
Now, we can run mutation testing by following the command.
mvn org.pitest:pitest-maven:1.15.3:mutationCoverage
After completion, we can go to the target/pit-report folder and check the detailed report about mutation testing.
Top comments (0)