Flaky unit tests in Angular

Introduction

A flaky test is an unstable test that randomly passes or fails despite no changes in the code or the test itself.

When unit tests are a part of the CI pipeline, flaky tests become the real problem. Tests unpredictably fail and every time devs have to spend precious time investigating and fixing. Also, flaky tests reduce confidence in tests in general.

In this article I want to research the nature of flaky unit tests in Angular and consider possible fixes.

Flaky test example

The simple component

Let's say we are developing a blog engine. The blog has the Backoffice, kind of an admin page, where authenticated users with admin permissions can set some settings, manage blog pages, etc.

Each page of the blog has a header. One of the header responsibilities is

• render the Backoffice link if the current user is has admin permissions
• hide the Backoffice link if the current user is not an admin

I'm going to implement it as simple as possible. You can find the code in Github Repo

Here is our typing

export interface User {
  name: string,
  isAdmin: boolean,
}

The service does nothing but storing the user data.

@Injectable({
  providedIn: 'root'
})
export class UserService {
  currentUser: User;
  constructor() { }
}

And finally the header component that consumes user data from the server and renders a link

@Component({
  selector: 'app-header',
  template: `
  <div *ngIf="user">
    <a class="backoffice-link" *ngIf="user.isAdmin" href="/backoffice"></a>
  </div>
`,
})
export class HeaderComponent implements OnInit {
  user: User;
  constructor(private userService: UserService) { }

  ngOnInit(): void {
    this.user = this.userService.currentUser;
  }
}

The tests for the simple component

Now I'm going to write some unit tests. Most of all, I want to cover <a>. It should be hidden for regular users, but visible for admins.

The mock for a regular non-admin user

import { User } from "./user.interface";

export const mockReguarUser: User = {
  name: 'John Doe',
  isAdmin: false,
}

And here are the tests. I guess, it's the most straightforward way to write unit tests for our case

describe('HeaderComponent', () => {
  let component: HeaderComponent;
  let fixture: ComponentFixture<HeaderComponent>;
  const userServiceStub = { currentUser: mockReguarUser }; // (1)

  beforeEach(async () => {
    await TestBed.configureTestingModule({
      declarations: [ HeaderComponent ],
      providers:[
        { provide: UserService, useValue: userServiceStub } // (2)
      ]
    })
    .compileComponents();
  });

  beforeEach(() => {
    fixture = TestBed.createComponent(HeaderComponent);
    component = fixture.componentInstance;
  });

  it('should not render Backoffice link for regular users ', () => {
    fixture.detectChanges(); // to trigger ngOnInit
    const link = fixture.debugElement.query(By.css('.backoffice-link'));
    expect(link).toBeFalsy(); //(3)
  });

  it('should render Backoffice link for admins users ', () => {
    userServiceStub.currentUser.isAdmin = true; //(4)
    fixture.detectChanges(); // to trigger ngOnInit
    const link = fixture.debugElement.query(By.css('.backoffice-link'));
    expect(link).toBeTruthy();
  });
});

Basically, I create a stub for the UserService, the mockReguarUser constant is used as user data (1). Then, I provide the stub to the component (2). The component consumes the service stub via DI engine, reads user data and render HTML according to user properties.

In (3) I check if the link is not displayed for a regular user. In (4) I change the isAdmin flag in mocked data to check how the header component handles admin users.

The tests look fine, but when I run Karma the strange things happen.

Initially, tests are green, but then I start hitting the reload button to rerun test suit. Surprisingly the particular test passes or fails in a random way. So, without any changes in the code I get different results many times. That's exactly what a flaky test is.

Researching a flaky test

First, let's figure out which test is unstable. The error message says

Looks like the the header component sometimes renders the Backoffice link for regular users, which it definitely should not do. I need to debug the test to find the root cause.

Unfortunately, the test result is unstable. It makes debugging too difficult. I need a reliable way to reproduce the test failure to be able to dive deeper. Here we need to learn about the random seed.

The seed

By default Jasmine runs unit tests in a random order. Random tests execution helps developers write independent and reliable unit tests. Basically it's what the I letter in F.I.R.S.T stands for. Read more about the FIRST stuff here

However this random order is controllable. Before running tests Jasmine generates a random number which is called the seed. Then the order of tests execution is calculated according to the seed. Also, Jasmine let us know which seed were used for the test run. Moreover, the seed can be provided via config to make Jasmine run tests in the same order over and over again.

That's what we can do to reproduce the failing test execution

1) Obtain the seed used for a failed test run. It can be taken from the browser's report

If a flaky test detected in CI and the browsers report is not available, the Jasmine order reporter can be used. The only thing is that you have to apply it beforehand. Then the seed can be found in logs of the CI pipeline

Chrome 115.0.0.0 (Windows 10) JASMINE ORDER REPORTER: Started with seed 05217

2) Once we know the seed that results in failing test order, it can be applied in Karma configs.

// karma.config.js
module.exports = function (config) {
  config.set({
    // ... other settings
    client: {
      jasmine: {
        random: true,
        seed: '05217' // <----------
      },
    },
  });
};

Now tests will be executed in the same order every time. In our case it means that the issue with the nasty unit test can be reproduced and investigated.

Studying the flaky test

We already discovered that the header components sometimes renders the Backoffice link for regular users. Let's figure out why. I simply put debugger in the ngOnInit method and run tests to check what's going on when the flaky test gets executed.

It turned out that the this.userService.currentUser.isAdmin is true when we run a test for a regular user. But the property is false in mockReguarUser we use in tests. How it becomes true?

The reason is the order of tests execution.

describe('HeaderComponent', () => {
  const userServiceStub = { currentUser: mockReguarUser };

  beforeEach(async () => {
    await TestBed.configureTestingModule({
      providers:[
        { provide: UserService, useValue: userServiceStub }
      ]
    })
    .compileComponents();
  });

  // (1) 
  it('should not render Backoffice link for regular users ', () => {
    fixture.detectChanges(); // to trigger ngOnInit
    const link = fixture.debugElement.query(By.css('.backoffice-link'));
    expect(link).toBeFalsy();
  });

  // (2) 
  it('should render Backoffice link for admins users ', () => {
    userServiceStub.currentUser.isAdmin = true;
    fixture.detectChanges(); // to trigger ngOnInit
    const link = fixture.debugElement.query(By.css('.backoffice-link'));
    expect(link).toBeTruthy();
  });
});

When the test (1) executed before (2) everything is fine. But when (2) executed first we face the problem.

Test (2) sets userServiceStub.currentUser.isAdmin to true.
But userServiceStub.currentUser is a reference "shared" between both tests. So, when the test (1) is executed next, it works with the modified mock! Having isAdmin = true results is unexpected behavior and the test fails.

In the specific order of execution the test (2) becomes a criminal and the test (1) is a victim.

I think now it's clear why tests unpredictably failed/passed when executed multiple times in a row.

Fixing the flaky test

Cloning mocks

We need to make tests more isolated from each other to fix the problem. Let's create a new copy of the mock for each test. Note how I clone mockReguarUser in beforeEach section to ensure that each test gets a separate mock. At the same time the original mock is kept intact.

describe('HeaderComponent', () => {
  let component: HeaderComponent;
  let fixture: ComponentFixture<HeaderComponent>;

  // use the mock regular user
  const userServiceStub: UserService = { currentUser: {} as User };

  beforeEach(async () => {
    userServiceStub.currentUser = {...mockReguarUser} // clone mock
    await TestBed.configureTestingModule({
      declarations: [ HeaderComponent ],
      providers:[
        { provide: UserService, useValue: userServiceStub }
      ]
    })
    .compileComponents();
  });

Now a test can modify its mocks whatever it wants. The changes will not affect other tests.

Deep cloning

The trick with cloning via the spread operator works fine because the mockReguarUser has no nested objects. But the operator creates a shallow copy. It is not enough for more complex mocks, since the nested objects will be copy by reference and the data still be shared among tests causing the same problem.

Lodash cloneDeep is quite useful to handle complex mock cloning. It would be as simple as

  userServiceStub.currentUser = cloneDeep(mockReguarUser)

Flaky tests in multiple components

Above we considered the relatively simple problem. The test that modifies mocks and the flaky test that unexpectedly fails due to the modifications both belong to the same component. And both tests sit in the same .spec.ts file.

In more complex apps this problem might be more complicated. Changing mocks made in suits for one component might cause test flakiness for other component. The components might even belong to different Angular modules.

In that case the investigation might be more difficult and solution probably will be more sophisticated. But the main idea is still the same. Most likely the problem can be fixed by applying cloning to prevent tests interaction via references in mocks.

Links

• my Github repo with the example of a flaky test
• Jasmine order reporter
• Lodash cloneDeep