I'm working on a big UI Testing Best Practices project on GitHub, I share this post to spread it and have direct feedback.
Speaking about UI Testing (remember that we are speaking about the UI only, not the underlying JavaScript code), there are three main test types:
- Component tests
- UI Integration tests
- End-to-end (E2E) tests
Component tests
The unit tests of a UI, they test every single component in an isolated
environment.
Developing components in isolation is important because it allows you to isolate them from the corresponding container/use. A component exists to isolate a single behavior/content (the Single responsibility principle) and therefore, coding it in isolation is profitable.
There are many ways and tools to develop components in isolation but
Storybook became the standard choice because of its effectiveness and its ecosystem.
Components have three types of contracts: the generated output (HTML), their visual aspect (CSS), and the external APIs (props and callbacks).
Testing every aspect could be cumbersome, that's where Storyshots comes in
handy. It allows you to automate:
• the snapshot tests: a snapshot is an output generated by your component, a string containing all the rendered HTML. If the generated HTML changes,
accidentally or not, the snapshot tests fail and you can choose if the changes were intentional or not.
• the visual regression tests: the visual aspect of the component compared pixel by pixel with the previous one, again, you are prompted to choose if you accept the changes or not.
These tests are launched by Storyshots automatically on every Storybook page (AKA "stories").
• the callback tests: a small React container app renders the component passing it some callbacks. Then, the user interactions are simulated and passed the callback is expected to be called.
React Testing Library is the standard library of choice for this kind of tests
• the interaction/state tests: some interactions with a component expect correct state management. This kind of test must be written from the consumer point of view, not from the inner one (ex. the value of the input field when the user fills it, not the inner component state). An interaction/state test should assert the input field value after the keyboard events triggered.
UI Integration tests
They run the whole app in a real browser without hitting a real server.
These tests are the ace in the hole of every front-end developer. They are blazing fast and less exposed to random failures or false negatives. Cypress is perfect for UI Integration tests.
The front-end application does not know that there is not a real server: every AJAX call is resolved in no time by the testing tool. Static JSON responses (called "fixtures") are used to simulate the server responses. Fixtures allow us to test the front-end state simulating every possible back-end state.
Another interesting effect: Fixtures allow you to work without a working back-end application. You can think about UI Integration tests as "front-end-only tests".
At the core of the most successful test suites, there is a lot of UI Integration tests, considering the best type of test for your front-end app.
End-to-end (E2E) tests
They run the whole app interacting with the real server. From the user
interactions (one of the "end") to the business data (the other "end"): everything must work as designed. E2E tests are typically slow because
• they need a working back-end application, typically launched alongside the front-end application. You can't launch them without a server, so you depend on the back-end developers to work
• they need reliable data, seeding and cleaning it before every test
That's why E2E tests are not feasible to be used as the only/main test type. They are pretty important because they are testing everything (front-end + back-end) but they must be used carefully to avoid brittle and hour-long test suites.
In a complete suite with a lot of UI Integration tests, you can think about E2E tests as "back-end tests". What flows should you test through them?
• the Happy Path flows: you need to be sure that, at least, the users are able to complete the basic operations
• everything valuable for your business: happy path or not, test whatever your business cares about (prioritizing them, obviously)
• everything that breaks often: weak areas of the system must be monitored too
Identifying/defining the type of test is useful to group them, to limit their scope, and to choose where to run them or not though the whole application and deployment pipelines.
Again, Cypress is my tool of choice for E2E tests.
Name them wisely
You can write a lot of different UI tests and it's a good habit to have a common way of naming the test files.
It's useful because often you need to run just a type of tests, the situations could be:
- during the development process, you need to run just some of them
- you're changing some related components and you need to check that the generated markup does not change
- you're changing a global CSS rule and you need to run only the visual tests
- you're changing an app flow and you need to run the whole app integration tests
- your DevOps colleague needs to be sure that everything is up and running and the easiest way to do that is launching just the E2E tests
- your building pipeline needs to run just the integration and E2E tests
- your monitoring pipeline needs a script to launch the E2E and monitoring tests
If you name your tests wisely, it will be really easy to launch just some kind of them.
Cypress:
cypress run --spec \"cypress/integration/**/*.e2e.*\"
Jest:
jest --testPathPattern=e2e\\.*$
A global way to name the test files does not exist, a suggestion could be to name them with:
- the subject you are testing
- the kind of test (
integration,e2e,monitoring,component, etc.) - the test suffix of choice (
test,spec, etc.) - the file extension (
.js,.ts,.jsx,.tsx, etc.)
all of them separated by a period.
Some examples could be
authentication.e2e.test.tsauthentication.integration.test.tssite.monitoring.test.js-
login.component.test.tsxetc.
Top comments (10)
Another great article, thank you!
From your experience, what are the advantages of using StoryShots over Chromatic tests?
Also, what are your opinions about using Playwright instead of Cypress? Playwright has become quite popular, and it seems much faster than Cypress. Furthermore, it has access to browser APIs that Cypress can't access, like memory usage and network requests. I'm interested to hear your thoughts about this :)
Hey Omri!
Consider that the post is quite old and the content is not update 😊
So I would go for Chromatic tests nowadays :) But not for the snapshot tests themselves (I shared here what I think about them and I do not suggest snapshot testing anymore) but for the component tests that's way more powerful, ha you already used them?
Playwright is way faster, it's more stable, and its UI mode is great! Anyway, what you see in the UI mode is not the live application but a series of snapshots, that means you can debug the HML+CSS but not the JS state, instead Cypress can. The day they Playwright fixes this problem I will jump immediately on it 😊 Playwright is great!!!
Anyway, I still feel the lack of a service like the Cypress Dashboard but it's just a matter of time or external SaaS 😊
What are your thougths about the topic? Which one you use?
Hey Stefano! :)
I actually didn't notice the publish date 😅
So if I get it right, you recommend Chromatic tests for the interactions + screenshots, but nothing more. Yes?
If I got you right, then it's pretty much what we do as well. We use a lot of Chromatic interaction tests in our components library (named Vibe). It works quite well for us, although it has some limitations (like the inability to write test suites).
Do you use Chromatic interaction tests a lot?
I wasn't aware of the difference between Cypress and Playwright in regards to JS debuggability, that's quite interesting. Currently we rely heavily on Cypress, but we would have wanted to shift to Playwright at some point. In addition to the advanced features that Playwright has, it can also run in parallel on AWS Lambda (for example). This can be very useful if you run many tests. Have you ever tried something like this?
Yes, today I suggest this
To be 100% sure, could you explain me what you mean with "test suites"? (sorry but in the testing world we all use different terms for different things, me included :D )
Maybe because it's not crucial for you and your devs 😊 you could do a small Playwright-migration POC, I think you won't regret it 😊
Sure! I'm probably using the wrong terms myself 😅 When I refer to Chromatic tests, I think of Chromatic's interaction tests. In these tests, you implement a
playfunction on your story, which interacts with it (type, click, etc) and assert on it. The thing is, that if you wish to have two tests on the same story (what I think of a "test suite"), you can't. Are you familiar with these tests? If so, was your experience similar? If not, how do you test your stories?You can rest assure that it's in our backlog 🥲
The question is: why do you need two tests for a single story? Think about it from the readers' perspective: a story tells something, and the test is just part of that story.
What I do is
As an example, look at this story/test. If you look at the sidebar, you can find that there are two stories that have been created just to write the tests. So, one test = one story. Storybook is a great tool to communicate to other devs and stakeholders what the component does and what you tested. What do you think?
Writing one story per one test is an interesting approach. I can see the benefits. However, please consider a case of a more complex component. For instance a dropdown with a search input. There are numerous use cases that should be tested (search, clearing the input with the "X" button, many cases of accessibility). If we'll write a story for each single test, we can easily end up with 10 or more identical stories. I find this problematic for a two important reasons:
I agree that this approach has many benefits as well. I just wish there was a way to write interaction tests in a similar way to how you write unittests: you declare a suite with some tests in it, and can use the before/after hooks for any setup you need. Then, you can run any slices of the suite as you need. When Chromatic integration tests have this support, I will be a happy developer 😊
WDYT?
I agree, the choice should be on you, not on the platform you use 😊
Great read! I’m using Cypress in conjunction with Cucumber Preprocessor plugin and found it super useful tagging my tests. Cypress will pick up all tests in .feature file that are tagged with @e2e. Also, tags come in handy when you’re developing your tests and don’t want to run the whole feature file.
Thanks for sharing Yuliya!! Cucumber is going to be the next thing to study for me 😊