I Went to Write Two Tests and Ended Up Fixing the App

The job sounded small. Two issues, same shape: write a Playwright end-to-end test. Log in, report something, check the confirmation, report it again, check the message. Maybe an afternoon each.

Four and a half months later, I had opened four pull requests, and not one of them was a test. Not yet. This is the story of why, and why I think it's the best argument for E2E tests I've ever lived through.

The project is Quote.Vote, an open-source, text-only platform for thoughtful dialogue (Next.js, an Express and Apollo GraphQL API, MongoDB, and a Prisma migration in progress). I'm a volunteer contributor there, and this is where things stand as I write this (November 12): the two bug-fix pull requests are merged, and the test pull request is open as a draft.

The assignment

On June 22 I was assigned two issues:

• E2E-MOD-006: report a post. A logged-in user reports content, picks a reason, sees a confirmation, and a repeat report is handled gracefully.
• E2E-MOD-007: report a user as a bot. Same idea, from a profile.

Before you can test anything, you have to run it. In July I did the local setup, and the app greeted me with a profile page that said "Invalid user". The server had answered 200 OK with {"data":{"user":null}}. No error anywhere. The GraphQL schema declared a user(username) query, and no resolver existed for it, so Apollo returned null with a straight face.

I fixed that, and I also found a seed script that wrote to a different database than the one the server read. Two small pull requests, both merged. Then the maintainers told me what I'd half suspected: the backend for reporting itself didn't exist yet. They opened a separate issue to build it, marked mine "Not Ready", and I stepped away.

Coming back after four months

In early November the maintainers asked if I was ready to pick the issues up again. I said yes, and my laptop immediately said no. In one day I hit:

• A local clone that was hundreds of files behind the real repository, so git status showed a wall of changes I hadn't made.
• A database container that crash-looped, because the "latest" Mongo image had moved a major version past my old data volume.
• A backend that needed a MongoDB replica set (Prisma uses transactions for writes), which was documented in a migration doc and not in the setup steps.
• A browser that couldn't talk to the API, because an old setting still allowed only the previous frontend port.
• A main branch whose frontend and backend deliberately don't match, because the real work lives on staging.

I wrote each one down in a notes file as I went. (Several turned out to be documentation gaps worth reporting.) None of that is the story, but it's worth saying: the first test you write is for your own setup.

Lesson one: read the issue against the real screen

The issue said: select a report reason. I opened the app. Reporting a post is a single click, with no reason picker. Reporting a user is a confirm dialog that says "Report Bot", and the reason picker component exists in the code but is never shown.

I could have built a test for the screen in the issue, one that could never pass. Instead I took screenshots and asked, with the evidence attached. The answer came back within days: test the flow that exists, and treat a reason picker as a separate product decision. A few minutes of comparing, and a test written against a screen that doesn't exist, avoided.

The bug: "Post not found" for a post that exists

I logged in as a second user, opened a post, and clicked Report post. The app said:

Post not found

The post was on the screen in front of me.

The reporting code checked for a duplicate with a Prisma filter, roughly "update this post unless reportedBy already contains this user."

updateMany({
  where: { id: postId, NOT: { reportedBy: { has: userId } } },
  data: { reportedBy: { push: userId }, reported: { increment: 1 } },
});

Posts created through the app's normal "create post" path have no reportedBy field at all. MongoDB leaves an unset list absent. And a filter on a list can't match a field that isn't there. So the update changed nothing, the code assumed "nothing changed means the post doesn't exist", and it threw NOT_FOUND.

I proved it on one post in the database: add an empty reportedBy: [], and the report worked, and the second report said "already reported". Remove the field, and "Post not found" came back.

A missing field is not an empty list. Humans read them the same. Databases don't.

A maintainer suggested the fix: one raw MongoDB update, where $ne also matches a missing field, and $push and $inc create absent fields.

q: { _id: { $oid: postId }, reportedBy: { $ne: userId } },
u: { $push: { reportedBy: userId }, $inc: { reported: 1 } },

The check and the write stay one atomic step, so I tested that too: five simultaneous reports from one user gave one success and four "already reported", and the database counted one.

Why had the unit tests stayed green? They mock the database, and every fake post in them already had reportedBy: []. The tests only ever tried the one shape that works.

The second finding: a repeat report is not a failure

Reporting a user twice produced a red error toast, a GraphQL error in the browser console, and a dialog that stayed open. The maintainers' view was fair: you already told us, that's not an error. The server now returns an ALREADY_REPORTED result and the app shows a calm, neutral message.

The third finding: tests that never ran

To change that dialog I looked at its existing tests. One ("reports user successfully") finished in 12 milliseconds. That's too fast to open a dialog.

They imported MockedProvider from Apollo, which Apollo Client 4 removed. In the test it was undefined, so the component crashed into its error boundary, and the tests passed through an "else" branch that expected the error screen. They had been passing without ever testing the feature.

I left that file alone and wrote new tests in their own file that drive the real menu and dialog. A test that passes in 12 ms deserves suspicion.

Accidental test-driven development

Test-driven development (TDD) is a simple loop: write a test that fails (red), write the smallest change that makes it pass (green), then tidy up. I'd always treated it like a gym membership: admirable, and mostly for other people.

Here I did it without planning to. The issue's steps were the failing test: follow them by hand and the confirmation never appears, because the app says "Post not found". So for each fix I wrote the unit test first and watched it fail (five failures for the post fix; two backend and three frontend for the bot fix), then changed the code until they passed (38 in the post test file, 55 in the user one).

The red step also caught my mistakes. My first version of a new frontend test failed for the wrong reason: I had copied the old tests' setup, which crashed the component. That's how I found out the old setup was broken in the first place.

What I'd tell a TDD skeptic: the failing test isn't a ritual, it's the bug report written in code. If you can't make the test fail the way the bug does, you haven't understood the bug yet.

And then, the actual tests

Finally, the thing I was assigned. Playwright drives a real Chrome browser like a very fast, very literal person. Headless means no window (fast, what CI uses). Headed means you can watch it. It's faster than you'd believe.

The design had one wrinkle: you can report a given post, or a given user as a bot, once per pair of accounts. A test that passes today would fail forever tomorrow. So each test creates a brand-new user through a small local-only seed script, and the tests repeat on the same database with nothing to reset. The spec also skips unless the API is explicitly local, because it writes test accounts.

The results from my machine: all 3 tests pass on desktop and 3 on mobile, and a second run on the same database passed again, 6 of 6. One more small thing surfaced: a failed report shows the same error toast twice, so Playwright's strict matching complained about two identical elements. I noted it as a finding and made the test accept the first copy.

What happens next

Both bug-fix pull requests were merged on November 12. The test pull request needed both, so it waited on a local branch, and then moved onto the updated staging with a single git rebase --onto (my commit was the only one that moved, since the maintainers' merge commits had kept the fixes intact). It's open now as a draft, and I'm adding the evidence: a Playwright report with all 6 results, two short screen recordings, and the step-by-step view of each test.

Here's a small tip I learned on the way: Playwright records video only for failures by default, and it starts recording before your setup finishes, so a raw video of a passing test opens on several seconds of a blank page. Run it in slow motion (slowMo), record every test, and trim the empty start. A reviewer will actually watch a seven-second clip.

One more thing to watch: GitHub only auto-closes issues when a pull request merges into the default branch. Mine targets staging, so the two issues won't close by themselves when it merges. I've mentioned that in the issue threads so nobody is surprised.

I'll update this post again when the test pull request is merged.

What I'd tell the next contributor

• Treat a test task as a product review. You're the first user the feature has had.
• Compare the issue with the real screen before writing a line. Attach screenshots when they disagree.
• Prove a bug before you report it. "Five simultaneous requests, one success" is harder to argue with than "I think there's a race".
• Be suspicious of fast, green tests.
• Write down your setup problems. Your notes file is a documentation PR waiting to happen.
• Ask before you build, and in the open. Maintainers answered in days, with a design, because I came with evidence.

If you're about to write E2E tests for a half-finished feature

1. Run the feature by hand first, as a user, with a second account.
2. Write down every way the screen differs from the issue.
3. Reproduce each bug in the smallest way you can, and keep the proof.
4. Fix the bugs in small separate pull requests, and say which one the test depends on.
5. Make the test independent of old data: a fresh user or fresh record each run.
6. Add a guard so it can't run against anything but a local server.
7. Run it twice on the same database before you trust it.

FAQ

What is test-driven development (TDD)?
A loop of three steps: write a test that fails (red), write the smallest code that makes it pass (green), then clean up (refactor). The failing test comes first, so it proves the bug or the missing feature exists before you fix it.

Can E2E tests find bugs that unit tests miss?
Yes. Unit tests mock the database and the network, so they only exercise the shapes of data you imagined. An end-to-end test uses real data created the real way, which is how a post with no reportedBy field showed up.

What's the difference between headed and headless in Playwright?
Headless runs the browser with no visible window, which is faster and is what CI servers use. Headed shows the window so you can watch. Playwright's --debug and --ui modes let you step through a test one action at a time.

Why does a MongoDB filter not match a missing field?
A list filter such as "does not contain X" is evaluated against an array. If the field doesn't exist there is no array, and Prisma's list filters don't match it. Raw MongoDB operators such as $ne do match a missing field, and $push and $inc create absent fields.

How do you test something that can only happen once per user?
Give each test its own fresh user, created by a small seed script, so the test never depends on earlier runs. Then it can repeat on the same database without cleanup.

How do you come back to an open-source issue after months away?
Re-read the whole thread first, since things change fast. Re-sync your clone and check which branch the project is currently developing on. Compare the issue text with the current app, and ask in the thread before building anything the screen contradicts.

Why would a test pass in 12 milliseconds?
Usually because it never ran the code it claims to test. A common cause is a test helper that no longer exists in an upgraded library, so the component falls into an error boundary and the assertions take a fallback path.

Update (Nov 13): @maciek_chmura_building asked two good questions about fixtures and the double toast. I tested both and wrote up the results, with screenshots, in this reply.

In short: both the component and the global error link show the toast, and three more post lists are missing on new posts but handled safely.

Backend engineer. I write about systems, architecture, and what I learn by poking at things. Portfolio 🚀