Bugs in open source projects are often highly visible and create a lot of trouble for maintainers and users alike.
Bugs in libraries can require both upstream and downstream maintainers make difficult decisions, too. Suppose an unintended breaking change is found in libfoo 1.5.0. Now maintainers not only need to fix that bug, they also need to decide whether to try "un-releasing" the broken version, how to notify the users of the problem and so on.
Myself I always test betas and code from the master branch whenever I can, with projects I use often. This behaviour is not entirely altruistic: I know that if a bug makes it into a released version before I see it, I will be in trouble as much as the maintainers.
As an open source maintainer, I also know that my downstream users will be in trouble if my dependencies have bugs.
It can be hard to find time for testing. However, I often see that people underestimate the importance of it. They think it wouldn't make a difference and they aren't good testers.
In reality, no one is a bad tester. Any testing is important, and every maintainer will (at least, should) appreciate any testing effort.
Here are some common misconeptions.
It works for me, thus the maintainers already know it works
They may know that it works for them. They have no way to know that it works for you until you tell them.
It's very nice and reassuring to hear from users that a new feature works for them as expected.
Someone else is already testing it
There's often a bizzare situation when everyone thinks someone else is already doing it, only to find out that no one did.
Even if someone is really testing it already, doesn't mean you should not. Chances are what they are doing isn't exactly the same, or their environment isn't exactly the same as yours.
Many bugs require specific environment settings to trigger: certain hardware, combination of options, dependency versions or something else. Even the largest and well-funded projects don't have resources to test every combination, so community help is required to catch and reproduce them.
The project has automated tests
Unit tests are great, but they can only test functions in isolation. Even automated integration/acceptance tests only catch problems that developers know may happen. Nothing catches the unknowns.
Even formal methods can only prove that the code matches developers' assumptions.
While some bugs are indeed simply programming errors, many more are caused by missing or incorrect assumptions about the ways the code interacts with the environment, or the ways different parts of the code interact with one another.
My workflow is so ordinary, how can my testing help?
You may think your setup and workflow is completely ordinary, and you need to be doing something special, outside of your normal workflow, to be a useful tester.
It isn't true! First, your workflow is almost surely different from that of the developers at least in some ways. Second, you have an excellent intuition for the right and wrong, and can spot even slightly odd behaviour easily.
Any maintainer will appreciate a dedicated tester who goes to great lengths to try the code in unusual settings, test uncommon paths, and push the code to its extremes.
However, even just doing what you are always doing, just with a development version instead of a stable one, and telling developers that it works for you is already valuable.
Top comments (0)