Gain and retain users for your project with user experimentation

As a developer, my work is highly depending on tools, as if keep finding better tools to be more efficient is part of my job. When I cannot find a tool that works for me, I create one. And I share it as I hope my hard work could help others.

However, I found that sharing my creation to other people even other team members were not as simple as imagined. There are various reasons to refuse my kind offer:

• They don't understand how this tool could help their work.
• They don't have time to learn and try out new stuff that looks not promising because it is made by an infamous person.
• They have no idea how to start.
• They worry that the tool would not be maintained well.
• They don't like its UI/UX etc.

Fair enough🤷🏻‍♀️. At this point, I require skills more than coding to "sell my product". Here are 2 ways that I found really helpful for resolving most of the refuse reasons mentioned above:

1. F2F user experimentation
2. User survey

The primary way: F2F user experimentation

Eyewitnesses can provide very compelling legal testimony, but rather than recording experiences flawlessly, their memories are susceptible to a variety of errors and biases.
from Eyewitness Testimony and Memory Biases

Observing how a user use your product by yourself can provide you a lot of unexpected information, even much more than interviewing in-person.

I got this conclusion from my first time user experimentation, and verified more times in the following experimentations I tried. I organized my wonderful first session for a tool that I would like to integrate, but it was not ready for quite some time.

What I did observe was that the project owner as well as the contributors wish to receive feedback passively from their target users. Yet they only received little feedback as the tool was not ready, and they were not that helpful as who provided those feedback were in fact not using the tool. As a result, the developer team figured out "the improvements" by their instincts and struggled in gaining users.

The session invited 2 major developers and 2 target users from a team who were eager to find such kind of solution to resolve their pain points.

The experimentation took around 1.5 hours and went on in the following flow:

1. Firstly, introduce the experimentation, why and how;
2. Then target user follow the instruction of doing specific things with the tool from scratch;
3. In the meantime, developer sit next to the target user, observe their behaviour, note down the issues user has met and provide guidance if necessary;
4. Lastly, 2 groups brainstorm together about the issues and possible improvements.

The tool developers were surprised at how buggy was the tool which they believed at least the main flow were well tested. And they also witnessed user could not easily understand how to even start from the UI/UX and the documents they provided.

If these were not seen by their own eyes, you may imagine how many rounds of communications required to make them understand what was going on and convince them to "waste" more effort on writing better documents.

As I expected, the developers found several points to improve, but still missed quite a few as they were busy on putting down the fire on the scene. In the end, me as the coordinator and a 3rd party helped noting down 50% more issues since I caught more details in my unique aspect.

It's effective to some point, while the downside is obvious as well - it takes precious coding time to design, prepare and follow up, yet each session only covers a small group. Since the sample size is small, the opinions they provide and the issues you found may not be the right thing to care about.

To obtain a larger size of sample, a well designed user survey and user behaviour tracking is more favourable, but we may not enjoy the effective observation and communication advantages from user experimentation.

Here I summarized some keys from my success and failure in the user experimentation.

When designing an experimentation:

• Select a specific feature or behaviour. Focusing on depth not breadth.
• Let user experience a complete process without disturbing if possible. And you may witness more.
• Plan for following up. By another experimentation? An interview? Or a survey? To close the feedback loop.

When conducting the experimentation:

• Involve the main devs, the decision makers. Witness is more convincing than being told.
• Devs and users should be in pairs, just like peer programming do. No one can observe so many objects at the same time.
• Note down observations immediately as much as possible. Or you will forgot at least half of them soon.
• Ask users' thoughts, discuss the possible improvements. Engage and open your mind.