Why I Love Zero Result Search as a Metric and You Should Too!

Have you ever "broken" Google search? Typed in something and received exactly zero results? Or maybe like a handful of results, with what almost looks like an apology from the search engine, that might as well be nothing? It's equal parts thrilling and annoying. Like HA! I got you Google. Until you realize, you didn't make any progress helping yourself.

It's likely your technical documentation or developer portal has a search capability.

Search gives you two really great opportunities, you can track what terms people are using and maybe even see what they are clicking on (click-through) after searching with those terms.

Check your assumptions, meet the community where they are

Ask yourself questions like the following:

• Are searchers using the terms you expected?
• Are searchers finding what you expect with the terms they are using?
• Are searchers finding what they expect with the terms they are using?
• Are searchers finding nothing with a term you expect them to use?
• Are searchers seeming to use terms you previously didn't think of and are they hitting a dead end?

This really all boils down to the core question - did you make assumptions about your community's knowledge that are incorrect?

Your searchers, your community members including potential customers(!!!), are using their terms, their language.

It is your job to educate or train them in the right direction.

This can be done with synonyms explicitly written in your content, a glossary, or more subtly, enhancing search metadata with synonyms.

Failure to guide them through this experience efficiently will result in the path of least resistance - finding a competing technology that does better/easier/more seamlessly.

If you can measure it...

Let's revisit that last question in the bulleted list. Depending on the tool you are using (slightly more on that below), you should be able to see when a term returns zero results - often called zero result search or no search result.

This is my favorite thing to track, both terms and a ratio.

Amara Graham

@missamarakay

What's your favorite developer experience (#DX) metric?

I'll start - zero result search or no result search rate 🤩

15:45 PM - 05 Feb 2021

In Developer Experience, my goal is to offer developers an efficient, effective, delightful experience and I would use none of those words to describe hitting the wall that is "zero results returned" when searching technical documentation.

I've already mentioned above some of the ways to improve this, including overt synonyms to educate your searchers or synonyms in metadata for a more subtle approach.

What I haven't mentioned is that zero result search is a great opportunity for a success metric you can clearly track and improve!

... you can improve it

I consider zero result search rich data because it often gives me a ratio (great for things like KPIs or other quantitative tracking metrics) and a curated list of terms I can iterate on and improve. It's literally like "hey, should people be searching for this? Why or why not? Better investigate."

It may even prioritize things for me by showing me just how many people attempted a given search!

Now, allow me to throw myself under the bus by sharing with you the data that has been collected and I've done exactly zero work towards improving.

This is a look at a 7-day range in our Algolia DocSearch for a project that hasn't officially launched. Dark blue, top line is total searches. Aqua, mostly middle line, is total users. Dusty purple, mostly bottom line, is no results rate.

Until we officially launch, I don't know what a good baseline is, as these early engagers have been asked to critique the overall experience and offering, as you do with pre-GA programs. They are in a mindset to break and challenging things, which is not to say our post-GA audience won't do this, but theoretically, they may have different goals in mind. This is why I've done no work to improve this, but I'm casually monitoring.

I do know the closer I can get the no results rate to 0% the better the developer experience will be, at least from one aspect of the searcher's perspective. I may have not solved their problem, but I have given them something to read that's hopefully relevant. And I didn't hit them with that "no results returned" page.

Let me show you a few terms that came up over a 3 month period, a great list to iterate on to improve your docs outside of the usual new or improved features.

So we've got some interesting terms in here like "maxmessagesize", "newthrowerrorcommand", and "encryption".

We previously had a bug with maxMessageSize where it wasn't respecting the value on write, which I only know because I did a Google search and found a release blog. Because of this history, and because maxMessageSize feels like an important value to understand in a system that does messaging, we should probably take care to explicitly document it somewhere. Writes herself a to-do.

You've convinced me, now what do I do?

You don't need to roll your own tools to track search terms and zero results search, but you may need to configure portions of existing tooling to do it.

You can capture this search data with purpose-built tools like Algolia (DocSearch) or configure Google Analytics and the massive voodoo magic that is that suite of tools.

If you are using a documentation framework or developer portal framework this may already be available to you in your admin console. For example, I know KnowledgeOwl makes this available and I've worked with tech writers who monitor this quite closely because they care about the experience.

So do a little investigating, see what you are currently tracking, or could track in your search experience, and go from there.

Are you measuring zero result search? Or have you looked at the data? I'd love to know what your threshold is for your no results ratio. Or if this post inspires you to start tracking this, let me know that too!