loading...
Cover image for YEStifications: Exploring how users engage with notification prompts in the Chrome UX Report
Google Web Dev

YEStifications: Exploring how users engage with notification prompts in the Chrome UX Report

rick_viscomi profile image Rick Viscomi ・6 min read

📢 example.com wants to show notifications

Notifications can be an effective way to keep users engaged, but many websites are unable to get users past the permission prompt. A common problem is prompting out of context, as soon as the page loads or without an explanation of how they will be used. So how do users tend to engage with these permission prompts? CrUX to the rescue.

The Chrome UX Report (CrUX) now includes data from 30,000 websites on users' engagement with notification prompts. The data is made available in a public BigQuery repository, which we'll explore in this post to see how users engage with prompts across the web. Let's dive in!

The state of notification permissions

For starters, let's get a feel for the number of websites we're working with in this dataset.

SELECT
  COUNT(DISTINCT origin)
FROM
  `chrome-ux-report.all.202001`
WHERE
  experimental.permission.notifications IS NOT NULL

The results show that there are 31,515 origins with permission data. An origin is the top-level address of a website, everything before the first "/" of the path, like https://www.example.com. User experience data from pages on the same website are all rolled up at the origin level.

Now let's look at the permission data for a single website, for example Slack.

SELECT
  SUM(experimental.permission.notifications.accept) AS accept,
  SUM(experimental.permission.notifications.dismiss) AS dismiss,
  SUM(experimental.permission.notifications.deny) AS deny,
  SUM(experimental.permission.notifications.ignore) AS `ignore`
FROM
  `chrome-ux-report.all.202001`
WHERE
  origin = 'https://app.slack.com'
Accept Dismiss Deny Ignore
92.10% 5.67% 1.96% 0.28%

Pie chart of the permission rates for app.slack.com

As you can see, the Slack app has a very good permission acceptance rate for their notification prompts at 92%. In other words, 92% of users who get the prompt end up granting Slack permission to show them notifications. About 6% of users close the prompt without explicitly deciding one way or the other, 2% of users explicitly disallow Slack from showing them notifications, and ~0% of users leave the prompt entirely untouched.

But how representative is Slack of the experience across the web? Let's zoom out and see how all 30,000 websites stack up.

SELECT
  origin,
  SUM(experimental.permission.notifications.accept) AS accept,
  SUM(experimental.permission.notifications.dismiss) AS dismiss,
  SUM(experimental.permission.notifications.deny) AS deny,
  SUM(experimental.permission.notifications.ignore) AS `ignore`
FROM
  `chrome-ux-report.all.202001`
WHERE
  experimental.permission.notifications IS NOT NULL
GROUP BY
  origin
ORDER BY
  `ignore` ASC,
  accept DESC

Graph of 30,000 websites' user engagement with notification permission prompts

What are we looking at here? The curved gray area on the top right of the chart shows the distribution of ignores across all websites with permission data. Next, the red area taking up the most space in the middle of the chart represents the proportion of user engagements that reject the permission prompt. The green area that curves down from the left edge of the chart represents the proportion of user engagements that accept the prompt. And the remaining yellow area between accept and deny is the dismissal rate.

We can draw a couple of insights from this visualization:

  • Most glaringly, there's a lot of red and not a lot of green in this chart. Users are highly likely to be denying notification permission prompts rather than accepting them. Slack is an extreme example of an outlier website with a high acceptance rate.

  • There seems to be an inverse correlation between deny and ignore rates. As sites have smaller deny rates, their ignore rates increase. We could interpret the lack of prompt interaction as an implicit rejection of permission.

So if Slack is such an outlier, what does a typical website look like?

SELECT
  AVG(accept / total) AS accept,
  AVG(dismiss / total) AS dismiss,
  AVG(deny / total) AS deny,
  AVG(`ignore` / total) AS `ignore`
FROM (
  SELECT
    SUM(experimental.permission.notifications.accept) AS accept,
    SUM(experimental.permission.notifications.deny) AS deny,
    SUM(experimental.permission.notifications.ignore) AS `ignore`,
    SUM(experimental.permission.notifications.dismiss) AS dismiss,
    SUM(experimental.permission.notifications.accept +
        experimental.permission.notifications.deny +
        experimental.permission.notifications.ignore +
        experimental.permission.notifications.dismiss) AS total
  FROM
    `chrome-ux-report.all.202001`
  WHERE
    experimental.permission.notifications IS NOT NULL)
Accept Dismiss Deny Ignore
16.71% 23.84% 40.74% 18.71%

Pie chart showing the average website's notification permission response rate

This is a more realistic view of the state of notification permissions. Keep in mind that we're averaging the UX for many websites together, even if they have drastically different numbers of users. So this is showing us that the average website has a notification prompt acceptance rate of 17%, 24% dismissal, 41% rejection, and 19% ignore.

What else can we learn about user behavior from this dataset? We could also slice this average distribution by users' device form factor.

SELECT
  device,
  AVG(accept / total) AS accept,
  AVG(dismiss / total) AS dismiss,
  AVG(deny / total) AS deny,
  AVG(`ignore` / total) AS `ignore`
FROM (
  SELECT
    form_factor.name AS device,
    SUM(experimental.permission.notifications.accept) AS accept,
    SUM(experimental.permission.notifications.deny) AS deny,
    SUM(experimental.permission.notifications.ignore) AS `ignore`,
    SUM(experimental.permission.notifications.dismiss) AS dismiss,
    SUM(experimental.permission.notifications.accept +
        experimental.permission.notifications.deny +
        experimental.permission.notifications.ignore +
        experimental.permission.notifications.dismiss) AS total
  FROM
    `chrome-ux-report.all.202001`
  WHERE
    experimental.permission.notifications IS NOT NULL
  GROUP BY
    device)
GROUP BY
  device
ORDER BY
  accept DESC
Device Accept Dismiss Deny Ignore
phone 22.76% 18.03% 57.33% 1.88%
tablet 18.75% 20.24% 58.34% 2.67%
desktop 6.11% 34.01% 11.61% 48.27%

Bar charts of phone, tablet, and desktop average notification permission prompt rates

Ok this is significant. There is a huge difference in engagement between phones and desktops. Phone users are very unlikely to ignore the prompt, whereas desktop users ignore the prompt about half the time on average. The simplest explanation is screen size; it's a lot harder to ignore a prompt when it's taking up a larger portion of your screen. Interestingly, despite the 57% rejection rate, phone users are the most likely to accept permission prompts, at 23%, compared to just 6% on desktop. One explanation may be that a phone is a much more natural and convenient form factor on which to receive notifications because it goes where you go.

Where we go from here

Users have been feeling the frustration of intrusive notification prompts and developers have only anecdotally heard about and experienced this frustration. Now we have actionable data to shed light on the quality of experience across the web. And it's clear from the CrUX data that relatively few websites are effectively enrolling users in notifications.

The Chrome team has announced that it is rolling out changes to how prompts are shown to users to minimize frustration, saying "sites with very low acceptance rates will be automatically enrolled in quieter prompts". So it's especially important for site owners to be aware of their prompt performance.

If your site requests permission to show notifications, add instrumentation so you can monitor the response rates. The CrUX dataset is not a substitute for your own UX monitoring, but it can be helpful for filling in gaps, seeing how similar websites are performing, and tracking the trends across the web. Read up on best practices for requesting permission to improve your response rates and get more yestifications than notifications. 🥁

The queries shown in this post can be run on BigQuery. You're strongly encouraged to remix the queries to discover your own insights. Read more about using CrUX on BigQuery to get started. If you have any questions or feedback about the data, the CrUX team would love to hear from you on any of these channels:

Posted on by:

rick_viscomi profile

Rick Viscomi

@rick_viscomi

I work on web transparency projects like the HTTP Archive and Chrome UX Report. I work towards a faster, better web for everyone.

Google Web Dev

Collected thoughts and posts on web development from the @ChromiumDev team.

Discussion

pic
Editor guide
 

Interesting stuff. I’ve become annoyed with the number of sites offering notifications and started to wonder if everyone else likes them. Seems like most people don’t like them- except for sites like slack where they make sense.

Wonder if the large number of ignores on PC are a result of multitasking, or because of increased screen space (you don’t really need to make the message go away), or something else.

 

Great insight, thanks !

It could be interesting to see the acceptance rate plotted against the time since page load when the notification was displayed, to confirm or invalidate the theory that users are more likely to dismiss popups that show up straight away vs when it is shown in a relevant situation.

 

Yes, that's a good idea. All of the websites in CrUX are also run through HTTP Archive where we audit them using Lighthouse. One audit specifically checks if the notification prompt happens on page load, so that might be a good heuristic to correlate with accept/deny rates in CrUX. Maybe fodder for a future post! :)

 

I think there are 2 things to consider that make a great difference:

  1. The actual use case for notifications. A messaging app like Slack has a pretty compelling use case - you want to get notified of messages you'd miss otherwise. Compare that to the notifications you receive on a tabloid website. Comparing sites with different use cases is going to skew the image you are going to get.

  2. Push vs. pull: does the site ask you to accept notifications with or without any user interaction? Users are more likely to reject permissions that are pushed on them. Unfortunately, the report does not add this kind of data.

 

Doesn't slack also not request for notification privileges in the browser until you enable it in slack?

 

A lot of pages and apps do that, so that would make another interesting property to watch for.