DevRel metrics – Counting things is terrible. Do it!

Upon seeing some activity numbers from a recent quarter, a leader I respect told me “I know everybody is busy.” They wanted to know what the impact of our activity was, not how much we did.

Of course! Anybody talking to you about metrics tells you that counting things is a terrible metric. We’ve all heard the horror stories of developer productivity being measured by lines of code. It doesn’t measure the right thing! But here’s the tricky part…

How do you get there? Where do you start? The ultimate metrics dashboard you saw in that webinar one time doesn’t just appear magically out of thin air.

You need to start somewhere.

My suggestion: Count. Things. Measure something. ANYTHING!

We’re going to go through 4 lessons you can use with the content marketing that your team is already doing.

Starting at the beginning

When our team first started including our content marketing results in our quarterly reports, it looked something like this:

• 32 videos
• 6 Technical Decks

Not very inspiring, but it was a start! Previously, we had not been measuring how much content we were creating. We had no idea. By trying to report on just these two numbers we had to solve a few things:

1. What videos were published to our channels between specific dates?
2. What PowerPoint files were created and also published during specific dates?

You might already be seeing why counting things is important. When you are not reporting on anything, you don’t have anything set up for measuring. By picking something to report on it forces you to start putting in place ways of measuring “stuff done”.

This also allows you to establish some baselines for future measuring. We continue to report on ‘counts’ of things today, though we also count things like blogs and doc contributions, live presentations, and webinars. It all started by building up a way to track and count those first two things.

Lesson #1: Use a counting metric to drive you to find a way to track what work the team is doing.

Trends are your friends

Another important evolution in counting things is looking at activity over time. Sure, you generated 8 blog posts and 3 documentation updates this quarter… but is that more or less than what you usually do? Once you have the baseline with your things you are counting, you can track these from one period of time to another to see how things change.

Isolated points of data are rarely helpful. You need to build up this data over time to be able to rule out one-off spikes and look for an overall trend.

Trend analysis – Comparing recent data points

Here is an example from one of my quarterly reports that shows activity across most of a year, with data points from the end of each quarter.

This simple dashboard allows me to get a sense of where the team is putting their efforts for types of content. In this bar chart, I can tell that a lot more time this quarter went into videos, and a lot less on presentations and blogs. Videos take a lot of time to edit and produce, and we get great engagement from them, so this isn’t a problem… but why the shift?

This graphic doesn’t answer that, but it starts to tell a story about the team that prompts me to dig. Why were there so many more presentations last quarter? Oh, right, our big annual event, Sitecore Symposium! The whole team was doing local sessions in the lead up to the event and then the big event sessions happened.

Also, a bunch of the new videos this quarter seem to be based on the content from that event. So we leveraged the presentation content from last quarter to turn into videos this quarter.

Once I’ve done this analysis, I can also get the sense that we’re going to need to go into presentation/blogging soon or else our video content output is likely to not have as much to drive it.

Lesson #2: Compare over time in your dashboards to be able to highlight areas that need investigating regarding team productivity.

Trend analysis – Comparing year over year

Looking across recent time periods is one way of looking at a trend, from quarter to quarter in the previous example. However, a lot of businesses have seasonal activities. Community events, marketing events, sales enablement… for our team, these tend to occur around the same time every year.

In the last example, I showed how the shift in activity by content type was affected by an annual event that drove more presentations and blogs. But what if I look against the same time frame from the year before?

Now I can start getting a better picture of how the activity level is for this time of year. Presentations dropped in comparison to the second quarter in the previous graph, but compared to last year’s data at the same time of year we are actually slightly up in number of presentations. Most things are roughly the same, except for videos.

Given the current state of things, seems like we are doing pretty well. The types of content are shifting around a little between categories, but the team is pretty consistent.

Except videos? What is going on there? Why are there so many more than the same time last year?

Well, I mentioned previously that we had recordings from the big event. We didn’t record our annual event the year before, so we weren’t able to release them in that year. That accounts for some of the disparity, but not everything.

This prompts me to dig in a bit more. This year, when we ran our Virtual Developer Day, we published the videos out to Master Sitecore a few weeks after the event, right at the end of the quarter. Usually we wait quite a while, sometimes to the next fiscal year. That definitely helps explain the sudden increase in video content in this quarter.

Lesson #3: Comparing against the ‘same thing’ across years is helpful to look for changes in seasonal activity levels. This type of graph will help you identify something strange that doesn’t normally happen at that time of year.

The gift that keeps on giving

Okay, so I said up front that counting things is a terrible metric and then went on to show several things that counting stuff helps with. So why is it so terrible? One word: VALUE.

Counting things allows you to look for general team activity, assuming regular-sized work loads, availability, team size, complexity, and consistent work/life balance (hello pandemic!). However, you have no idea if what was delivered was any good. Did anybody watch those videos? Does anybody read your blogs? Did you help even a single person do their job? Counting things gives you none of this.

It’s a good start on your steps towards a full view of your team. The great thing is that these metrics are not lost investment! I still use these activity counts today. Why use such a horrible metric? ROI calculations!

Do you know what your Return on Investment (ROI) is for your content? Let’s say you know exactly how much people engage with your videos, and how many views you are getting, and which videos work and don’t. Ask yourself these questions:

1. What effort did it take to get to that level?
2. How much effort was required to reach a similar level at another point in time?
3. Is there a better balance of time that could be spent to get the same results?
4. If there is an increase in content activity, is the increase in value equivalent to the increase in effort?

Lesson #4: All those things you counted and the trends on activity you graphed out help you balance against the aggregate value being delivered. You can now dig into where the team can focus time and perhaps relax effort a little.

“Counts of things” may be a terrible type of metric on its own, but counting things sure can be a big help!