You’ve built an API to solve technical problems, but you know that’s just the beginning. In addition to helping developers use it, you need to understand how they use it. You want to measure its performance and popularity, and make adjustments based on what you discover.
Maybe some developers are seeing a lot of errors when making API calls or it’s taking too long for them to get to the first “Hello World.” Perhaps the number of developers converting to paying customers is below your expectations. You want to understand the usage of your API and ensure that customers use your API in the long term.
At this point, you might reasonably look to your API logs. You’re seeking deep insights into your API and its users, but by default what you’ll find is raw usage. How can you visualize not just the usage, but also insights that help you take action? This post will cover how you might approach this on your own, as well as how Moesif can help you move more quickly in your API program.
Your API Logs Contain a Wealth of Actionable Information
The amount of valuable information you can obtain from your API logs may amaze you. For example, you can use API log data to answer a multitude of questions about your API, such as:
- What are the request and response payloads being sent?
- Which API version is used the most?
- Where is the API experiencing higher latency?
- What is the sequence of transactions leading to errors on POST /*/count/events?
- Which SDKs are used to access the API the most?
You can also use API log data to answer questions about the users of your API, like:
- What are the top endpoints used by each of my customers?
- Who are my top customers by API usage?
- Which users are scraping large amounts of data from the API?
- Which customers are running into 401 unauthorized errors?
- What is the API activity for a specific customer?
Your API logs contain valuable information that you can use to the benefit of your customers and your business. But how do you access and analyze that log data?
The first tool many API product development teams turn to for API logging and metrics is Kibana.
You Probably Pipe Your Logs into Kibana
If you want to gain insights from your API log data, then you’re probably piping those logs into Kibana. If you’re not familiar, Kibana is one of the most popular, if not the most popular, tools for log visualization. It’s part of the Elastic Stack (formerly ELK Stack), which consists of three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine built on Apache Lucene. Logstash is a server-side data processing pipeline. And Kibana lets you explore and visualize Elasticsearch data.
Like any stack, you’ll need to put the pieces together. While they work well together, these are three separate tools also used apart. You’ll need to stitch together the customer journey data, from UI to API. These traces, as they’re called in Kibana terms, are important for turning data into something actionable. It’s not as simple as connecting logs and being done—you need to determine what’s needed, handle collisions, and assume the ongoing cost of growing indices.
Kibana is a log visualization tool, which makes it an appropriate place to start when thinking about API analytics. However, the primary use case for Kibana is infrastructure monitoring and metrics. It’s not designed specifically for API products, so there are some drawbacks to using Kibana for API logs and metrics.
Pros and Cons of Kibana for API Logs and Metrics
If you’ve built an API product, you may already use Kibana to gain insights into its performance or how developers use it. Perhaps you’re considering using Kibana because of its popularity when it comes to log visualization. Either way, it helps to know some of the pros and cons of using Kibana for API logs and analytics.
Kibana Pros
The pros of Kibana include:
- Open source and free to use.
- Great at visualizing API logs.
- You can explore massive volumes of log data.
- Many useful features. Although it should be noted that some features are available separately and some are currently experimental or in beta.
- Layered on top of Elasticsearch (also a con), making it ideal for use on high-cardinality, high-dimension log data- a must-have feature for API logs.
And now onto the cons.
Kibana Cons
Kibana's cons include:
- Compatible with Elasticsearch and Logstash only. If you want to use Kibana with other databases, you’re out of luck
- Designed for infrastructure metrics and not specifically for API products. You have to customize Kibana for API log use cases.
- To connect user behavior and API activity into a single journey is incredibly manual and prone to errors over time. You need to use the Kibana Query Language (KQL) or Lucene query syntax (Kibana legacy query language) for queries, so there is a moderate to high learning curve.
- Maintaining the Elastic Stack takes a lot of effort. For example, you must periodically upgrade each part of the stack and make sure upgrades won’t break any plugins you’re using or require that you rewrite any of your visualizations.
Visualization is only a starting point when it comes to gaining insights from your API logs. And while Kibana is great at visualizing logs, it often leaves you wondering, “what’s next?” What action should I take based on this data?
Move Beyond Kibana and Get Actionable Insights from Your API Logs with Moesif
Moesif is more than log visualization, we extract actionable information from your API logs and we guide you as to what to do with it. Moesif is a user behavior API analytics and monitoring service that features a fast query engine and structured event-based data. We designed our platform for API products, so we support a wide variety of API protocols including REST, GraphQL, JSON-RPC, Hypermedia (HATEOAS), and SOAP.
With Moesif, you can filter and aggregate billions of API calls and user actions on pretty much any field, even high cardinality fields like a session token or user id.
The Moesif platform includes many features and capabilities that you won’t find in Kibana and Elastic Stack, such as:
- High-cardinality, high-dimension API metrics that are compatible with any database, not just Elasticsearch.
- Automatic analysis of REST and GraphQL APIs.
- Automatic insights on query parameters and HTTP text payloads like JSON and XML.
- Track API calls, user actions and behavior, and website activity.
- Embeddable API logs and charts.
For more details, see the complete comparison of Moesif vs. Kibana.
Moesif lets you to obtain valuable information from your API logs and helps you understand what your data is telling you through custom and pre-built dashboards.
Understand What Your API Log Data is Telling You
Extracting data from your API logs won’t help much if you don’t understand what that data is telling you. Both Kibana and Moesif provide custom dashboards that help you make sense of the data obtained from API logs. However, unlike Kibana, Moesif features pre-built dashboards designed specifically for API products as well as embed templates and public share links. You can use our pre-built dashboards or build your own custom dashboards to gain a better understanding of what your API log data is telling you.
For example, a fintech company (and Moesif customer) created a dashboard that lets them track key API and user metrics, which includes:
- Recent API errors
- HTTP status requests
- Product usage
- Daily active users (DAU)
- Most active users
Through the dashboard, the fintech API product team sees who their customers are, how they integrated the API with applications, and what users of their API are experiencing. The team also gains insights into how well their API is performing and how many developers moved from a trial or sandbox version of the API to a production version.
Don’t Miss Out on Valuable Insights
You need to move beyond log visualizations and towards user behavior API analytics so that you can gain actionable insights from your API log data.
Want to learn more about how Moesif can help you gain deep insights from your API logs?
Get started building great APIs with Moesif API Analytics. Learn more.
This article was originally written for the Moesif Blog by Matt Tanner, Head of Developer Relations at Moesif.
Top comments (0)