How to get mobile app logs to Elasticsearch using Bugfender

When we speak to our customers, they tell us that Bugfender is great for capturing application logs and then pinpointing the logs of a specific user. By pulling all the logs from a developer’s mobile and web apps and centralizing them in a single dashboard, Bugfender gives that developer a 360-degree sweep of their entire user base.

But what if you want to use this data somewhere else? What if you could provide this data within an Elasticsearch database?

By doing that, you could open up heaps of new possibilities. You could go far beyond fixing bugs: you could perform analytics, for example, or correlate security events. With Elasticsearch, you can perform a free-text search to make aggregations, histograms, and statistics, and you can hook up to Kibana, which allows you to view security information and event management (SIEM).

Well, if this prospect has whetted your appetite, then great news: there is a way to use the Bugfender API to download your logs and push them to an Elasticsearch database, in real time.

(Elasticsearch is merely an example. You could use the same technique to push Bugfender to your favorite database, like MySQL, Postgres, BigQuery, HBase, Solr, or anything else).

Getting the logs from Bugfender

Bugfender offers you an API that can you can use to gather logs in real time. Logs come in pages, and cursors for the next and previous logs come with each page, like this:

{
   "data": [
     {
       "uuid": "a774e285-78d4-54bb-9dba-dca684f11400",
       "device.udid": "6e306bed-4d7b-46f3-acee-fe6339b3c6fb",
       "device.name": "PLK-AL10",
       "device.type": "PLK-AL10",
       "version.version": "1.4.6",
       "version.build": "2020110306",
       "language": "en",
       "os_version": "6.0",
       "timezone": "CAT",
       "text": "This is a test log.",
       "method": "test",
       "file": "SourceFile",
       "line": 1,
       "log_level": 0,
       "tag": "SynchronizationJob",
       "time": "2021-04-20T09:34:24.157Z",
       "thread_id": "21960",
       "thread_name": "AndroidJob-2",
       …
     },
     …
   ],
   "previous": "https://dashboard.bugfender.com/api/…",
   "next": "https://dashboard.bugfender.com/api/…",
 }

Then you can push each page to your database. In fact, scrolling through pages is as easy as calling the “previous” and “next” URLs.

Downloading the sample project

To make things easier, we created a sample project: https://github.com/bugfender/bugfender-integration-elasticsearch

This project essentially performs two steps, ad infinitum:

• Gets a batch of logs from Bugfender
• Pushes the logs to Elasticsearch

This project also takes care of other details, like error handling and retrying, task cancellation and resuming, and authentication.

We encourage you to download it and adapt it to your needs. It might be useful already as-is, but it might also serve as a starting point for your own project. For example, you might want to do some transformations of the data downloaded from Bugfender before pushing it to your own database.

The core of the integration lies here:

// syncOnePage synchronizes one page of logs, returns error if something failed
 func (i *Integration) syncOnePage(ctx context.Context) error {
     // get a page from Bugfender
     logs, err := i.bugfenderClient.GetNextPage(ctx)
     if err != nil {
         return err
     }
     if ctx.Err() != nil {
         return ctx.Err()
     }
     // put it in the destination
     err = i.destination.WriteLogs(ctx, logs)
     if err != nil {
         return err
     }
     if i.verbose {
         log.Printf("Wrote %d logs", len(logs))
     }
     return ctx.Err()
 }

https://github.com/bugfender/bugfender-integration-elasticsearch/blob/main/pkg/integration/integration.go#L66-L85

You might also find these parts interesting:

• The Bugfender client, which handles authentication with OAuth2 automatically and provides a cursor mechanism to paginate through your logs.
• You can customize the log query if you only want to get some of the logs by checking the makeFirstPageURL function.
• You can use the LogWriter interface to implement your own log destination. For example, we have implemented a ConsoleDestination that prints the logs to the console (duh!)

Running the sample project

If you do not have an API key yet, contact us to get one. You can follow the instructions here: https://support.bugfender.com/en/articles/4667823-api-access.

Then run like this (newlines added for readability):

./bugfender-integration-elasticsearch
     -app-id=1234
     -client-id=your_client_id
     -client-secret=your_client_secret
     -state-file state.json
     -es-index logs
     -es-nodes http://127.0.0.1:9200

The state-file parameter is useful to remember the synchronization progress. You can stop the synchronization with Ctrl+C and running the command again will pick up where it left off.

Exploring the API

As we have seen, the Bugfender API can be handy to extract logs for further processing. You can run analytics, merge logs from multiple sources or archive them in your preferred long-term storage solution. But the API can do more than what we covered in this article.

If you would like to learn more, you can see the full documentation here: https://dashboard.bugfender.com/api/.

We can’t wait to see what you build with it!