In this story, we will look at a log capturing app using the following technologies:
- MongoDB
- Java + Spring Boot
- Swagger UI
This demo application showcases the ability of MongoDB to store data without schema. All log entries can have arbitrary fields and we can search logs via those arbitrary fields. To make the app easy to test, we’ll visualize its REST API via Swagger UI.
The Challenge of Schemaless Data
In this story we’ll build an app which captures log data from 3rd party apps. The main challenge here: we don’t know how the log data from these 3rd party apps looks like. Still, we want to store the data in a compact format, plus we want to support queries for this log data.
Let’s look at a concrete example. Here’s two sample logs we want to store, in JSON format:
We have one microservice PAYMENT
which creates logs during a payment process. On the right hand, we have a log from another app called LOGIN
. Some of the log fields are common like severity
, but some are unique to an app like creditCardProvider
which only exists in the PAYMENT app.
The SQL Approach
If we wanted to store this data in an SQL store, we’d have a hard time to figure out a neat solution. Either we’d create a huge table which has a column for every log field we ever see. But this would result in many null entries and we would waste storage. Another solution is to store all data in JSON format, so you’d only have two rows in your table: the primary key and the JSON document. However, with this solution, you wouldn’t be able to search by JSON fields in a fast way.
The NoSQL Approach
NoSQL Document Stores come to our rescue: they store data in JSON format without any fix schema. Basically, a document store contains JSON documents – and nothing else. Even the ID of a given document is contained in the JSON itself.
Now, we can do really fast querying by JSON field, so you can do something like this:
> db.logEvents.find({_appType:"PAYMENT"})
This would return the first log above. More interestingly, you can perform the following query:
> db.logEvents.find({creditCardProvider:"VISA"})
The document store is robust to the fact that not every log contains the field creditCardProver
!
Implement the Log Capturing App
Enough talk - let’s get started implementing our app!
MongoCollection Configuration
We use MongoCollection from the official MongoDB Java Driver for DB access. Let’s go ahead and create a Bean with a properly configured MongoCollection:
@Configuration
public class MongoConfiguration {
@Bean
public MongoCollection<Document> getMongoCollection() {
MongoClient mongoClient = MongoClients.create();
MongoDatabase db = mongoClient.getDatabase("mydb");
return db.getCollection("logEvents");
}
}
MongoDB Queries
Let’s use the above declared Bean to search entries in the MongoDB by id
:
public class MongoDbService {
@Autowired
MongoCollection<Document> collection;
public Optional<Document> findById(String id) {
Document doc = collection.find(eq("_id", new ObjectId(id))).first();
return Optional.ofNullable(doc);
}
}
Using the @Autowired
annotation, we can pull in our Bean of type MongoCollection
. This class offers many query capabilities like searching by id, by key-value pair, and even more complex queries like greater-than etc. On my Github repo, you can see other query samples.
One more note: all DB data is represented as Documents, which are described in the official documentation as follows:
The
Document
class can represent dynamically structured documents of any complexity ...Document
implements Map<String, Object>
Java Date Model
Inside our logging app, we don’t want to use Documents, but use a custom POJO instead:
import lombok.Data;
@Data
public final class LogEvent {
private Map<String, String> logEvent;
}
We represent each log entry as a map with String
keys and values.
Side note: the code snippet above uses Lombok which offers a neat way to create setter, getters, and constructors.
Document to LogEvent Mapper
Our DB layer represents logs as Document
, but our service layer uses LogEvent
. Therefore, we need a mapper method:
public LogEvent toLog(Document document) {
Map<String, String> map = new HashMap<>();
document.forEach((key, val) -> map.put(key, val.toString()));
return new LogEvent(map);
}
REST Controller
Now it’s time to create the controller layer. We’ll use REST Controllers, which can be nicely created using Spring Boot’s @RequestMapping
annotation:
@RequestMapping(method = RequestMethod.GET, path = "/api/log/{id}")
public ResponseEntity<LogEvent> getLogById(
@PathVariable(value = "id") String id) {
return mongoDbService.findById(id)
.map(documentToLogMapper::toLog)
.map(ResponseEntity::ok)
.orElse(ResponseEntity.notFound().build());
}
Everything we’ve coded so far comes together now!😎 So let’s go through the above code snippet step by step! First, our REST controller will serve HTTP GET requests to the URI "api/log/{id}", id
is the identifier of the log event. Second, the REST call will return a LogEvent
in JSON format. Third, we use functional programming in the method body. We start with the DB query, pass the result to our mapper, and finally check if everything went fine – and if not: return a 404 NOT_FOUND result.
Spring Boot Service
Only one thing is missing to make our app executable:
@SpringBootApplication
public class LogCaptureApp {
public static void main(String[] args) {
new SpringApplication(LogCaptureApp.class).run();
}
}
That’s it, that’s all necessary code😊 Now your logging app can serve REST requests to read logs!
Swagger UI Documentation
One more thing – let’s add a simple GUI to our log capturing app. We use Swagger UI for this purpose. Swagger UI allows to visualize your API without any further hustle. All you need is configure Swagger:
@Configuration
@EnableSwagger2
public class SwaggerUIConfiguration {
@Bean
public Docket apiDocu() {
return new Docket(DocumentationType.SWAGGER_2)
.apiInfo(apiInfo())
.select()
.apis(RequestHandlerSelectors.basePackage("com.example.controller"))
.paths(PathSelectors.any())
.build();
}
private ApiInfo apiInfo() {
return new ApiInfoBuilder()
.title("Log capturing app with MongoDB + SpringBoot")
.build();
}
}
Run the App
The complete code can be found on Github:
pmgysel / mongodb-logging-app
Demo app for MongoDB: manage log events of varying schema type
My Github repo contains more functionality than what we saw in this story, like searching log events by date range and storing log events from JSON data.
So, if you have implemented the log app as you read through this story, go ahead and run it now. If not, no problem, you can use my fully working example from Github.
Also, make sure you have a MongoDB Server running locally on the standard port. Then, compile and run the app:
$ git clone https://github.com/pmgysel/mongodb-logging-app.git
$ cd mongodb-logging-app
$ mvn clean package
$ java -jar target/mongodb-logging-app-0.0.1-SNAPSHOT.jar
Go to the Swagger UI page in your favorite web browser: http://localhost:8080/swagger-ui.html
You can use the log capturing app as follows:
-
createLogRadom
: Create random log entries to get some data into the DB - Alternatively, create arbitrary log entries using the
createLog
endpoint. Swagger UI will show sample request to make your life easy 😊 - Now search for logs by key-value pair (
getLogByOneField
) or via date range (getLogByDateRange
) - Sample search:
Conclusion
So, in this story, we built a fully working log capturing app in Java. Since our requirements were to support flexible log entries, we chose MongoDB as database solution.
This scenario is very well suited to NoSQL Document Stores. Note that every application calls for another database – if your data has a fix schema, you might be better off with a traditional SQL store.
Feel free to drop a comment with your ideas! Also, like ❤️ this story if it was helpful for you!
Top comments (2)
Congratulations on the post. Are you using the new swagger lib for spring? These days I was building an api, in the latest version of spring and it didn't work at all. The solution was to use: springdoc-openapi-ui version 1.2.32.
Hey man thanks! So my app only uses Swagger UI, and only for API visualization. I've quickly looked at springdoc, as you mentioned, it's a neat looking project, and it can generate the API similar to Swagger. So you enjoyed working with springdoc?
Also, there's always two things to consider: API documentation generation, but additionally it's nice to generate the Java REST Controller. I haven't included the second step in this project though!