Adding Observability - Tracking Exceptions

This post will show how to easily observe exceptions in a Spring Boot application, using Honeycomb and Aspect Oriented Programming.

Observability

Observability is understanding how your system is running in production without deploying changes. It’s about being able to answer questions you didn’t know you wanted the answer to, without changing code. Two things that enable observability are: exporting non-aggregated data (i.e. events not metrics) and high cardinality labels for those events. Events contain lots of data about what happened in a system which can then be queried or aggregated later. A high cardinality label, for example, would be user ID. That’s useful because if your system is showing slow responses, you can group the response times by user ID and see what proportion of users are being affected. If it’s only a small subset of users you can then investigate only those users - you’ve delved into the heart of the problem much quicker than you could have with low cardinality metrics.

To improve observability in a personal side project app, I decided to use Honeycomb. Honeycomb is super easy to export data into and has an intuitive UI for querying and graphing (I’m not affiliated with Honeycomb, it’s simply something I’ve wanted to try for a while).

Observing Exceptions

Another high cardinality label would be exception data. Adding exception data to events is also very useful for understanding why things are happening in your system and particular exceptions might be the cause or differentiator for you to be able to find issues and the causes of them.

I’m going to show how to easily add exception data to your events with Aspect Oriented Programming.

Aspect oriented programming (AOP)

Aspect oriented programming allows you to write code for cross-cutting concerns (non-functional requirements (NFRs) that are needed for all business functionality e.g. logging, transactions, observability) without adding extra code into the business logic - which would distract from others being able to read the intent of the logic. Instead, you can define actions to be taken before, after or around method calls in your system and keep business logic and NFRs separate.

Terminology: Pointcuts and Advices

Pointcuts are simply a filter, without which an advice would run on every method call (or join point as it’s called in AOP). An advice is the action that runs before, after, or around a method call.

Adding fields to spans after an exception

This is using Spring AOP. So firstly add the Spring Boot starter for AOP into build.gradle:

implementation 'org.springframework.boot:spring-boot-starter-aop'

Then add a new Aspect class (make sure to annotate with @Aspect):

@Aspect
public class BeelineAspect {

    private final Beeline beeline;

    public BeelineAspect(Beeline beeline) { // 0
        this.beeline = beeline;
    }

    @Pointcut("within(@org.springframework.stereotype.Repository *)" +
            " || within(@org.springframework.stereotype.Service *)" +
            " || within(@org.springframework.web.bind.annotation.RestController *)") // 1
    public void springBeanPointcut() {
    }

    @Pointcut("within(technology.wick.repository..*)" +
            " || within(technology.wick.service..*)" +
            " || within(technology.wick.web.rest..*)") // 2
    public void applicationPackagePointcut() {
    }

    @AfterThrowing(pointcut = "applicationPackagePointcut() && springBeanPointcut()", throwing = "e") // 3
    public void traceAfterThrowing(JoinPoint joinPoint, Throwable e) {
        beeline.getActiveSpan().addField("exception.message", e.getMessage()) // 4
            .addField("exception.cause", e.getCause())
            .addField("exception.name", e.getClass().getSimpleName())
            .addField("exception.happenedIn", joinPoint.getSignature().getDeclaringTypeName() + "." + joinPoint.getSignature().getName())
        ;
    }
}

1. Beeline gets autowired in since it is exposed as a bean by the beeline-spring-boot-starter.
2. Add pointcut to match any method call in beans annotated with @Repository, @Service or @RestController.
3. Then add a pointcut to match any method call in beans in the repository, service and web packages.
4. Use those pointcuts in an advice to run only after throwing an exception.
5. Add exception data to the current span using Beeline (Honeycomb’s interface to modify spans and traces).

The pointcut methods are purposefully empty because the actual action we want to take is in the advice.

This aspect can then be enabled with an @Configuration class, which exposes the aspect as a bean.

@Configuration
@EnableAspectJAutoProxy
public class AspectConfiguration {

    @Bean
    public BeelineAspect beelineAspect(Beeline beeline) {
        return new BeelineAspect(beeline);
    }
}

Conclusion

If you have the rest of the Honeycomb configuration set up, you can now query for exception events in the dashboard.

AOP is a nice way to add observability for exceptions since you don’t have to touch any of your current code. This can also be applied to other tracing providers like OpenTelemetry.