Mask logs using logstash logback in java with regex

In today's data-driven world, data security is most important. Logging frameworks play a crucial role in application monitoring and debugging, but they can inadvertently expose sensitive information which should not have been. Log masking is a technique that effectively obfuscates sensitive data in log messages, safeguarding confidential information.

Understanding Logback

Logback is a powerful and mostly used logging framework in Java applications. It offers flexible configuration options, including the ability to format log events as JSON objects. It is a successor to the Log4j framework, has quickly gained popularity due to its features and ease of use. It is comprised of Logger, Encoders, Layout, Appender, Encoder.

Logger: Logger is a context for log messages. Applications will interact with this class to create log messages.

Encoders: Encoders are introduced in logback 0.9.91 and responsible for transforming an event into a byte array as well as writing out that byte array into an OutputStream. Encoders introduced as Layouts are only capable to transform an event into a String which restricts their scope to non-binary output.

Layout: The layout is responsible for formatting the logging request according to the user's wishes, whereas an appender takes care of sending the formatted output to its destination.

Appenders: In logback speak, an output destination is called an appender. This places log messages in their final destinations. A Logger can have more than one Appender. Currently, appenders exist for the console, files, remote socket servers, to MySQL, PostgreSQL, Oracle and other databases, JMS, and remote UNIX Syslog daemons.

About Logstash Logback Encoder

The logstash-logback-encoder library is a valuable tool for enhancing your Spring Boot applications' logging capabilities. It provides a convenient way to format log messages in a structured JSON format, making them readily consumable by log aggregation and analysis tools like Logstash. JSON format provides a structured and machine-readable way to log information, making it ideal for advanced log analysis and security measures. Benefits of logstash

• JSON Customization Logstash allows you to customize the JSON output to include specific fields and metadata.

• Dynamically Fields It also allows Dynamically add fields to log events based on application context.

• Improved Readability JSON format offers a clear and human-readable structure for log events.

• Enhanced Search and Analysis Log aggregation tools can easily parse and query JSON logs.

• Machine Parsing JSON logs are ideal for automated analysis and alerting systems.

Solution to mask the data in logs

Here main goal is to provide a solution to mask the data which is customizable and configurable at runtime.

Here is our simple requirement to

1. mask password completely in log.
2. mask phone number and login name except last 5 in log.

Step 1
Create spring boot application. This solution will work with any java based application with little customization.

Step 2
Configure all regular expressions to mask the data. Please remember regex are costly in terms of resource utilization. Make sure you are tuning your regex. Regex groups will allows us to select required substring from string.

Step 3
Create a class and implement MessageJsonProvider. This interface is from logstash and allows us to customize the message before printing to the appender. writeTo method in this interface will be called for every log message.

• in start() method read all regex and prepare LogMasker which contains all MaskingRule. This method is from AbstractJsonProvider and simply marks the process started as true.

• MaskingRule will hold regex pattern and a function. This function replaces the identified string from in the log.
  

@Data
public class MaskingMessagingProvider extends MessageJsonProvider {

    public static final String DEFAULT_RULES_DELIMITER = ",";
    private LogMasker logMasker;
    private String rules;

    public MaskingMessagingProvider() {
        super();
    }

    @Override
    public void start() {
        super.start();
        this.logMasker = LogMasker.create(StringUtils.tokenizeToStringArray(rules, DEFAULT_RULES_DELIMITER));
    }

    @Override
    public void writeTo(JsonGenerator generator, ILoggingEvent event) throws IOException {

        if (isStarted()) {
            JsonWritingUtils.writeStringField(generator, getFieldName(), logMasker.mask(event.getFormattedMessage()));
        }
    }
}

class LogMasker {

    private MaskingRule[] masks;

    public LogMasker(MaskingRule[] masks) {
        super();
        this.masks = masks.clone();
    }

    public static LogMasker create(String[] rules) {

        return new LogMasker(Arrays.stream(rules).map(rule -> MaskingRule.create(rule)).toArray(MaskingRule[]::new));
    }

    public String mask(String input) {
        String transformed = input;
        for (MaskingRule m : masks) {
            transformed = m.mask(transformed);
        }
        return transformed;
    }
}

class MaskingRule {
    public static final int REG_EX_DEFAULT_GROUP_SELECTOR = 2;
    public static final String DEFAULT_REPLACEMENT = "*";

    private Pattern pattern;
    private UnaryOperator<String> replacement;

    public MaskingRule(Pattern maskPattern, UnaryOperator<String> replacement) {
        super();
        this.pattern = maskPattern;
        this.replacement = replacement;
    }

    public static MaskingRule create(String rule) {
        return new MaskingRule(Pattern.compile(rule), (in) -> MaskingRule.maskDataWithReplacement(in, DEFAULT_REPLACEMENT));
    }

    public String mask(String transformed) {
        Matcher matcher = pattern.matcher(transformed);
        StringBuffer sb = new StringBuffer();
        while (matcher.find()) {
            matcher.appendReplacement(sb, replacement.apply(getDataToBeMasked(matcher)));
        }
        matcher.appendTail(sb);
        return sb.toString();
    }

    private static String maskDataWithReplacement(String input, String replacement) {
        int repetition = !StringUtils.hasLength(input) ? 0 : input.length();
        return String.join("", Collections.nCopies(repetition, replacement));
    }

    private static String getDataToBeMasked(Matcher matcher) {
        if (matcher.groupCount() > 1) {
            return matcher.group(REG_EX_DEFAULT_GROUP_SELECTOR);
        }
        return matcher.groupCount() > 0 ? matcher.group(1) : "";
    }
}

Step 4
Configure class in logback-spring.xml file.

<configuration>
    <springProperty scope="context" name="rules" source="app.logging.masking.rules"
                    defaultValue=""/>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <provider class="com.daya.logging.logstash.MaskingMessagingProvider">
                    <rules>${rules}</rules>
                    <rulesDelimiter>${rulesDelimiter}</rulesDelimiter>
                    <ruleDelimiter>${ruleDelimiter}</ruleDelimiter>
                </provider>
                <threadName/>
                <timestamp/>
                <logLevel/>
                <loggerName/>
                <mdc/>
                <version/>
                <stackTrace/>
            </providers>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>

Steps 5
Run the application. For simplicity, i have taken a string which is holding data and printing it at application start up.

@SpringBootApplication
@Slf4j
public class LogDataMaskingApplication {

    public static void main(String[] args) {
        SpringApplication.run(LogDataMaskingApplication.class, args);
        LogDataMaskingApplication.maskingTest();
    }

    public static void maskingTest() {
        String data = "{\"loginName\":\"maskingtest\",\"phoneNumber\":\"9898981212\",\"password\":\"Masking@123\"}";
        log.info(data);
    }

}

This is very basics solution and having lot of scope for enhancement as per requirement like message digest etc...

You can find code at GitHub.

Please drop a comment if you have any question.