Why Encoding Matters in Binary String Extraction

If you’ve ever used tools like strings to extract readable text from binary files, you know how crucial correct encoding interpretation is. Misinterpreting the encoding can lead to garbled output or missed strings, especially with non-ASCII data. The LLVM project's llvm-strings tool historically lacked an explicit encoding option, limiting its accuracy and compatibility with GNU strings — until a recent update introduced a dedicated --encoding flag.

This article dives into the problem, the technical fix, and the broader lesson for anyone working with binary string extraction.

The Problem: Implicit and Inflexible Encoding Assumptions

Before this change, llvm-strings assumed a single default character set without giving users control over encoding interpretation. This approach worked reasonably well for simple ASCII data but faltered with non-ASCII bytes or when running in different locales.

The root cause was that the tool did not differentiate between:

• ASCII-only strings (where bytes with the high bit set should be rejected)
• Locale-dependent character sets (where the interpretation depends on the environment’s locale settings)
• UTF-8 encoded strings (which can use multiple bytes per character)

Without explicit encoding selection, llvm-strings could misinterpret bytes, either including invalid characters or missing valid multi-byte sequences. This divergence from GNU strings behavior caused interoperability issues and made the tool less reliable in internationalized contexts.

The Fix: Introducing the --encoding Option

The core of the solution was to add a new command-line option --encoding with three possible values:

• s: ASCII-only mode — reject any byte with the high bit set (non-ASCII bytes).
• S: Locale-based mode — interpret bytes according to the current locale’s character set. This is the default and is POSIX-compliant.
• utf8: UTF-8 mode — interpret bytes as UTF-8 encoded characters, correctly handling multi-byte sequences.

How It Works Under the Hood

The implementation involved updating the llvm-strings.cpp source to:

1. Parse the --encoding argument: The tool now accepts this option and validates the provided value.

2. Adjust string extraction logic: Depending on the selected encoding, the string extraction algorithm changes:

• In ASCII-only mode (s), the code verifies each byte is below 0x80 and rejects sequences containing higher bytes.
• In locale-based mode (S), the tool uses locale-aware character classification functions to determine valid characters.
• In UTF-8 mode (utf8), the extraction logic validates UTF-8 sequences, ensuring multi-byte characters are correctly identified and extracted.

1. Add tests and documentation: Comprehensive tests were added to verify correct behavior across all encoding modes, including edge cases with invalid or mixed encodings.

By explicitly handling these encoding modes, llvm-strings aligns more closely with GNU strings and POSIX expectations, improving accuracy and user control.

Takeaway: Explicit Encoding Handling Is Essential

This change highlights an important lesson for developers working with binary data and string extraction:

Relying on a single default encoding without explicit user control can lead to subtle bugs and interoperability issues, especially in internationalized environments.

Explicit encoding options empower users to tailor string extraction to their data’s character set, whether it’s simple ASCII, locale-specific encodings, or UTF-8. This flexibility is crucial for tools that operate on diverse binary inputs and need to be robust across platforms and locales.

If you’re building or maintaining similar tools, consider how encoding assumptions affect correctness and usability.

This article is based on the merged PR titled [llvm-strings] Add --encoding={s,S,utf8} option in the llvm/llvm-project repository. Read the full breakdown on nossletter.tech.

Originally published on Nossletter.