DEV Community

Cover image for How to Make JSON String Comparisons in Unit Tests Less Brittle
Anthony Fung
Anthony Fung

Posted on • Originally published at webdeveloperdiary.substack.com

How to Make JSON String Comparisons in Unit Tests Less Brittle

We previously looked at how we can extract longer texts used as expected values into separate files and load them as and when needed in our unit tests. As an example, we looked at a string representing the body of a welcome email for an (imaginary) online shop.

In this article, we’ll look at another use case where we can use this technique. We’ll also look at how we can make the checks a bit more robust too.

JSON Data and Unit Tests

Transferring data is a necessary part of building a useful system. Before being sent, it’s often converted into a common format understood by all subsystems involved. Due to its simplicity and human readability, JSON is a popular choice. We’ll use it in these examples, but the concept should also be applicable to other text-based data formats too, e.g. XML.

If we wanted to check that a method sends a particular piece of data, one option would be to compare its output against an expected value. When writing our test, we have two options:

  1. Declare the expected JSON directly in test code.

  2. Add it to a separate file and read from it while the test is running.

Option (1) can make the data difficult to read from the programmer’s perspective unless the transformed object is simple and has only a few properties. Line spacing and formatting aside, JSON properties need to be surrounded with double quotes; this conflicts with how strings are represented in C#. To work around this, we can:

  • Escape JSON double quotes by prefixing them with \.

  • Wrap JSON data containing double quotes inside double quotes in a C# verbatim string literal, i.e. a string starting with @.

  • Use raw string literals, available in C# 11/.NET 7 (or greater). With this option, we can express a multi-line value that includes double quotes inside a block that starts and ends with a series of double quotes.

To avoid this problem, we can declare the expected value in a separate file; this is Option (2) in the list of options previous presented. However, if we formatted the data to make it readable, we’ll have one more issue we need to resolve. Let’s say we have the following class.

public class MyObject
{
    public string Property1 { get; set; }
    public string Property2 { get; set; }
}
Enter fullscreen mode Exit fullscreen mode

It’s used in the following test. In this example, we instantiate a MyObject directly, but an instance could also have been created from a service we want to test.

[Test]
public void SerializedJsonIsAsExpected()
{
    // Arrange

    var myObject = new MyObject
    {
        Property1 = "Value1",
        Property2 = "Value2"
    };

    // Act

    var json = JsonConvert.SerializeObject(myObject);

    // Assert

    var expected = ReadEmbeddedResource("ExpectedJson.json");
    Assert.That(json, Is.EqualTo(expected));
}

private static string ReadEmbeddedResource(string resourceName)
{
    var assembly = Assembly.GetExecutingAssembly();
    using var stream =
        assembly.GetManifestResourceStream(resourceName);
    using var reader = new StreamReader(stream);
    var result = reader.ReadToEnd();
    return result;
}
Enter fullscreen mode Exit fullscreen mode

ExpectedJson.json contains the following text.

{
      "Property1": "Value1",
      "Property2": "Value2"
}
Enter fullscreen mode Exit fullscreen mode

When run, the test fails with the message:

Expected string length 53 but was 43. Strings differ at index 1.
Expected: "{\r\n\t"Property1": "Value1",\r\n\t"Property2": "Value2"\r\n}"
But was:  "{"Property1":"Value1","Property2":"Value2"}"
------------^
Enter fullscreen mode Exit fullscreen mode

Formatting and String Comparisons

We formatted ExpectedJson.json to make it easier to both read and edit if necessary (this becomes more important with real-world data which is likely to be much more complex). However, the failing test shows that doing so causes problems when comparing string values. To resolve this, we can minify both sets of JSON data before the assertion. In the following code, we’ve added a Minify method where we strip out newline terminators, tabs, and spaces.

[Test]
public void SerializedJsonIsAsExpected()
{
    // Arrange

    var myObject = new MyObject
    {
        Property1 = "Value1",
        Property2 = "Value2"
    };

    // Act

    var json = JsonConvert.SerializeObject(myObject);

    // Assert

    var expected = ReadEmbeddedResource("ExpectedJson.json");
    var minifiedJson = Minify(json);
    var minifiedExpected = Minify(expected);
    Assert.That(minifiedJson, Is.EqualTo(minifiedExpected));
}

private static string Minify(string json)
{
    var minified = json.Replace("\r", "")
        .Replace("\n", "")
        .Replace("\t", "")
        .Replace(" ", "");

    return minified;
}

private static string ReadEmbeddedResource(string resourceName)
{
    var assembly = Assembly.GetExecutingAssembly();
    using var stream =
       assembly.GetManifestResourceStream(resourceName);
    using var reader = new StreamReader(stream);
    var result = reader.ReadToEnd();
    return result;
}
Enter fullscreen mode Exit fullscreen mode

This implementation of Minify should be sufficient for most cases. However, we can deserialize and reserialize the data for a more consistent and robust approach.

private static string Minify(string json)
{
    var minified = JsonConvert.SerializeObject(
        JsonConvert.DeserializeObject(json));

    return minified;
}
Enter fullscreen mode Exit fullscreen mode

Summary

Comparing JSON in unit tests can be trickier than expected. Tests can become harder to read and fail due to formatting. But you can work around these issues by following a few tips.

JSON property names must be surrounded by double quotes, conflicting with how string values are typically represented in C#. However, you can either escape the double quotes or use a more specialised string representation. Another option is to extract the JSON into a separate file, which also makes it more readable.

Formatting can also affect your test results. New line and indentation characters will make string values differ, even when your object and data values are otherwise identical. However, you can minify your JSON before making any comparisons. Doing this to both your expected and test values will remove any formatting differences and let you compare the actual data.


Thanks for reading!

This article is from my newsletter. If you found it useful, please consider subscribing. You’ll get more articles like this delivered straight to your inbox (once per week), plus bonus developer tips too!

Top comments (4)

Collapse
 
tbroyer profile image
Thomas Broyer

I don't get it.

If you're testing your own JSON serializer, then you should take formatting into account, but this is a niche situation.

If you're testing that you're serializing appropriate values, then either test the values before serialization, or parse the JSON and compare the parsed value against the expected one.

Collapse
 
ant_f_dev profile image
Anthony Fung

Hi Thomas

Thanks for reading.

It’s more the case of your second point: testing that the correct values are in place. For smaller data, you’re absolutely right – do property matching in object form instead of raw JSON.

I used to work somewhere that produced some data models; I can’t remember exactly how many properties there were, but it was somewhere in the region of 50-150 per type. And there were several types of these objects, with varying property names. We could have written the corresponding assertions, but opted for comparing the JSON given that the data changed with iterations on the model.

Another use case is checking that larger API responses are correct.

These tests were typically part of the end-to-end set and were admittedly brittle. But they served as an early warning to investigate if something went wrong.

Collapse
 
tbroyer profile image
Thomas Broyer

Oh I wasn't talking about "compare objects property-by-property" or even "parse JSON to objects and then assert property-by-property", that could indeed quickly be unreadable and unmaintainable. But maybe "compare value objects using their equals operator/method", or "parse JSON to maps and lists and primitive types" so you can easily compare them equal with the expected response, that you can also parse from JSON if you want (I don't do C# but, in Java, maps and lists compare equal if their content compare equal, so you can assertEquals(parse(expectedJson), parse(actualJson)), and records in Java 17+ automatically have an equals method generated for you comparing all fields).

Comparing parsed values that way also doesn't care in which order key-value pairs are serialized for an object, or the exact representation of a number or even string (whether it escapes some characters or not, and how).

That being said, using your model objects in tests to represent your expected response would help you maintain your tests, as a change in the model makes the test fail to compile (rather than just fail when run so you have to understand why it fails and adjust it); but that's another subject 😉

Thread Thread
 
ant_f_dev profile image
Anthony Fung

This could work in some cases – thanks for the ideas!

When comparing value types (e.g. when a model is represented as a struct and data values are built-in/primitive types) it's straightforward to do an Is.EqualTo comparison.

When they're reference types (e.g. a class) a comparison will fail by default, as they refer to different objects. But it is possible to change this behaviour by either providing a custom comparer object in the assertion call, or by overriding the object's Equals method. As the latter means GetHashCode should also be overridden, it’s generally easier to provide a custom comparer.

However, where the data values are built-in types, you’re right in saying that it's possible to compare using the Dictionary type (I'm guessing it's the equivalent of a Java Map). (It’s the first time I’ve come across this approach, so I had to try it out.)

For anyone interested, here’s a sample in C# to do this:

 [Test]
 public void ObjectCompare()
 {
     var myObject = new MyObject
     {
         Property1 = "Value1",
         Property2 = "Value2",
         Property3 = 3
     };

     var expected = ReadEmbeddedResource("ExpectedJson.json");
     var expectedAsDictionary =
         JsonConvert.DeserializeObject<Dictionary<string, object>>(expected);

     Dictionary<string, object?> myObjectDictionary = myObject.GetType().GetProperties().ToDictionary(
         x => x.Name,
         x => x.GetValue(myObject));

     Assert.That(myObjectDictionary, Is.EqualTo(expectedAsDictionary));
 }
Enter fullscreen mode Exit fullscreen mode