Verifying Assembly Redirects Before Runtime

This week's "yak shaving" project was updating a library dependency in across several large .NET solutions in a monorepo. It took about as long as we expected, but not for the reasons we expected - this is a tale about assembly binding redirects.

The library (Npgsql) was bumped two major versions, from 2.x to 4.x. The primary advantage of the update in our case was added .NET Core support, allowing us to incrementally port services in this monorepo over to .NET Core.

Our dependency structure looks like this:

- Npgsql
  - SharedBusinessLogic
    - Web UI layer
    - Some async services that generate reports
    - An unrelated UI project that doesn't use Npgsql directly

There were a few database queries that broke but overall the update seemed to go fine, tests were passing locally and on CI.

However, after merging and deploying the branch to one of our test environments, one of our projects started throwing errors on certain routes:

What happened?

An updated graph of dependencies after the update:

- System.Runtime.CompilerServices.Unsafe
- System.Numerics.Vectors
- System.Memory
- System.Threading.Tasks.Extensions
  - Npgsql
    - SharedBusinessLogic
      - Web UI layer
      - Some async services that generate reports
      - An unrelated UI project that doesn't use Npgsql directly

The Npgsql dependency now targets netstandard, which means we can reference it via .NET Core and .NET Framework (legacy) without compatibility hacks. However, to accomplish this, some of the dependencies that were previously referenced in the installed .NET runtime are now referenced via NuGet packages. By convention, most of these are prefixed System. in the dependency list.

However, some of our other app dependencies already had references to these System. assemblies, but for different versions.

Confused? Let's talk a bit about how this works.

In .NET Core projects assembly binding redirects are not necessary, but many of our projects still run on .NET 4.8, also referred to as .NET Framework, or netfx.

At runtime, the .NET Framework tries to resolve a specific version of any dependencies requested. If the exact version is not present in the bin folder, an assembly binding redirect is required to "force" the runtime to resolve to a specific version, or an exception will be thrown. These binding redirects are specified in a config file and checked at runtime.

For further reading, I recommend Nick Craver's excellent post on binding redirects here.

To complicate matters further, not all assemblies are loaded and checked at application startup. We found that only some of our server-side routes caused the Npgsql assembly to load, so building and running just the home route was not enough.

OK, so we'll update these bindings now that we know that the code crashes. But how can we know about this problem before running every possible route on our web app?

Our first attempt was to use the msbuild /warnaserror flag. With this switch, warnings will be printed for NuGet level restore problems (downgrade warnings), and binding redirects that are mismatched or missing entirely. More information here.

C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2084,5): warning MSB3247: Found conflicts between different versions of the same dependent assembly. 

In Visual Studio, double-click this warning (or select it and press Enter) to fix the conflicts; otherwise, add the following binding redirects to the "runtime" node in the application configuration file:

<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1"><dependentAssembly><assemblyIdentity name="System.Memory" culture="neutral" publicKeyToken="cc7b13ffcd2ddd51" /><bindingRedirect oldVersion="0.0.0.0-4.0.1.1" newVersion="4.0.1.1" /></dependentAssembly></assemblyBinding> 

In our case, we could not use /warnaserror yet because we wanted to progressively roll out this validation, and additionally:

• Alert on binding redirects that are targeting an old version of the library
• Warn on outdated redirects for assemblies not present in the bin folder (deleted dependencies)
• Run the tool on an entire git repository locally to quickly sanity check repos with multiple solution files
• Generate missing bindings without copy-pasting text from msbuild's output

We ended up putting together a Powershell script to handle this. It has no dependencies and has been tested on Powershell 5. Check it out here if you're interested

The basic idea is this:

• Find any web.config or app.config files within a folder, recursively
• For each resolved config, open the bin folder and load every DLL, building a map of versions and publicKeyTokens
• Check to make sure the lists match
• If they don't, fail the build
• ???
• Profit

With increased confidence in our build tooling, we can be more proactive about moving shared projects to netstandard, and ultimately ship more projects using .NET Core.

Happy shipping!