Determining the RGB "Distance" Between Two Colors

[NOTE: The live web app that encompasses this functionality can be found here: https://www.paintmap.studio. All of the underlying code for that site can be found here: https://github.com/bytebodger/color-map.]

At this point in the series, I've done things that are, quite honestly, pretty simple (from a programmatic sense, at least). Loading an image in React isn't exactly rocket science. Transferring that image into a <canvas> element has been done a million times before. Pixelating the image and repainting the canvas requires a bit of code - but it really ain't that hard.

Here's the point where I hit an unexpected hurdle. I now have a complete inventory of paints with their associated RGB equivalents. (Or... at least as close as I could determine. That particular journey was outlined in the first article in this series.) I also have an image that's been loaded into memory, and I've parsed through the image, pixel-by-pixel, in order to create a pixelated version of the original. So now that I have the pixelated blocks, I need to look at the color of each block and determine which color in my inventory of paints most-closely matches the block.

The "easy" approach

When I set about creating the underlying code for Paint Map Studio, I thought that the color-matching aspect of the equation would be simple. I was working off a few basic assumptions:

1. Now that I have the pixelated version of the image, each block is represented by RGB values.
   

2. I've captured all of the RGB values for every paint in my inventory.
   

3. RGB values are nothing more than... numbers. And as long as I have the numbers associated with the pixelated block, and I have the numbers associated with every paint in my inventory, it should be elementary to do some simple math and determine which paint is closest to the color in the original block.
   

Then, I started noticing some... quirks.

First, let's look again at the pixelated image that I'm trying to match against my inventory of paints. We generated it in the last article, but I'll display it again here:

Then let's look at my inventory of paints. They look like this:

Obviously, if those paint colors are represented onscreen, then I must have the underlying RGB values for each color. (You can see the full list of paints here: https://www.paintmap.studio/palettes. There are more than 200 of them.)

RGB matching

Previously, we weren't trying to match the pixelated blocks against any kinda reference color. So if we want to do that, we'll need to add some more logic to the pixelate() function. The new function looks like this:

const pixelate = () => {
   const { height, width } = canvas.current;
   const stats = {
      colorCounts: {},
      colors: [],
      map: [],
   };
   const blockSize = 10;
   loadPalettes();
   for (let y = 0; y < height; y += blockSize) {
      const row = [];
      for (let x = 0; x < width; x += blockSize) {
         const remainingX = width - x;
         const remainingY = height - y;
         const blockX = remainingX > blockSize ? blockSize : remainingX;
         const blockY = remainingY > blockSize ? blockSize : remainingY;
         const referenceColor = calculateAverageColor(context.current.getImageData(x, y, blockX, blockY));
         referenceColor.name = '';
         const closestColor = getClosestColorInThePalette(referenceColor);
         row.push(closestColor);
         if (Object.hasOwn(stats.colorCounts, closestColor.name))
            stats.colorCounts[closestColor.name]++;
         else {
            stats.colorCounts[closestColor.name] = 1;
            stats.colors.push(closestColor);
         }
         context.current.fillStyle = `rgb(${closestColor.red}, ${closestColor.green}, ${closestColor.blue})`;
         context.current.fillRect(x, y, blockX, blockY);
      }
      stats.map.push(row);
   }
   return stats;
};

This is pretty similar to the pixelate() function that I showed you in the last article, with a few key additions:

1. Before we enter the for loops, I've added a call to loadPalettes(). This will grab the RGB data from all of the existing paints in my inventory.
   

2. After we calculate the average color, we need to take the resulting RGB object and pass it into getClosestColorInThePalette(). This will compare the RGB values of the pixelated blocks to all of the RGB values for the existing paints.
   

I'm not going to bother illustrating the code in loadPalettes(). It grabs a static object that contains all of the RGB values for my paint inventory. Those values look like this:

{red: 77, green: 33, blue: 33, name: 'Liquitex: Alizarin Crimson Hue Permanent'},
{red: 63, green: 55, blue: 48, name: 'Liquitex: Raw Umber'},
{red: 220, green: 208, blue: 33, name: 'Liquitex: Yellow Light Hansa'},
{red: 201, green: 31, blue: 27, name: 'Liquitex: Cadmium Red Medium'},
{red: 205, green: 195, blue: 168, name: 'Liquitex: Parchment'},
{red: 16, green: 145, blue: 139, name: 'Liquitex: Bright Aqua Green'},
{red: 10, green: 113, blue: 180, name: 'Liquitex: Brilliant Blue'},
{red: 95, green: 67, blue: 135, name: 'Liquitex: Brilliant Purple'},
{red: 141, green: 190, blue: 49, name: 'Liquitex: Brilliant Yellow Green'},
{red: 141, green: 105, blue: 44, name: 'Liquitex: Bronze Yellow'},
{red: 106, green: 50, blue: 35, name: 'Liquitex: Burnt Sienna'},
{red: 61, green: 44, blue: 38, name: 'Liquitex: Burnt Umber'},
// and on, and on, and on...

Much more important is what happens inside getClosestColorInThePalette(). That's shown here:

const getClosestColorInThePalette = (referenceColor = rgbModel) => {
   const key = `${referenceColor.red},${referenceColor.green},${referenceColor.blue}`;
   // use the existing calculation if we already determined the closest color
   // for this given referenceColor
   if (closestColors[key])
      return closestColors[key];
   let closestColor = {
      blue: -1,
      green: -1,
      name: '',
      red: -1,
   };
   let shortestDistance = Number.MAX_SAFE_INTEGER;
   // loop through every paint color - this was already loaded inside loadPalettes()
   palette.forEach(paletteColor => {
      // if we already found an exact match, then short-circuit the comparisons
      if (shortestDistance === 0)
         return;
      // calculate the "distance" between referenceColor and this particular paint color
      const distance = Math.abs(paletteColor.red - referenceColor.red)
         + Math.abs(paletteColor.green - referenceColor.green)
         + Math.abs(paletteColor.blue - referenceColor.blue);
      // if this paint color is "closer" than any of the others we examined, save it
      // as the paint that is currently the "closest"
      if (distance < shortestDistance) {
         shortestDistance = distance;
         closestColor = paletteColor;
         closestColors[key] = paletteColor;
      }
   });
   return closestColor;
};

Imagine that our reference block has RGB values of rgb(201, 182, 19). Then imagine that we're looping through the colors in the palette and we're comparing it to Liquitex: Alizarin Crimson Hue Permanent. That paint has RGB values of rgb(77, 33, 33). We can determine the difference between both colors by adding the difference between each color's red, to the difference between each color's blue, to the difference between each color's green.

Therefore, the formula for determining the RGB difference between the reference color and Liquitex: Alizarin Crimson Hue Permanent would be:

const distance = Math.abs(77 - 201)
  + Math.abs(33 - 182)
  + Math.abs(33 - 19);

So the distance between the reference color and Liquitex: Alizarin Crimson Hue Permanent would be: 287.

Moving onward through the array of paint colors, we see that Liquitex: Bright Aqua Green has RGB values of rgb(16, 145, 139). Therefore, the formula for determining the RGB difference between the reference color and Liquitex: Bright Aqua Green would be:

const distance = Math.abs(16- 201)
  + Math.abs(145 - 182)
  + Math.abs(139 - 19);

So the distance between the reference color and Liquitex: Bright Aqua Green would be: 342.

This means that, according to our RGB calculations, Liquitex: Alizarin Crimson Hue Permanent is "closer" to our reference color than Liquitex: Bright Aqua Green. Obviously, this doesn't necessarily mean that either of those two paint colors are particularly "close" to the reference color. At least, not as judged by the human eye. But mathematically speaking, Liquitex: Alizarin Crimson Hue Permanent is a closer match to the reference color than Liquitex: Bright Aqua Green.

The idea here is that, once you've cycled through every single paint in the inventory (more than 200 different colors), the algorithm will find the "closest" paint color to the reference block. And the hope is that the closest color will be a pretty good match to the human eye.

So how does it perform? Well... let's take a look.

Oof...

When we use the RGB color difference algorithm shown above to match all of the blocks in the target image to our inventory of known paint colors, this is what we get:

That's... not great. It's still pretty obvious that we're looking at a (pixelated) image of a woman's face - a woman with brown skin tone. But the algorithm has done some wonky things. Her face looks as though it's been smeared with pink-and-tan paint.

When I first started doing this, results like these baffled me. I reasoned that, with 200+ colors available in my inventory, I should be able to come up with pretty close matches for nearly any color in the image. But there were a few things that I didn't understand.

First, 200+ colors is not really a lot of colors with which to recreate something as nuanced as a human face. When you're setting up to do some painting, that may feel like a huge inventory, but it's nothing compared to the millions of colors that we perceive in any given photo. So one problem is the breadth of my palette. However, I'm not going to dive into that issue yet, because that's the subject for a future article.

Second, although RGB values feel to us (as programmers) as though they're nothing more than numbers, and numbers feel as though they should be easy to manage programmatically/mathematically, the human eye does not perceive color in the same way that our program does when we we're trying to parse through the image algorithmically. It's entirely possible for two colors to be relatively "close" in terms of their RGB values - but for our eyes to perceive those colors as being quite different.

For example, if you look closely at her hair, you'll see that there's a lot of dark green in the translated image. If you look at the original image, you probably won't perceive any green in her hair. But when the algorithm tried to find the "closest" RGB match for some of those darker shades of grey in her hair, the closest match it could find was a dark... green.

So aside from the problems we encounter with our limited palette (again, this will be covered in a future article), we also have problems with the basic algorithm that we're using to find the "closest" colors. Some colors that are relatively close in the RGB color space look as though they are not close at all to the human eye.

To be fair, calculating the basic RGB difference between colors is not always inherently bad. For some target images, it may be perfectly fine to use this basic algorithm. But when trying to match against more nuanced values - like those found on a human face - the RGB difference algorithm often falls short.

What images would work best for this sort of basis RGB analysis?

1. Images that feature a limited color palette.
   

2. Images that feature subjects with high contrast.

For example, this image:

Fares far better with only a basic RGB analysis:

Every pixel on those balls is perfectly matched to one of the colors in my paint inventory. And there aren't too many parts of the image that look outright "wrong". This happens because those balls feature much higher contrast and it's easier for the basic algorithm to match them against the core set of paints.

Also, the color depth of this image isn't really that... deep. Although there are many different colors in the image (red, blue, green, yellow, etc.), there aren't many different shades of each color. So it's easier to match an image like this against my base set of paints.

But when we take a low contrast image like this:

We get strange color-matching again:

This happens because the image has, overall, very little contrast.
Although there are still thousands of colors in the source image, many of those colors are extremely "close" to all of the other colors. (Also, notice how those odd greens have popped up in our processed image again.)

In the next installment...

Thankfully, there are more ways than simple RGB difference calculations to accomplish color matching. In the next installment, I'll show different models we can use to find better matches. These include calculations based on different color spaces (XYZ, CMYK, and L*a*b), and a fancy-sounding formula called Delta-E 2000.