The Great Orphan Object Hunt: How I Fixed Our Cloud Storage Woes

As an SRE working with a cloud provider and object storage, I've learned firsthand how tricky it can be to keep a Ceph cluster running smoothly. And while most of the time, everything hums along just fine, every once in a while, we encounter a problem that leaves us scratching our heads.

Recently, we started getting calls from users who were confused about their usage metrics. They were seeing that the sum of all objects in their buckets was many fewer than what we were showing in our panel, and naturally, they were concerned. After some digging, we discovered that there were orphan objects stuck in the cluster - objects that couldn't be shown by any client, like s3cmd, boto3, or mc.

Now, with billions of objects in our cluster, finding and removing these orphan objects was no easy task. I tried using normal tools, but they just weren't up to the job. So, I had to get creative.

I started by creating a bash script that would help me identify and remove these orphan objects. Here's the code:

#!/bin/bash

#set -e

if [ -z "$1" ]
  then
    echo "No bucket name provided"
    exit 1
fi
BUCKET=$1

BUCKET_ID=$(radosgw-admin bucket stats -b $BUCKET | jq -r '.id')
if [ -z "$BUCKET_ID" ]
  then
    echo "Failed to get bucket ID for $BUCKET"
    exit 1
fi

INDEX_POOL=$(radosgw-admin zone get | jq '.placement_pools[].val.index_pool' -r)
DATA_POOL=$(radosgw-admin zone get | jq '.placement_pools[].val.storage_classes[].data_pool' -r)
ORPHANS=$(radosgw-admin bucket check -b $BUCKET | jq -r ".[]" )
if [ -z "$ORPHANS" ]
  then
    echo "No orphan objects found in $BUCKET"
    exit 0
fi

UPLOAD_ID=""
for object in $ORPHANS
do
  NEW_ID=$(echo $object | awk -F '.' '{print $(NF-1)}')
  UPLOAD_ID="$UPLOAD_ID $NEW_ID"
done

# You must Remove this objects from data pool
ORPHAN_OBJECTS_DATA=$(radosgw-admin bucket radoslist -b $BUCKET | egrep --text  "($(echo $UPLOAD_ID | tr ' ' '|' ))")
if [ -n "$ORPHAN_OBJECTS_DATA" ]
  then
    while read -r object
    do
      echo rados -p $DATA_POOL rm $object || true
    done <<<  "${ORPHAN_OBJECTS_DATA// /\\ }"
  fi

# You must Remove this omapkeys from index pool
SHARDS=$(radosgw-admin bucket stats -b $BUCKET | jq -r ".num_shards")
for (( COUNTER=0; COUNTER<$SHARDS; COUNTER+=1 ))
do
  ORPHAN_OMAP=$(rados -p $INDEX_POOL listomapkeys .dir.${BUCKET_ID}.${COUNTER} | egrep --text "($(echo $UPLOAD_ID | tr ' ' '|'))")
  if [ -n "$ORPHAN_OMAP" ]
    then
      while read -r omap
      do
        echo rados -p $INDEX_POOL rmomapkey .dir.${BUCKET_ID}.${COUNTER} $omap
      done <<< "${ORPHAN_OMAP// /\\ }"
    fi
done

# Fix bucket
echo radosgw-admin bucket check -b $BUCKET --fix 

This script uses radosgw-admin to check for orphan objects in a specified bucket, then removes them from both the data pool and the index pool. It's not a perfect solution, but it worked for us - and it might just work for you too!

But why do orphan objects even happen in the first place?

Well, it's a complicated issue, and one that's beyond the scope of this post. But suffice it to say that they can be caused by a number of factors, including bugs in the system, network issues, or even user error.

So, what can you do if you find yourself in a similar situation?

My advice would be to start by using a script like the one I've provided here to identify and remove any orphan objects in your cluster. But beyond that, there are a few best practices you can follow to help prevent orphan objects from occurring in the first place:

• Keep your software up-to-date: Orphan objects can sometimes be caused by bugs in older versions of your storage software. Make sure you're always running the latest stable release.
• Monitor your cluster closely: Regularly monitoring your cluster's performance can help you catch orphan objects early, before they become a big problem.
• Educate your users: Make sure your users understand how to use your storage system correctly to minimize the risk of orphan objects caused by user error.

In conclusion, the Great Orphan Object Hunt was a challenging but ultimately rewarding experience. By using a bit of creativity and some handy tools, we were able to fix our cloud storage woes and keep our users happy. And with a bit of luck, we'll be able to avoid orphan objects altogether in the future!

UPDATE:

https://tracker.ceph.com/issues/16767