Deploying Twemproxy to GCP

Now that the cloud resources have been provisioned and the docker image for the Twemproxy server has been created, it's time to deploy the proxy to GCP and make a few changes to the client services that will be communicating with the proxy.

Pushing the Twemproxy Docker Image to Google Container Registry

First we need to push the docker image to Google Container Registry (GCR). You may have to enable the GCR API for the project containing the Twemproxy server if you have not used the container registry service for any other applications in the project. You can enable the API for GCR using the Cloud SDK.

gcloud services enable containerregistry.googleapis.com

To push an image to a container registry in GCP, Docker must be authenticated on your machine. Detailed information about configuring authentication can be found here with the recommended method being the gcloud credential helper.

You will need to have a service account created with the "Storage Admin" role as well as a key file with the service account's credentials in JSON format in order to use the credential helper. With those two things Docker can be authenticated with the following commands.

# make sure you are logged in before running the Docker authentication commands
gcloud auth login 

gcloud auth activate-service-account your-service-account-name@your-project-name.iam.gserviceaccount.com --key-file=path/to/key-file.json

gcloud auth configure-docker

With Docker authenticated you can push the image to GCR and a new container registry will be created for you.

For the image to be pushed successfully it must be tagged either when it is created or before you push the image to GCR with the registry name gcr.io/your-project-name/ followed by the name of the image and the version. If a version is not specified the latest version will be used.

# if you are creating the image for the first time, you can tag it with the flag '-t'
docker build . -t gcr.io/your-project-name/twemproxy:v1

# or if the image already exists, use the 'docker tag' command
docker tag twemproxy gcr.io/your-project-name/twemproxy:v1

docker push gcr.io/your-project-name/twemproxy:v1

Creating an Image Template from the Docker Image

Next an image template must be created. This is the virtual machine image that will be used by the managed instance group (MIG) when creating new instances of the Twemproxy server.

To create the image template you will need to have the digest of the Docker image you just pushed to GCR. The digest is a unique id generated by GCP to distinguish different versions of the same Docker image. You can find the digest manually using the browser-based console or you can script out the process of retrieving the digest which is useful for automating the deployment process.

(All script examples will be written in Powershell but can be adapted as Linux shell scripts)

$Digest = (gcloud container images list-tags "gcr.io/your-project-name/twemproxy" `
    --limit=1 `
    --format="get(digest)")[0]

$Image = "gcr.io/your-project-name/twemproxy@{0}" -f $Digest

This script retrieves a list of the twemproxy Docker image tags with a limit of "1" so that only the first one is retrieved. This should be the most recent image pushed to the registry. The --format flag reduces the output to only the digest property. You can run this command with the format changed to --format=json if you would like to see all properties available. Once the digest is retrieved it is appended to the fully qualified name of the image in your registry.

You can now use the Docker image name to create an image template with the Docker container.

gcloud compute instance-templates create-with-container twemproxy-v1 `
    --container-image=$Image `
    --machine-type=f1-micro `
    --tags=fw-allow-lb-and-hc,fw-allow-twemproxy-6380 `
    --no-address `
    --scopes=cloud-platform `
    --network=your-network-name `
    --subnet=your-subnet-name `
    --region=us-east1

Note that the tag names following the --tags flag must match the target tags defined by the firewall rules created in the previous blog post.

Performing a Rolling Update on the Managed Instances Group

Lastly, we will need to perform a rolling restart of the MIG so that the newly created template with the Twemproxy server is used instead of the placeholder template used in the Terraform code.

gcloud compute instance-groups managed rolling-action start-update twemproxy-mig `
    --version="template=twemproxy-v1" `
    --zone=us-east1-b

Once the rolling update completes your Twemproxy server will be live and ready to test. For your microservices to communicate with the Twemproxy server instead of directly with a Memorystore instance you may need to make some changes to the configuration of any Redis client libraries used by your code.

For example in C# the Nuget package StackExchange.Redis has a configuration object that must register the Twemproxy server to work correctly. The only difference between using this package directly against Memorystore and indirectly through Twemproxy is the Proxy property of the ConfigurationOptions object. The API for setting and retrieving data from the cache remains the same.

private IConnectionMultiplexer CreateConnectionMultiplexer(string ip, int port)
{
    var config = new ConfigurationOptions
    {
        AsyncTimeout = 1000,
        EndPoints = {{ip, port}},
        Proxy = Proxy.Twemproxy
    };

    return ConnectionMultiplexer.Connect(config);
}

You will also need to set the IP and port number to match that of the load balancer for the Twemproxy MIG.

Test and Configure to Meet Your Requirements

Now you can redeploy your client services with any changes to the code and start testing your newly created Memorystore cluster! Every use case is different so you may run into a few issues that I did not encounter or you may need to go back and further configure the Twemproxy server to meet your requirements.

Before concluding this series on using Twemproxy with Memorystore in GCP there is one last script you may find useful when automating your deployment process. Whenever you add, remove, or recreate an instance of Memorystore the list of IP addresses in the YAML file used to configure the Twemproxy server will need to be updated. Rather than doing this manually, you can use a script.

The first step is to remove the existing list of IP addresses from the YAML file.

Get-Content -Path .\nutcracker.yml | Where-Object {$_ -notmatch ":6379:1"} | Set-Content .\tmp.yml
Get-Content -Path .\tmp.yml | Set-Content .\nutcracker.yml
Remove-Item .\tmp.yml

Then use the gcloud SDK to retrieve a list of the IP addresses of your deployed Memorystore instances and append them to the end of the YAML file. For this to work it's important that the servers property of the configuration YAML be the last one listed.

gcloud redis instances list --region us-east1 --format "value(host)" `
| ForEach-Object {"        - {0}:6379:1" -f $_} `
| Out-File -FilePath .\nutcracker.yml -Append -Encoding utf8

With the scripts in this blog post you should have most if not all the tools you need to completely automate the process of redeploying your Twemproxy server whenever you need to change the configuration or scale up the number of Memorystore instances.