Subscribe to GCP Topic in Azure from C#

Today in this Blog I would like to share

How to subscribe google cloud publisher subscriber (Pubsub) topics into Azure services (azure function) using Microsoft visual studio C#

Overview
Pub/Sub is an asynchronous messaging service designed to be highly reliable and scalable. The service is built on a core Google infrastructure component that many Google products have relied upon for over a decade. Google products including Ads, Search and Gmail use this infrastructure to send over 500 million messages per second, totaling over 1TB/s of data. This article describes the salient design features that enables Pub/Sub to provide this type of scale reliably.

The Basics of a Publish/Subscribe Service
Pub/Sub is a publish/subscribe (Pub/Sub) service: a messaging service where the senders of messages are decoupled from the receivers of messages. There are several key concepts in a Pub/Sub service:

Message: the data that moves through the service.

Topic: a named entity that represents a feed of messages.

Subscription: a named entity that represents an interest in receiving messages on a particular topic.

Publisher (also called a producer): creates messages and sends (publishes) them to the messaging service on a specified topic.

Subscriber (also called a consumer): receives messages on a specified subscription.

The following diagram shows the basic flow of messages through Pub/Sub:

Publishers A and B push a message to a single topic, where two subscriptions listen for messages and push them to multiple subscribers.

In this scenario, there are two publishers publishing messages on a single topic. There are two subscriptions to the topic. The first subscription has two subscribers, meaning messages will be load-balanced across them, with each subscriber receiving a subset of the messages. The second subscription has one subscriber that will receive all of the messages. The bold letters represent messages. Message A comes from Publisher 1 and is sent to Subscriber 2 via Subscription 1, and to Subscriber 3 via Subscription 2. Message B comes from Publisher 2 and is sent to Subscriber 1 via Subscription 1 and to Subscriber 3 via Subscription 2

Prerequisites
Now as we have some idea about the architecture of this google service, let's create a topic in google cloud.

first you need to have google cloud account where we can create a GCP topic and subscriber.

You can register free with your gmail account to create a google cloud account.

Register here

Once you have google cloud account you can login to cloud console using this URL

By default a google cloud project will be created, you create a new one using Project drop down

Once you are in your project you can select Pubsub service from the left menu bar

Or you can search on top search bar with keyword "Topic"

Once you select the Pubsub service you will be navigated to Topic service Dashboard where you can find all the created topics and it's details.

Create Topic
To create new topic click on Create Topic button

Once you create a topic a default subscription will be created.

Now let's publish a new message to this topic manually (in your actual application it might possible some process will publish a message)

click on message tab

Now on the dashboard you can select subscription (default one in this case) and view the messages

Next we requires to create a service account and role to access the Pubsub service

Create Service account

To create a Service Account search IAM in the top search bar select IAM

Select Service account option from the left menu, Click on 'Create new Service Account' button to create a new service account.

Enter the detail and click on Create Account and Proceed

Select Role

Select Pub/Sub Publisher and Pub/Sub Subscriber

Click on Continue and click 'Done' (Don't make any changes to Grant users access to this service account option)

after this an account will be created now we need to create a Key for this service account

Select service account and click on Key tab

Click on 'Add Key' button to add new key for this service account

This will save a private key on your work station

This is the key which you can use in your program

{
  "type": "service_account",
  "project_id": "xenon-raceway-268218",
  "private_key_id": "9620fb3dd81a33e2bbe0e8defd327858633459ef",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC2Brf0+VVIv2vs\nUVAUSPKiP57z/9NOp/7qJyVgJsnk7brqvpU7RiJpCF84wmp4Emlgxh3vbwZa2+BU\n9MtCcd3Xb4JVnA0dxSafRIRQE6Fljp8KtKk42VZg2mggwdHhCXO1wfCbWtd5yng6\nIGV0OxWlzQbATmk+HvHZoOVZ1bWv22Baf2rF5w3msj9i4Jh5wU8fAQHX9+oWEVJRGCEZslB\nJ+vaWVVYluxfDpmv7YCZ3gs=\n-----END PRIVATE KEY-----\n",
  "client_email": "spn-pub-sub@xenon-raceway-268218.iam.gserviceaccount.com",
  "client_id": "116457436621716168449",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/spn-pub-sub%40xenon-raceway-268218.iam.gserviceaccount.com"
}

This json is the one which will be used to in the program to access to service data.

You can find more details about the environment variables here

https://cloud.google.com/docs/authentication/getting-started

Using above mentioned steps you can complete the setup of your Infra structure for PubSub topic

Lets create a new Console application or azure function application with C# as language

The code requires to use google cloud C# SDK using the NuGet packages

Now you can write azure function code to write following code to read the message from service topic from google.

Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", Path.Combine(Environment.CurrentDirectory, "credentials.json"));
    string projectId = "xenon-raceway-268218";
    string topicId = "order-topic";
    string subscriptionId = "order-topic-sub";
    ProjectName projectName = ProjectName.FromProject(projectId);

    var subscriptionName = new SubscriptionName(projectId, subscriptionId);
    while (true)
    {
        var subscriber = await SubscriberClient.CreateAsync(subscriptionName);
        try
        {
            await subscriber.StartAsync((msg, ct) =>
            {
                Console.WriteLine(msg.Data.ToStringUtf8());
                return Task.FromResult(SubscriberClient.Reply.Ack);
            });
        }
        catch (Exception e)
        {
            Console.WriteLine($"Exception: {e}");
        }
    }

This is how the output will looks like

References

https://cloud.google.com/pubsub/docs/reference/libraries#client-libraries-usage-csharp

https://cloud.google.com/dotnet/docs/reference/Google.Cloud.PubSub.V1/latest/index

I hope you have found this useful.