Managing Kafka/AutoMQ clusters more conveniently with Kafka Web UI
With the rapid development of big data technology, Kafka, as a high-throughput, low-latency distributed messaging system, has become a core component of real-time data processing in enterprises. However, managing and monitoring Kafka clusters is not an easy task. Traditional command-line tools and scripts, although powerful, are complex and not intuitive for developers and operations personnel. To address these challenges, Kafka Web UI emerged, providing users with a more convenient and efficient way to manage Kafka clusters.

Over more than a decade of development, Apache Kafka® has accumulated a very rich ecosystem. As the successor to Apache Kafka®, AutoMQ can fully leverage the products in its ecosystem due to its complete compatibility with Kafka. AutoMQ Business Edition offers a very powerful control plane. If you are using AutoMQ, you can also manage AutoMQ clusters with products like Kafdrop and Redpanda Console [1].

Today, we will share how to monitor the state of AutoMQ clusters using Redpanda Console [1] to enhance system maintainability and stability.

Integrating AutoMQ with Redpanda Console
Redpanda Console is a Kafka Web UI interface provided by Redpanda [2] for monitoring and managing Redpanda or Kafka clusters. It offers an intuitive user interface where users can easily view cluster states, monitor performance metrics, and manage topics and partitions. This console is designed to simplify the daily operations of data flow systems, enabling users to more effectively maintain and monitor their clusters.
Thanks to AutoMQ's complete compatibility with Kafka, it can be seamlessly integrated with Redpanda Console. By utilizing Redpanda Console, AutoMQ users can also benefit from an intuitive user interface to monitor the real-time status of AutoMQ clusters, including topics, partitions, consumer groups, and their offsets. This monitoring capability not only enhances the efficiency of issue diagnosis but also helps optimize cluster performance and resource utilization.
This tutorial will teach you how to start the Redpanda Console service and use it in conjunction with AutoMQ clusters to achieve cluster state monitoring and management.

Prerequisites

• Deploy AutoMQ Cluster
• Prepare Redpanda Console Environment Deploy AutoMQ Cluster Please refer to the official AutoMQ documentation: Cluster Deployment | AutoMQ [3]. Deploy Redpanda Console There are two ways to deploy Redpanda Console: Docker deployment and release version deployment. Docker deployment is simpler, and if you want to quickly and easily experience the integration of AutoMQ and Redpanda Console, it is recommended to choose Docker for deployment. If you have special requirements, such as login authentication, SASL authentication, TLS configuration, and log level settings, you can opt for the release version deployment. Below, I will introduce both configuration methods separately. Docker Deployment Redpanda Console can be deployed via Docker, as referenced in Quick Start [4]. In this process, after setting up the AutoMQ cluster, you will know all the addresses and ports of the Broker nodes that are listening. Therefore, you can establish a connection between Redpanda Console and the AutoMQ cluster by specifying the KAFKA_BROKERS parameter in the Docker startup command. The Docker container startup command is as follows: docker run -p 8080:8080 -e KAFKA_BROKERS=192.168.0.4:9092,192.168.0.5:9092,192.168.0.6:9092 docker.redpanda.com/redpandadata/console:latest
• -p 8080:8080: Specify the port mapping for accessing the Redpanda Console service.
• KAFKA_BROKERS: This should be specified as the Broker addresses of your AutoMQ cluster.

Release Deployment
You need to download and extract the suitable version from the GitHub Releases page of Redpanda Console: Release Redpanda Console [5], into a specified folder, such as /opt. The command is as follows:

ubuntu Linux

cd /opt
sudo curl -L -o redpanda_console.tar.gz https://github.com/redpanda-data/console/releases/download/v2.6.0/redpanda_console_2.6.0_linux_amd64.tar.gz

unzip, get redpanda_console

sudo tar -xzf redpanda_console.tar.gz

config set

sudo mkdir -p /etc/redpanda

write config

sudo vim /etc/redpanda/redpanda-console-config.yaml
An example of the content of the redpanda-console-config.yaml configuration file is as follows:
kafka:
#Brokers is a list of bootstrap servers with
#port (for example "localhost:9092").
brokers:
- broker-0.mycompany.com:19092
- broker-1.mycompany.com:19092
- broker-2.mycompany.com:19092
Note: Please ensure that the server where you are currently installing Redpanda Console can access the servers where the Broker nodes specified in the configuration file are located.
For more detailed settings, you can refer to the official documentation: Redpanda Console Configuration [6]. After completing the configuration, you need to set environment variables so that the Redpanda Console executable can obtain the configuration file information and start the Redpanda Console:

set env

export CONFIG_FILEPATH="/etc/redpanda/redpanda-console-config.yaml"

/opt/ run console

./redpanda-console
You will get the following results:
./redpanda-console
{"level":"info","ts":"2024-07-10T09:52:52.958+0800","msg":"started Redpanda Console","version":"2.6.0","built_at":"1717083695"}
{"level":"info","ts":"2024-07-10T09:52:52.963+0800","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"}
{"level":"info","ts":"2024-07-10T09:52:54.780+0800","msg":"successfully connected to kafka cluster","advertised_broker_count":1,"topic_count":2,"controller_id":0,"kafka_version":"at least v3.6"}
{"level":"info","ts":"2024-07-10T09:53:05.620+0800","msg":"Server listening on address","address":"[::]:8080","port":8080}
Access the console page
fter completing the above deployment operations, you can access the console service by entering the address (e.g., http://{console_ip}:8080) in the browser. The display effect is as follows:
Cluster Overview
The Cluster Overview page provides users with a macro perspective, displaying core information of the AutoMQ cluster. This includes but is not limited to:

• Cluster Health Status: Shows the current health status of the cluster, aiding users in quickly identifying issues.
• Storage Usage: Displays the data storage usage within the cluster, making it easier for users to manage and plan their storage.
• Version Information: Shows the current version of the AutoMQ cluster, facilitating tracking and upgrades.
• Number of Online Brokers: Displays the real-time number of online Brokers, which is a critical metric.
• Number of Topics and Replicas: Provides information on the number of Topics and Replicas, helping users understand the scale of the cluster and data replication status.

Monitoring the cluster status is crucial for ensuring the stability and performance of the messaging queue system. By monitoring the real-time status of the cluster, storage usage, version information, the number of online Brokers, as well as the number of Topics and Replicas, operations personnel can quickly identify and resolve potential issues, preventing system failures from impacting business operations. Additionally, these metrics aid in capacity planning and resource management, ensuring the system can handle future data growth. Moreover, knowing the cluster's version information helps users to perform timely software upgrades, leveraging the latest features and security fixes, thereby enhancing overall system reliability and efficiency.
**Topic Overview
**On the Topic list page, users can see a list of all Topics in the current AutoMQ cluster, including key information for each Topic, such as the number of partitions and replication strategy. Users can quickly browse and manage Topics through this page.

Topic Details
After clicking on a specific Topic, users will be taken to the detailed page of that Topic, where they can explore and manage various aspects:

• Message List: Browse and search messages within the Topic, which is very useful for message tracking and debugging.
• Consumer Information: Displays information about the consumers and Consumer Groups currently subscribed to the Topic, facilitating monitoring of consumption status.
• Partition Status: Shows detailed information for each partition, including key metrics such as Leader and ISR.
• Configuration Information: List the configuration details of the Topic, supporting modifications to optimize performance or behavior.
• ACL (Access Control List): Manage access permissions for the Topic to ensure data security. Additionally, Redpanda Console supports users in manually creating and publishing messages, which is highly valuable for testing or injecting messages in specific scenarios.

Monitoring Topic details allows us to gain deep insights into the operation of the message queue. By browsing the message list, we can track and debug messages, monitor consumer information to assess consumption status, understand partition states to ensure data distribution and high availability, manage configuration information to optimize performance, and set access controls to ensure data security. These features help in timely identification and resolution of issues, thereby improving the overall efficiency and reliability of the system.
Summary
This article introduces the integration process of Redpanda Console with AutoMQ, demonstrating how this powerful tool simplifies and enhances the management of AutoMQ clusters. It is hoped that this article provides practical references for users aiming to improve the efficiency and functionality of message queue management.

References
[1] Redpanda Console: https://redpanda.com/redpanda-console-kafka-ui
[2] Redpanda: https://redpanda.com/
[3] Cluster Deployment of AutoMQ: https://docs.automq.com/zh/docs/automq-opensource/IyXrw3lHriVPdQkQLDvcPGQdnNh
[4] Quick Start: https://github.com/redpanda-data/console?tab=readme-ov-file#quick-start
[5] Release Redpanda Console: https://github.com/redpanda-data/console/releases/tag/v2.6.0
[6] Redpanda Console Configuration: https://docs.redpanda.com/current/reference/console/config/#example-redpanda-console-configuration-file
[7] Kafdrop Github: https://github.com/obsidiandynamics/kafdrop