Kafka is a powerful distributed streaming platform, and managing topics and access control lists (ACLs) is a crucial part of Kafka administration. To make this task easier, I've created a Python library called Kafka Ease. Kafka Ease provides utilities to automate the creation of Kafka topics and ACLs, streamlining the administrative process.
- Create Kafka topics with a specified number of partitions and replication factor.
- Define ACLs (Access Control Lists) for Kafka topics, controlling who can perform various operations on them.
To install Kafka Ease use the following command:
pip install kafka-ease
Kafka Ease uses YAML or JSON files for configuration. The configuration file must contain two main sections:
acls. Here's an example:
topics: - name: topic.name num_partitions: 1 replication_factor: 1 cleanup_policy: delete retention_days: 7 acls: - resource_type: TOPIC resource_name: topic.name principal: User:kafka-user host: "*" operation: READ permission_type: ALLOW pattern_type: LITERAL
- name: Topic name
- num_partitions: Number of partitions
- replication_factor: Replication factor
cleanup_policy: Cleanup policy (
- retention_days: Delete retention days
resource_type: Resource type (
- resource_name: Resource name
principal: Principal (
- host: Host (*)
operation: Operation (
permission_type: Permission type (
pattern_type: Pattern type (
To validate your configuration file, use the following command:
kafka-ease apply -f acl.yaml --only-validate
Validating file acl.yaml... File: acl.yaml YAML file detected 2 topics found. 2 ACLs found. File acl.yaml is valid.
To apply your configuration to Kafka, use a command like this:
kafka-ease apply -f acl.yaml \ --kafka-brokers kafka:9093 \ --security-protocol SASL_PLAINTEXT \ --sasl-mechanism SCRAM-SHA-256 \ --sasl-username kafka-admin \ --sasl-password SECRET
Applying file... Kafka brokers: kafka:9093 Security protocol: SASL_PLAINTEXT SASL mechanism: SCRAM-SHA-256 SASL username: kafka-admin SASL password: *************** File: acl.yaml YAML file detected 1 topics found. 1 ACLs found. Topic topic.name updated Removing old ACLs 3 ACL User:kafka-user synced File synced successfully.
For more details and options, check out the documentation.
In conclusion, Kafka Ease is a valuable Python library that simplifies Kafka topic and ACL management. With its features for creating and configuring topics and access control lists, it significantly streamlines administrative tasks for Kafka administrators. This tool can be seamlessly integrated into your CI/CD (Continuous Integration/Continuous Deployment) pipeline to enhance the Kafka configuration deployment process.
By incorporating Kafka Ease into your CI/CD pipeline, you can achieve the following benefits:
- Automation: Kafka Ease automates the creation and management of Kafka topics and ACLs, reducing the manual effort required for these tasks.
- Consistency: CI/CD pipelines ensure that Kafka configurations are consistently applied across different environments, reducing the risk of configuration errors.
- Version Control: With Kafka Ease configuration files under version control, changes to Kafka configurations can be tracked and reviewed, enhancing collaboration among team members.
- Testing: Before applying configurations to production Kafka clusters, you can thoroughly test them in staging or testing environments, ensuring the configurations work as expected.
- Security: Kafka Ease supports security protocols like SASL, allowing you to securely manage Kafka configurations in your CI/CD pipeline.
By integrating Kafka Ease into your CI/CD workflow, you can achieve a more efficient and controlled process for managing Kafka topics and ACLs across your Kafka clusters, ultimately enhancing the reliability and security of your Kafka infrastructure.
GitHub Repository: Kafka Ease on GitHub
Happy Kafkaing! 🚀