I encountered challenges when managing Celery asynchronous tasks across three different codebases: the backend REST API, the frontend web app, and the dashboard app, each residing in its own repository and each app ran on a separate AWS EC2 instance with its dedicated RabbitMQ, and things worked smoothly.
However, when we brought all the apps under a single instance, we ran into issues with Celery asynchronous tasks during development. Tasks collided between apps, causing disruptions. This problem stemmed from using the same RabbitMQ with its default guest account.
Why did it happen?
It occurred because the codebase was from 2013, and during fast development sprints, it was easier and faster to work on separate repositories and instances.
We directly used Python functions to call Celery with .apply_async and .delay, without using the async call with topic and queue.
The RabbitMQ vHosts
To address this, we implemented a simple yet effective solution: creating separate virtual hosts (vHosts) for each app within one RabbitMQ instance.
Key Takeaways
1. Password Pattern Management
Ensuring a correct password pattern (devoid of characters like oO0|.,@ ‘ “) became pivotal. Initially, RabbitMQ’s guest account didn’t require credentials. However, the shift to separate vhosts and vhost users made credentials indispensable, emphasizing the need for legible connection URL formats.
2. RabbitMQ Vhost Implementation
The introduction of RabbitMQ vhosts added an additional task for our DevOps team. Managing vhost setup and credentials became an essential step. Despite the extra maintenance, the clarity it brought to connection URLs justified the effort.
3. Reduced AWS Costs
Consolidating from three instances to one yielded significant reductions in AWS costs. The average cost shrunk from 3x to 1x, offering tangible savings.
Incorporating Connection URL in Code
To enhance readability and maintain a secure connection, we implemented the following in our code:
# Constructing Connection URL with Vhost Information
AMQP_URL_AND_PASSWORD = 'amqp://vhost_user:vhost_password@' + str(os.environ.get('RABBITMQ_IP')) + ':5672/vhost_name'
This addition ensured a clear and secure connection URL, aligning with our strategy of effective RabbitMQ and Celery management.
Adding Vhost and User via RabbitMQ Commands
For readers looking to replicate our setup, the following commands can be used to add a vhost and user to RabbitMQ:
# Adding a new vhost for the web app Celery instance
sudo rabbitmqctl add_vhost <vhost_name>
# Creating a new user for the web app Celery instance with a secure password
sudo rabbitmqctl add_user <vhost_user> <vhost_password>
# Setting permissions for the new user on the created vhost
sudo rabbitmqctl set_permissions -p <vhost_name> <vhost_user> ".*" ".*" ".*"
These commands help establish a segregated environment for the web app Celery instance, enhancing security and manageability.
Conclusion
Our experience in merging three projects into a single instance wasn’t without challenges. The emergence of async task collision issues prompted a reevaluation of our approach. The adoption of RabbitMQ vhosts provided an effective solution, enhancing management, security with credentials, and driving substantial cost savings. This journey underscores the importance of addressing fundamental infrastructure concerns, even during rapid development sprints.
P.S:
We specialize in procuring products and fostering application developments from overseas. Our robust team excels in mobile app development, encompassing Android, iOS, and Flutter platforms, as well as web app and dashboard development.
Our track record includes successfully conceptualizing, developing, and delivering applications and infrastructure that have seamlessly scaled from initial stages to serving millions of users.
For further inquiries and detailed discussions, please feel free to connect with us:
Top comments (0)