DEV Community


Posted on

Apache Airflow Installation - mysql+celery

pre requisites: python 3.7 or greater

Execute the below ones one after another

pip install apache-airflow==1.10.6
pip install 'apache-airflow[celery]'
pip install 'apache-airflow[mysql]'
brew install rabbitmq

start the rabbitmq-server in the background
rabbitmq-server -detached

make the mysql config changes
sudo chown -R _mysql:mysql /usr/local/var/mysql

start the mysql server
sudo mysql.server start

mysql setup

mysql -uroot

set the password for the root user with the below

mysql>ALTER USER 'root'@'localhost' IDENTIFIED BY 'new password';

create the DB, the user and grant the required privileges

mysql>CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
mysql>create user ‘airflow’@’localhost’ identified by ‘airflow’;
mysql>grant all privileges on * . * to 'airflow'@'localhost';
mysql>flush privileges;

airflow initialization

airflow initdb

airflow config file update

Update the airflow.cfg file (should be available in ~/airflow/ directory.

sql connection as:
sql_alchemy_conn = mysql://root:airflow@localhost/airflow
there should be a sql alchemy connection string and you can comment it and add the above

executor as:

Save the file and exit.

run the below on their separate windows

airflow webserver
airflow scheduler
airflow worker

airflow webserver window will show you the url for airflow UI.

note: airflow initialization (airflow initdb) should load the example dags. You can turn them off in your airflow.cfg file, if you don't want to. command: load_examples = False

Top comments (0)