Apache Airflow Installation - mysql+celery

pre requisites: python 3.7 or greater

Execute the below ones one after another

pip install apache-airflow==1.10.6
pip install 'apache-airflow[celery]'
pip install 'apache-airflow[mysql]'
brew install rabbitmq

start the rabbitmq-server in the background
rabbitmq-server -detached

make the mysql config changes
sudo chown -R _mysql:mysql /usr/local/var/mysql

start the mysql server
sudo mysql.server start

mysql setup

mysql -uroot

set the password for the root user with the below

mysql>ALTER USER 'root'@'localhost' IDENTIFIED BY 'new password';

create the DB, the user and grant the required privileges

mysql>CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
mysql>create user ‘airflow’@’localhost’ identified by ‘airflow’;
mysql>grant all privileges on * . * to 'airflow'@'localhost';
mysql>flush privileges;
mysql>quit

airflow initialization

airflow initdb

airflow config file update

Update the airflow.cfg file (should be available in ~/airflow/ directory.

sql connection as:
sql_alchemy_conn = mysql://root:airflow@localhost/airflow
there should be a sql alchemy connection string and you can comment it and add the above

executor as:
CeleryExecutor

Save the file and exit.

run the below on their separate windows

airflow webserver
airflow scheduler
airflow worker

airflow webserver window will show you the url for airflow UI.

note: airflow initialization (airflow initdb) should load the example dags. You can turn them off in your airflow.cfg file, if you don't want to. command: load_examples = False