DEV Community

Cover image for How to become more productive using Makefile
Ousseynou Diop
Ousseynou Diop

Posted on

How to become more productive using Makefile

Originally posted on my blog

Introduction

You want to become more productive, automate your workflow, there are many tools to help you achieve your goal.

make gives you the flexibility to run and compile a program from source.
In this guide, we will use it to automate our development workflow.
Note: I will use Django as an example, but the technics are applicable to other technologies.
In this guide, I will use Makefile to automate many things in Django like database migration, superuser, deployment, and more.
At the end of the guide, you'll be more productive in your development.
Before you continue, ensure that make is installed in your system.

Installation

In Order to use make we need two(2) things one(1) we need to install it in our system and finally we need to create a file called Makefile.

To install make in your system you can follow these links :
OSX
Windows

Basic examples

To explore make let's begin with this simple program, go to any directory, and create a Makefile.
Note: Don't give it an extension.

$ mkdir test_make && cd test_make
$ touch Makefile
Enter fullscreen mode Exit fullscreen mode

Open the file with your text editor and put this

say_hello:
        echo "Hello world"
Enter fullscreen mode Exit fullscreen mode

run the file by typing make inside the directory test_make

$ make
Enter fullscreen mode Exit fullscreen mode

What is the output?
Here is what I get.

echo "Hello world"
Hello world
Enter fullscreen mode Exit fullscreen mode

Django integration

If you're a Django developer, you know how tedious is to repeat again and again these commands :

python manage.py runserver
python manage.py makemigrations
python manage.py migrate
...
Enter fullscreen mode Exit fullscreen mode

Let's automate them with make

Clone the example project here and follow the README guide.

git clone https://github.com/xarala221/consume-restfull-api-with-django && cd consume-restfull-api-with-django
Enter fullscreen mode Exit fullscreen mode

Run the application to ensure that everything is working fine.

python manage.py migrate
python manage.py runserver
Enter fullscreen mode Exit fullscreen mode

Open your browser and go to http://localhost:8000.

Inside the project, folder created a file called Makefile and add these lines.

touch Makefile
Enter fullscreen mode Exit fullscreen mode

The final file will look like this

SHELL := /bin/bash

help:
    @$(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | sort | egrep -v -e '^[^[:alnum:]]' -e '^$@$$'

install:
    pipenv install

activate:
    pipenv shell

run:
    python manage.py runserver

migration:
    python manage.py makemigrations

migrate:
    python manage.py migrate

superuser:
    python manage.py createsuperuser

heroku:
    git push heroku master

deploy:
    docker-compose build
    docker-compose up -d

down:
    docker-compose down

Enter fullscreen mode Exit fullscreen mode

you'll be more productive with these tips.

Conclusion

Become more productive and make things done is very important, in this guide you've learned how to automate your workflow by using make.

What do you think about this? Let us know in the comment section.

Top comments (20)

Collapse
 
tartley profile image
Jonathan Hartley • Edited

I love Makefiles and use them in almost all my projects, but I think the main benefits of them is walking DAGs of dependencies, which the article doesn't really mention.

For example, if your toolchain generates files:

source-file
   |
   tool-a
   |
intermediate-file
   |
   tool-b
   |
final-file

Then you can define some Makefile targets:

# final-file depends upon the state of intermediate-file
final-file: intermediate-file
     tool-b intermediate-file >final-file

# intermediate file depends on source-file
intermediate-file: source-file
    tool-a source-file >intermediate-file

Then the command 'make final-file' will run tool-b to generate the intermediate file, and knows that it first needs to run tool-a to generate the intermediate-file, and (the clever bit) it only runs each step if the output of the step is older than any of its inputs.

This means you eliminate needless typing (from running a tree of pre-requisite steps), and you eliminate needless (and possibly slow) rebuilding of files which don't need it. This is the core value of make, and if you can't take advantage of this, then there isn't much value over a set of script files.

I'd classify my current employer (Canonical) as high on the technically savvy scale, and my team uses a Makefile or two on every project. Even in cases where we can't make use of the above dependency analysis, it's nice for us to standardize on one approach to controlling how to build/test/etc projects.

Collapse
 
xarala221 profile image
Ousseynou Diop

Very interesting, thanks.

Collapse
 
tartley profile image
Jonathan Hartley

I added some missing "dependencies" (the parts after the colon) in my Makefile example.

Collapse
 
simo97 profile image
ADONIS SIMO

Great article, i use to handle some of those task with custom django commands... but it really depends on what the task has to achieve, for example initializing app with default data... but it's a good idea to use makefile also.

But i am not sure if it can do everything, for example can it help you to deploy on remote server via SSH like fabfile ?

Collapse
 
xarala221 profile image
Ousseynou Diop

Thank you, friend.
Yes, you can do that.
I use this file to deploy my app to a docker container or Heroku server.

Collapse
 
simo97 profile image
ADONIS SIMO

yeah i use it also but for VPS deployment and sometime on EC2....

Collapse
 
hammertoe profile image
Matt Hamilton

TABS! Make sure you mention you need to use tabs not spaces to indent the start of a command. That is something that still gets me two decades of using Makefile's later.

Collapse
 
xarala221 profile image
Ousseynou Diop

Good to know.
Thank you.

Collapse
 
gwutama profile image
Galuh Utama • Edited

But you don’t really save time though since the commands are short already. I question the productivity improvements here and would use bash aliases instead.

Make was invented for compiling (C/C++) codebases which involves a lot of long commands for compiler, linker, finding libraries, dependency tracking, etc. And it’s still a pain in the ass to maintain makefiles for those purposes. Nowadays I see less and less makefiles at work and in the opensource community.

Collapse
 
stefanjarina profile image
Štefan Jarina • Edited

Hi, yeah, the example is a little bit too simplistic, however with a bit more involved example, one can start seeing the benefits:

include Makefile.settings

.PHONY: init build clean publish log jenkins slave

# Jenkins settings
export DOCKER_GID ?= 100
export JENKINS_USERNAME ?= admin
export JENKINS_PASSWORD ?= password
export JENKINS_SLAVE_VERSION ?= 2.2
export JENKINS_SLAVE_LABELS ?= DOCKER

# AWS settings
# The role to assume to inject temporary credentials into your Jenkins container
AWS_ROLE ?= `aws configure get role_arn`
# KMS encrypted password - the temporary credentials must possess kms:decrypt permissions for the key used to encrypt the credentials
export KMS_JENKINS_PASSWORD ?=

init:
    ${INFO} "Creating volumes..."
    @ docker volume create --name=jenkins_home

build:
    ${INFO} "Building image..."
    @ docker-compose build --pull
    ${INFO} "Build complete"

jenkins: init
    @ $(if $(and $(AWS_PROFILE),$(KMS_JENKINS_PASSWORD)),$(call assume_role,$(AWS_ROLE)),)
    ${INFO} "Starting Jenkins..."
    ${INFO} "This may take some time..."
    @ docker-compose up -d jenkins
    @ $(call check_service_health,$(RELEASE_ARGS),jenkins)
    ${INFO} "Jenkins is running at http://$(DOCKER_HOST_IP):$(call get_port_mapping,jenkins,8080)..."

publish:
    ${INFO} "Publishing images..."
    @ docker-compose push
    ${INFO} "Publish complete"

slave:
    ${INFO} "Checking Jenkins is healthy..."
    @ $(if $(and $(AWS_PROFILE),$(KMS_JENKINS_PASSWORD)),$(call assume_role,$(AWS_ROLE)),)
    @ $(call check_service_health,$(RELEASE_ARGS),jenkins)
    ${INFO} "Starting $(SLAVE_COUNT) slave(s)..."
    @ docker-compose up -d --scale jenkins-slave=$(SLAVE_COUNT)
    ${INFO} "$(SLAVE_COUNT) slave(s) running"

clean:
    ${INFO} "Stopping services..."
    @ docker-compose down -v || true
    ${INFO} "Services stopped"

destroy: clean
    ${INFO} "Deleting jenkins home volume..."
    @ docker volume rm -f jenkins_home
    ${INFO} "Deletion complete"

log:
    ${INFO} "Streaming Jenkins logs - press CTRL+C to exit..."
    @ docker-compose logs -f jenkins
Enter fullscreen mode Exit fullscreen mode

what I see is that makefiles are less used with some languages, but are still quite heavily used with others.
Though I agree that I also haven't seen complex makefile in a while, but that is probably because more and more languages have their own tooling.

  • JS: npm scripts, gulp, grunt, etc.
  • C#: Powershell (Invoke!), maybe Cake
  • F#: FAKE
  • Rust: cargo custom tasks
  • JVM languages had their own for a long time also: sbt (Scala), gradle (a lot of JVM langs)
  • Python: Fabric, but here I've seen a lot of makefiles :-)
  • Ruby: Rake
  • Elixir: mix with custom tasks
  • Go: maybe Task, but here I've also seen quite a lot of makefiles

Can't tell for the rest, these are the ones I am familiar with.

Collapse
 
stefanjarina profile image
Štefan Jarina • Edited

Small updates:

PYTHON:

Fabric was used for this in it's 1.x version.

Since version 2.0 Fabric was split essentially into 2 packages with separate functionalities:

  • Fabric 2.0 - now used only for remote shell/commands using ssh
  • Invoke! - task execution tool & library extracted from Fabric 1.x into it's own library

GENERIC:
There is now another tool, that is generic and can be used for many languages, it is even cross-platform

Haven't used it though, in most cases I go with language native solution and if there is none, I usually either:

  • still write Makefile if only linux/mac/docker/kubernetes is needed
  • use Powershell 7 if I need it to run on MAC/Linux/Windows
Collapse
 
mintypt profile image
mintyPT

What do you have inside your Makefile.settings?

Thread Thread
 
stefanjarina profile image
Štefan Jarina

Hi,

The code in this file is probably a bit more involved and harder to decipher, but it is just parsing functions really.
To get useful data from docker and/or env variables

Makefile.settings:

YELLOW := "\e[1;33m"
NC := "\e[0m"
INFO := @bash -c 'printf $(YELLOW); echo "=> $$1"; printf $(NC)' MESSAGE
ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))

SHELL = bash

# Slave arguments
ifeq ($(firstword $(MAKECMDGOALS)),$(filter $(firstword $(MAKECMDGOALS)),slave))
  SLAVE_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
  SLAVE_COUNT = $(if $(SLAVE_ARGS),$(firstword $(SLAVE_ARGS)),1)
endif

# Docker host settings
DOCKER_HOST_IP := $(shell echo $$DOCKER_HOST | awk -F/ '{printf $$3}' | awk -F: '{printf $$1}')
DOCKER_HOST_IP := $(if $(DOCKER_HOST_IP),$(DOCKER_HOST_IP),localhost)

# Image and Repository Tag introspection functions
# Syntax: $(call get_image_id,<docker-compose-environment>,<service-name>)
# Syntax: $(call get_repo_tags,<docker-compose-environment>,<service-name>,<fully-qualified-image-name>)
get_container_id = $$(docker-compose $(1) ps -q $(2))
get_image_id = $$(echo $(call get_container_id,$(1),$(2)) | xargs -I ARGS docker inspect -f '{{ .Image }}' ARGS)
get_container_state = $$(echo $(call get_container_id,$(1),$(2)) | xargs -I ID docker inspect -f '$(3)' ID)
filter_repo_tags = $(if $(findstring docker.io,$(1)),$(subst docker.io/,,$(1))[^[:space:]|\$$]*,$(1)[^[:space:]|\$$]*)
get_repo_tags = $$(echo $(call get_image_id,$(1),$(2)) | xargs -I ID docker inspect -f '{{range .RepoTags}}{{.}} {{end}}' ID | grep -oh "$(call filter_repo_tags,$(3))" | xargs)

# Port introspection functions
# Syntax: $(call get_port_mapping,<service-name>,<internal-port>)
get_raw_port_mapping = $$(docker-compose ps -q $(1) | xargs -I ID docker port ID $(2))
get_port_mapping = $$(echo $$(IFS=':' read -r -a array <<< "$(call get_raw_port_mapping,$(1),$(2))" && echo "$${array[1]}"))

# Service health functions
# Syntax: $(call check_service_health,<docker-compose-environment>,<service-name>)
get_service_health = $$(echo $(call get_container_state,$(1),$(2),{{if .State.Running}}{{ .State.Health.Status }}{{end}}))
check_service_health = { \
  until [[ $(call get_service_health,$(1),$(2)) != starting ]]; \
    do sleep 1; \
  done; \
  if [[ $(call get_service_health,$(1),$(2)) != healthy ]]; \
    then echo $(2) failed health check; exit 1; \
  fi; \
}

# AWS assume role settings
# Attempts to assume IAM role using STS
# Syntax: $(call assume_role,<role-arn>)
get_assume_session = aws sts assume-role --role-arn=$(1) --role-session-name=admin
get_assume_credential = jq --null-input '$(1)' | jq .Credentials.$(2) -r
define assume_role
    $(eval AWS_SESSION = $(shell $(call get_assume_session,$(1))))
    $(eval export AWS_ACCESS_KEY_ID = $(shell $(call get_assume_credential,$(AWS_SESSION),AccessKeyId)))
    $(eval export AWS_SECRET_ACCESS_KEY = $(shell $(call get_assume_credential,$(AWS_SESSION),SecretAccessKey)))
    $(eval export AWS_SESSION_TOKEN = $(shell $(call get_assume_credential,$(AWS_SESSION),SessionToken)))
endef
Enter fullscreen mode Exit fullscreen mode
Thread Thread
 
mintypt profile image
mintyPT

Just the first 3 lines are worth it

Collapse
 
xarala221 profile image
Ousseynou Diop

Great content, Thank you.

Collapse
 
xarala221 profile image
Ousseynou Diop

Thank you,
I find this very interesting.

Collapse
 
patarapolw profile image
Pacharapol Withayasakpunt

I have thought about this a while ago, and Make seems to be both quite standard, and preinstalled (if not using Windows).

Collapse
 
ryands17 profile image
Ryan Dsouza

Great article! I'm trying it for a workflow for Node apps :)

Collapse
 
xarala221 profile image
Ousseynou Diop

Great.
Thank you.

Collapse
 
jingxue profile image
Jing Xue

It's an interesting approach, but why use make for a bash script's job?