DEV Community

How to become more productive using Makefile

Ousseynou Diop on June 27, 2020

Originally posted on my blog Introduction You want to become more productive, automate your workflow, there are many tools to help you ...

Read full post

Jonathan Hartley • Jul 6 '20 • Edited

I love Makefiles and use them in almost all my projects, but I think the main benefits of them is walking DAGs of dependencies, which the article doesn't really mention.

For example, if your toolchain generates files:

source-file
   |
   tool-a
   |
intermediate-file
   |
   tool-b
   |
final-file

Then you can define some Makefile targets:

# final-file depends upon the state of intermediate-file
final-file: intermediate-file
     tool-b intermediate-file >final-file

# intermediate file depends on source-file
intermediate-file: source-file
    tool-a source-file >intermediate-file

Then the command 'make final-file' will run tool-b to generate the intermediate file, and knows that it first needs to run tool-a to generate the intermediate-file, and (the clever bit) it only runs each step if the output of the step is older than any of its inputs.

This means you eliminate needless typing (from running a tree of pre-requisite steps), and you eliminate needless (and possibly slow) rebuilding of files which don't need it. This is the core value of make, and if you can't take advantage of this, then there isn't much value over a set of script files.

I'd classify my current employer (Canonical) as high on the technically savvy scale, and my team uses a Makefile or two on every project. Even in cases where we can't make use of the above dependency analysis, it's nice for us to standardize on one approach to controlling how to build/test/etc projects.

Ousseynou Diop • Jul 7 '20

Very interesting, thanks.

Jonathan Hartley • Jul 7 '20

I added some missing "dependencies" (the parts after the colon) in my Makefile example.

ADONIS SIMO • Jun 27 '20

Great article, i use to handle some of those task with custom django commands... but it really depends on what the task has to achieve, for example initializing app with default data... but it's a good idea to use makefile also.

But i am not sure if it can do everything, for example can it help you to deploy on remote server via SSH like fabfile ?

Ousseynou Diop • Jun 27 '20

Thank you, friend.
Yes, you can do that.
I use this file to deploy my app to a docker container or Heroku server.

ADONIS SIMO • Jun 27 '20

yeah i use it also but for VPS deployment and sometime on EC2....

Matt Hamilton • Jun 27 '20

TABS! Make sure you mention you need to use tabs not spaces to indent the start of a command. That is something that still gets me two decades of using Makefile's later.

Ousseynou Diop • Jun 27 '20

Good to know.
Thank you.

Galuh Utama • Jun 27 '20 • Edited

But you don’t really save time though since the commands are short already. I question the productivity improvements here and would use bash aliases instead.

Make was invented for compiling (C/C++) codebases which involves a lot of long commands for compiler, linker, finding libraries, dependency tracking, etc. And it’s still a pain in the ass to maintain makefiles for those purposes. Nowadays I see less and less makefiles at work and in the opensource community.

Štefan Jarina • Jun 27 '20 • Edited

Hi, yeah, the example is a little bit too simplistic, however with a bit more involved example, one can start seeing the benefits:

include Makefile.settings

.PHONY: init build clean publish log jenkins slave

# Jenkins settings
export DOCKER_GID ?= 100
export JENKINS_USERNAME ?= admin
export JENKINS_PASSWORD ?= password
export JENKINS_SLAVE_VERSION ?= 2.2
export JENKINS_SLAVE_LABELS ?= DOCKER

# AWS settings
# The role to assume to inject temporary credentials into your Jenkins container
AWS_ROLE ?= `aws configure get role_arn`
# KMS encrypted password - the temporary credentials must possess kms:decrypt permissions for the key used to encrypt the credentials
export KMS_JENKINS_PASSWORD ?=

init:
    ${INFO} "Creating volumes..."
    @ docker volume create --name=jenkins_home

build:
    ${INFO} "Building image..."
    @ docker-compose build --pull
    ${INFO} "Build complete"

jenkins: init
    @ $(if $(and $(AWS_PROFILE),$(KMS_JENKINS_PASSWORD)),$(call assume_role,$(AWS_ROLE)),)
    ${INFO} "Starting Jenkins..."
    ${INFO} "This may take some time..."
    @ docker-compose up -d jenkins
    @ $(call check_service_health,$(RELEASE_ARGS),jenkins)
    ${INFO} "Jenkins is running at http://$(DOCKER_HOST_IP):$(call get_port_mapping,jenkins,8080)..."

publish:
    ${INFO} "Publishing images..."
    @ docker-compose push
    ${INFO} "Publish complete"

slave:
    ${INFO} "Checking Jenkins is healthy..."
    @ $(if $(and $(AWS_PROFILE),$(KMS_JENKINS_PASSWORD)),$(call assume_role,$(AWS_ROLE)),)
    @ $(call check_service_health,$(RELEASE_ARGS),jenkins)
    ${INFO} "Starting $(SLAVE_COUNT) slave(s)..."
    @ docker-compose up -d --scale jenkins-slave=$(SLAVE_COUNT)
    ${INFO} "$(SLAVE_COUNT) slave(s) running"

clean:
    ${INFO} "Stopping services..."
    @ docker-compose down -v || true
    ${INFO} "Services stopped"

destroy: clean
    ${INFO} "Deleting jenkins home volume..."
    @ docker volume rm -f jenkins_home
    ${INFO} "Deletion complete"

log:
    ${INFO} "Streaming Jenkins logs - press CTRL+C to exit..."
    @ docker-compose logs -f jenkins

what I see is that makefiles are less used with some languages, but are still quite heavily used with others.
Though I agree that I also haven't seen complex makefile in a while, but that is probably because more and more languages have their own tooling.

JS: npm scripts, gulp, grunt, etc.
C#: Powershell (Invoke!), maybe Cake
F#: FAKE
Rust: cargo custom tasks
JVM languages had their own for a long time also: sbt (Scala), gradle (a lot of JVM langs)
Python: Fabric, but here I've seen a lot of makefiles :-)
Ruby: Rake
Elixir: mix with custom tasks
Go: maybe Task, but here I've also seen quite a lot of makefiles

Can't tell for the rest, these are the ones I am familiar with.

Štefan Jarina • Feb 9 '21 • Edited

Small updates:

PYTHON:

Fabric was used for this in it's 1.x version.

Since version 2.0 Fabric was split essentially into 2 packages with separate functionalities:

Fabric 2.0 - now used only for remote shell/commands using ssh
Invoke! - task execution tool & library extracted from Fabric 1.x into it's own library

GENERIC:
There is now another tool, that is generic and can be used for many languages, it is even cross-platform

Bazel: bazel.build/

Haven't used it though, in most cases I go with language native solution and if there is none, I usually either:

still write Makefile if only linux/mac/docker/kubernetes is needed
use Powershell 7 if I need it to run on MAC/Linux/Windows

mintyPT • Feb 9 '21

What do you have inside your Makefile.settings?

Štefan Jarina • Feb 9 '21

Hi,

The code in this file is probably a bit more involved and harder to decipher, but it is just parsing functions really.
To get useful data from docker and/or env variables

Makefile.settings:

YELLOW := "\e[1;33m"
NC := "\e[0m"
INFO := @bash -c 'printf $(YELLOW); echo "=> $$1"; printf $(NC)' MESSAGE
ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))

SHELL = bash

# Slave arguments
ifeq ($(firstword $(MAKECMDGOALS)),$(filter $(firstword $(MAKECMDGOALS)),slave))
  SLAVE_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
  SLAVE_COUNT = $(if $(SLAVE_ARGS),$(firstword $(SLAVE_ARGS)),1)
endif

# Docker host settings
DOCKER_HOST_IP := $(shell echo $$DOCKER_HOST | awk -F/ '{printf $$3}' | awk -F: '{printf $$1}')
DOCKER_HOST_IP := $(if $(DOCKER_HOST_IP),$(DOCKER_HOST_IP),localhost)

# Image and Repository Tag introspection functions
# Syntax: $(call get_image_id,<docker-compose-environment>,<service-name>)
# Syntax: $(call get_repo_tags,<docker-compose-environment>,<service-name>,<fully-qualified-image-name>)
get_container_id = $$(docker-compose $(1) ps -q $(2))
get_image_id = $$(echo $(call get_container_id,$(1),$(2)) | xargs -I ARGS docker inspect -f '{{ .Image }}' ARGS)
get_container_state = $$(echo $(call get_container_id,$(1),$(2)) | xargs -I ID docker inspect -f '$(3)' ID)
filter_repo_tags = $(if $(findstring docker.io,$(1)),$(subst docker.io/,,$(1))[^[:space:]|\$$]*,$(1)[^[:space:]|\$$]*)
get_repo_tags = $$(echo $(call get_image_id,$(1),$(2)) | xargs -I ID docker inspect -f '{{range .RepoTags}}{{.}} {{end}}' ID | grep -oh "$(call filter_repo_tags,$(3))" | xargs)

# Port introspection functions
# Syntax: $(call get_port_mapping,<service-name>,<internal-port>)
get_raw_port_mapping = $$(docker-compose ps -q $(1) | xargs -I ID docker port ID $(2))
get_port_mapping = $$(echo $$(IFS=':' read -r -a array <<< "$(call get_raw_port_mapping,$(1),$(2))" && echo "$${array[1]}"))

# Service health functions
# Syntax: $(call check_service_health,<docker-compose-environment>,<service-name>)
get_service_health = $$(echo $(call get_container_state,$(1),$(2),{{if .State.Running}}{{ .State.Health.Status }}{{end}}))
check_service_health = { \
  until [[ $(call get_service_health,$(1),$(2)) != starting ]]; \
    do sleep 1; \
  done; \
  if [[ $(call get_service_health,$(1),$(2)) != healthy ]]; \
    then echo $(2) failed health check; exit 1; \
  fi; \
}

# AWS assume role settings
# Attempts to assume IAM role using STS
# Syntax: $(call assume_role,<role-arn>)
get_assume_session = aws sts assume-role --role-arn=$(1) --role-session-name=admin
get_assume_credential = jq --null-input '$(1)' | jq .Credentials.$(2) -r
define assume_role
    $(eval AWS_SESSION = $(shell $(call get_assume_session,$(1))))
    $(eval export AWS_ACCESS_KEY_ID = $(shell $(call get_assume_credential,$(AWS_SESSION),AccessKeyId)))
    $(eval export AWS_SECRET_ACCESS_KEY = $(shell $(call get_assume_credential,$(AWS_SESSION),SecretAccessKey)))
    $(eval export AWS_SESSION_TOKEN = $(shell $(call get_assume_credential,$(AWS_SESSION),SessionToken)))
endef