Dealing with dependence on generated source files in a Makefile

I think I finally got it right today.

I have a static website, with pages generated from markdown documents. Some of them are written by hand, but most are generated with a python script from a flat database in a json file. The build process is automated with a Makefile.

I wanted to trigger the generator only when necessary, that is, when either the json or the script were changed. There are too many markdown docs, however, to hardcode their names in the Makefile. Normally, this is solved using a wildcard filemask, which enumerates relevant file names in a source directory, and prepending the path to the target directory:

ALL_MDS_IN_OBJ = $(wildcard $(OBJ)/*.md)
ALL_HTML_IN_BIN = $(addprefix $(BIN)/, $(notdir $(ALL_MDS_IN_OBJ:.md=.html)))

default: $(ALL_HTML_IN_BIN)

$(BIN)/%.html: \
    $(OBJ)/%.md \
    | $(BIN)

    ... convert mds to html with pandoc ...

$(OBJ)/%.md: \
    $(SRC)/source_data.json \
    generate-mds-from-json.py \
    | $(OBJ)

    ./generate-mds-from-json.py

The problem is, when make is run for the for the first time after cloning the repo or doing make clean, there are no markdown files in $(OBJ) yet. The wildcard doesn't match anything, so make happily announces that nothing needs to be done and exits.

To cope with that, I was running the generator manually before make each time I changed the json file or the script, which, of course, in the long run turned out to be tedious and error-prone.

Then I saw this question on stackoverflow and was finally enlightened.

As ChrisW explained, making the generator run for the first time could be forced by introducing a dependency on a sentinel file kept separately from the build artifacts, whose sole purpose was to contain the timestamp of the last generator run; The file would be touched if the generation was successful, and deleted on make clean.

MDS_SENTINEL = .mds_sentinel

default: \
    $(MDS_SENTINEL) \
    $(ALL_HTML_IN_BIN) \
    ... other deps ...

$(MDS_SENTINEL): \
    $(SRC)/source_data.json \
    generate-mds-from-json.py

    ./generate-mds-from-json.py \
        && touch $(MDS_SENTINEL)

clean:
    rm -r $(MDS_SENTINEL) $(OBJ)/* $(BIN)/*

But this was still not sufficient; As mentioned above, the wildcard function would still be evaluated before running the generator, resulting in empty files list and an early exit. It would be necessary to invoke make twice - first time to generate the intermediate files, second time to process them further. And compared to ./generate-mds-from-json.py && make, having to do make && make was not that huge win.

Luckily, this problem was solved in the same thread by Paul Roub, who suggested running make recursively from the recipe. This inner make would have wildcards expanded after all files are generated and process the files correctly.

So the final solution looked something like:

MDS_SENTINEL = .mds_sentinel

ALL_MDS_IN_OBJ = $(wildcard $(OBJ)/*.md)
ALL_HTML_IN_BIN = $(addprefix $(BIN)/, $(notdir $(ALL_MDS_IN_OBJ:.md=.html)))

default: \
    $(MDS_SENTINEL) \
    $(ALL_HTML_IN_BIN) \
    ... other deps ...

$(MDS_SENTINEL): \
    $(SRC)/source_data.json \
    generate-mds-from-json.py

    ./generate-mds-from-json.py \
        && touch $(MDS_SENTINEL) \
        && make mds

mds: $(ALL_HTML_IN_BIN)

clean:
    rm -r $(MDS_SENTINEL) $(OBJ)/* $(BIN)/*

This post was originally published on blog.tznvy.eu