DEV Community

Ujjwal Dubey
Ujjwal Dubey

Posted on

A Morning Smoke Test for Agency-Built Automations: One Python Script, Five Checks

When an outside team hands over an automation, the client usually gets a runbook and a demo. What they rarely get is a tiny script that answers one question every morning: is it still working?

This is the pattern we leave behind as an AI automation agency that hands over checks, not just code. Adapt the checks to your own stack.

The five checks

  1. The last successful run happened recently.
  2. No run is stuck in "processing".
  3. The error log is quiet since yesterday.
  4. Credentials are still valid (a cheap read call succeeds).
  5. The approval queue is not piling up.
import json, sys, datetime as dt

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

def smoke(runs, approvals, max_gap_hours=6, max_pending=20):
    now = dt.datetime.utcnow()
    problems = []
    ok = [r for r in runs if r["status"] == "ok"]
    if not ok:
        problems.append("no successful run on record")
    else:
        last = max(dt.datetime.fromisoformat(r["ts"]) for r in ok)
        if (now - last).total_seconds() > max_gap_hours * 3600:
            problems.append(f"last success at {last.isoformat()}")
    stuck = [r for r in runs if r["status"] == "processing"]
    if stuck:
        problems.append(f"{len(stuck)} runs stuck in processing")
    errors = [r for r in runs if r["status"] == "error"]
    if errors:
        problems.append(f"{len(errors)} errors in log")
    if len(approvals) > max_pending:
        problems.append(f"{len(approvals)} items waiting for approval")
    return problems

if __name__ == "__main__":
    issues = smoke(load("runs.jsonl"), load("approvals.jsonl"))
    print("OK" if not issues else "\n".join(issues))
    sys.exit(1 if issues else 0)
Enter fullscreen mode Exit fullscreen mode

Check 4 depends on your tools, so add one cheap read call per integration.

Where to run it

A scheduled job that emails the owner only when the script exits non-zero. Silence means healthy.

For the thinking behind choosing what to automate at all, see our blog on workflow audits and custom builds.

Top comments (0)