Developing Custom Nodes for n8n with Docker

The n8n automation / orchestration / workflow tool already covers a massive number of services without adding anything to it, but there may come a time when writing your own "custom-node" makes sense. The custom-node tutorial hits the high points, but doesn't include (as of this writing) any specifics about how to develop a custom node and test it in a self-hosted instance of n8n running in a Docker container. That's the subject of this post.

Prerequisites

• n8n running in Docker, preferably started using docker-compose
• An n8n custom-node (typescript/npm) project that is already working

Quick Clarification

This is about developing custom-nodes for n8n, which are like plugins for the product, not to be confused with Node.js, even though n8n runs in Node.js. Assuming if you're still reading you probably know the difference, but if you thought this was about coding in Node.js, stop now. This isn't the droid you're looking for.

Goal

Map one or more custom-node projects directly into an n8n Docker container such that n8n will find the "compiled" files and load the custom-node(s) at startup.

Dev Process Steps

(i.e. "How This Works Once You Have It Set Up")

For each "development cycle"...

1. Make changes to the custom-node source.
2. rm -rf dist (when necessary, in the dev project directory, to remove old/renamed file versions)
3. npm run build - i.e. re-compile
4. docker-compose restart n8n - i.e. trigger code reload in n8n

Setup

• Add one or more docker volumes, mapped to the dev/source directories docker-compose.yaml
  • The important part here is that the volumes map into the container at a path under /Users/node/.n8n/custom

    ...
    volumes:
      - /Users/myuserid/devstuff/custom_node_dev_project_1:/Users/node/.n8n/custom/node_modules/my-awesome-custom-node
      - /Users/myuserid/devstuff/custom_node_dev_project_abc:/Users/node/.n8n/custom/node_modules/my-other-custom-node

• That's it. Go write code.

Explanations (...if you care...)

(Read the rest of this [later] if you want to know reasons.)

npm link

The tutorial says to use npm link to map a custom-node project into n8n, but that assumes n8n is running from source, not in Docker.

• Notes:
  • Getting set up to run n8n from source is more time-consuming and has its own challenges, vs. running n8n in docker.
  • Docker volumes don't permit access to symlinked files, so npm link won't work for this anyway.
  • Trying to run npm link from the n8n custom node starter project already causes an error because the project (in package.json) is set up to require pnpm in the preinstall script, which runs implicitly from npm link and causes a conflict between pnpm and npm. :(
    • The error message is npm ERR! command sh -c npx only-allow pnpm
    • See: package.json

Effective Directory Structure

The directory structure mapped into the docker volumes mimics what npm link my-awesome-custom-node and npm link my-other-custom-node would have created, if they were run within the n8n runtime's .n8n/custom directory. Within the running Docker container, it ends up looking like this:

├── .n8n
    ├── custom
        ├── node_modules
            ├── my-awesome-custom-node
            │   ├── dist
            │   ├── ...other stuff...
            ├── my-other-custom-node
                ├── dist
                ├── ...other stuff...

Normal Docker install for a Custom-Node

Per the instructions for Install(ing) Private Nodes, you would just "Copy the node and credential folders from within the dist folder into your container's ~/.n8n/custom/ directory."

There are few issues with that strategy though.

1. Without some additional directory structure, which isn't specified in the instructions, it would only allow one custom-node to be presented into the n8n Docker runtime.
2. If the project is based on the custom node starter project, from the n8n github repo, you'd be copying the nodes and credentials directories/folders, not node and credential directories... but that could be just something that didn't get corrected/updated in the docs.
3. More importantly... copying stuff into a container's filesystem is transient. As soon as the container gets rebuilt/reset, all that stuff is gone. Mapping a volume to a directory on the host filesystem survives container resets, and doesn't require any overwriting. After each npm run build of the project, the container (i.e. n8n) just needs to be restarted to load the changes.

Links

• https://n8n.io

Things that might lead here...

npx only-allow pnpm

• Search finds stuff like this: https://stackoverflow.com/questions/63165630/trouble-with-npm-preinstall-script

ERR_PNPM_LINK_BAD_PARAMS  You must provide a parameter

• Search finds stuff like this: https://github.com/pnpm/pnpm/issues/4296

npm install n8n-nodes-my-custom-nodes

• Search finds stuff like this: https://community.n8n.io/t/npm-install-n8n-nodes-my-custom-nodes-throws-error/6528