Projen - External Project Types

Extensibility is king.

Look what happened with the Github gh cli just recently with extensions in the 2.0 release. I even wrote one of them p6m7g8/gh-parallel to parallel clone a Github organization. Right now there are 191 extensions.

This post isn't about Github though. Projen is now an AWS Open Source project officially! Along with 28 other Github Organizations..... You can find them all like such:

curl -s https://aws.github.io | \
    grep https://github.com | \
    grep -v project*name |
    sed -e 's,.\_com/,,' -e 's,".*,,' -e 's,/,,' | \
    sort

So how do you make one?

Well first, you need a use case. In this case, I wanted to make an list for projen. It then occurred to me there are hundreds of awesome lists. I had never made an awesome list before. Turns out currently most folks would use yeoman. Unfortunately, while it works great, it can definitely add some more end-2-end automation like CI/CD, dependency updates, security updates, auto-approvals, merging, auto linting......

Ok, I feel like this has value.

Step 1: Install node.js

I will not cover this here. A good option is nodenv. For the purposes of this exercise you need to be on node 14.x - 16.x (not 12.x or 17.x).

Also install yarn v1 (not v2) globally

Step 2: Install projen

I like to wrap npm:

p6_js_npm_global_install () {
    local mod="$1"
    npm uninstall -g "$mod"
    npm install -g "$mod"
    npm list -g --depth 0
    nodenv rehash
}

This has the consequence of getting me the newest one each time too.

p6_js_npm_global_install 'projen'

npm list --depth 0 -g  | grep -E 'projen|yarn'
├── projen@0.53.14
└── yarn@1.22.18

Word to the wise, projen has changed quite a bit since March 2021 when it was at v0.17.x. You'll need to be on v0.53.x+ for this. I'm doin this with v0.53.14 which is current right now.

This is what caused me the most grief, because this issue got buried in my yarn.lock file which projen will use by default over npm. This wasn't projen's fault but mine for screwing up my published versions numbers on npmjs.org. Take my advice bump yourself to a clean new version if you see some of these errors coming up. It cost me 4 days.

Step 3: Initialize the project

mkdir p6-projen-project-awesome-list
cd p6-projen-project-awesome-list

I intend to provide this module in every language JSII supports so the lowest I can go is the projen type jsii.

The --projenrc-ts option isn't well documented but it does work fluidly, so why give up type checking on it especially when I was having issues getting this to work.

In fact, its only documented here in a Github Issue Comment - https://github.com/projen/projen/issues/14#issuecomment-871318062

projen new jsii --projenrc-ts

It made a local git repo for us, and it created v0.0.0. Thats not a bug. You should publish this version first or it will randomly do it later and you will be very confused. If it does, a 2nd publish will
"Do The Right Thing" (tm).

Step 4: Put it on Github!

gh repo create p6m7g8/p6-projen-project-awesome-list  -d "projen external project for Awesome Lists" -h "https://p6m7g8.github.io" --public --source . --push

Clearly, the picture is after I was done. I'm lazy, so feel free to blame me.

projen doesn't yet do other repository config, but see cdktf which uses terraform to see how you can manage teams, topics, and other things.
I have an example below for which I will write a future post about using cdktf with projen to manage Github Organizations soup to nuts.

Example(wip): https://github.com/p6m7g8/p6-cdktf-github-p6m7g8

Step 5: Open .projenrc.ts

It's time to break out VSCode.

I've already figured out the kinks for you, so we're going to replace this with something that gives me the values I wanted above in my use-case. I'm not going to dive into these because most of this is irrelevant for external project types; however, do note that you need to make sure:

• default branch is main not master
• correct permissions to allow issue creation and auto approving in your GitHub teams/users.

import { cdk } from "projen";

const project = new cdk.JsiiProject({
  name: "p6-projen-project-awesome-list",
  author: "Philip M. Gollucci",
  authorAddress: "pgollucci@p6m7g8.com",
  repositoryUrl: "https://github.com/p6m7g8/p6-projen-project-awesome-list.git",
  description: "Projen External Project for awesome-lists",
  stability: "experimental",
  keywords: ["awesome lists", "projen", "list", "awesome", "constructs"],

  defaultReleaseBranch: "main",
  projenrcTs: true,
  gitpod: true,
  devContainer: true,
  codeCov: true,
  prettier: true,
  releaseFailureIssue: true,
  autoApproveUpgrades: true,
  autoApproveOptions: {
    allowedUsernames: ["p6m7g8-automation"],
  },

  deps: ["projen@^0.53.14"],
  peerDeps: ["projen@^0.53.14"],

  publishToPypi: {
    distName: "p6-projen-project-awesome-list",
    module: "p6_projen_project_awesome_list",
  },

  publishToMaven: {
    javaPackage: "com.github.p6m7g8.P6ProjectProjenAwesomeList",
    mavenGroupId: "com.github.p6m7g8",
    mavenArtifactId: "p6-projen-project-awesome-list",
  },

  publishToNuget: {
    dotNetNamespace: "P6m7g8.P6AwesomeList",
    packageId: "P6m7g8.P6AwesomeList",
  },

  publishToGo: {
    moduleName: "github.com/p6m7g8/p6-projen-project-awesome-list", // why doesn't this default to repositoryUrl?
  },
});

project.synth();

Step 6

Regenerate it and build it

alias pj='npx projen'
pj && pj build

Hooray, it still works; but, we don't have any code it in it yet. There are a couple things you need to do for projen to think this is an External Project Type:

Step 7:

• You must extend an existing projen class in this case cdk.JsiiProject (or another External Project Type; start simple)
• You must include the @pjid keyword
• You must call the super() constructor
• You can NO longer call this.buildTask.reset(), instead, this.postCompileTask.spawn(awesomeLintTask);
  • Its immutable now

import { cdk, SampleFile } from "projen";
/**
 * Configurable knobs for Awesome Lists
 */
export interface AwesomeListProjectOptions extends cdk.JsiiProjectOptions {
  /**
   * What e-mail address to list for the Code of Conduct Point of Contact
   *
   * @default - `project.authorAddress`
   */
  readonly contactEmail?: string;
}

/**
 * Awesome List project
 *
 * @pjid awesome-list
 */
export class AwesomeList extends cdk.JsiiProject {
  constructor(options: AwesomeListProjectOptions) {
    super({
      ...options,
      readme: {
        filename: "readme.md",
        contents: readmeContents(),
      },
      stability: "experimental",
      keywords: ["awesome lists", "list", "awesome", "constructs"],

      defaultReleaseBranch: "main",
      gitpod: true,
      releaseToNpm: false,
      projenrcTs: true,
      devContainer: true,
      codeCov: true,
      prettier: true,
      releaseFailureIssue: true,
      autoApproveUpgrades: true,
      autoApproveOptions: {
        allowedUsernames: ["p6m7g8-automation"],
      },
    });

    new SampleFile(this, "code-of-conduct.md", {
      contents: this.codeOfConduct().replace(
        "CONTACTEMAIL",
        options.contactEmail ?? "noreply@example.com"
      ),
    });

    new SampleFile(this, "contributing.md", {
      contents: this.contributing(),
    });

    this._awesomeLint();
  }

  private _awesomeLint() {
    this.addDevDeps("awesome-lint");

    const awesomeLintTask = this.addTask("awesome-lint");
    awesomeLintTask.exec("npx awesome-lint");

    this.postCompileTask.spawn(awesomeLintTask)
  }

  // Actual content remove to keep this short
  private codeOfConduct(): string {
    return `content`;
  }

  private contributing(): string {
    return `content`;
  }
}

function readmeContents(): string {
  return `content`;
}

You might be wondering why I disabled publishtoNpm above. While the project type itself will be published. The Awesome List has no reason to be published, these only exist as Github readme.md files.

Step 8: Commit and Pull request

git add -A .
gh pr create -a $USER -f

This PR isn't exact, because I'm doing this post facto and I struggled. None-the-less, this is what the diff should look like roughly:

• https://github.com/p6m7g8/p6-projen-project-awesome-list/pull/6/files

Step 9: Merge the PR

The .mergify.yml will not be active until it's on main. So merge this by hand once the build finishes successfully.

• https://github.com/p6m7g8/p6-projen-project-awesome-list/actions/runs/2045203257

gh pr merge -d -s 6 ## 6 is the number of pr above
git pull

• https://github.com/p6m7g8/p6-projen-project-awesome-list/actions/runs/2045209274

Step 10: Look it up on on npmjs.org

• https://npmjs.org/packages/p6-projen-project-awesome-list

You should now iterate until you think you're happy.

Step 11: Lets Use it

If you goto the main projen README.md, you'll see these instructions for using your new module:

• The --from is just the name of your module. The second vuejs-ts is OPTIONAL. You only need it if you put more than 1 project type in the same repo, but will not hurt you.

mkdir awesome-projen
cd awesome-projen
projen new --from p6-projen-project-awesome-list@1.0.2 --projenrc-ts

Step 12: Lather, Rinse, Repeat -- aka Use it

At this point, its just another projen generated repository and you can treat it as such.
Don't forget to make a repo and publish v0.0.0. After that, if you update the project repository (p6-projen-project-awesome-list) the auto-upgrade flow will automatically update the down stream repo (in this case awesome-projen)

The generated .projenrc.ts for the AwesomeList looks like such:

Don't panic. All those options are in the super() hidden in the constructor of AwesomeList()

Step 13: v1.0.0+

Go into .projenrc.ts in p6-projen-project-awesome-list add:

majorVersion: 1

to the constructor. This will bump the next version to 1.0.0 using Semantic Versioning.

• https://github.com/projen/projen/issues/982 is the best explanation I've found of this.

Step 14: You are Done!

Rejoice.

A tale of my prior struggles in a gist

• https://gist.github.com/pgollucci/f0c58f0d92a2c59b8dcb6044d42fa67d#file-gist-txt

Where to go for help

• cdk.dev slack in #projen where you can also talk to me if you like.