Turning GitHub Into Your Own Registry

Occasionally I have ideas. They're not necessarily good ideas and occasionally I make the mistake of acting on them. What you're about to read is one of those times.

Look I'm not saying I turned @github into an alternate registry for @rustlang packages, but actually I did. Now if they add actual support to the package repos beta that would be neat. pic.twitter.com/GadqKxXSbJ

— Michael Gattozzi (@mgattozzi ) June 11, 2019

So how do you, dear reader, do this weird (sort of useless) thing? Follow me and you'll be in a world of pure configuration.

Index me like one of your git repos

Well the first step is you've got to actually set up an index. The way cargo alternative registries work is that it assumes you have an index to tell it what crates you have available as well as where to get the things. First up you'll want to set up a git repo on GitHub. Now I haven't tested this with private repos but I have a suspicion it won't work for authentication reasons. The token aspect of cargo login for a registry might work. It probably won't. For our purposes assume we're using a public repo for everything.

Create a file in the top level of your repo called config.json it should look something like this (replace mgattozzi with your github username):

{
    "dl": "https://github.com/mgattozzi/{crate}/releases/download/v{version}/{crate}-{version}.crate"
}

There is another field for the web api if you are running a crates.io instance but we aren't here so we can just drop it. What we're telling cargo is that all the crate files it needs will be in some repo we own, with the crate name being the repo name, and as part of tagged releases. This allows us to set a deterministic url for downloading, version our crates by release, and have it work for all of our own personal crates. Now the v in /v of the url doesn't need to be there. It's just the prefix I use to tag releases. As long as you're consistent and but the version as part of it when you tag your release you should be good to go.

Cool so we have a config file. What about a way to actually list crates? Well there's a file format and a way to store the file. Here are the rules for the file hierarchy in the docs:

The rest of the index repository contains one file for each package, where the filename is the name of the package in lowercase. Each version of the package has a separate line in the file. The files are organized in a tier of directories:

• Packages with 1 character names are placed in a directory named 1.
• Packages with 2 character names are placed in a directory named 2. -Packages with 3 character names are placed in the directory 3/{first-character} where {first-character} is the first character of the package name.
• All other packages are stored in directories named {first-two}/{second-two} where the top directory is the first two characters of the package name, and the next subdirectory is the third and fourth characters of the package name. For example, cargo would be stored in a file named ca/rg/cargo.

So in this example my crate's name is abrade, so my file would be under ab/ra/abrade.

Here's what it looks like:

{"name": "abrade","vers": "0.1.0","deps": [],"cksum": "65b07c5782e771125d9feda1ec08d908087997051339c492220346a14d6dd936","features": {},"yanked": false,"links": null}

Note each version for a crate has an entry and has to be on it's own line otherwise it just won't work. This means pretty multiline JSON is a no go. The cksum field is a SHA256 of the .crate file we'll need to generate. After doing that step (which we'll cover next) commit the file and the config and push it to GitHub. The docs for the registryare a bit light but go more indepth as to what an entry should look like. There's also the actual indexfor crates.io that you can look at for more examples.

Box

Okay so how do you generate a .crate file anyways? Luckily for us cargo has a subcommand for us! After you know what commit you'll use for the release is and you've pushed it to GitHub, just run cargo package. If you look under target/package you should see a file {crate}-{version}.crate. In my case it was abrade-0.1.0.crate. That's it. Next thing you have to do is tag a release on GitHub and make this .crate file as part of the release. Finally to use it in our own code.

Me, My Registry, and I

Okay so in whatever project you have you will want to specify it should use your alternative registry. In your Cargo.toml put something like this (depending on your crate):

[dependencies]
abrade = { version = "0.1.0", registry = "my-registry" }

Now cargo doesn't know what my-registry is, so we need to tell it. Open up this file as we need to modify it ~/.cargo/config (or the equivalent on Windows). Now this file might not exist if you haven't done this before. That's fine! You'll need to add something like this to it:

[registries]
my-registry = { index = "https://github.com/mgattozzi/mgattozzis-crates-on-github-index.git" }

This is the http url for your index. Replace my username/repo with yours for your index. Then that's it. cargo build and you should see that it pulls it down and compiles it!

You can find my repos for reference below:

• Index : https://github.com/mgattozzi/mgattozzis-crates-on-github-index
• Crate: https://github.com/mgattozzi/abrade

You can find more about the config file for cargo at this link.

Cargo Culting

Should you actually do this? I mean no probably not, but you can and that's the cool thing. There's some limitations (like only your crates) and it's a manual process since you can't just cargo publish but that's an automatable thing. Still if you want the option is there if you want to be a bit of a renegade, but you'd be better of just using crates.io or setting up a registry on your own server. I hope you enjoyed following along a bad idea. It's just always fun to see systems used in unconventional ways.