It's a common scenario we lost our local projects because of problems in our machine. Our Laptop/Desktop can break down and we lost a lot of data. It's a painful situation, recovery not only all data, photos, software's, etc. But, recovery all our repositories too.

To our happiness, all repositories are storage/save in cloud through GitHub if we use Github as hosting service.

However, if we work simultaneously on various projects could be a little boring clone all repositories one-by-one.

Automate tasks is a great way to make a better use of our time. And shell script is a good resource to achieve this type of automation.

In this article we will create a shell script that automatically clone all repositories of your GitHub account.

Shell Script

Shell script is a script language used in command line environment. It allow us automate terminal commands in Unix based systems like Linux and MacOS.

Let's start

This article assumes that you have a basic knowledge about terminal commands, so is not the focus here explain what instructions like touch, mkdir, etc do.

First of all, we need to create a new script file.

 clone-github-repositories.sh

Note that the ".sh" is the extension of shell script files, without it the interpreter cannot understand that this file is a shell script.

After, we need open this file in a text editor of your preference (vi, nano, etc).

 clone-github-repositories.sh

In the editor we need write in the first line something very important to our script: indicate the interpreter of the script. In my case, I will use bash, but, exists others like sh, ksh, etc.

So, the first line of our script will be the following:

 #!/bin/bash 

We will create a directory where we'll clone all our repositories. I will call this directory "projects", but you can name it like you want.

So the next line of our script will be the this command:

 mkdir projects 

Like I said, we will clone our repositories inside projects directory, so we need enter to the projects directory, the following line of script will be:

 cd projects 

Okay, so until now we have some simple instructions in our script, we need have access to all repositories from our GitHub account, we could go to our GitHub profile and copy the URL repositories one-by-one, but, this is not an agile way, right? Here, enter in scene the GitHub REST API.

The GitHub REST API allow us make integrations, automation and much more.

Our next line in script will be a request to GitHub API using curl:

 -s https://api.github.com/users/{your-github-username}/repos 

Replace {your-github-username} by your GitHub username.

If you run this command solo in terminal the return will be a JSON content with a array of objects containing all properties of your repositories. Below an example of the JSON that return from the above command.

Between the many properties we have two especially important: ssh_url and clone_url.

This properties contains the URL's that we can use to clone the repositories. If you use ssh feature to clone your projects, you can use the "ssh_url", else, the "clone_url" will work the same way. In my case, I use SSH to clone my repositories.

Now, we have to work with some resources to manipulate content of files, because we don't need use all information that return from GitHub API.

Here, "grep" is our best friend. grep will search inside a file by a string/pattern that we inform.

We will refactor our script line to the following:

 curl -s https://api.github.com/users/{your-github-username}/repos | grep ssh_url 

Remember that if you don't use SSH you must change the "ssh_url" by "clone_url".

The output of this instruction will be something like this (if you run this directly in terminal):

To clone a project we need only the URL, so we need filter this grep return.

We will use the command "cut" with a delimiter to remove the unnecessary sections from the lines.

Refactoring our instruction, now the command will be this:

 curl -s https://api.github.com/users/{your-github-username}/repos | grep ssh_url | cut -d'"' -f 4 

-d is a flag for field delimiter and we use the double quotes (") as delimiter.

-f is a flag for fields that select only the field (or range of fields) informed. And 4 is the specified index of field line that we want extract.

If you run this instruction in terminal the output is like this:

Now, we have the exact URL to clone our repositories.

So, we will refactor the line to save the output list in a variable.

 repos=$(curl -s https://api.github.com/users/{your-github-username}/repos | grep ssh_url | cut -d'"' -f 4) 

repos is the name of variable and the "$" symbol indicate to the shell that this is a variable, we have a list of URL's in this variable.

We will need apply some programming logic to do the task, iterable the repos list and for each repository of the list we will clone the repository.

 for repo in $repos
do
  git clone $repo
done 

The full script is below:

#!/bin/bash

mkdir projects
cd projects

repos=$(curl -s https://api.github.com/users/gniches/repos | grep ssh_url | cut -d'"' -f 4)

for repo in $repos 
do
    git clone $repo
done

It's done. Now we need run the script to test if our script is working as expected.

 bash clone-github-repositories.sh 

If everything is ok, now you have a directory called projects with all your repositories inside.

Concluding

In this article we learned how to create a shell script to clone all our repositories from GitHub account using GitHub API.

Shell script is particularly useful to automate process and tasks, like monitoring servers, create periodic backups of databases, send e-mails and much more.

Reviews, suggestions and/or contributions to this article are encouraged and welcome.

References

GitHub REST API Documentation