DEV Community

Cover image for Initializing a `git` repository with `git-lfs` (large file storage)
Mikołaj Buchwald
Mikołaj Buchwald

Posted on

Initializing a `git` repository with `git-lfs` (large file storage)

#git #programming #datascience #gitlfs

As it happens, you would like to initialize a git repository with git-lfs (large file storage) functionality to store some really large files. For someone a file of a 10MB size may seem large, for somebody else it would have to be as 1GB in size, but regardless of that, this tutorial explains how to efficiently initialize a repository with git-lfs.

Prerequisites

First, make sure that you have git lfs installed (how-to install for: macOS, Linux). In short, for macOS: brew install git-lfs.

Initializing git-lfs

Assuming that you have not initialized your repository with the "normal" git command yet, run the following commands:

git init
git lfs install
Enter fullscreen mode Exit fullscreen mode

For the sake of the example (to show you some difficulties you may encounter), let's assume that the files that you wish to add to git-lfs are stored in two separate directories (e.g., data and supplementary_materials). The tree of your repository looks somehow like that:

.
├── ...
├── README.md
├── data
│   ├── data1.png
│   ├── data2.zip
│   └── label.txt
├── images
│   └── some_pict.png
└── supplementary_materials
    ├── first_subdir
    │   └── somefile.png 
    ├── second_subdir
    │   └── somefile.png 
    └── third_subdir
        └── somefile.zip
Enter fullscreen mode Exit fullscreen mode

First, some basic stuff about synchronizing with your remote repository:

git remote add origin https://github.com/mikbuch/your-repository.git
Enter fullscreen mode Exit fullscreen mode

Then:

git branch -M main
git pull origin main
Enter fullscreen mode Exit fullscreen mode

Now, there are basically two ways of adding files to git-LFS.

The first (recommended) method

Create .gitattributes file:

vim .gitattributes
Enter fullscreen mode Exit fullscreen mode

and add the following lines:

data/**/* filter=lfs diff=lfs merge=lfs -text
supplementary_materials/**/* filter=lfs diff=lfs merge=lfs -text
Enter fullscreen mode Exit fullscreen mode

Note! In our example we have two separate directories that you wish to add to git-lfs in which you have some subdirectories. (1) The syntax has to be exactly like the one in the code above, or else you won't add all files in all subdirectories. (2) If you have several directories, from the point of view of your current working directory, you have to list them all, one by one.

The second (not recommended) method

Use the following commands to add directory and all files in this directory and its subdirectories:

git lfs track data/**/*
git lfs track supplementary_materials/**/*
Enter fullscreen mode Exit fullscreen mode

Note! The command git lfs track supplementary_materials/** will not be sufficient to add all files in all subdirectories.

Why this method is not recommended?: It will list/add all tracked files to your .gitattributes (one by one!), e.g.:

➜  git:(main) ✗ cat .gitattributes

data/data1.png filter=lfs diff=lfs merge=lfs -text
data/data2.zip filter=lfs diff=lfs merge=lfs -text
data/label.txt filter=lfs diff=lfs merge=lfs -text
...
supplementary_materials/first_subdir/somefile.png filter=lfs diff=lfs merge=lfs -text
supplementary_materials/second_subdir/somefile.png filter=lfs diff=lfs merge=lfs -text
supplementary_materials/third_subdir/somefile.zip filter=lfs diff=lfs merge=lfs -text
...
Enter fullscreen mode Exit fullscreen mode

What you would like to have in your .gitattributes is just:

data/**/* filter=lfs diff=lfs merge=lfs -text
supplementary_materials/**/* filter=lfs diff=lfs merge=lfs -text
Enter fullscreen mode Exit fullscreen mode

It is much more clearer approach. See the first (recommended) method described above.

Preview the .gitattributes file 

cat .gitattributes
Enter fullscreen mode Exit fullscreen mode

Add all files to your repository: 

git add --all
Enter fullscreen mode Exit fullscreen mode

Show which files are tracked with (git lfs ls-files) 

git lfs ls-files
Enter fullscreen mode Exit fullscreen mode

Note! git lfs ls-files works only after you have added the files to the repository with git add.

Show which files were added to the repository (regardless of whether LFS or not):

git status
Enter fullscreen mode Exit fullscreen mode

Make sure that the files are tracked with git-LFS:

du -sh .git
du -sh .git/lfs
Enter fullscreen mode Exit fullscreen mode

Note on GitHub

If you are using GitHub, please make sure that you are using the proper SSH file:

cat ~/.ssh/config
Enter fullscreen mode Exit fullscreen mode

In GitHub, also make sure that your key is added to you profile (not just one repository!). See the output of a command:

ssh -T -ai ~/.ssh/my-key.pem git@github.com
Enter fullscreen mode Exit fullscreen mode

Source: https://superuser.com/a/232406/950943

If you used this key for one repository only you will see somethink like this:

Hi mikbuch/your-repository! You've successfully authenticated, but GitHub does not provide shell access.
Enter fullscreen mode Exit fullscreen mode

You should see output like this:

Hi mikbuch! You've successfully authenticated, but GitHub does not provide shell access.
Enter fullscreen mode Exit fullscreen mode

Committing and pushing to the server

Commit the files (both LFS and "normal"):

git commit -a -m "Initializing commit with git LFS"
Enter fullscreen mode Exit fullscreen mode

Finally, you can push your reporitory to the remote server:

git push -u origin main
Enter fullscreen mode Exit fullscreen mode

In order to reset (if needed/optional) during the process:

rm -rf .git
rm .gitattributes
Enter fullscreen mode Exit fullscreen mode

And that's it, I hope you will find this tutorial useful, have a nice day!

Additional resources

Source for the cover photo image: Photo by Wesley Tingey on Unsplash

Top comments (0)

Read next

mikeyoung44 profile image

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters

Mike Young -

mikeyoung44 profile image

New Research Breaks Through AI Language Model Safeguards, Exposing Security Risks

Mike Young -

mikeyoung44 profile image

xLSTM: Fast and Efficient Large Recurrent Action Model for Robotics

Mike Young -

mikeyoung44 profile image

New AI Revolution: Designing a Global Multi-Agent Network with Large Language Models

Mike Young -