Using Vim As Your Shell Command-Line Scratch

Using (n)vim as your command-line scratch helps you to efficiently fix, create, and run ad-hoc commands.

If you are not familiar with vim, I suggest you start playing with the vimtutor first. I’ll try my best to explain each keys that I have pressed. In this blog, I’m using Neovim, but you can still follow along even if you’re using Vim. I have aliased my nvim to vim with alias vim=nvim. Please follow along with the demonstrations, so you can get used to this workflow.

Fixing Ad-Hoc Commands

Have you experienced running a command and realizing that you made a mistake after executing it? You missed some syntax or forgot adding a sudo, and much more?

Commands that requires privilege

You are copying a file to another directory, or installing a package, but you forgot that it requires a sudo privilege.

I - insert mode before the first character
Escape or Ctrl + C - to go to the normal mode
ZZ or :wq- to save and exit

Note that in vim, keys are case-sensitive, this means that the lowercase i is different to the capital I.

As you have noticed, we were able to edit our previously executed command with fix command (fc), it opened the vim editor for us with the command as our content. If we exit from the editor, it’ll run the command that we have edited.

For this case, it is better to use sudo-bang-bang (sudo !!) if you need to add sudo before the previous command.

!! - will use your previously executed command

If we view the man page of fc. It checks for the FCEDIT and EDITOR environment variables. If both of them are unassigned, it’ll use vi.

-e ename
...
If ename is not given, the value of the FCEDIT variable is used,
and the value of EDITOR if FCEDIT is not set.
If neither variable is set, vi is used.
When editing is complete, the edited commands are echoed and executed.

I have exported my EDITOR variable in my config file to use neovim.

export EDITOR=nvim

You can assign nano, or code here to use VS Code as your fc editor, but I do not recommend it. You’ll see why it’s better to use vim later on.

Commands with typo

Another use case is if you made a typo in the middle of the command, or missed any quotation marks.

Arrow Keys Spam

When we try to fix our command, we will spam the thing out of the left arrow key to navigate to the middle of the command to fix it. This workflow is so slow, and even annoying to deal with.

fc to the rescue!

Invoke the fc command to fix the previous command that you have executed. This will open a vim editor which we can easily perform search and replace commands. This would be more useful if we have really long commands and arguments.

:s/ooo/o - perform a substitute ooo to o

:s/hir/here - perform a substitute hir to here

For more information about the vim substitute command, please check the help file with :h :s.

Commands with shell expansion

Let’s say you want to say hello to the current USER.

Hmmmmm 🤔, our USER is not printing, but it has value as we have checked it. We have found out that we were using single quotes ('), and this means that shell expansion never happened, what we need is to surround them with double quotes (").

We can still use :s/'/" here, but we can also leverage the power of vim with plugins!

I have navigated to the text to replace the surrounding quotes, then used tpope/vim-surround plugin.

cs'" - it means change surrounding single quote (') to double quote (").
                c      s                         '                   "

If you want to delete a quote, you can use ds', or ds" for double quotes.

Commands that you want to edit without executing

We have learned that we can use fc to edit or fix our previous command. What if you noticed a typo, and you do not want to execute the command? In that case, what we can do is press ctrl + a to move the cursor before the first character in your command line and type pound (#), this results into making the command as a comment.

If the command with pound (#) in front is executed, it won’t execute the command itself, but instead it’ll result into being a shell comment. The good thing with this is you can use fc to open it in your editor and edit it. You can do this because the commented-out command was already added to your command history, and fc will edit the previously executed command. Just remove the # on your edit.

Commands with long parameters

If we want to run a postgres container with a command from their documentation, we will run the following command:

docker run -d --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -e PGDATA=/var/lib/postgresql/data/pgdata -v /custom/mount:/var/lib/postgresql/data postgres

After we have pasted this in our terminal, it’s hard for us to update the parameter variables. We will use so many arrow key presses just to navigate to the environment variable values and edit it. We can comment this for now and run the fc command. By doing this, we can leverage the vim editor to manipulate the text.

x - delete the # symbol

:s/some-/my- - substitute the container name

/secretciwmyverysecret
  - '/secret' search and navigate to the first 'secret' word match
  - 'ciw' delete the inner word, and go into insert mode
  - 'myverysecret' literal word to replace the password value

Add new lines before the -e and -v

:s/ -[ev]/\r\0/g
  - ':s/' substitute command
  - ' -[ev]' search for text that has ' -e' or ' -v'
  - '/\r' add a carriage return
  - '\0' substitute with the first captured group
  - '/g' substitute globally, not just the first match

With this substitute, we were able to format our command, but there is a problem, in shell with multiple lines, we need to suffix every line with a backslash (\). We can achieve that with performing a normal command.

:%norm A \
  - ':%norm' perform a normal command for the whole buffer
  - 'A' go to insert mode to the end of the line
  - ' \' add literal space and backslash

The normal command helps you to perform it on multiple lines as if you’re actually typing it directly on the buffer, to learn more about it, check the help file for normal using :h norm (norm command is just a shorter version of normal command).

We were now able to update the values of a command with multiple long parameters, and also properly formatted it. If you noticed that my line numbers are not linearly increasing, I’m using a relative numbers, you can learn more about it with :h relativenumber. I highly recommend setting set rnu config together with set nu so you can easily jump to arbitrary lines relative to your current line.

Commands that you are currently fixing and want to abort it

If you have edited your previous command and you do not want to execute it. If you use ZQ or :wq! to exit vim without saving, it’ll still execute your command.

What happened? It turns out that when the editor exits with 0, it’ll execute your command, and using :wq! gracefully exits vim.

echo $?
0

How to solve this? You need to use :cq to exit vim with error code. You can add a key bind for this, or you can just delete the content of your vim buffer with dd and exit.

Vim Buffers and Shell

Fixing your previous command is cool, but what if you want to perform a command operation to the output of your previous command, and update its content? You can use Vim to achieve that.

If we perform an operation and pipe it to our Vim buffer, it’ll use the stdin as your buffer content.

echo hello | nvim -
# look ma, no dash
echo hello | nvim

There are times that you need to use dash (-) to pipe the standard output to standard input of the next command. In nvim, the dash is not required.

Sort and remove duplicate names

We have a file with a list of names, but there are duplicates. We want to remove the duplicates, and sort them.

cat names.txt
John Doe
John Doe
David William
Joseph Thomas
David William
Joseph Thomas
Joseph Thomas

We can pipe the output of the cat names.txt, or just directly open it with vim.

vim names.txt

Then we can now perform a command operation to update the content.

:%!sort -u
David William
John Doe
Joseph Thomas

Anything after the ! character is the shell command.

Of course, we do not need to open it in vim to remove the duplicates, it is to simply demonstrate that you can perform commands on the current buffer and replace it with the output of the shell command.

Sort and remove duplicate names of some part of the buffer

We can also specify some area that we want to perform the operation using the visual mode of vim.

cat names2.txt
John Doe
John Doe
David William
Joseph Thomas
David William
Joseph Thomas
Joseph Thomas
# please do not modify the following lines
Christopher Daniel
Mark Anthony
Christopher Daniel
Paul Steven
Mark Anthony
Paul Steven

First, we need to select the range with the visual line (V), and run the :!sort -u command.

Note that the '<,'> characters are automatically added by vim, this represents the selected line range.

Formatting and filtering JSON content

cat characters.json
[ {
"id": 1,
          "name": "Tomoko Kuroki",
    "age": 15,
"height": "147 cm"
  }, {
    "id": 2,
          "name": "Marin Kitagawa",
    "age": 15,
"height": "164 cm" },
  {
          "id": 3,
"name": "Kaguya Shinomiya", "age": 17,
                "height": null
  } ]

This is a JSON file with a content that is not properly formatted. We also want to remove some noise, we only want the id, and the name properties of the characters. We can do this by using the jq command. We can also use any vim plugins like prettier for different file types.

We have formatted it with :%!jq .

:%!jq .
[
  {
    "id": 1,
    "name": "Tomoko Kuroki",
    "age": 15,
    "height": "147 cm"
  },
  {
    "id": 2,
    "name": "Marin Kitagawa",
    "age": 15,
    "height": "164 cm"
  },
  {
    "id": 3,
    "name": "Kaguya Shinomiya",
    "age": 17,
    "height": null
  }
]

Filtering the properties that we only want with :%!jq '[.[] | {id, name}]'. This is really helpful to analyze some JSON data, and want to play with the properties.

:%!jq '[.[] | {id, name}]'
[
  {
    "id": 1,
    "name": "Tomoko Kuroki"
  },
  {
    "id": 2,
    "name": "Marin Kitagawa"
  },
  {
    "id": 3,
    "name": "Kaguya Shinomiya"
  }
]

Pressing u to undo the changes to the buffer. You can learn more about jq from their manual.

We were able to easily parse the JSON without leaving our editor!

Searching file contents

You can also pipe the output of your grep to vim, so you can easily navigate to the matched files. If the cursor is under a file path, and pressing gf will open another buffer with the file and the cursor is on the line number that was created in our grep result.

grep -Hnri 'search keyword here' . | nvim
H - print file name
n - add line number
r - recursive search for each directory
i - ignore case

Now that we have the matched searches in our buffer with their associated file names and line numbers, we can use gf here to navigate to the file under the cursor. This can help you preview and analyze the files, you can get back to the previous buffer with ctrl + 6. See :h buffers for more information.

Writing to Shell

Can we do it the other way around? Using the content of your buffer to execute them in your shell?

One common misconception when you run vim <file>, is that your’e editing the actual file, but in fact, you are actually opening a buffer of that file, and not editing the file itself.

This is different to the other editors where they will immediately create the file for you.

When you save it with :w, you are just overwriting the file with your current buffer, and if the file does not exist, it’ll create it. This is equivalent to :w % or :w yourfile.txt

With this in mind, what if you write the buffer to your shell? Well...you guessed it right (pun not intended 😉), if you write the buffer to your shell, you can execute commands from your Vim buffer!

I am using zsh here, but any shell should work.

If we write the buffer to a shell, it’ll invoke the command to a sub-shell.

:w !zsh
:w !$SHELL

cat file-that-contains-shell-script.txt

echo 'i have echo'
echo $(pwd)
touch newfile.txt

We can also pick which line to execute.

:.w !zsh

Notice the dot(.) before w, this means “write the current line of the cursor to the shell”.

cat sample-line.txt
echo hello1
echo hello2
echo hello3
echo hello4
echo hello5

Shell Level

If we print out the SHLVL environment variable, we are currently at level 1. Invoking this command inside vim will result into the level 2. This means that it is executing it to the sub-shell.

Bindings Configuration

Typing :w !zsh or :w !$SHELL again and again is too much, I recommend adding your own bindings similar to what I have

" write to shell
let mapleader=" "
nnoremap <silent> <leader>xs :.w !$SHELL<CR>
xnoremap <silent> <leader>xs :.w !$SHELL<CR>
nnoremap <silent> <leader>xS :w !$SHELL<CR>

You can put this into your .vimrc file or the init.vim for neovim. Then you can just press the leader key which is set to space and xs.

File Analysis

From our previously found files in the “Searching file contents” section, we decided that we want to delete those files. We need to filter out the file paths first, and there are multiple ways to achieve this. We can use cut here, but for now, let’s use awk with a colon separator.

:%!awk -F: '{print $1}'
./subfolder0/somefile.txt
./subfolder0/anotherfile.txt
./subfolder1/alpha.txt
./subfolder3/beta.txt
./subfolder4/magic.txt
./sample-line.txt
./sample-line.txt
./sample-line.txt
./sample-line.txt
./sample-line.txt

There are duplicate files, let’s perform a unique sort and prepare to delete with rm.

:%!sort -u
./sample-line.txt
./subfolder0/anotherfile.txt
./subfolder0/somefile.txt
./subfolder1/alpha.txt
./subfolder3/beta.txt
./subfolder4/magic.txt

Now that we have added the rm command on each line, and writing them to your shell will remove the files.

Shell Intellisense

We can improve our workflow by adding intellisense using the Language Server Protocol (LSP). LSP is a protocol that can help provide some auto completion, quick documentation, go to definition, and many other features. You can use any LSP provider, at the time of this writing, I’m using coc.nvim for my LSP.

Setting up your own LSP and their configuration is out of scope for this blog, you can check their documentation for it.

In order for our LSP to work, there are multiple ways to enable it, but the easiest one I can think of is to set the file type of the buffer that we are currently working on. You can check the current file type with set filetype?, and assign a file type using set filetype=sh. Now that we have an sh filetype, our LSP now knows that we are editing a shell buffer, which will provide us the language features of a shell.

Command Auto-Completion and Documentation

Path Auto-Completion

This is just the tip of LSP, and you can check the language server of what features it can offer for you to help in your shell scratch workflow.

Other Use Cases

In this section, I will share some of my workflow that I use day to day.

Workflows in Git

Long Branch Names

Whenever we create new features, or fixing a bug. It’s better to create new branch from the main or any base branch. We usually create branches using the ticket name feat/ZERO-1234, but browsing these branches does not provide any context as to what is the purpose of that branch, unless you memorized the ticket number and their description.

We can add more details to the branch name, but the problem with this branch names is that the checkouts, rebase, and renames is a bit of pain to type, unless you have auto complete. You can pipe the branch output to vim, and perform the checkouts.

feat/ZERO-123-create-some-cool-feature-here-that-no-one-usees
fix/ZERO-456-the-bug-that-will-consume-your-days

The aliases that you have configured in your Git will work here like git config --global alias.co checkout.

Deleting multiple branches

Deleting multiple branches with grep is a bit of a hassle, unless you run the grep first to see the list of the branches. What we can do is pipe the output to vim, and perform any searches.

By doing this, you have a faster feedback of what branches would be deleted, you can even easily remove any branch that was part of the initial grep.

Reading Command Outputs

APIs mostly use JSON as their payload. We can easily create them using jo. We can read the command output and put it to your current buffer. For example, we want to create a JSON object with a lower case uuid value for its id property, and a simple name.

:read !jo id=$(uuidgen | tr A-Z a-z) name='zaerald'

Anything after the ! will run a shell command.

Conclusion

You can now efficiently edit previously executed commands using fc. Using vim with plugins and LSP can help improve your workflow. Sky is the limit! Using vim as your command-line scratch is much fun to use. Just play with it, and don’t forget to have fun!

You can check my dotfiles in github.com/zaerald/dotfiles. Please note that my dotfiles continues to evolve, and there’s a higher chance that it is different now compared at the time of this writing.

This blog post is originally posted here.