Using LibUV in Neovim

Off The Main Loop: Async Actions within Neovim

Neovim embeds the libuv library in the editor and exposes lua (and to some extent vimscript through the jobstart function) bindings for interacting with the library’s API. This allows actions in the editor to happen asynchronously, keeping the main editor loop clear for user input. What this means practically for users is that actions which take a long time such as grepping through large projects, generating ctags, or linting can now be done in the background without blocking the user’s ability to keep editing text.

Let’s explore how to use the libuv lua bindings in Neovim to create useful asynchronous actions. I will cover two examples: using the pandoc program to convert a markdown file to html, and using ripgrep to search within a project. Both of these examples use the libuv bindings differently, but still illustrate well the capabilities of libuv inside Neovim.

First Example: Convert Markdown Files with Pandoc

Pandoc is a powerful tool for converting files into and different formats. I write many Markdown documents and it is useful to convert them into HTML for my blog. In order to do this in an asynchronous way, we will use vim.loop from Neovim’s lua library. I would recommend going through the documentation on vim.loop as you code along with this post. Let’s start by creating a basic lua function outline in our packagepath:

-- in ~/.config/nvim/lua/markdown.lua
local M = {}
local loop = vim.loop
local api = vim.api

function M.convertFile()
  local shortname = vim.fn.expand('%:t:r')
  local fullname = api.nvim_buf_get_name(0)
 -- loop logic goes here
end

return M

In this snippet, we first get the shortened name of the file (i.e. vimlooppost in vimlooppost.md), and the full name of the file (i.e. ~/blog/posts/vimlooppost.md). We will use both of these variables later when we spawn the pandoc process. In order to get a better understanding of what vim.loop.spawn does, let’s head over to the libuv documentation. Here we can see that it takes an options table, an onexit callback, and returns the processId and the handle of the spawned process. Using this information, let’s build out the rest of the convertFile function:

-- in ~/.config/nvim/lua/markdown.lua
local M = {}
local loop = vim.loop
local api = vim.api

function M.convertFile()
  local shortname = vim.fn.expand('%:t:r')
  local fullname = api.nvim_buf_get_name(0)
  handle = vim.loop.spawn('pandoc', {
    args = {fullname, '--to=html5', '-o', string.format('%s.html', shortname), '-s', '--highlight-style', 'tango', '-c', '--css=pandoc.css'}
  },
  function()
    print('DOCUMENT CONVERSION COMPLETE')
    handle:close()
  end
  )
end

return M

Pandoc receives as arguments the strings listed in the args table, and when the process finishes, we echo a success message and close the handle. Using this in our vimrc is as simple as:

" in ftplug/markdown.vim

nnoremap <leader>c :lua require'markdown'.convertFile()<CR>

Second Example: Async Grep

Much like in the first example, we will be relying on an external program to do the heavy lifting, ripgrep. Unlike in the first example, we don’t only want to kick off a background process, but also use the values generated in this process inside of Neovim. For that, we will need to pass a set of file descriptors to vim.loop.spawn:

-- in ~/.config/nvim/lua/tools.lua
local M = {}
local loop = vim.loop
local api = vim.api

function M.asyncGrep(term)
  local stdout = loop.new_pipe(false) -- create file descriptor for stdout
  local stderr = loop.new_pipe(false) -- create file descriptor for stdout
  handle = loop.spawn('rg', {
    args = {term, '--vimgrep', '--smart-case'},
    stdio = {stdout,stderr}
  },
  function()
    stdout:read_stop()
    stderr:read_stop()
    stdout:close()
    stderr:close()
    handle:close()
  end
  )
  loop.read_start(stdout, onread) -- TODO implement onread handler
  loop.read_start(stderr, onread)
end
return M

Our function, asyncGrep will take a search term as an argument which it then passes to ripgrep in the loop.spawn call. After the process is spawned, we need to start reading the output into our file descriptors, which will then call the onread callback which we have yet to implement. Let’s implement the onread callback now:

-- in tools.lua
local results = {}
local function onread(err, data)
  if err then
    -- print('ERROR: ', err)
    -- TODO handle err
  end
  if data then
    table.insert(results, data)
  end
end

The onread callback takes data written to our file descriptor by the ripgrep process and appends a table called results. We now want to add functionality to our asyncGrep function that will allow us to use these results to set the quickfix list:

function M.asyncGrep(term)
  local stdout = loop.new_pipe(false) -- create file descriptor for stdout
  local stderr = loop.new_pipe(false) -- create file descriptor for stdout
  local function setQF()
    vim.fn.setqflist({}, 'r', {title = 'Search Results', lines = results})
    api.nvim_command('cwindow')
    local count = #results
    for i=0, count do results[i]=nil end -- clear the table for next search
  end
  handle = loop.spawn('rg', {
    args = {term, '--vimgrep', '--smart-case'},
    stdio = {stdout,stderr}
  },
  function()
    stdout:read_stop()
    stderr:read_stop()
    stdout:close()
    stderr:close()
    handle:close()
    setQF()
  end
  )
  loop.read_start(stdout, onread) -- TODO implement onread handler
  loop.read_start(stderr, onread)
end

If you run this function, you will encounter this error message: lua/tools.lua:122: E5560: vimL function must not be called in a lua loop callback. In order for any vim functions to be called within a lua loop callback, they need to be wrapped in vim.schedule_wrap. Wrapping vim functions in vim.schedule_wrap is necessary since it schedules the callbacks to be invoked when it is safe, bridging the gap between the libuv event loop and the internal Neovim main loop. To learn more about vim.schedule_wrap, check out :h schedule_wrap. Let’s fix this error and see what our function looks like when it’s all put together:

-- in ~/.config/nvim/lua/tools.lua
local M = {}
local loop = vim.loop
local api = vim.api
local results = {}

local function onread(err, data)
  if err then
    -- print('ERROR: ', err)
    -- TODO handle err
  end
  if data then
    table.insert(results, data)
  end
end

function M.asyncGrep(term)
  local stdout = vim.loop.new_pipe(false)
  local stderr = vim.loop.new_pipe(false)
  local function setQF()
    vim.fn.setqflist({}, 'r', {title = 'Search Results', lines = results})
    api.nvim_command('cwindow')
    local count = #results
    for i=0, count do results[i]=nil end -- clear the table for the next search
  end
  handle = vim.loop.spawn('rg', {
    args = {term, '--vimgrep', '--smart-case'},
    stdio = {stdout,stderr}
  },
  vim.schedule_wrap(function()
    stdout:read_stop()
    stderr:read_stop()
    stdout:close()
    stderr:close()
    handle:close()
    setQF()
  end
  )
  )
  vim.loop.read_start(stdout, onread)
  vim.loop.read_start(stderr, onread)
end

return M

Let’s use this newly created async function in our vimrc:

" in init.vim
command! -nargs=+ -complete=dir -bar Grep lua require'tools'.asyncGrep(<q-args>)

Now we can call :Grep searchTerm and get results without blocking the main editor loop!

What’s Next?

Having libuv bindings in Neovim unlocks a lot of potential for extending the functionality of your editor. The ability to asynchronously spawn other process can be used for linting, file watching, formatting, and much more. Free yourself from main loop blockage and try experimenting with vim.loop.