My Omnipytent plugin for Vim is a central part of my workflow, but it seems to be a bit hard for other people to grok it. More specifically - to understand why would they need something like that. So - I'm making this post to explain the rationale behind Omnipytent and to demonstrate it's worth.

The problem: running commands

TL;DR - this section explains why I needed to create a plugin for something most developers just... do. If you don't care about justifications, just skip it and go directly to the next session - "Using Omnipytent".

This may look like a solved problem - you have ran commands before you ever heart of Omnipytent. Heck - you probably ran commands before you even heard of Vim! Why do you need a plugin for that? That's what the command line is for!

As Vimmers we tend to adhere the "Unix is my IDE" philosophy - every development task has a command line tool, and we just run it with the arguments we need. So - my project can be built and ran with simple commands. And I know these commands. But...

• Do I really have to type the same command each and every time I want to run it?
• And if I need to build with different arguments - do I add them every time I type the build command? Or do I change the actual build file to make them the default?
• I want to run a specific test - do I need to type it's name every time? Or, I can paste it - but then I need to keep it in the clipboard, or copy it each time...
• I need to run the tool with certain arguments(yes, this should be a test, I'll make this a test, I promise!) - do I type these arguments every time?

You get the idea - I'm lazy and I don't want to keep typing the same command line commands over and over again. What can I do?

So... map some keys?

An obvious choice - if there is a command you use a lot and want quick access to, just set a keymap. A simple solution - but for me, at least, it was not flexible enough:

• I work on several projects in different languages and environments, and I need different commands to build each project. The usual Vim solutions is to use :nnoremap <buffer>s :autocmds or in filetype plugins - but what keys will I set, for example, for XML files? I may want to build while in one of those!

• Even in the same environment, I need different ways to run(and sometimes build) different projects. Each project has it's own entry point, and unless you only work on single-file scripts that entry point will not be the file in your current buffer. Having different config for each project does not scale.

• Even in the same project, I often want to change these commands. Build with a different flag, run with different arguments. Changing the keymap each time is too much trouble.

No... I want something I can easily change - without touching my .vimrc!

Just use the command line history like everyone else

A straightforward solution - if you use a command a lot, it's going to be in the history, so just Ctrl+r in Bash to find it.

That's good and all when you work a single project, but with many projects - they are just going to override each other's history. So... I'd still like something better. Also, I may be risking getting my Vim card revoked - but I really don't want to context-switch to a terminal emulator every time I want to build or run what I'm working on. And Vim's own history as not as easy to navigate - not to mention I need it for command-mode commands...

You spoiled brat! Put these commands in .sh files and get it over with!

That's another common solution(though apparently not as common as the first two) - creating simple scripts for the commands. From the shell, or from Vim's :!, it's easy to tab-complete and launch these scripts. And I can even set keymaps to the different scripts, and have the same keymap do project-specific stuff in different projects!

But... all these files scattered all over the project create a huge mess. So we need to:

Have a single file containing all these commands

Now we are getting somewhere! It can be a simple bash script with case on the first argument, and you could just set it to have as many commands as you want. Or - you could go fancy and abuse a build system, which usually have nicer syntax, and use their tasks as commands.

So that's what I did - I chose Rake, because it wasn't colliding with existing build systems(it's mainly used with RoR, and I didn't really need to have build and run commands there(at least not at first)), and I get to write my tasks in Ruby instead of Bash. Yey!

But I still wanted more. What if, I thought, I had a plugin to ease the creation of new tasks, autocomplete task names for me, help me easily use a different file than the one rake targets without additional args, etc. etc.

And then I realized - Rake is a Ruby library! If I can load it into Vim with the Ruby interface, I can run my tasks inside Vim, and they'll have access to my Vim instance! This opens a new world of possibilities - I can make a task run with regular :!, or with Erroneous to fill the quickfix list, or in a VimShell terminal(Vim did not have :terminal back then, and Neovim was not even conceived), or load a log file in a buffer, or... or anything I wanted!

And that's how Integrake was born. And when I had to abandon Ruby so I can move to Neovim - I rewrote the whole thing in Pyhton - and that's Omnipytent.

Using Omnipytent

Tohttps://thepracticaldev.s3.amazonaws.com/i/229bjqis5lejr6ma3rhc.gif demonstrate Omnipytent I will use this example Java Spring project.

Simple tasks and commands

So, after cloning the repository, let's say I want to run tests. This is a Maven project, so we run tests with mvn test. I'm going to do it with Omnipytent:

So... what happened here?

First thing first - I have set these in my .vimrc:

let g:omnipytent_filePrefix = '.idanarye'
let g:omnipytent_defaultPythonVersion = 3

This means I want my tasks file to begin with my name and be hidden(.idanarye) and I want to use Python 3(you can write your tasks files in Python 2 or Python 3). The tasks file is supposed to be personal, so we won't have to invest effort in making it portable to allow other developers to use it. We don't check it in to source control, and even if we do - other developers that happen to use Omnipytent will have their own tasks file with a different name.

So, with Vim opened in the repository's root, I run :OPedit test and it opens the task file for that project - .idanarye.omnipytent.3.py. Because the file did not exist before it added some imports, and because I wanted to edit a non-existing task it created a skeleton for that task - and all that's left is to write the task's body:

import vim
from omnipytent import *


@task
def test(ctx):
     BANG << 'mvn test'   

So, what do we have here? Imports are imports - we have vim - the built-in interface for Vim from Pyhton - and a start import from omnipytent with the common things you are going to want to use in a tasks file. One of the is task - a decorator we use to - surprise surprise - create tasks. The other is BANG - a Shell Command Executor. Shell command executors are handles for running shell commands - BANG specifically is using Vim's bang command(:!). There are other shell command executors, and you can define your own - it's all in the docs. The << operator in shell command executors can be used to execute a string as a shell command. And once I save the tasks file and run :OP test - Omnipytent executes the code of the test task and runs all the tests.

The << operator executes the string as is. You can also use it as a function - and it'll quote each argument(which is better if you get them from a variable):

@task
def test(ctx):
     BANG('mvn', 'test')

The third way to use shell command executors is with Plumbum - a shell combinator library for composing shell commands with Pythonic syntax. If you have it installed, and import omnipytent.interation.plumbum in your tasks file, you can use the shell command executors like Plumbum's background and foreground modifiers BG and FG. Since you usually want Plumbum's local to start the commands, you can import that from omnipytent.interation.plumbum:

from omnipytent.integration.plumbum import local
local['mvn']['test'] & BANG

Since we are going to be using Maven quite a lot, might as well bind local['mvn'] globally:

import vim
from omnipytent import *
from omnipytent.integration.plumbum import local

mvn = local['mvn']


@task
def test(ctx):
    mvn['test'] & BANG

OK - but :! is not a very convenient way to run a test - certainly not with a tool that spans you with text like Maven. How about we run it with a terminal emulator instead? It's simple - all we have to do is use a different shell command executor - TERMINAL_PANEL - and it'll create a Vim 8 or Neovim's terminal emulator:

OK... but why?

So far, we didn't really need Omnipytent - couldn't we just run these commands, right from Vim's command mode? Well, yes, but running them with Omnipytent has two advantages:

1. This is a Maven project, so the command is :!mvn test. If it was a Gradle project, I'd need :!gradle test. And with Ant I'd need :!ant test. Or maybe :!ant junit? Ant is free-spirited like that, so it can be anything.

   And what about flags you sometimes need to set? And that's just Java - other languages have their own various build systems...

   With Omnipytent, it's always :OP test - because you don't depend on what build system the project's creator picked and how they chose to configure it - you always create your own Omnipytent task test to run it. You can even map a key to it, and it'll work with any project(after your created the task for it). Personally, I mapped many short generic verbs to an "Omnipytent leader" followed by their first letter:

   

   They don't all do something on all my projects, but as projects get big I tend to have many different useful tasks and it's nice to have keymaps available for them.

2. Sometimes commands need arguments - like the test's name when you want to run a specific test. You can't bind that in your global .vimrc because it's constantly changing - but it's easy to edit an Omnipytent file:
   

import vim
from omnipytent import *
from omnipytent.integration.plumbum import local

mvn = local['mvn']


@task
def test(ctx):
    mvn['test']['-Dtest=ClinicServiceTests#shouldFindOwnersByLastName'] & TERMINAL_PANEL

So - we can use :OP test(or the key(s) we mapped to it) to run this test, and when we want to work on a different test - we can just edit the tasks file.

OK - but what if we want something more dynamic? Maybe we don't want to edit the tasks file each time, and prefer to give the test to the command? We can do that too - with task arguments:

Task arguments and completion

@task
def test_specific(ctx, testname):
    mvn['test']['-Dtest=' + testname] & TERMINAL_PANEL

And now, we just need to give that argument to our task with:

:OP test_specific ClinicServiceTests#shouldFindOwnersByLastName

Aaaaannnd... we are back to square one - because if we are going to type the test's name anyways we could have just used:

:terminal mvn test -Dtest=ClinicServiceTests#shouldFindOwnersByLastName

So, why use Omnipytent? As you may have guessed from the subsection's title - completion! Omnipytent already gives you command mode completion for task names, and if you want to you can easily define completions for the task arguments.

If I were to create a generic completion plugin for Java tests, I would need to make it super-robust to account for the different styles and conventions. Maybe even run Maven/Ant/Gradle with some injected target that emits them. But here I just need them for one specific project - so I don't need to put all that effort, and can just depend on the characteristics of the tests:

• They are all inside src/test/java.
• They are all void methods.
• They all have @Test in the line above them.

So - all I have to do is grep for the @Test lines, get the lines after them, and extract the filename and the method name from those lines. I can quickly write something like this:

import re
pattern = re.compile(r'(\w+)\.java-.*void (\w+)')
for line in local['rg']['-e', '@Test']['--after-context', 1]['src/test/java']().splitlines():
    m = pattern.search(line)
    if m:
        class_name, test_name = m.groups()
        yield '%s#%s' % (class_name, test_name)

This is definitely not plugin-grade - but for a personal search snippet just for me and just for this project it's acceptable. And now all that's left is to make it the completion function for test_specific:

@task
def test_specific(ctx, testname):
    mvn['test']['-Dtest=' + testname] & TERMINAL_PANEL


@test_specific.complete
def test_specific__completion(ctx):
    import re
    pattern = re.compile(r'(\w+)\.java-.*void (\w+)')
    for line in local['rg']['-e', '@Test']['--after-context', 1]['src/test/java']().splitlines():
        m = pattern.search(line)
        if m:
            class_name, test_name = m.groups()
            yield '%s#%s' % (class_name, test_name)

Interacting with Vim

Running tests is nice - but we also want to build the project, don't we? "use :make" - a thousand Vim users would scream at once(if... they were reading this at the same time). OK, let's use :make:

Oh, right - my &makeprg is set to Gradle for Java files, and this is a Maven project. Well - I don't want to change my .vimrc to use Maven - so let's use an Omnipytent task!

I run :OPedit compile to scaffold the compile task, and write this:

@task
def compile(ctx):
    with OPT.changed(makeprg='mvn', errorformat=r'[ERROR] %f:[%l\,%v] %m'):
        CMD.make.bang('compile')

What's going on here?

• OPT - the helper for setting Vim options. We could use OPT.makeprg or OPT['makeprg'] to get and set the&makeprg` option.
• OPT.changed(...) a context manager for temporarily changing the values of some Vim option. In this case - &makeprg and &errorformat.
• CMD - the helper for running Vim commands.
• CMD.make - Vim's :make command - can be used like a function.
• CMD.make.bang - this is :make!(because I don't like to get jumped to the first error)

All together - when we run the compile task, it'll temporarily set &makeprg and &errorformat, run :make! compile, and then set &makeprg and &errorformat back. This will result with executing mvn compile and running it's output through the proper error format to populate the quickfix list:

Of course - instead of CMD.make you can use :make alternatives - e.g. you can install dispatch.vim and use CMD.Make. Or you can use CMD to do other things, unrelated to building the project...

Extensions

Writing the error format in each tasks file makes little sense. Chances are I'll use the same error format in many different projects. Same thing may be true for other things we define in our tasks files.

To allow easy reuse of such things, Omnipytent supports an extension mechanism. A plugin can put a Python source file under it's omnipytent/ directory, and it'll become a child module of omnipytent.ext. For example, my MakeCFG plugin exposes such interface - a makecfg function for setting &makeprg and &errorformat for entries in it's database.

So - if I have MakeCFG installed, I can write my compile task like this:

`

@task
def compile(ctx):
    from omnipytent.ext.makecfg import makecfg
    with makecfg('mvn'):
        CMD.make.bang('compile')

Combining tasks together

During development we often want to interact with the application we are working on. The one we chose is a web application, so we will want to run it, send commands to it, and stop it. Omnipytent can automate that as well!

Let's start with running. The README says we need to use ./mvnw spring-boot:run - so let's write a task to run it in a Vim terminal:

@task.window
def launch(ctx):
    mvn['spring-boot:run'] & TERMINAL_PANEL

Noticed anything new? Instead of @task I've used @task.window. This creates a special type of test called window task. Inside window tasks you can create new Vim windows which can be used in other tasks(we'll see that later). On it's own, it acts like a normal task - expect:

• If you go to a different window during that task(you are expected to create one), when the window task is over you will be moved back to the window where you started.
• If you run the task when the window it created last time is still open - it will be closed before the task runs.

OK - so we can start the server whenever we want, and we will only have one running at a time. But what about when we don't need it? Do we have to kill it manually? No - we write a task:

@task(launch)
def kill(ctx):
    from omnipytent.util import other_windows
    with other_windows(ctx.dep.launch):
        CMD.bdelete.bang()

There are several new things here:

1. @task gets an argument - launch! This makes the launch task a dependency of the kill task - so it will be invoked whenever we call kill.
2. ctx.dep.launch - that weird ctx argument we always had in our tasks is the task's execution context - it provides methods for interacting with Omnipytent itself, and is useful when we want to combine tasks -like we do now. ctx.dep is the access point for Python objects passed to us from dependencies - in this case, because launch is a window task it automatically passes the window object(:help python-window) it created.
3. other_windows is a context manager which allows us to travel to other windows and promises to return us to where we started. It also accepts a window object argument, and when it does it brings us to that window - so we can do stuff in it.

Window tasks have a special behavior when used as dependencies - when the window they were supposed to create already exists, they don't execute and instead pass the same thing they passed before. So when we call kill, launch will pass to it the window that the previous launch task created. We then go to that window with other_windows and delete that buffer to terminate the program and close the window:

OK - so we have our server running - how do we use it? This server accepts JSON requests for finding a vet - let't create something to query it:

@task.window
def queries_terminal(ctx):
    shell = local['sh'] & TERMINAL_PANEL.vert.size(50)
    ctx.pass_data(shell)


@task(queries_terminal)
def find_vet(ctx, name):
    import json
    name = json.dumps(name)
    cmd = local['curl']['-s']
    cmd = cmd['localhost:8080/vets.json']
    cmd = cmd | local['jq']['.[] | map(select(.firstName == %s))' % name]
    cmd & ctx.dep.queries_terminal

OK... this is starting to get complex. queries_terminal creates a terminal we can use to run our queries with curl. To make the results easier to read, it makes it a vertical terminal this time(.vert) and sets it to 50 columns(.size(50)). The it calls ctx.pass_data with the result of the terminal-opening command? What's going on here?

The result of a terminal opening is a shell command executor you can use for interacting with the terminal. We then use ctx.pass_data to pass it to dependent tasks. A window task will automatically pass the window - but in this case we want to pass the terminal handler so that dependent tasks will be able to run things in it.

Which leads us to find_vet, that constructs a curl&jq command to find a vet with a given name, and executes this command using ctx.dep.queries_terminal - the shell command executor we got from queries_terminal.

Let's see it in action(I've moved the server's terminal to a tab because screen real estate):

Easy!

JSON is nice, but apparently PetClinic also supports XML. What if we want to tinker with both? We can make it an argument, or duplicate the task, or... use an options task!

@task.options
def query_format(ctx):
    json = dict(suffix='.json',
                filter=lambda name: local['jq']['.[] | map(select(.firstName == %s))' % name])
    xml = dict(suffix='.xml',
               filter=lambda name: local['xmllint']['--xpath', '//vetList[firstName=%s]' % name, '-']
               | local['xmllint']['--format', '-'])


@task(queries_terminal, query_format)
def find_vet(ctx, name):
    import json
    name = json.dumps(name)
    cmd = local['curl']['-s']
    cmd = cmd['localhost:8080/vets' + ctx.dep.query_format['suffix']]
    cmd = cmd | ctx.dep.query_format['filter'](name)
    cmd & ctx.dep.queries_terminal

WHOA! What's that? Don't be alarmed - most of it are just shell pipes stuff for filtering the data. Let's focus on the main new thing - @task.options. This creates an options task - a task used for choosing an option. This task uses a weird syntax - every local variable it creates is an option. In this case - xml and json.

If you run find_vet without picking an option first, Omnipytent will prompt you to pick one. After that it'll remember your choice - but you may change it by invoking query_format directly(with an argument to pick the option or one to get prompted).

Conclusion

If you know some basic Vimscript, you could have created commands for all the things I demonstrated. But... you probably wouldn't. Too much hassle for things you can just type in the terminal. And even if you would, you wouldn't go the extra mile to add completion and choice-cache. Too much work for something you can only use in one project...

Omnipytent's power is not in allowing you to do things - it's in making these things more accessible. When adding tasks is so easy(just :OPedit <task-name> and code it in Python), they suddenly worth the effort - even if you are only going to run a task a few times.

So automate your workflow - because you can!

Followup:

Omnipytent 1.3.0: Async Tasks and Selection UIs

Idan Arye ・ Nov 15 '18

#vim #workflow #automation