💻 CliWrap — Execute Shell Commands with Extensive Support for Piping

Hey there!

Recently I've been working on an update for my library, CliWrap, which can be used to automate execution of shell commands from C#. Today I released version 3.0 which is a major change with a lot of improvements, mainly the new support for piping. ⭐

You can download the library from NuGet. In this post I will give a short overview of what you can do with this library. 🚀

Executing a command

The following is a basic example that shows how to asynchronously execute a command by specifying command line arguments:

using CliWrap;

var result = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteAsync();

// Result contains:
// -- result.ExitCode        (int)
// -- result.StartTime       (DateTimeOffset)
// -- result.ExitTime        (DateTimeOffset)
// -- result.RunTime         (TimeSpan)

In this scenario, all of the streams are redirected into CliWrap's equivalent of /dev/null, avoiding unnecessary memory allocations. This approach is useful if you want to execute a command, but don't care about what it writes to the console. You still get the returned exit code, which is usually enough to determine whether the command ran successfully or not.

By default, ExecuteAsync() will throw a CommandExecutionException if the underlying process returned a non-zero exit code. You can choose to disable this check.

Command configuration and execution are not coupled in any way so you can separate them:

var cmd = Cli.Wrap("path/to/exe").WithArguments("--foo bar");
var result = await cmd.ExecuteAsync();

Every call to ExecuteAsync() is stateless and will spawn a new process for each execution.

Executing a command with buffering

You can also execute a command while buffering its outputs in memory:

using CliWrap.Buffered;

var result = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteBufferedAsync();

// Result contains:
// -- result.StandardOutput  (string)
// -- result.StandardError   (string)
// -- result.ExitCode        (int)
// -- result.StartTime       (DateTimeOffset)
// -- result.ExitTime        (DateTimeOffset)
// -- result.RunTime         (TimeSpan)

Calling ExecuteBufferedAsync() is similar to ExecuteAsync(), but the returned result contains two extra fields: StandardOutput and StandardError. These contain the aggregated text data produced by the underlying command. This approach is useful if you want execute a command and then inspect what it wrote to the console. Note, however, that some commands may produce really large outputs or even write binary content instead of text, in which case it's better to use direct piping methods (explained later).

By default, this method will assume that the underlying command uses Console.OutputEncoding for writing text to the console. If it doesn't, you can override it using one of the overloads:

// Treat both stdout and stderr as UTF8-encoded text streams
var result = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteBufferedAsync(Encoding.UTF8);

// Treat stdout as ASCII-encoded and stderr as UTF8-encoded
var result = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteBufferedAsync(Encoding.ASCII, Encoding.UTF8);

Getting process ID

The promise returned by ExecuteAsync() and ExecuteBufferedAsync() is in fact not Task<T> but CommandTask<T>. It's a special object that similarly can be awaited, but on top of that it contains information about the ongoing execution.

You can inspect the task object and get the ID of the underlying process that is represented by this command execution:

var task = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteAsync();

var processId = task.ProcessId;

await task;

Lazily mapping the result of an execution

Additionally, you can transform the result of CommandTask<T> lazily with the help of Select() method:

// We're only interested in the exit code
var exitCode = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteAsync()
    .Select(r => r.ExitCode);

// We're only interested in stdout
var stdOut = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteBufferedAsync()
    .Select(r => r.StandardOutput);

Configuring arguments and other options

CliWrap has a fluent interface with various overloads to help configure different options related to command execution. For example, here are three alternative ways you can configure command line arguments:

// Set arguments directly (no formatting, no escaping)
var command = Cli.Wrap("git")
    .WithArguments("clone https://github.com/Tyrrrz/CliWrap --depth 10");

// Set arguments from a list (joined to a string, with escaping)
var command = Cli.Wrap("git")
    .WithArguments(new[] {"clone", "https://github.com/Tyrrrz/CliWrap", "--depth", "10"});

// Build arguments from parts (joined to a string, with formatting and escaping)
var command = Cli.Wrap("git")
    .WithArguments(a => a
        .Add("clone")
        .Add("https://github.com/Tyrrrz/CliWrap")
        .Add("--depth")
        .Add(10));

While all of these approaches can be used interchangeably, the last two take care of escaping automatically for you. Moreover, the builder approach has some overloads to help with formatting, which makes it the preferred approach in most situations.

Besides command line arguments, you can also configure other aspects, such as environment variables and working directory:

var command = Cli.Wrap("git")
    .WithWorkingDirectory("path/to/repo/")
    .WithArguments(a => a
        .Add("commit")
        .Add("-m")
        .Add("my commit"))
    .WithEnvironmentVariables(e => e
        .Set("GIT_AUTHOR_NAME", "John")
        .Set("GIT_AUTHOR_EMAIL", "john@email.com"));

Additionally, you can use WithValidation() to configure whether the command will throw an exception in case the execution finishes with a non-zero exit code:

var commandNoCheck = Cli.Wrap("git").WithValidation(CommandResultValidation.None);
var commandWithCheck = Cli.Wrap("git").WithValidation(CommandResultValidation.ZeroExitCode); // (default)

Note that each call to any of these mentioned WithXyz() methods returns a completely new immutable object, with the corresponding property set to the specified value. That means you can safely re-use parts of your commands as you see fit:

var command1 = Cli.Wrap("git")
    .WithWorkingDirectory("path/to/repo/")
    .WithArguments("--version")
    .WithEnvironmentVariables(e => e
        .Set("GIT_AUTHOR_NAME", "John")
        .Set("GIT_AUTHOR_EMAIL", "john@email.com"));

var command2 = command1.WithArguments("pull");

In the above example, command1 and command2 are separate objects, sharing all configuration options except command line arguments.

Timeout and cancellation

Command execution is asynchronous by nature because it involves a completely separate process. Often you may want to implement an abortion mechanism to stop the execution before it finishes, either by a manual trigger or a timeout.

To do that with CliWrap, you simply need to pass a CancellationToken that represents the cancellation signal:

using var cts = new CancellationTokenSource();

// Cancel automatically after a timeout of 10 seconds
cts.CancelAfter(TimeSpan.FromSeconds(10));

// If this is canceled, a TaskCanceledException is thrown
var result = await Cli.Wrap("path/to/exe").ExecuteAsync(cts.Token);

When an execution is canceled, the underlying process is killed and the ExecuteAsync() method throws an exception of type OperationCanceledException (or its derivative, TaskCanceledException). You will have to catch this exception to recover from cancellation.

All other execution models like ExecuteBufferedAsync() similarly accept cancellation tokens as well.

Executing a command as an event stream

Besides executing a command as a task, CliWrap also supports an alternative model, in which an execution is represented as an event stream. In this scenario, the underlying command may trigger the following events:

• StartedCommandEvent -- received just once, when the command starts executing. Contains the process ID.
• StandardOutputCommandEvent -- received every time the underlying process writes a new line to the output stream. Contains the text as string.
• StandardErrorCommandEvent -- received every time the underlying process writes a new line to the error stream. Contains the text as string.
• ExitedCommandEvent -- received just once, when the command successfully finishes executing. Contains the exit code.

There are two ways you can start a command and listen to its events. One of them is through an asynchronous pull-based stream:

using CliWrap.EventStream;

var cmd = Cli.Wrap("foo").WithArguments("bar");

await foreach (var cmdEvent in cmd.ListenAsync())
{
    switch (cmdEvent)
    {
        case StartedCommandEvent started:
            _output.WriteLine($"Process started; ID: {started.ProcessId}");
            break;
        case StandardOutputCommandEvent stdOut:
            _output.WriteLine($"Out> {stdOut.Text}");
            break;
        case StandardErrorCommandEvent stdErr:
            _output.WriteLine($"Err> {stdErr.Text}");
            break;
        case ExitedCommandEvent exited:
            _output.WriteLine($"Process exited; Code: {exited.ExitCode}");
            break;
    }
}

The ListenAsync() method starts the command and returns an object of type IAsyncEnumreable<CommandEvent>, which you can iterate over using the await foreach construct introduced with C# 8. In this scenario, back-pressure is performed by locking the pipes until an event is processed, which means there's no buffering of data in memory.

Alternatively, you can also start a command as an observable push-based stream instead:

using CliWrap.EventStream;
using System.Reactive;

await cmd.Observe().ForEachAsync(cmdEvent =>
{
    switch (cmdEvent)
    {
        case StartedCommandEvent started:
            _output.WriteLine($"Process started; ID: {started.ProcessId}");
            break;
        case StandardOutputCommandEvent stdOut:
            _output.WriteLine($"Out> {stdOut.Text}");
            break;
        case StandardErrorCommandEvent stdErr:
            _output.WriteLine($"Err> {stdErr.Text}");
            break;
        case ExitedCommandEvent exited:
            _output.WriteLine($"Process exited; Code: {exited.ExitCode}");
            break;
    }
});

In this case, Observe() starts the command and returns an IObservable<CommandEvent>. You can use the set of extensions provided by Rx.NET to transform, filter, throttle, and otherwise manipulate the stream. There is no locking in this scenario so the data is pushed as soon as it's available.

Both ListenAsync() and Observe() also have overloads that accept custom encoding and/or a cancellation token.

Piping

Most of the features you've seen so far are based on CliWrap's core model of piping. It lets you redirect input and output streams of the underlying process to form a complex execution pipeline.

To facilitate piping, Command object has three methods:

• WithStandardInputPipe(PipeSource source)
• WithStandardOutputPipe(PipeTarget target)
• WithStandardErrorPipe(PipeTarget target)

By default, every command is piped from PipeSource.Null and is itself piped to PipeTarget.Null, which are CliWrap's equivalents of /dev/null. You can change that and, for example, have the command pipe its standard input from one file and redirect its standard output to another:

await using var input = File.OpenRead("input.txt");
await using var output = File.Create("output.txt");

await Cli.Wrap("foo")
    .WithStandardInputPipe(PipeSource.FromStream(input))
    .WithStandardOutputPipe(PipeTarget.ToStream(output))
    .ExecuteAsync();

The exact same thing can be expressed in a terser way using pipe operators:

await using var input = File.OpenRead("input.txt");
await using var output = File.Create("output.txt");

await (input | Cli.Wrap("foo") | output).ExecuteAsync();

Besides raw streams, PipeSource and PipeTarget both have factory methods that help express different piping directions:

• PipeSource.Null -- represents an empty pipe source.
• PipeSource.FromStream() -- pipes data from any readable stream.
• PipeSource.FromBytes() -- pipes data from a byte array.
• PipeSource.FromString() -- pipes from a text string (supports custom encoding)
• PipeSource.FromCommand() -- pipes data from standard output of another command.
• PipeTarget.Null -- represents a pipe target that discards all data.
• PipeTarget.ToStream() -- pipes data into any writeable stream.
• PipeTarget.ToStringBuilder() -- pipes data as text into StringBuilder (supports custom encoding).
• PipeTarget.ToDelegate() -- pipes data as text, line-by-line, into Action<string> or Func<string, Task> (supports custom encoding).
• PipeTarget.Merge() -- merges multiple pipes into one.

The pipe operator also has overloads for most of these. Below you can see some examples of what you can do.

Pipe a string into stdin:

await ("Hello world" | Cli.Wrap("foo")).ExecuteAsync();

Pipe stdout as text into a StringBuilder:

var stdOutBuffer = new StringBuilder();
await (Cli.Wrap("foo") | stdOutBuffer).ExecuteAsync();

Pipe a binary HTTP stream into stdin:

using var httpClient = new HttpClient();
await using var input = await httpClient.GetStreamAsync("https://example.com/image.png");

await (input | Cli.Wrap("foo")).ExecuteAsync();

Pipe stdout of one command into stdin of another:

await (Cli.Wrap("foo") | Cli.Wrap("bar") | Cli.Wrap("baz")).ExecuteAsync();

Pipe stdout and stderr into those of parent process:

await using var stdOut = Console.OpenStandardOutput();
await using var stdErr = Console.OpenStandardError();

var cmd = Cli.Wrap("foo") | (stdOut, stdErr);
await cmd.ExecuteAsync();

var cmd = Cli.Wrap("foo") |
    (Console.WriteLine, Console.Error.WriteLine);

await cmd.ExecuteAsync();

Pipe stdout into a file and stderr into a StringBuilder:

await using var file = File.Create("output.txt");
var buffer = new StringBuilder();

var cmd = Cli.Wrap("foo") |
    (PipeTarget.ToStream(file), PipeTarget.ToStringBuilder(buffer));

await cmd.ExecuteAsync();

Pipe stdout into multiple files simultaneously:

await using var file1 = File.Create("file1.txt");
await using var file2 = File.Create("file2.txt");
await using var file3 = File.Create("file3.txt");

var target = PipeTarget.Merge(
    PipeTarget.ToStream(file1)
    PipeTarget.ToStream(file2)
    PipeTarget.ToStream(file3));

await (Cli.Wrap("foo") | target).ExecuteAsync();

Pipe a string into a command, that command into another command, and then into parent's stdout and stderr:

var cmd = "Hello world" | Cli.Wrap("foo")
    .WithArguments("print random") | Cli.Wrap("bar")
    .WithArguments("reverse") | (Console.WriteLine, Console.Error.WriteLine);

await cmd.ExecuteAsync();

As you can see, piping enables a wide range of different use cases. It's not only used for convenience, but also to improve memory efficiency when dealing with large and/or binary inputs and outputs. With the help of CliWrap's pipe operators, configuring pipelines is really easy -- just imagine doing the same with System.Diagnostics.Process manually.

The different execution models which we saw earlier, ExecuteBufferedAsync(), ListenAsync() and Observe() are all based on the concept of piping, but these approaches are not exclusive. For example, you can create a piped command and start it as an event stream:

await using var input = File.OpenRead("input.txt");
await using var output = File.Create("output.txt");

var cmd = input | Cli.Wrap("foo") | output;

await foreach (var cmdEvent in cmd.ListenAsync())
{
    // ...
}

This works because internally these methods call PipeTarget.Merge to add additional pipe targets while preserving those configured earlier.