This post provides a secure and highly transparent way for port forwarding or tunneling a single TCP connection, using only nc and curl. Moreover, it also provides more secure tunneling with end-to-end encryptions using openssl and socat.
# server host
curl -sSN https://ppng.io/aaa | nc localhost 22 | curl -sSNT - https://ppng.io/bbb
# client host
curl -sSN https://ppng.io/bbb | nc -lp 2222 | curl -sSNT - https://ppng.io/aaa
Why port forwarding from anywhere?
Today's application is everywhere. We often want to access an application on our remote machine. It is useful to make the remote application as if it is in local. To achieve that, port forwarding or tunneling is widely used for accessing a private remote port from a local machine. A common approach for secure port forwarding is using ssh -L. This is a very secure connection over SSH. However, exposing SSH port in public may not be allowed because of firewall or NAT. We sometimes need to port forward seamlessly no matter where we are. Besides, since SSH is very powerful, it allows us to use a shell by default even when we want to use only port forwarding without arbitrary command execution. In addition, some environments may not allow to install and run an SSH server, which usually needs the root permission.
Why nc and curl?
Some tools allow us to port forward from anywhere. However, as far as I know, they require us to install their own dedicated command. Some of them are open source projects, but I wanted a more transparent way because a shorter chain of trust should be better.
I found a higher transparent way that uses existing and widely used commands: nc and curl. The original idea of a port forwarding with socat, curl, and Piping Server was proposed in a Japanese article by @Cryolite. Meanwhile, this post introduces a way using nc (netcat), which is more widely used and installed than socat, and the way allows simple integration with other commands using Unix pipe. It is very helpful when using an untrustable server because we can integrate it with an encryption command such as openssl.
Piping Server
Here is an introduction to Piping Server. I developed Piping Server, which streams any data infinitely between every device over pure HTTP/HTTPS.
nwtgck
/
piping-server
Infinitely transfer between every device over pure HTTP with pipes or browsers
Piping Server
Infinitely transfer between every device over HTTP/HTTPS
Transfer
Piping Server is simple. You can transfer as follows.
# Send
echo 'hello, world' | curl -T - https://ppng.io/hello
# Get
curl https://ppng.io/hello > hello.txt
Piping Server transfers data to POST /hello or PUT /hello into GET /hello. The path /hello can be anything such as /mypath or /mypath/123/. A sender and receivers who specify the same path can transfer. Both the sender and the recipient can start the transfer first. The first one waits for the other.
You can also use Web UI like https://ppng.io on your browser. A more modern UI is found in https://piping-ui.org, which supports E2E encryption.
Stream
The most important thing is that the data are streamed. This means that you can transfer any data infinitely. The demo below transfers an infinite text stream with seq inf.
Ideas
…Piping Server is an HTTP server. A sender and recipient who specify the same path such as /mypath can transfer. The image below is the concept of Piping Server.
The image above shows the sender who does POST /mypath and the recipient who does GET /mypath can transfer. Both the sender and the recipient can start the transfer first. The first one waits for the other. After the sender and recipient connection is established, /mypath is protected not to be connected until the connection is closed. Closing the path frees the path and anyone can start a new transfer at the path.
Both POST and PUT methods are the same effect in Piping Server. Here is a simple example of transferring "hello" to a receiver.
# sender: PUT /mypath
$ echo hello | curl -T - https://ppng.io/mypath
# receiver: GET /mypath
$ curl https://ppng.io/mypath
hello
The server transfers any kind of data including text and binaries. In addition, the data through a Piping Server are infinitely streamed. The data are only on memory just for a moment and never saved. I posted The Power of Pure HTTP, which shows how powerful HTTP is and what Piping Server can do.
Public servers
Here are some available public servers.
https://piping.nwtgck.repl.co may be faster depending on your location. You can host one using docker run -p 8181:8080 nwtgck/piping-server instantly or other ways including portable binaries: https://github.com/nwtgck/piping-server/wiki/How-to-self-host-Piping-Server.
Port forwarding command
Here is the main part of this post.
Suppose you have two machines named "server host" which serves 22 port and "client host", and you want to use the 22 port in the server host as a new 2222 port in the client host. The following commands forward the 22 port in the server host to the 2222 port in the client host.
# server host
curl -sSN https://ppng.io/aaa | nc localhost 22 | curl -sSNT - https://ppng.io/bbb
# client host
curl -sSN https://ppng.io/bbb | nc -lp 2222 | curl -sSNT - https://ppng.io/aaa
-
-sslient without progress message -
-Sshows error even when-sused -
-NSmooth streaming, disalbing buffering of the output stream
Cheat sheat
Though it's very cool to remember and type the commands, we can go to https://piping-server-command.nwtgck.org to copy the commands.
GNU nc vs OpenBSD nc vs socat
netcat (nc) has some variants. The ways of listening are different as follows.
- GNU nc:
nc -lp 2222 - OpenBSD nc:
nc -l 2222 - socat:
socat TCP-LISTEN:2222,reuseaddr -
# client host (GNU nc)
curl -sSN https://ppng.io/bbb | nc -lp 2222 | curl -sSNT - https://ppng.io/aaa
# client host (OpenBSD nc)
curl -sSN https://ppng.io/bbb | nc -l 2222 | curl -sSNT - https://ppng.io/aaa
# client host (socat)
curl -sSN https://ppng.io/bbb | socat TCP-LISTEN:2222,reuseaddr - | curl -sSNT - https://ppng.io/aaa
SSH demo
Here is a demo video of forwarding 22 port in my "basil" server to my local 2222 port.
SSH is very powerful because you use can create a SOCKS proxy with ssh -D 1080 ... like a simple VPN, do rsync for fast resumable file transfer, use GUI with ssh -X ... and mount a remote file system with SSHFS.
The example is about SSH since it is literally a well-known protocol. This method allows any TCP port to be forwarded for a single connection not only SSH.
Mechanism
The goal of tunneling is as follows, forwarding the 22 port of "server program" to the 2222 port of "client program."
The diagram below illustrates how the two commands above connect each other. All outgoing connections are HTTPS. This means that the tunneling requirements are the same as web browsing. Therefore, you can do the tunneling anywhere.
You can control the timing of port forwarding. Each server host side and client host side can start the command first. Each side can close the tunnel by Ctrl+C at any time and the other side command exits at that time, and this frees the paths. Piping Server guarantees the transferred protocol data is only on memory and never stored. The server also guarantees that after port forwarding is established at paths, no one can establish and get the data at the paths until the pre-established tunnel is closed.
High transparency
I believe "The less you trust in, the more secure you are."
Some tunneling tools require installing additional dedicated CLI. In contrast, the way uses only nc (netcat) and curl commands. They are widely used and accepted, and most people know how they work and have already trusted them. You just use the common commands, not need to install and trust extra tools. We can obviously understand that the commands never send any extra information, such as private information or credentials. This means there is no black box at all. Besides, the commands are short enough to remember.
All communication outside is complete in HTTP/HTTPS. The protocol is also familiar, widely used, accepted, and trusted already. Piping Server is open on GitHub, developed in TypeScript and Node.js. Other implementations in Rust and Go are provided as open-source. The server is designed to keep simple as possible to verify the source and reduce the potential of bugs.
In order to improve transparency more and get independent from the pre-hosted public servers, you can self host Piping Server. When you prefer to limit traffic with basic auth or path, etc., Rich Piping Server, which uses internally the original Piping Server as a library, is useful.
We need to trust a few existing widely accepted command-line tools and Piping Server. Even if the source of the server is public, the server might have vulnerabilities or might be modified maliciously. To avoid those concerns, I will introduce end-to-end encryption (E2E encryption) for higher-level security. E2E encryption allows us not to trust the server and make us secure.
Universal end-to-end encryption
The section describes secure tunneling even if a Piping Server is untrustable. As you know, all HTTPS traffic to the server is encrypted. However, what if the server is compromised or malicious? Tunneling SSH is one of the solutions. This section provides a more universal end-to-end encryption way for any protocol.
The idea is simple, which decrypts incoming streams and encrypts outgoing streams as follows.
# server host (idea)
curl -sSN https://ppng.io/aaa | <decrypt> | nc localhost 22 | <encrypt> |curl -sSNT - https://ppng.io/bbb
# client host (idea)
curl -sSN https://ppng.io/bbb | <decrypt> | nc -lp 2222 | <encrypt> | curl -sSNT - https://ppng.io/aaa
The real commands using openssl are as follows. Although openssl commands are used, any kind of encryption commands can be replaced with existing commands or future-invented commands.
# server host
curl -sSN https://ppng.io/aaa | stdbuf -i0 -o0 openssl aes-256-ctr -d -pass "pass:mypass" -bufsize 1 -pbkdf2 | nc localhost 22 | stdbuf -i0 -o0 openssl aes-256-ctr -pass "pass:mypass" -bufsize 1 -pbkdf2 | curl -sSNT - https://ppng.io/bbb
# client host
curl -sSN https://ppng.io/bbb | stdbuf -i0 -o0 openssl aes-256-ctr -d -pass "pass:mypass" -bufsize 1 -pbkdf2 | nc -lp 2222 | stdbuf -i0 -o0 openssl aes-256-ctr -pass "pass:mypass" -bufsize 1 -pbkdf2 | curl -sSNT - https://ppng.io/aaa
-
stdbuf -i0 -o0 <command>disables buffers for real-time streaming. -
aes-256-ctris a stream cipher. -
-pbkdf2derives key from the password using PBKDF2. This is optional but makes it more secure. - The password is
mypass. You can replaced with better one.
The commands below allow us to input password and hides passwords from commands.
# server host
read -p "password: " -s pass && curl -sSN https://ppng.io/aaa | stdbuf -i0 -o0 openssl aes-256-ctr -d -pass "pass:$pass" -bufsize 1 -pbkdf2 | nc localhost 22 | stdbuf -i0 -o0 openssl aes-256-ctr -pass "pass:$pass" -bufsize 1 -pbkdf2 | curl -sSNT - https://ppng.io/bbb; unset pass
# client host
read -p "password: " -s pass && curl -sSN https://ppng.io/bbb | stdbuf -i0 -o0 openssl aes-256-ctr -d -pass "pass:$pass" -bufsize 1 -pbkdf2 | nc -lp 2222 | stdbuf -i0 -o0 openssl aes-256-ctr -pass "pass:$pass" -bufsize 1 -pbkdf2 | curl -sSNT - https://ppng.io/aaa; unset pass
These commands are also in the cheat sheet.
Web browser, not only terminal and the commands
High interoperability matters because it realizes fewer dependencies and less lock-in. This section shows the interoperability, replacing nc and curl with JavaScript.
Piping SSH and Piping VNC run on Web browser. This means a Web browser is an alternative frontend of curl ... | nc -lp ... | curl ... and a terminal. Those apps have SSH and VNC (RBF protocol) implementations in JavaScript, thanks to authors of based projects. This means that they work on your local device and never process on the server side. The server is only a Piping Server only for data transferring. See my previous post for detail.
Those apps require "fetch() upload streaming", which is only available now in Chromium-based browsers >= 85 with "Experimental Web Platform features" enabled. Every major vendors has positive reaction to the "fetch() upload streamingl". The video below is an example using Piping SSH.
VNC is widely used for remote desktop. The example below has two windows, a Ubuntu on a virtual machine and Piping VNC on Chrome on my local machine. It shows relatively smooth remote desktop.
The port forwarding way has high interoperability because the forwarding command, curl ... | nc ... | curl ... , in the example videos is the same even when a client runs on Web browser. In this example, fetch() in JavaScript replaces nc, curl, and terminal. As another example, java.net.URLConnection and java.net.Socket in Java and net.Dial() and http in Go language can replace them.
E2E encryption VNC
Although the transport to Piping Server is encrypted with TLS because of HTTPS, E2E encryption for VNC is secure if the server is untrustable.
You can put a check in "Encrypt with OpenSSL AES CTR" below in Piping VNC. It has the same effect as openssl aes-256-ctr -d -pass "pass:mypass" -bufsize 1 -pbkdf2 -iter 100000 -md sha256 mentioned in the "Universal end-to-end encryption" section above.
Piping VNC automatically generates a command you should type on the server host side.
How is it possible in Web browser? I made https://github.com/nwtgck/openssl-aes-ctr-stream-npm for using OpenSSL-compatible AES CTR encryption/decryption in Web browser. The concept is very simple as follows.
// Encrypt the ReadableStream `uploadReadableStream`
const encryptedUploadReadableStream = opensslAesCtrStream.aesCtrEncryptWithPbkdf2(
uploadReadableStream,
options
);
// Transfer the encrypted stream
fetch(pipingServerUrl, {
method: "POST",
body: encryptedUploadReadableStream,
});
(simplified version of https://github.com/nwtgck/piping-vnc-web/blob/8f2df75c7452b515fcc2ba36fe9cbf6218c673fc/core/websock.js#L205-L214)
openssl-aes-ctr-stream implementation is shorter than 200 lines: https://github.com/nwtgck/openssl-aes-ctr-stream-npm/blob/d071752009b0c6262f6c27722b438cb73cdb7cab/src/index.ts. The major browsers provide Web Crypto API. openssl-aes-ctr-stream uses AES-CTR in the Web Crypto API.
Here is a demo video of E2E encryption VNC. Because of the E2E encryption, the screen is laggier than without it.
https://piping-vnc.nwtgck.org
Because the protocol of OpenSSL is open, we can stay highly interoperable with implementation in the other language.
Universal end-to-end encryption over TLS
This section introduces the way to encrypt pre-listened TCP port using TLS with socat. This is also a universal E2E encryption way. The advantages of the way are that you can use well-trusted TLS and encryptions with sharing certificates instead a password.
In the server host, create certificates as follows. It generates server.crt, server.key and server.pem.
# server host
openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3653 -sha256 -nodes -subj '/CN=localhost/' && cat server.key server.crt > server.pem
In a client host, create certificates as follows. It generates client.crt, client.key and client.pem.
# client host
openssl req -x509 -newkey rsa:4096 -keyout client.key -out client.crt -days 3653 -sha256 -nodes -subj '/CN=localhost/' && cat client.key client.crt > client.pem
Share server.crt to the client host and share client.crt to the server host respectively.
server host
Suppose the server host serves 8181 port and you want to forward the 8181 to a client with encryption. Open two terminals in server host and run the following commands. Make sure to run socat in the directory which has server.pem and client.crt.
socat openssl-listen:4433,reuseaddr,cert=server.pem,cafile=client.crt tcp:localhost:8181
curl -sSN https://ppng.io/aaa | nc localhost 4433 | curl -sSNT - https://ppng.io/bbb
client host
Open two terminals in the client host and run the following commands. Make sure to run socat in the directory which has client.pem and server.crt.
curl -sSN https://ppng.io/bbb | nc -lp 5433 | curl -sSNT - https://ppng.io/aaa
socat tcp-listen:8282,reuseaddr openssl-connect:localhost:5433,openssl-commonname=localhost,cert=client.pem,cafile=server.crt
Now, the 8282 port in the client host is the decrypted TCP port forwarded to 8181 in the server host. When the protocol of the 8181 port in the server host is HTTP, you can use curl localhost:8282 in the client host.
Here is a graph to describe how this E2EE works.
[server host 8181 <--> 4433] <-- TLS over HTTPS --> [Piping Server] <-- TLS over HTTPS --> [5433 <--> 8282 client host]
The way is based on Securing Traffic Between two Socat Instances Using SSL found on the socat official page. If you encountered Invalid argument error in macOS with socat, you can upgrade socat 1.7.4.2 or later, which is fixed.
Universal end-to-end encryption over SSH
You can simply use ssh -L as follows when a machine on server host side has SSH server. SSH also multiplexes TCP connections over single SSH connection.
Suppose you want to forward 8080 port in server host with encryption to 8181 port in client host. Suppose the server host listens SSH on 22 port. The command below forwards 22 to 2222 and 8080 to 8181.
# server host
curl -sSN https://ppng.io/aaa | nc localhost 22 | curl -sSNT - https://ppng.io/bbb
# client host 1/2
curl -sSN https://ppng.io/bbb | nc -lp 2222 | curl -sSNT - https://ppng.io/aaa
# client host 2/2
ssh -p 2222 -L 8181:localhost:8080 <server host user>@localhost
Now, you can access localhost:8181 in the client host.
bonus: Multiple TCP connections
The way introduced here allows forwarding a single TCP connection. A single TCP connection is enough for SSH and VNC (RFB protocol). This feature sometimes contributes to greater security since one connection is always guaranteed. However, for instance, HTTP generally requires multiple TCP connections even when HTTP/2, which multiplexes request and response streams, is available.
A simple solution is to multiplex TCP requests over Unix pipe. In order to multiplex TCP requests, Yamux is available created by Hashicorp, who creates Vagrant, Terraform, and so on. The protocol of Yamux is inspired by SPDY, which is the basis of HTTP/2 specification. The protocol spec is found in https://github.com/hashicorp/yamux/blob/master/spec.md. libp2p, which is used in IPFS, also uses Yamux as one choice of multiplexes and maintains Go and Rust versions of Yamux libraries.
I made a yamux CLI and distribute its portable binaries for multi-platforms. You can install it from https://github.com/nwtgck/yamux-cli and replaces nc with yamux as follows.
# server host
curl -sSN https://ppng.io/aaa | yamux localhost 8080 | curl -sSNT - https://ppng.io/bbb
# client host
curl -sSN https://ppng.io/bbb | yamux -l 8181 | curl -sSNT - https://ppng.io/aaa
You can also combine this yamux and E2E commands to make it more secure.
This topic is in a bonus section since its way requires an additional CLI. The way of the section "Universal end-to-end encryption over SSH" also allows multiple TCP connections because ssh -L also multiplexes TCP connections.
bonus: UDP
TCP is a perfect fit for Unix pipe because TCP delivers streams. In contrast, UDP delivers datagrams, so boundaries are needed to deliver multiple datagrams over Unix pipe. I successfully run dig command for forwarded DNS using nc -u. However, multiple datagrams are not processed. To solve this problem, I made an experimental option to yamux cli for UDP.
# server host (experimental)
curl -sSN https://ppng.io/aaa | yamux -u localhost 53 | curl -sSNT - https://ppng.io/bbb
# client host (experimental)
curl -sSN https://ppng.io/bbb | yamux -ul 1053 | curl -sSNT - https://ppng.io/aaa
"Experimental" means that I may change the way of transferring over Yamux. Datagrams from the same address are delivered over the same Yamux stream. The data structure is simple, which has datagram length in 4 bytes in network byte order and raw datagram.
In my experiment, HTTP/3, HTTP over QUIC, can be port forwarded over Piping Server. Actually, I realized HTTP/3 Piping Server over Piping Server.
Links
- Japanese article of port forwarding using socat and curl
- GitHub repository of Pipnig Server
- GitHub repository Piping Server in Rust
- How to self host Piping Server
- Command cheat sheet
The cover image by Fabian Jung on Unsplash
Top comments (0)