Hi all. Today, I'd like to show you the power of pure HTTP stream, which allows us to do screen sharing, video chatting, real-time text messaging, SSH, VNC, and so on. The demo videos below show SSH and VNC over pure HTTP. Let's dive into this!

SSH on Web browser	VNC on Web browser

Why HTTP?

HTTP is everywhere for everyone and everything, which allows us to communicate in any situation. HTTP is one of the most mature protocols and wildly used. You can find HTTP in web browser in personal computers including Windows, Mac and Linux, smartphones and tablets including Android, iPhone and iPad, curl, wget commands in your terminal, some IoT devices, automation tools such as Shortcuts iOS app and Microsoft Flow, Web Hooks and so on. Additionally, HTTP is an evolving protocol as HTTP/2, HTTP/3 and getting faster.

Usuary, an HTTP request has a short life, used for fetching HTML/CSS/JavaScript, media and API requests. So, this post introduces long-life HTTP request, which does real-time communications over pure HTTP stream without WebSocket and WebRTC.

Piping Server

I made Piping Server, which allows us to transfer data between every device.

# Send
echo 'hello, world' | curl -T - https://ppng.io/hello

# Get
curl https://ppng.io/hello > hello.txt

Piping Server is simple. A sender and recipient who specify the same path such as /hello can transfer. The image below is the concept of transfer.

The image shows the sender who POSTs /mypath and the recipient GETs /mypath can transfer. Both the sender and the recipient can start the transfer first. The first one waits for the other. Both POST and PUT methods are the same in Piping Server.

You can transfer a text by using fetch() in JavaScript like below.

// Send
fetch("https://ppng.io/hello", {
  method: "POST",
  body: "hello, world"
});

// Get
const res = await fetch("https://ppng.io/hello");
console.log(await res.text());
// => "hello, world"

You can also use curl command like below.

You can also transfer binary data such as an image and a video like below. As you can see, the recipient just opened the URL on the browser to get the image.

HTTP is everywhere. So we can transfer data freely without any extra tools.

Infinitely transfer

The most notable feature of Piping Server is allowing you to transfer infinite data. The demo below shows inputting text streams into the web browser.

Send folder

You can transfer a folder (a directory), which has multiple files as follows.

# Send folder
tar c ./mydir | curl -T - https://ppng.io/mypath

# Get folder
curl https://ppng.io/mypath | tar xv

The files are packing while uploading in the sender and unpacking while downloading in the recipient. Stream makes this possible without creating a temporary file.

It is easy to end-to-end encrypt your data and transfer as follows.

• send: ... | openssl aes-256-cbc | curl -T ...
• get: curl ... | openssl aes-256-cbc -d

It is also easy to reduce the size by compression as follows.

• send: ... | gzip | curl -T ...
• get: curl ... | zcat

You can transform data as you want such as gpg, zip or tools invented in the future. Combining pipe is efficient in terms of both time and memory. Unix pipe is an amazing way to combine software. The name of Piping Server is derived from Unix pipe.

The most common use case of Piping Server is a file transfer. For transferring files, you can use Piping UI, which allows you to transfer securely with end-to-end encryption over many devices.

Transfer huge data for a long time

Here are simple experiments to transfer data over HTTP using local and remote Piping Server.

The demo video below shows 45TB is transferred for 2,092 hours (87 days) over HTTP via remote Piping Server, using cat /dev/zero | curl -T- ....

The image below shows transferred 1,110TB (≈ 1PB) for 64 days over HTTP via local Piping Server.

These experiments show a huge amount of data can be continuously transferred over a single HTTP request and a single HTTP request lives long enough.

Infinite stream for Web browser

Infinite stream sending over HTTP had come to Web browser at last!

Google Chrome 85 or above has the feature as origin trial. Open chrome://flags and enable "Experimental Web Platform features" as follows

Other main browsers such as Firefox and Safari are also interested in this feature.

Uploading a Request made from a ReadableStream body by yutakahirano · Pull Request #425 · whatwg/fetch

In a nutshell, this feature allows us to send ReadableStream as follows.

fetch("https://example.com", {
  method: "POST",
  body: <ReadableStream here!>
});

Simple text messaging

Here is a simple text messaging on Web browser with fetch() and ReadableStream.

The code below creates ReadableStream from user input and sends the input stream to Piping Server. The recipient just opens the URL on the browser and sees streamed text messages.

const readableStream = new ReadableStream({
  start(ctrl) {
    const encoder = new TextEncoder();
    window.myinput.onkeyup = (ev) => {
      if (ev.key === 'Enter') {
        ctrl.enqueue(encoder.encode(ev.target.value+'\n'));
        ev.target.value = '';
      }
    }
  }
});

fetch("https://ppng.io/mytext", {
  method: 'POST',
  body: readableStream,
  headers: { 'Content-Type': 'text/plain;charset=UTF-8' },
  allowHTTP1ForStreamingUpload: true,
});

(full: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/text_stream.html)

allowHTTP1ForStreamingUpload in the code is a temporary property in Chrome to allow us to use this feature over HTTP/1.1 (see: 4c75c0c9f730589ad8d6c33af919d6b105be1462 - chromium/src - Git at Google).

Screen sharing

You can share your screen in almost the same way as the text streaming above. Get MediaStream and convert to ReadableStream and send the stream to Piping Server with fetch().

The function mediaStreamToReadableStream() below converts MediaStream to ReadableStream.

(async () => {
  // Get display
  const mediaStream = await navigator.mediaDevices.getDisplayMedia({video: true});
  // Convert MediaStream to ReadableStream
  const readableStream = mediaStreamToReadableStream(mediaStream, 100);

  fetch("https://ppng.io/myvideo", {
    method: 'POST',
    body: readableStream,
    allowHTTP1ForStreamingUpload: true,
  });
})();

// Convert MediaStream to ReadableStream
function mediaStreamToReadableStream(mediaStream, timeslice) {
  return new ReadableStream({
    start(ctrl){
      const recorder = new MediaRecorder(mediaStream);
      recorder.ondataavailable = async (e) => {
        ctrl.enqueue(new Uint8Array(await e.data.arrayBuffer()));
      };
      recorder.start(timeslice);
    }
  });
}

(full: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/screen_share.html)

The recipient just opens the HTML below with one <video> tag.

<!-- viewer -->
<video src="https://ppng.io/myvideo" autoplay muted></video>

This way is friendy to command-line tools too. You can also view the screen with curl https://ppng.io/myvideo | ffplay -. You can also send your screen with ffmpeg command. See Capture/Desktop – FFmpeg for more info.

Voice and video chatting

For voice or video chatting, all you need to do is to replace the code, const mediaStream = above with:

// Voice
const mediaStream = navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: true } })

// video + voice
const mediaStream = navigator.mediaDevices.getUserMedia({ video: true, audio: { echoCancellation: true } })

(voice: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/simple_phone.html)

(video + voice: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/video_chat.html)

Then, you can use the mediaStreamToReadableStream() to converts those MediaStreams to ReadableStreams to specify body: in fetch().

Video filtering

You can get MediaStream from the canvas. The function below creates a video and a canvas in memory and transforms a MediaStream to another one. JSManipulate is used. You may create a filter app like Snap Camera.

// Filter for sepia
async function sepiaMediaStream(mediaStream) {
  const memVideo = document.createElement('video');
  memVideo.srcObject = mediaStream;
  await memVideo.play();

  const width = memVideo.videoWidth;
  const height = memVideo.videoHeight;
  const srcCanvas = document.createElement('canvas');
  const dstCanvas = document.createElement('canvas');
  srcCanvas.width = dstCanvas.width = width;
  srcCanvas.height = dstCanvas.height = height;
  const srcCtx = srcCanvas.getContext('2d');
  const dstCtx = dstCanvas.getContext('2d');

  (function loop(){
    srcCtx.drawImage(memVideo, 0, 0, width, height);
    const frame = srcCtx.getImageData(0, 0, width, height);

    JSManipulate.sepia.filter(frame);
    dstCtx.putImageData(frame, 0, 0);
    setTimeout(loop, 0);
  })();

  return dstCanvas.captureStream();
}

(full: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/screen_share_with_filter.html)

(demo video: https://youtu.be/VcKJR8D8IFA)

Compression

Compress data with gzip as follows. In Chrome, you can easily compress a stream with readableStream.pipeThrough(new CompressionStream('gzip')).

const readableStream = new ReadableStream({
  pull(ctrl) {
    // random bytes
    ctrl.enqueue(window.crypto.getRandomValues(new Uint32Array(128)));
  }
}).pipeThrough(new CompressionStream('gzip'))

fetch("https://ppng.io/mytext", {
  method: 'POST',
  body: readableStream,
  allowHTTP1ForStreamingUpload: true,
});

(full: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/gzip_inifinite_stream.html)
The sample code sends infinite random bytes with compression over Piping Server.

End-to-end encryption for infinite stream

You can safely transfer your stream even if a server is untrustable. You can encrypt any ReadableStream with the code below using OpenPGP.js.

// Encrypt ReadableStream with password by OpenPGP
async function encryptStream(readableStream, password) {
  const options = {
    message: openpgp.message.fromBinary(readableStream),
    passwords: [password],
    armor: false
  };
  const ciphertext = await openpgp.encrypt(options);
  return ciphertext.message.packets.write();
}

https://youtu.be/lxpxeB_0UDk is a demo video of end-to-end encrypted screen sharing over Piping Server.

Service Worker is used on the viewer-side. The purpose of using Service Worker is for getting a decrypted video at https://localhost:8080/e2ee_screen_share/swvideo#myvideo. Service Worker is used as a proxy. See the full code for detail: https://github.com/nwtgck/piping-server-streaming-upload-htmls/tree/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/e2ee_screen_share.

Web browsers have Web Crypto, which can generate keys safely and do Diffie-Hellman key exchange an untrustable channel. For example, Piping UI, which is a file transfer tool, exchanges public keys and encrypts a file by using ECDH and OpenPGP.js.

Access the repository below to get other examples using fetch() upload streaming feature with Piping Server.
https://github.com/nwtgck/piping-server-streaming-upload-htmls

SSH over HTTP

As you see, any data can be streamed over HTTP. So, this means a protocol can be over HTTP via Piping Server.

Why Piping Server?

There are some environments that can not release ports public. For such kind of environment, when you have the only outward-connection to HTTP/HTTPS ports, you can use SSH. A possible example is for GitHub Actions, which does not support SSH debug like CircleCI (see: SSH in GitHub Actions over Piping Server).

SSH client in JavaScirpt

I found a wonderful project, SSHy whose JavaScript speaks SSH. The data communication is over WebSocket, so I just need to switch WebSocket to HTTP with fetch(). Unfortunately, although SSHy is not actively maintained now, this is a perfect fit for my proof of concept to speak SSH over HTTP using Piping Server. We could port OpenSSH by using Emscripten, write Rust and compile to Web Assembly, or do something in the future.

By using SSHy, it is possible to SSH with only Web browser and Piping Server. The data streamed to Piping Server is securely encrypted since the communication is SSH.

How to SSH over Piping Server?

Create two sets of connections over Piping Server for duplex communication. One of them is for sending data to your peer. The other one is for receiving data from your peer. On HTTP/2, multiple HTTP requests are bundled into one TCP connection.

The command below is an example to forward 22 port over HTTP via Piping Server. This way was proposed by @Cryolite in a Japanese great post https://qiita.com/Cryolite/items/ed8fa237dd8eab54ef2f. The data to the 22 port is downloading from /path1 and the data from the 22 port is uploading to /path2.

# server-host
socat 'EXEC:curl -NsS https\://ppng.io/path1!!EXEC:curl -NsST - https\://ppng.io/path2' TCP:127.0.0.1:22

The way makes possible NAT traversal without releasing port public over HTTP.

The command below creates the tunnel with the command above. The 22 port is forwarded to 31376 port in your another machine.

# client-host
socat TCP-LISTEN:31376 'EXEC:curl -NsS https\://ppng.io/path2!!EXEC:curl -NsST - https\://ppng.io/path1'

You can do ssh -p 31376 <user>@localhost in the machine in another terminal. This is a versatile way to forward a port to another device, not only SSH.

Transport implementations of SSHy

The implementation below sends bytes over WebSocket.
https://github.com/stuicey/SSHy/blob/82941c8ae15359fd387109dcee3a218808df0bb0/index.html#L259-L264
ws, WebSocket instance has a user-defined method, sendB64() and send Base64-encoded string. A proxy server called stuicey/wsproxy is used, which is for proxying WebSocket to TCP (in this case SSH).

The implementation below receives bytes over WebSocket.
https://github.com/stuicey/SSHy/blob/82941c8ae15359fd387109dcee3a218808df0bb0/index.html#L233-L236

SSH over Piping Server

These sending and receiving parts are replaced fetch() and a way of using Piping Server. The code below is the replaced implementation.
https://github.com/nwtgck/piping-ssh-web/blob/287e89ef05173e69d1302b29acf2abbe858ee78b/index.html#L187-L219

The application is called Piping SSH. Here is a demo video. In it, logging in Ubuntu machine from the web browser and type ls and htop command.

• Application: https://piping-ssh.nwtgck.org
• GitHub: https://github.com/nwtgck/piping-ssh-web

TIPS: Keep-alive of SSH

In Chrome, an HTTP request is stopped when no bytes arrived for 60 seconds. To resolve the issue, you can set /etc/ssh/sshd_config as follows in your SSH server setting.

# /etc/ssh/sshd_config
# ...
ClientAliveInterval 20
ClientAliveCountMax 3
# ...

VNC over HTTP

VNC (Virtual Network Computing) is widely used for controlling the computer remotely.

Here is the demo video. The front window is a Chrome, Web browser and the back one is a controlled machine on Ubuntu on VirtualBox.

• Application: https://piping-vnc.nwtgck.org
• GitHub: https://github.com/nwtgck/piping-vnc-web

For Ubuntu 20.04 users, to enable VNC, you can turn on Settings > Sharing and run gsettings set org.gnome.Vino require-encryption false to avoid an error, "Failed when connecting: Unsupported security types (types: 18)".

VNC is also available for Windows. Here is a demo controlling Windows 10 from Chrome. It was more smooth on a real Windows machine since the windows machine in the demo below was running on VirtualBox. UltraVNC is running on the Windows machine.

The fetch-uploading feature is also available on Android Chrome. The demo below controls Windows 10 by an Android smartphone.

For windows users, you can a download tunneling tool over Piping Server here: https://github.com/nwtgck/go-piping-tunnel. It is convenient to create a simple .bat file as follows.

.\piping-tunnel server -p 5900 path1 path2

piping-tunnel has the same feature of the socat + curl command. For mac users, you can install by brew install nwtgck/piping-tunnel/piping-tunnel.

How it works

The application is fully based on noVNC, which is a VNC client written in JavaScript. Only transport implementations are replaced with the way of using fetch and Piping Server instead of WebSocket.

Here is the diff for replacing WebSocket transportation with fetch and Piping Server.
https://github.com/nwtgck/piping-vnc-web/commit/1e1f2863160bfab8c9fbfc4c6970cd2b31135bfd

Network in Web browser

Here is the network in the Chrome DevTools. There are only two pure HTTPS connections. v86 is uploading and 7vk is downloading. As you can see the download size of v86 is increasing. Although 7vk is uploading, the view in the current Chrome says "pending".

fetch() upload streaming

I have been following this feature. Here are useful links to get information about the fetch() upload streaming feature.

• whatwg: Fetch Standard
• whatwg issue: Uploading a Request made from a ReadableStream body by yutakahirano · Pull Request #425 · whatwg/fetch
• Chromium commits: 688906 - Streaming upload support - chromium
• fetch() upload streaming - Chrome Platform Status
• web.dev: Streaming requests with the fetch API
• Firefox: 1387483 - [Meta-Bug] Support ReadableStream as Request.body in fetch API
• Firefox: 1469359 - Support ReadableStream as Request.body in fetch API in necko
• Safari: 203617 – "ReadableStream uploading is not supported" when fetch()ing a Request that has been logged to console

Public Piping Server

Here are public Piping Servers.

• https://ppng.io
  • alias: https://piping.ml
  • These aliases can be different servers in the future.
• https://piping.glitch.me
• https://piping-47q675ro2guv.runkit.sh
• https://ppng.herokuapp.com (NOTE: Heroku does not support streaming)

Self-hosted Piping Server

Run Piping Server on http://localhost:8080 as follows using Docker.

docker run -p 8080:8080 nwtgck/piping-server

Single binaries are also available on https://github.com/nwtgck/piping-server-pkg.

Here are easier ways to get public your Piping Server is to use Glitch and Runkit.

• remix from https://glitch.com/~piping and serve it
• clone from https://runkit.com/nwtgck/piping/ and serve it

Piping Server with JWT authentication

To restrict users to use Piping Server, you can use https://github.com/nwtgck/jwt-piping-server with an example using Auth0.

Piping Server in Rust

Piping Server is also written in Rust. This is the fastest Piping Server now.

GitHub: https://github.com/nwtgck/piping-server-rust

Base posts

Here are my posts based on this post.

• Data Streaming between Every Device over HTTP/HTTPS
• (Japanese): https://scrapbox.io/nwtgck/Web%E3%83%96%E3%83%A9%E3%82%A6%E3%82%B6%E4%B8%8A%E3%81%A7%E7%B4%94%E7%B2%8B%E3%81%AAHTTP%E3%81%A0%E3%81%91%E3%81%A7%E5%8D%98%E6%96%B9%E5%90%91%E3%83%AA%E3%82%A2%E3%83%AB%E3%82%BF%E3%82%A4%E3%83%A0%E9%80%9A%E4%BF%A1%E3%82%92%E5%8F%AF%E8%83%BD%E3%81%AB%E3%81%99%E3%82%8BHTTP%E3%81%AE%E3%82%B9%E3%83%88%E3%83%AA%E3%83%BC%E3%83%9F%E3%83%B3%E3%82%B0%E3%82%A2%E3%83%83%E3%83%97%E3%83%AD%E3%83%BC%E3%83%89%E3%81%8C%E9%81%82%E3%81%AB%E3%82%84%E3%81%A3%E3%81%A6%E3%81%8F%E3%82%8B
• (Japanese): https://scrapbox.io/nwtgck/SSH%E6%8E%A5%E7%B6%9A%E3%82%92Web%E3%83%96%E3%83%A9%E3%82%A6%E3%82%B6%E3%81%AE%E7%B4%94%E7%B2%8B%E3%81%AAHTTP%E4%B8%8A%E3%81%A7%E5%AE%9F%E7%8F%BE%E3%81%99%E3%82%8B
• (Japanese): https://scrapbox.io/nwtgck/%E3%83%AA%E3%83%A2%E3%83%BC%E3%83%88PC%E6%93%8D%E4%BD%9C%E3%82%92Web%E3%83%96%E3%83%A9%E3%82%A6%E3%82%B6%E3%81%AE%E7%B4%94%E7%B2%8B%E3%81%AAHTTP%E4%B8%8A%E3%81%A7%E5%AE%9F%E7%8F%BE%E3%81%99%E3%82%8B%EF%BC%88VNC%EF%BC%89

More

The link below is the repository of Piping Server.
GitHub: https://github.com/nwtgck/piping-server

Get more information from the link below about Piping Server such as end-to-end encrypted file transfer, with basic auth, real-time drawing and so on.
https://github.com/nwtgck/piping-server/wiki/Ecosystem-around-Piping-Server