DEV Community

Kazuki Higashiguchi
Kazuki Higashiguchi

Posted on • Updated on

Reverse HTTP proxy over WebSocket in Go (Part 3)

Series introduction

In my previous post I talked about how to establish a WebSocket connection in Go.

In this post, I will be starting to talk about how to relay TCP connection from "App" to the peer of WebSocket in Go.

  • Start a WebSocket server (Part 1)
  • Establish a WebSocket connection (Part 2)
  • Relay TCP connection from "App" to the peer of WebSocket (Part 3 | Part 4)
  • Relay TCP connection in WebSocket data to "internal API"
  • Keep a established connection

Reverse HTTP proxy over WebSocket

A reverse HTTP proxy over WebSocket is a type of proxies, which retrieves resources on behalf on a client from servers and uses the WebSocket protocol as a "tunnel" to pass TCP communication from server to client.

A network diagram for reverse proxy over WebSocket

I'll use root-gg/wsp as a sample code to explain it. I'll use the forked code to explain because maintenance has stopped and the Go language and libraries version needed to be updated.

GitHub logo hgsgtk / wsp

HTTP tunnel over Websocket

Relay TCP connection to the peer of WebSocket

In the part 1, I presented a demonstration.

A terminal image when sending a HTTP request from app

HTTP communication is relayed by the following route.

app -[1]-> wsp server -[2](WebSocket)-> wsp client -> internal API
Enter fullscreen mode Exit fullscreen mode

As a pre requirement, a WebSocket connection has already been established between wsp server and wsp client before relaying.

The way to establish WebSocket connection is explained in part 2, but next we need to pool the connection on the server side and use it when relaying.

These flow are divided into three parts to explain it.

  1. Receive requests to be proxied ([1] in the relay flow)
  2. Pool the WebSocket connection on the server for relaying
  3. Relay TCP connection to the peer WebSocket ([2] in the relay flow)

This post describes the 1st and 2nd flow. In part 4, I'll explain the 3rd flow.

1. Receive requests to be proxied

To receive requests to be proxied to "internal API" finally, there are mainly two kinds of API interfaces.

  1. Expose an endpoint (i.e. /requests) that accept HTTP requests
  2. Work as a HTTP proxy used by "app" (i.e. curl -x {wsp server's address} http://localhost:8081)

In the demo example, wsp server (WebSocket server) chose 1st option and exposes the endpoint whose path is /requests.

$ curl -H 'X-PROXY-DESTINATION: http://localhost:8081/hello' http://127.0.0.1:8080/request
Enter fullscreen mode Exit fullscreen mode

Let's read the HTTP handler code in Go (code), which waits the request to /requests/.

func (s *Server) request(w http.ResponseWriter, r *http.Request) {
    // Parse destination URL
    dstURL := r.Header.Get("X-PROXY-DESTINATION")
    if dstURL == "" {
        wsp.ProxyErrorf(w, "Missing X-PROXY-DESTINATION header")
        return
    }
    URL, err := url.Parse(dstURL)
    if err != nil {
        wsp.ProxyErrorf(w, "Unable to parse X-PROXY-DESTINATION header")
        return
    }
    r.URL = URL

    // (omit)
}
Enter fullscreen mode Exit fullscreen mode

wsp defined the custom HTTP header X-PROXY-DESTINATION to specify the final destination URL. For example, when you want to access http://localhost:8081/hello, add a HTTP header X-PROXY-DESTINATION: http://localhost:8081/hello in your request.

2. Pool the WebSocket connection on the server for relaying

We need to make sure that the http handler can recognize and use the connected WebSocket connections.

In this code, an HTTP server that exposes an endpoint /request is set up at a given address. The handler mapped to /request is as follow (code):

func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
    // - 1. Upgrade a received HTTP request to a WebSocket connection
    // (omit)
    // - 2. Wait a greeting message from the peer and parse it
    // (omit)

    // 3. Register the connection into server pools.
    // s.lock is for exclusive control of pools operation.
    s.lock.Lock()
    defer s.lock.Unlock()

    var pool *Pool
    // There is no need to create a new pool,
    // if it is already registered in current pools.
    for _, p := range s.pools {
        if p.id == id {
            pool = p
            break
        }
    }
    if pool == nil {
        pool = NewPool(s, id)
        s.pools = append(s.pools, pool)
    }
    // update pool size
    pool.size = size

    // Add the WebSocket connection to the pool
    pool.Register(ws)
}
Enter fullscreen mode Exit fullscreen mode

First, the server needs to have a WebSocket connection pooled so that it can be used for relay. One design idea to achieve this is to prepare three entities Server, Pool, and Connection as shown below.

A Miro image describing the relationship map

In this code, the Server struct have a field pools which is a pointer of the slice of Pool struct.

type Server struct {
    // (omit)

    pools []*Pool

    // (omit)
}
Enter fullscreen mode Exit fullscreen mode

The Pool struct represents the connection from ws client. The definition is as follow:

type Pool struct {
    server *Server
    id     PoolID

    size int

    connections []*Connection
    idle        chan *Connection

    done bool
    lock sync.RWMutex
}
Enter fullscreen mode Exit fullscreen mode

The Connection struct manages a single WebSocket connection because wsp supports multiple connections from a single wsp client at the same time.

type Connection struct {
    pool         *Pool
    ws           *websocket.Conn
    status       ConnectionStatus
    idleSince    time.Time
    lock         sync.Mutex
    nextResponse chan chan io.Reader
}
Enter fullscreen mode Exit fullscreen mode

Then, going back to the handler code, we can see that the server checks whether requesting connection is already registered, and create a new pool if it's not registered in current pools.

func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
    // (omit)

    var pool *Pool
    // There is no need to create a new pool,
    // if it is already registered in current pools.
    for _, p := range s.pools {
        if p.id == id {
            pool = p
            break
        }
    }
    if pool == nil {
        pool = NewPool(s, id)
        s.pools = append(s.pools, pool)
    }

    // (omit)
}
Enter fullscreen mode Exit fullscreen mode

Each ws client issues an unique ID, which is a common design when you want to pool and recognize external devices. By issuing an ID, the server is able to recognize meta information such as ws client source IP and so on.

At the end of the handler function, the server registers a new connection to the created pool.

func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
    // (omit)

    pool.Register(ws)
}
Enter fullscreen mode Exit fullscreen mode

Pool.Register is as follow:

func (pool *Pool) Register(ws *websocket.Conn) {
    // (omit)

    log.Printf("Registering new connection from %s", pool.id)
    connection := NewConnection(pool, ws)
    pool.connections = append(pool.connections, connection)
}
Enter fullscreen mode Exit fullscreen mode

This handler process registered an idle connection that can be used for relay. NewConnection() function mark a new connection as a usable connection for relay in the pool and start a thread (technically, start a new goroutine running) that keeps reading messages over the connected WebSocket connection from the wsp client.

func NewConnection(pool *Pool, ws *websocket.Conn) *Connection {
    // Initialize a new Connection
    c := new(Connection)
    c.pool = pool
    c.ws = ws
    c.nextResponse = make(chan chan io.Reader)
    c.status = Idle

    // Mark that this connection is ready to use for relay
    c.Release()

    // Start to listen to incoming messages over the WebSocket connection
    go c.read()

    return c
}
Enter fullscreen mode Exit fullscreen mode

At initialization, a new Connection is created by state Idle.

Enum in Go

The status Idle means it is opened but not working now. Here is a state diagram describing the behavior of Connection.

A state diagram of Connection

In this repository, I made the defined type ConnectionStatus and constants Idle, Busy, and Closed.

// ConnectionStatus is an enumeration type which represents the status of WebSocket connection.
type ConnectionStatus int

const (
    // Idle state means it is opened but not working now.
    // The default value for Connection is Idle, so it is ok to use zero-value(int: 0) for Idle status.
    Idle ConnectionStatus = iota
    Busy
    Closed
)
Enter fullscreen mode Exit fullscreen mode

By the way, if you want not to define ConnectionStatus with Zero-Value, you can skip the zero-value as follows:

// Pattern1: Assign zero-value to blank
const (
    _ State = iota 
    A = iota
    B
)

// Pattern2: Start from 1
const (
    C State = iota + 1
    D
)
Enter fullscreen mode Exit fullscreen mode

Conclusion

This post explained how to relay TCP connection from "App" to the peer of WebSocket, especially implementation to receive requests to be proxied and to pool the WebSocket connection on the server for relaying.

I will explain the rest points in part 3 and beyond.

  • Relay TCP connection from "App" to the peer of WebSocket (rest)
  • Relay TCP connection in WebSocket data to "internal API"
  • Keep a established connection

Discussion (0)