The Situation: Practicing Data Ingestion in Rails (and Trying to be Async)

So, for reasons that (ahem) may or may not be job-interview-related, I'm practicing building a Rails server for ingesting, parsing, and storing data in a database.

My goal was to make two Rails apps, and have them talk to each other:

1. A server that broadcasts hashes containing randomly-generated characters (ideally repeating every 1 second, indefinitely)
2. A server that listens for and receives the hashes, and parses them as they're received to store in a database

I intended to implement this behavior asynchronously with the ActionController::Live module. The intended behavior is for the Broadcast server to emit a hash (as a string) every 1 second, and for the Receiver server to parse and store each hash as they come in. (For my tests, I have this looping 5 times.)

My problem is that the character-hashes are rendered 1-by-1 when testing in my browser and in the Broadcast server's console...

working fine in Chrome...

working fine in console...

...but in the Receiver server's console, all the character-hashes come at once in the HTTP response!

all the JSON is received as one response body!

So why is the async behavior (apparently) working in some places, and not in others?

Broadcaster server

The Broadcaster server is a simple Rails app that uses a BroadcasterController with ActionController::Live and its server-side event (SSE) module.

The index method generates a random character_hash, writes it to the current SSE response.stream in the variable sse, and pauses for 1 second to illustrate async behavior.

# Broadcaster app
# /app/controllers/broadcaster_controller.rb

class BroadcasterController < ApplicationController
    include ActionController::Live

    def index
        name_array = ["Ryu", "Peco", "Rei", "Momo", "Garr", "Nina"]
        hp_array = [132, 71, 15, 1, 0, 325]
        magic_array = ["Frost", "Typhoon", "Magic Ball", "Ascension", "Rejuvinate", "Weretiger"]

        response.headers['Content-Type'] = "text/event-stream"
        sse = SSE.new(response.stream)

        begin
            5.times do      
                character_hash = {
                    "uuid": SecureRandom.uuid,
                    "name": name_array.sample,
                    "hp": hp_array.sample,
                    "magic": magic_array.sample
                }
                sse.write({ character: character_hash })
                sleep 1
            end
        rescue IOError
            # client disconnected
        ensure
            sse.close
        end
    end
end

# Broadcaster app
# /config/routes.rb

Rails.application.routes.draw do
  get 'broadcaster' => 'broadcaster#index'
end

Once we start the server with rails s, we can use curl -i http://localhost:3000/broadcaster in the command line to send a Get request to the index method. The response will return each character with a 1 second delay in-between:

Since navigating to http://localhost:3000/broadcaster in the browser will also send a Get request, we see the same behavior here in Chrome:

So far, so good...

Receiver server

The other Rails app is a Receiver server that sends a Get request to the Broadcaster at http://localhost:3000/broadcaster, and parses its response to store the received characters in the database.

We also have it puts a readout to show us the res.body characters arriving all at once, instead of asynchronously as we saw above.

# Receiver app
# /app/controllers/listener_controller.rb

require 'net/http'

class ListenerController < ApplicationController
    def index
        url = URI.parse('http://localhost:3000/broadcaster')
        req = Net::HTTP::Get.new(url.to_s)
        res = Net::HTTP.start(url.host, url.port) { |http| http.request(req) }

        puts <<-READOUT

    res.body:
    ==============================
    #{res.body}

        READOUT

        char_array = res.body.split("\n\n")
        char_array.each do |data_str|
            data_hash = eval(data_str.slice!(6..-1))    # slice to remove leading "data: " substring
            char_hash = data_hash[:character]
            Character.create("uuid": char_hash[:uuid], "name": char_hash[:name], "hp": char_hash[:hp], "magic": char_hash[:magic])
        end
    end
end

# Receiver app
# /config/routes.rb

Rails.application.routes.draw do
  get 'listener' => 'listener#index'
end

Thus, when we start the server with rails server -p 3001 and send a Get request with curl -i http://localhost:3001/listener, we call the ListenerController's index method.

Here, index sends a Get request to our Broadcaster server at localhost:3000/broadcaster. But instead of seeing the asynchronous behavior we saw before, it all arrives at once:

So, instead of parsing each character as they come in as separate objects, we have to split the res.body into separate strings. And of course, we have to wait until all 5 characters are finished generating before we receive them. So much for scaling it up to send an unlimited number of characters!

Where I'm At

From the research I've done, I think the async behavior is being limited by Rails' use of the standard HTTP request/response cycle as the basis for ActionController::Live. As such, each request only gets one response, and that's why all the characters have to come back as one res.body string!

Per this excellent article by Eric Bidelman covering SSEs in HTML5, I thought I was moving toward implementing long polling...but apparently not.

Further, the tutorials I'm following usually expect us to build a JavaScript event listener to catch the async data from the Broadcaster server. So, is it just browser-magic that's making the ActionController::Live async behavior work in Chrome?

But then, why do the characters still appear to be coming in asynchronously when we use curl directly on localhost:3000/broadcaster...

...and NOT when using curl indirectly through localhost:3001/listener?

Any help, advice, or insight is greatly appreciated! <3

Links/Tutorials used:

• Main tutorial followed: https://tenderlovemaking.com/2012/07/30/is-it-live.html

• https://stackoverflow.com/questions/17678068/long-polling-push-events-between-rails-and-ruby-client/17707334#17707334

• https://stackoverflow.com/questions/35057075/rails-actioncontrollerlive-sends-everything-at-once-instead-of-async

• https://stackoverflow.com/questions/21763527/asynchronous-responses-in-ruby-on-rails-4-using-sse

• https://www.html5rocks.com/en/tutorials/eventsource/basics/

• https://medium.com/@fbazzarella/streaming-data-with-server-sent-events-b0694e6e0b32