Intro
I have been working with dozens of different Web APIs (mostly JSON) as a Ruby developer. Some of them can provide an official Ruby gem that you can install and use in order to interact with the API in your Ruby code. Others don't have anything like It at all and you need to implement the interaction with the API yourself. In this case, there are a lot of helpful API ruby gems that are easier to use than the default Ruby URI module https://github.com/ruby/uri. Let's list a few of them:
- Faraday - https://github.com/lostisland/faraday. Personally, I use It, just because historically it was linked to every project I worked with. In addition, you can install the Faraday Curl gem https://github.com/mauricio/faraday_curl to get nice Curl requests in logs.
- HTTPParty - https://github.com/jnunemaker/httparty. Heard a lot about It, but never used it.
- RESTClient - https://github.com/rest-client/rest-client for folks who prefer Sinatra's actions syntax.
- etc...
Those tools are just tools that you can use in order to implement the connection and interaction with any Web API. But the problem is that every API has Its interface, authorization rules, rate limiter, the way It handles errors, etc. So if in your project you need to connect 2 totally different APIs - you need to implement the interaction to each one yourself (assuming that those APIs have no gems available). If that API interaction code will not be properly encapsulated in Ruby classes, every developer in your team will need to carefully read API documentation in order to use the code that triggers those API calls. With every added API to your project, this issue will be a bigger and bigger problem with time. In this article, I will provide my own idea of how to create a simple, but powerful Ruby class pattern to hide HTTP interaction for any API in a Ruby class. This idea is based on the work that I did with my colleague years ago. Every such class will have the same external interface. As a bonus - my solution includes a way to do async HTTP requests.
Pattern
We should start with the list of Ruby gems that I am gonna use In my example:
gem 'concurrent-ruby'
gem 'faraday'
gem 'faraday_curl'
-
concurrent-ruby
- to add async requests ability. https://github.com/ruby-concurrency/concurrent-ruby -
faraday
- to do HTTP requests (you can use any approach to do HTTP requests as I mentioned in the Intro section) -
faraday_curl
- to get nice Curl requests in logs (totally optional)
The next step is to create a module that I am gonna call FaradayConnector - this module will have all the needed logic in order to make any kind of HTTP requests to Web API (JSON by default). This module will be used to mix in other modules (API-specific ones). I will add more clarity on that later in the article:
require 'faraday'
require 'faraday_curl'
require 'concurrent'
module FaradayConnector
class ServerError < RuntimeError
attr_reader :response
def initialize(response)
@response = response
super
end
end
class TimeoutError < RuntimeError; end
class ClientError < RuntimeError; end
def request
return @_request if defined?(@_request)
# creating a Promise for async approach
@_request = Concurrent::Promises.future { do_request }
end
def process
return @_process if defined?(@_process)
request
@_process = do_process
end
def as_json(_options = {})
process
end
protected
def do_request
# implement the real request in Child
end
def do_process
# implement additional response decorations in Child
request.value!
end
def url
# must be added in Child
raise 'Undefined url'
end
def auth
# must be added in Child or use nil, if API has no Authorization
raise 'Undefined auth'
end
def additional_headers
{}
end
def content_type
'application/json'
end
def request_type
:url_encoded
end
def get(path, params = {})
handle_request { connection.get(path, params) }
end
def post(path, body = {})
formatted_body = json_content? ? body.to_json : body
handle_request { connection.post(path, formatted_body) }
end
def delete(path, params = {})
handle_request { connection.delete(path, params) }
end
def put(path, body = {})
formatted_body = json_content? ? body.to_json : body
handle_request { connection.put(path, formatted_body) }
end
def timeout
45
end
def connection
@connection ||= Faraday.new(url: url) do |faraday|
faraday.request request_type
faraday.headers['Authorization'] = auth if auth
faraday.headers['Content-Type'] = content_type
faraday.headers = faraday.headers.merge(additional_headers) if additional_headers
faraday.options.timeout = timeout
faraday.response(:logger)
faraday.request :curl, Logger.new($stdout), :info
faraday.adapter Faraday.default_adapter
end
end
def handle_request
response = handle_errors { yield }
parse_response(response)
end
# just an easier way to handle HTTP errors
def handle_errors
response = yield
e = if [502, 504].include?(response.status)
TimeoutError.new(response)
elsif [500, 503].include?(response.status)
ServerError.new(response)
elsif [400, 401, 404, 422].include?(response.status)
ClientError.new(response)
end
return response unless e
raise e
end
def parse_response(response)
return {} unless response.body
json_content? ? JSON.parse(response.body) : response
rescue JSON::ParserError
{}
end
def json_content?
content_type == 'application/json'
end
end
Now, assume we need to connect 2 different APIs (I just googled 2 public ones to show as an example):
-
OpenMeteo
- https://open-meteo.com/ to get weather forecast -
SportMonks
- https://docs.sportmonks.com/football/api/ to get stats for Football teams
Every API has Its own Authorization rules, OpenMeteo has none, and SportMonks uses an Authorization token. We need to create 2 folders with modules called ApiConnector
:
-
open_meteo/api_connector.rb
require './faraday_connector'
module OpenMeteo
module ApiConnector
include FaradayConnector
def url
'https://api.open-meteo.com'
end
def auth
# no auth for this example
end
end
end
-
sport_monks/api_connector.rb
require './faraday_connector'
module SportMonks
module ApiConnector
include FaradayConnector
def url
'https://api.sportmonks.com'
end
def auth
# a token goes here based on the API documentation
# https://docs.sportmonks.com/football/welcome/authentication
'some-token'
end
end
end
With these 2 easy steps, we created a codebase that can successfully authorize those 2 APIs we are using.
Time to create some HTTP requests. Let's start with OpenMeteo. On their home page, there are 2 examples of 2 different endpoints - forecast and historical. The next step is to create a separate class for every endpoint using the already created ApiConnector
module.
-
forecast
module OpenMeteo
class Forecast
include ApiConnector
attr_reader :latitude, :longitude
def initialize(latitude, longitude)
@latitude = latitude
@longitude = longitude
end
def do_request
get('v1/forecast',
latitude: latitude,
longitude: longitude
)
end
def do_process
request.value!
# additional data manipulations goes here
end
end
end
-
historical
module OpenMeteo
class Historical
include ApiConnector
attr_reader :latitude, :longitude, :start_date, :end_date
def initialize(latitude, longitude, start_date, end_date)
@latitude = latitude
@longitude = longitude
@start_date = start_date
@end_date = end_date
end
def do_request
get('v1/era5',
latitude: latitude,
longitude: longitude,
start_date: start_date,
end_date: end_date
)
end
def do_process
request.value!
# additional data manipulations goes here
end
private
def url
# you can change url when needed easily
'https://archive-api.open-meteo.com/'
end
end
end
There are 3 essential methods here:
-
initialize
- a class constructor is only needed when an API request has input parameters -
do_request
- a place where you are triggering an HTTP request (available methods are get, post, put, delete). Remember this method returns Concurrent::Promise class instances. -
do_process
- calling value! method onConcurrent::Promise
a class instance is essential to get the request-response.
Note: You probably noticed the url method override in the second file, we need this because a second API call has a different URL, and with this pattern, it's very easy to do)
This is the way to use those classes to do requests:
forecast = OpenMeteo::Forecast.new(42.361145, -71.057083)
historical = OpenMeteo::Historical.new(42.361145, -71.057083, '2024-01-01', '2024-01-30')
# Async part
forecast.request
historical.request
puts 'historical', historical.process
puts 'forecast', forecast.process
With the async approach, it means that we are doing both requests in 2 different threads, so they will work faster. When you call process method on any of It, promise will wait until all requests will be finished before returning any results, this is the way to synchronize code back.
The output will look like this:
I, [2024-05-09T14:34:30.756703 #74264] INFO -- request: GET https://api.open-meteo.com/v1/forecast?latitude=42.361145&longitude=-71.057083
I, [2024-05-09T14:34:30.756857 #74264] INFO -- request: Content-Type: "application/json"
User-Agent: "Faraday v2.9.0"
I, [2024-05-09T14:34:30.756822 #74264] INFO -- request: GET https://archive-api.open-meteo.com/v1/era5?end_date=2024-01-30&latitude=42.361145&longitude=-71.057083&start_date=2024-01-01
I, [2024-05-09T14:34:30.761810 #74264] INFO -- request: Content-Type: "application/json"
User-Agent: "Faraday v2.9.0"
I, [2024-05-09T14:34:31.084956 #74264] INFO -- response: Status 200
I, [2024-05-09T14:34:31.085185 #74264] INFO -- response: date: "Thu, 09 May 2024 18:34:31 GMT"
content-type: "application/json; charset=utf-8"
transfer-encoding: "chunked"
connection: "keep-alive"
content-encoding: "gzip"
I, [2024-05-09T14:34:31.090995 #74264] INFO -- response: Status 200
I, [2024-05-09T14:34:31.091058 #74264] INFO -- response: date: "Thu, 09 May 2024 18:34:31 GMT"
content-type: "application/json; charset=utf-8"
transfer-encoding: "chunked"
connection: "keep-alive"
content-encoding: "gzip"
historical
{"latitude"=>42.355007, "longitude"=>-71.12906, "generationtime_ms"=>0.0050067901611328125, "utc_offset_seconds"=>0, "timezone"=>"GMT", "timezone_abbreviation"=>"GMT", "elevation"=>9.0}
forecast
{"latitude"=>42.36515, "longitude"=>-71.0618, "generationtime_ms"=>0.00095367431640625, "utc_offset_seconds"=>0, "timezone"=>"GMT", "timezone_abbreviation"=>"GMT", "elevation"=>9.0}
As we can see, both requests are running at the same time, so we are sure the async approach is working. If we just run process methods for both instances one by one, we will have a synchronized approach, that will look like this:
forecast = OpenMeteo::Forecast.new(42.361145, -71.057083)
historical = OpenMeteo::Historical.new(42.361145, -71.057083, '2024-01-01', '2024-01-30')
puts 'historical', historical.process
puts 'forecast', forecast.process
I, [2024-05-09T14:37:15.830733 #74386] INFO -- request: GET https://archive-api.open-meteo.com/v1/era5?end_date=2024-01-30&latitude=42.361145&longitude=-71.057083&start_date=2024-01-01
I, [2024-05-09T14:37:15.830765 #74386] INFO -- request: Content-Type: "application/json"
User-Agent: "Faraday v2.9.0"
I, [2024-05-09T14:37:16.154027 #74386] INFO -- response: Status 200
I, [2024-05-09T14:37:16.154220 #74386] INFO -- response: date: "Thu, 09 May 2024 18:37:16 GMT"
content-type: "application/json; charset=utf-8"
transfer-encoding: "chunked"
connection: "keep-alive"
content-encoding: "gzip"
historical
{"latitude"=>42.355007, "longitude"=>-71.12906, "generationtime_ms"=>0.0050067901611328125, "utc_offset_seconds"=>0, "timezone"=>"GMT", "timezone_abbreviation"=>"GMT", "elevation"=>9.0}
I, [2024-05-09T14:37:16.157443 #74386] INFO -- request: GET https://api.open-meteo.com/v1/forecast?latitude=42.361145&longitude=-71.057083
I, [2024-05-09T14:37:16.157511 #74386] INFO -- request: Content-Type: "application/json"
User-Agent: "Faraday v2.9.0"
I, [2024-05-09T14:37:16.495799 #74386] INFO -- response: Status 200
I, [2024-05-09T14:37:16.496003 #74386] INFO -- response: date: "Thu, 09 May 2024 18:37:16 GMT"
content-type: "application/json; charset=utf-8"
transfer-encoding: "chunked"
connection: "keep-alive"
content-encoding: "gzip"
forecast
{"latitude"=>42.36515, "longitude"=>-71.0618, "generationtime_ms"=>0.00095367431640625, "utc_offset_seconds"=>0, "timezone"=>"GMT", "timezone_abbreviation"=>"GMT", "elevation"=>9.0}
Now let's get back to the second API that we planned to connect. It has a lot more examples that you can check out here https://docs.sportmonks.com/football/api/demo-response-files. To be very simple, let's say we want to get data about some team with an id 8
.
module SportMonks
class Team
include ApiConnector
attr_reader :team_id
def initialize(team_id)
@team_id = team_id
end
def do_request
get("v3/football/teams/#{team_id}")
end
def do_process
request.value!
# additional data manipulations goes here
end
end
end
team = SportMonks::Team.new(8)
team.request
team.process
Same interface, but a different API.
Summary
Feel free to use this pattern for your API needs. I have been using this for years for a wide variety of services. I prefer using this, then connecting API-specific gems. From my perspective, this approach has a few benefits:
- integrated async way of doing requests
- same external interface for every API you will use in your codebase, which makes It very easy to use by any developer in your team
- once ApiConnector is created with configured authorization, you will need to think about It ever again. (always use a secure way to store tokens or other secrets). If you don't know what to use, I can suggest dotenv gem - https://github.com/bkeepers/dotenv
- flexible request configuration that is encapsulated in one class with a method override approach
PS: It's a copy of my original post on Medium - https://medium.com/@zozulyak.nick/ruby-class-pattern-to-work-with-api-requests-with-built-in-async-approach-bf0713a7dc96
Top comments (3)
Nicely done. Thanks for sharing.
Great write-up.
Good post!