Greeting
Hello #devElixir!!! Welcome to #FullStackElxpro
Here we discuss strategies and tips for your Elixir learning journey. This is from scratch until your first wave as an elixir developer
Do you want to learn Elixir in three months? https://elxpro.com/sell
I am Gustavo and today's theme is: Distributed Systems in Elixir
_ps: You can follow the article with video
Which are Distributed Systems?
A distributed system is a collection of computer programs that utilize computing resources at several different central computing points to achieve a shared goal. Distributed systems centralized bottlenecks or central points of failure of a system.
In this case, in the language, we chose (Elixir). We will use the resources of NODE
https://hexdocs.pm/elixir/1.12/Node.html
Why is it so important to understand?
What is the principle of programming in Elixir? And what are the advantages?
Everyone who chooses to work with Elixir is always the same subject:
- Functional Language
- competition
- Parallelism
The important thing is what I see that could be a trend in the future in advanced Elixir developers and use mainly of concurrency and parallelism in distributed systems to decrease the cost in a company.
And how does Elixir help us with distribution?
A list of frameworks is below, but before using these frameworks, you need to understand a very important concept.
MNESIA - https://elixirschool.com/en/lessons/storage/mnesia
PUBSUB - https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html
LibCluster - https://github.com/bitwalker/libcluster
And reading this article: https://dashbit.co/blog/you-may-not-need-redis-with-elixir
I could talk about MNESIA, Pubsub, and Libcluster, but before you understand and before you start distributing systems with Elixir you need to understand what's behind all these frameworks.
The why and in case you still don't understand the distribution of systems It is low cost and uses performance of computer systems and hardware of a fantastic language that is not mainly and Elixir
How did I come to that conclusion?
I arrived at this final computation, this solver different one Feature had three applications running (in Kubernetes) I needed to read a file and send this to an external service, and an external service that saves a file to the database when the service was running on staging, sometimes these files were not read, so we tried to understand what the data was doing sometimes.
Our problem is that we had 3nodes running and the file was sometimes sent to the batch at the time of background processing, and when trying to read the file it was not healthy.
The simplest solution for this case would be to save this data in Storage (S3). When we talk about a functional language and a powerful language that facilitates concurrency, parallelism and distribution, 3 minutes to read files is a lot of time and that's where the idea for the next version came from using distributed systems and few resources. These resources that require a lot of knowledge of how Elixir works behind the scenes that I've come to complete and I want to share my experience with you and how you can make decisions after this article.
Getting Started with Elixir Distribution
This article is an example of how to use Nodes and RPC. Our first step is to know how to creators and RPC (remote procedure call).
Let's start with IEX
iex --sname gus #this way you create a shortname
iex --name gus@ip #this way you create a name
the biggest difference is access between the internet and local.
Let's learn some of the commands above:
Node.ping: app_name
You will do a ping checking if it is possible to connect with the APP
Node.connect: app_name
You will connect to a Node.
list of nodes
you will see the list of connections you have. What's interesting is that in this case when you're going to connect the nodes, they create communication between all of them. as you see in the Last Node.list
Calling functions.
Before we start talking about processes, something that Elixir gives us for free is a :rpc. something that is very complex in other applications.
Let's see the scenario below:
In the example above we have the RPC that follows the following pattern to be called:
:rpc.call app_name, MODULE, :function, [parameters]
And look at the module and when it exists, the stop for the interesting node that is always not there.
Using processes in Calls.
You can create a project in Elixir using
mix new product_stock
defmodule ProductStock do
def handle_stock(stock) do
receive do
{:add, product} ->
stock_updated = [product] ++ stock
handle_stock(stock_updated)
:status ->
IO.inspect stock
handle_stock(stock)
end
end
end
Let's start with the start of the application:
❯ iex --sname product_stock -S mix
in this example we start a normal application in elixir but with a short-name
a process has been created and registered with a name.
> pid = generate ProductStock, :handle_stock, [[]]
> Process.register pid, :product_stock
ps: The article below will help you understand more about spawn, send and receive
https://www.youtube.com/watch?v=em4QECkQx4s&t=801s
And now we start sending data to update our stock, but in the same module, we started the application.
> send :product_stock, :status
> send :product_stock, {:add, %{name: "nike", qty: 30}}
> send :product_stock, :status
> send :product_stock, {:add, %{name: "adidas", qty: 35}}
What is interesting to note is what is done after connecting to the server application. And how are we going to update the stock. You will notice that to call a process in another app you need to include in your send the following structure.
send {:pid_name, :app_name}, message
And when calling N times the service always goes to the app server as shown in the image above.
Handling Cross-Process Responses
defmodule ProductStock do
def handle_stock(stock) do
receive do
{:add, from, product} ->
stock_updated = [product] ++ stock
send(from, stock_updated)
handle_stock(stock_updated)
{:status, from} ->
send(from, stock)
handle_stock(stock)
end
end
end
In the example above, we either update our code to always return the response to some process whatever. We use functions like Process.info(PID, :messages) to know if you have messages and flush to read as messages.
About communication of Nodes and Processes
defmodule ProductStock do
def handle_stock(stock) do
receive do
{:add, from, product} ->
stock_updated = [product] ++ stock
log(from, :add)
send(from, stock_updated)
handle_stock(stock_updated)
{:status, from} ->
log(from, :status)
send(from, {:show, stock})
handle_stock(stock)
end
end
defp log(pid, :add) do
IO.inspect("#{:erlang.pid_to_list(pid)} added a new item on stock")
end
defp log(pid, :status) do
IO.inspect("#{:erlang.pid_to_list(pid)} checked the stock")
end
end
In the image, then we apply what we learned above from this entire article. What has changed is the communication between the nodes and processes now whenever a PID calls a pid you can see that it has a number like: <19214.112.> which translates to: .
Let's share a little bit about distribution.
Imagine that you have a central to stock the products of several stores, and every time a customer requests a product in the store, the stores check if the central has the product, and return the message to the customer. Follow the example below.
defmodule ProductStock do
def handle_stock(stock) do
receive do
{:add, from, product} ->
stock_updated = [product] ++ stock
log(from, :add)
send(from, stock_updated)
handle_stock(stock_updated)
{:status, from} ->
log(from, :status)
send(from, stock)
handle_stock(stock)
end
end
defp log(pid, :add) do
IO.inspect("#{:erlang.pid_to_list(pid)} added a new item on stock")
end
defp log(pid, :status) do
IO.inspect("#{:erlang.pid_to_list(pid)} checked the stock")
end
end
defmodule Pos do
def ask() do
receive do
{:ask, from, product} ->
stock_address = :"server@Gustavos-MacBook-Pro"
stock_info = {:server, stock_address}
IO.inspect("#{:erlang.pid_to_list(from)} wants to know if get a product")
send(stock_info, {:status, self()})
return_itens(from, product)
end
ask()
end
def return_itens(from, product) do
receive do
products ->
case Enum.find(products, &(&1.name == product)) do
nil -> send(from, {:print, "There is no Product #{product}"})
product -> send(from, {:print, product})
end
end
end
def show_msg do
receive do
{:print, msg} -> msg
end
end
end
In the example above, if you tested the code. You got the following result:
- The server saves the products (process state)
- Stores, change state (adding or searching for products)
- And customers consult the products.
I hope I helped, the learning continues and Monday we will have this article.
References
https://elixir-lang.org/getting-started/mix-otp/distributed-tasks.html
Top comments (2)
Thank you so much. This is exatly what I was looking for.
Wow. Finally found someone blogging about distributed systems in elixir.