So, if you have read my intro post when I started this blog like forever ago (Feb 7, 2021 to be exact), then you know that I’m super excited about a project called Atomizer. I’ve been slowly stringing you (my readers) along for the last few posts to build up some background so that I can reference it when I start talking about MPDC (Massively Parallel Distributed Computing).
When I built out Atomizer with a team of students (shout out to Matthew, Megan, and Nick) at the University of Illinois Springfield, we tested it with what is known as a Monte Carlo π simulation. The Monto Carlo π simulation showed that the system functioned as a distributed computing framework using a mix of node- and cluster-level parallelization. Monte Carlo π simulations are commonly used to explain how algorithms can be parallelized and distributed, so it worked well for those purposes.
What is a Monte Carlo π simulation?
Assume that you had a square dartboard, and there was a circle inside that square that met the edges of the square (See Figure 1).
If you throw darts at the dart board then calculate the number of darts that landed in the circle compared to the total number of darts thrown, then you can estimate the value of π.
How is π estimated using this simulation?
π = 4 x ( # darts in circle / total # of tosses)
The equation for this simulation can be parallelized by making each throw of the dart its own process. It 1) calculates its “toss” or random x/y coordinates, 2) determines if the toss is in the circle, and 3) reports back to the original process. This algorithm allows for each toss to happen in its own “simulation.”
Once the parent process receives the results of all processes, it can then calculate the final π value.
Try It Out Yourself!
So, in honor of π day 2021, I’ve set up Atomizer so you can try a Monte Carlo π simulation yourself!
Prerequisites:
Once you have the above applications installed on your system it’s super easy to run your own simulation using an Atomizer cluster.
Clone the Atomizer Test Agent Repository
From your terminal or git UI clone the following repo and move into the atomizer-test-agent directory:
git clone https://github.com/devnw/atomizer-test-agent
cd ./atomizer-test-agent
Execute the Docker Compose File to Start the Cluster
Execute the following code in a terminal window from the atomizer-test-agent directory. Figure 2 shows what this cluster looks like graphically.
NOTE: You can adjust the number of nodes docker-compose starts by adjusting the --scale atomizer=3 argument from 3 to whatever number you want.
# Pull down the docker images from docker hub for this compose file
docker-compose pull
# Spin up the rabbitmq instance and 3 node Atomizer cluster (pre-loaded with the Monte Carlo π simulation)
docker-compose up --scale atomizer=3
If docker-compose did not work, click here to visit the docker-compose trouble shooting section.
NOTE: You may see messages similar to those below when executing this command. These can be ignored as it indicates that the RabbitMQ instance has not yet fully started listening on the network for the Atomizer nodes to connect to. If you see these kinds of messages, wait 30 seconds or so before moving on to the next step.
# These are normal log messages to see which indicate
# the Atomizer nodes are working on connecting to the
# RabbitMQ instance
waiting for host [atomrabbit:5672], attempt 3 failed
waiting for host [atomrabbit:5672], attempt 4 failed
waiting for host [atomrabbit:5672], attempt 5 failed
Interact with the Cluster through the Test Console
From _ another _ terminal window install the atomizer-test-console using the following command specific to your version of Go:
# For Go 1.16
go install atomizer.io/test-console@latest
# For Other Go Versions
go get atomizer.io/test-console@latest
After executing these commands the test-console should have been installed by the Go tool. From here execute the following command, which will connect to the cluster:
test-console
NOTE: If this doesn’t work, click here to visit the troubleshooting section.
The cluster uses default connection information for RabbitMQ so the test-console should immediately connect and you should see something similar to this:
Enter Tosses: _
Enter the number of tosses you want the simulation to run and hit enter.
Results
The results of the simulation should be displayed fairly quickly depending on the number of tosses you entered. Look at the test-console once the calculation finishes, and it should show something similar to the example below:
Enter Tosses: 500000
Pi Estimation: 3.151984
Since Monte Carlo π is a simulation, it is important to point out that the π estimation that results are not going to be exact and will not be the same every time – even if the number of tosses is the same. Also, the π estimation will be more accurate the higher the number of tosses.
Fair warning though, the Atomizer system is intended to run on multiple systems not a singular system, and it will very quickly consume a large portion of your system’s CPU if the number of tosses is high.
What is happening under the covers?
Monitoring the Cluster
The docker-compose file that is included in the atomizer-test-agent repository has disabled logging. This was done to maximize the throughput of the Atomizer cluster, since IO can greatly impede performance if it’s unnecessary to the functionality of the application.
If you want to see what the cluster is doing, though, you can re-enable logging by updating the docker-compose.yml file and adding - LOGLEVEL=INFO in the environment section of the file like this:
atomizer:
image: 'atomizer/test-agent:latest'
depends_on:
- "rabbit-mq"
environment:
- CONNECTIONSTRING=...
- QUEUE=...
- LOGLEVEL=INFO
This setting creates several listeners in the atomizer-test-agent which will display events as they pass through the communication elements of the AMQP system as well as the Atomizer engine itself. Once you’ve made this change, then re-run the docker-compose command to stand up your cluster and re-run the simulation with the test-console.
NOTE: If you stop and re-start the cluster, you will need to stop and re-start the test-console in order to re-connect to the new RabbitMQ instance.
RabbitMQ Management Console
The docker-compose.yml file is set up to use a specific version of RabbitMQ which exposes the RabbitMQ management console. If you want to view the management console and see the messages as they go through RabbitMQ while running a simulation, then navigate to http://localhost:8080/
At the login screen enter the default credentials guest / guest and click Login.
Under the Queues section you will see that there is an atomizer queue, and, once you start a simulation, you will see a couple more queues show up in the list, similar to what you see in Figure 4.
Each of the three queues are important. The primary queue – where requests are pulled from for each node – is the atomizer queue. Then, one of the other two queues is the queue which the test-console listens to, and the third is the one the montecarlopi atom listens to for the results of the toss simulations.
The coordination between these queues allows for messages to flow to the appropriate destinations so that results can be calculated and viewed by the user.
Cleanup
After you are done running the simulation, don’t forget to cleanup your environment by running the following commands:
docker-compose stop
docker-compose rm
Troubleshooting
Docker Compose Not Working
If the docker-compose command is not working as expected, here is a set of docker commands you can run instead to stand up a three node cluster.
NOTE: The following commands add logging to stdout. If you want to disable logging and improve performance, remove -e LOGLEVEL=INFO from the command.
# Setup a Docker network for the cluster to communicate on
docker network create atomizer_nw
# Setup the RabbitMQ container instance
docker run -d --rm=true --hostname my-rabbit --name some-rabbit -p 8080:15672 -p 5672:5672 --network=atomizer_nw rabbitmq:3-management
# Setup each Atomizer node in a different terminal, or run each command with the `-d` flag to ignore it's output
docker run --rm=true -e CONNECTIONSTRING=amqp://guest:guest@some-rabbit:5672/ -e QUEUE=atomizer -e LOGLEVEL=INFO --network=atomizer_nw atomizer/test-agent:latest
docker run --rm=true -e CONNECTIONSTRING=amqp://guest:guest@some-rabbit:5672/ -e QUEUE=atomizer -e LOGLEVEL=INFO --network=atomizer_nw atomizer/test-agent:latest
docker run --rm=true -e CONNECTIONSTRING=amqp://guest:guest@some-rabbit:5672/ -e QUEUE=atomizer -e LOGLEVEL=INFO --network=atomizer_nw atomizer/test-agent:latest
Test Console Not Working
If the atomizer test-console did not work when you tried to run it, then pull the repository down and build it directly using the following commands:
git clone https://github.com/devnw/atomizer-test-console
cd ./atomizer-test-console
go build .
./test-console
This will build the test-console and allow for direct execution.
Top comments (0)