Visualizing Bitcoin to USD Exchange Rates using FastAPI, Prometheus, Grafana, Deploy with jenkins

Visualizing Bitcoin to USD Exchange Rates using FastAPI, Prometheus, Grafana, Deploy with jenkins On Localhost Ubuntu Server 20.04

In this article, we’ll explore how to visualize the exchange rate of Bitcoin to USD using FastAPI, Prometheus, Grafana, and Docker. We will create a simple FastAPI application to import exchange rate data from an API, store it in a database, and expose it as metrics using Prometheus. Then, we’ll use Grafana to create dashboards that visualize the data, and deploy the whole setup using Docker and Jenkins. Inspired by the article of amlanscloud

Setup Jenkins Agent/Slave Using SSH [Password & SSH Key]

Overview

Here’s an outline of our plan:

1. Data Source: The process commences by acquiring data from a public data source. In this case, we’ll use an API that offers free exchange rates for Bitcoin to USD. The API delivers the exchange rate between Bitcoin and USD at the moment the API is called. By invoking the API multiple times, we can obtain time series data for the fluctuations in the exchange rate.

2. Data Importer App: An importer app reads the data from the data source API mentioned above. This app operates daily, invoking the data source API to acquire the exchange rate. Subsequently, the importer app stores this rate in a database. Each day’s exchange rate is represented by a row of items in the database.

3. Prometheus Data Scraper: This component is a data scraper job defined within Prometheus. The Prometheus data scraper fetches data from an API endpoint and incorporates it as a metric in Prometheus. A custom API has been developed for this purpose, which, upon invocation, retrieves the day’s exchange rate from the database and returns the data in a format that can be easily read and imported as a metric in Prometheus.

4. Visualize Data: Prometheus stores the data scraped as a distinct metric. This metric serves as a data source for creating visualization dashboards on Grafana. These dashboards display various trends in the exchange rate fluctuations as documented by the metrics.

5. Deployment: We will deploy the entire setup using Jenkins, Docker, and Docker Compose.

   from fastapi import FastAPI, Response
   from fastapi.responses import PlainTextResponse, JSONResponse
   import uvicorn
   from dotenv import load_dotenv
   import os
   import requests
   import json
   from prometheus_client import Gauge, CollectorRegistry, generate_latest
   from datetime import date
   import redis

   load_dotenv()

   def scrape_data():
   qry_url = f'{os.environ.get("STOCK_URL")}?function=CURRENCY_EXCHANGE_RATE&from_currency=BTC&to_currency=USD&apikey={os.environ.get("API_KEY")}'
   response = requests.request("GET", qry_url)
   respdata = response.json()
   rate = respdata['Realtime Currency Exchange Rate']['5. Exchange Rate']
   float_rate = "{:.2f}".format(float(rate))
   return float_rate

   registry = CollectorRegistry()
   exchange_rate_btc_usd = Gauge('exchange_rate_btc_usd', 'Exchange rate between BTC and USD', registry=registry)

   app = FastAPI()

   @app.get("/", response_class=JSONResponse)
   async def root():
   return {"message": "Hello World"}

   @app.get("/exchangemetrics", response_class=PlainTextResponse)
   async def get_exchange_metrics():
   r = redis.Redis(host="redis", port=os.environ.get('REDIS_PORT'), password=os.environ.get('REDIS_PASSWORD'), db=0)
   scraped_data = 0.0
   try:
   todays_date = str(date.today())
   redis_key = f'exchange_rate-{todays_date}'
   tmpdata = r.get(redis_key)
   scraped_data = tmpdata.decode("utf-8")
   exchange_rate_btc_usd.set(float(scraped_data))
   except Exception as e:
   print(e)
   print('responding default value')
   return Response(generate_latest(registry), media_type="text/plain")

   @app.get("/getexchangedata", response_class=PlainTextResponse)
   async def get_exchange_data():
   r = redis.Redis(host="redis", port=os.environ.get('REDIS_PORT'), password=os.environ.get('REDIS_PASSWORD'), db=0)
   scraped_data = 0.0
   respdata = "error"
   try:
   scraped_data = scrape_data()
   todays_date = str(date.today())
   r.set(f'exchange_rate-{todays_date}', scraped_data)
   respdata = "done"
   except Exception as e:
   print(e)
   print('responding default value')
   todays_date = str(date.today())
   r.set(f'exchange_rate-{todays_date}', scraped_data)
   return respdata

   if name == "main":
   uvicorn.run("main:app", host="0.0.0.0", port=5000, reload=True)

FastAPI application that fetches the Bitcoin to USD exchange rate from an external API and stores the data in a Redis database. It also exposes the exchange rate data as Prometheus metrics. Let’s break down the code:

1. Import necessary modules: The code starts by importing the required modules such as FastAPI, uvicorn, dotenv, os, requests, json, prometheus_client, datetime, and redis.

2. Load environment variables: The load_dotenv() function is called to load the environment variables from the .env file.

3. Scrape exchange rate data: The scrape_data() function fetches the exchange rate data by making a GET request to the external API using the requests library. It then extracts the exchange rate and formats it as a float with two decimal places.

4. Initialize Prometheus Gauge: A Prometheus Gauge named exchange_rate_btc_usd is initialized to store the exchange rate data.

5. Create FastAPI app: A FastAPI application named app is created.

6. Define endpoints:

• GET /: A simple Hello World endpoint that returns a JSON response.

• GET /exchangemetrics: This endpoint fetches the exchange rate data for the current day from the Redis database and sets the value of the Prometheus Gauge exchange_rate_btc_usd. It then returns the Prometheus metrics as a plain text response.

• GET /getexchangedata: This endpoint fetches the exchange rate data by calling the scrape_data() function, stores the data in the Redis database, and returns a plain text response indicating whether the operation was successful.

1. Run FastAPI app: The FastAPI app is run using uvicorn with the host set to “0.0.0.0” and port 5000. The reload=True parameter enables hot-reloading during development.

Prometheus

It sets up Prometheus to scrape metrics from two different sources: Prometheus itself and a FastAPI application. Let’s break down the configuration:

1. Global settings: The global settings apply to all the scrape jobs defined in the configuration.

• scrape_interval: The interval at which Prometheus scrapes metrics from the targets. In this case, it's set to 15 seconds.

1. Scrape configs: The scrape_configs section defines the jobs that Prometheus will use to scrape metrics from different sources.

• job_name: 'prometheus': The first job is named 'prometheus' and is configured to scrape metrics from the Prometheus server itself.

• static_configs: This section specifies the target for this job. In this case, it's set to scrape metrics from the Prometheus server running on 'localhost:9090'.

• job_name: 'fastapi_app': The second job is named 'fastapi_app' and is configured to scrape metrics from the FastAPI application.

• metrics_path: The path to the metrics endpoint in the FastAPI application, which is '/exchangemetrics' in this case.

• static_configs: This section specifies the target for this job. In this case, it's set to scrape metrics from the FastAPI application running on 'fastapi_app:5000' (the application's hostname and port).

  global:
  scrape_interval: 15s

  scrape_configs:

  • job_name: 'prometheus' static_configs:
    • targets: ['localhost:9090']
  • job_name: 'fastapi_app' metrics_path: '/exchangemetrics' static_configs:
    • targets: ['fastapi_app:5000']

Jenkins

which defines a Jenkins pipeline for building and deploying a project. The pipeline consists of four stages, and there’s a post section to execute actions after all stages have completed. Let's break down the pipeline:

1. Agent: Specifies that the pipeline will run on a Jenkins agent with the label ‘ubuntu’.

2. Stages: Contains the sequential stages to be executed in the pipeline.

• Stage 1: Build and Deploy prometheus, grafana, and redis:

• This stage changes the working directory to /home/stefen/deploy/adminer.

• It prints the current working directory and its contents.

• Then, it runs the docker-compose up -d command to deploy Prometheus, Grafana, and Redis using Docker Compose.

• Stage 2: Build and Deploy API:

• This stage waits for 30 seconds before proceeding.

• It changes the working directory to /home/stefen/deploy/api.

• It prints the current working directory and its contents.

• Then, it runs the docker-compose up --build -d command to build and deploy the FastAPI application using Docker Compose.

• Stage 3: Fetch and Print getexchangedata:

• This stage makes an HTTP request to the FastAPI application’s /getexchangedata endpoint.

• It prints the response received from the API.

• Stage 4: Fetch and Print Exchangemetrics:

• This stage makes an HTTP request to the FastAPI application’s /exchangemetrics endpoint.

• It prints the response received from the API.

1. Post: Specifies actions to be executed after all the stages have completed, regardless of their success or failure.

• In this case, it prints the URLs for Prometheus, Grafana, Redis, and the FastAPI application.

  pipeline {
  agent {
  label 'ubuntu'
  }
  stages {
  stage('Build and Deploy prometheus, grafana, and redis') {
  steps {
  dir('/home/stefen/deploy/adminer') {
  sh 'pwd' // Ajout de la commande pwd pour vérifier le répertoire de travail
  sh 'ls -la'
  sh 'docker-compose up -d'
  }
  }
  }
  stage('Build and Deploy API') {
  steps {
  sleep(time: 30, unit: 'SECONDS') // Wait for 30 seconds
  dir('/home/stefen/deploy/api') {
  sh 'pwd' // Check the current working directory
  sh 'ls -la'
  sh 'docker-compose up --build -d'
  }
  }
  }
  stage('Fetch and Print getexchangedata') {
  steps {
  script {
  def response = httpRequest 'http://localhost:5000/getexchangedata'
  echo "Response: ${response.content}"
  }
  }
  }
  stage('Fetch and Print Exchangemetrics') {
  steps {
  script {
  def response = httpRequest 'http://localhost:5000/exchangemetrics'
  echo "Response: ${response.content}"
  }
  }
  }
  }
  post {
  always {
  script {
  def prometheusURL = 'http://localhost:9090'
  def grafanaURL = 'http://localhost:3000'
  def redisURL = 'http://localhost:6379'
  def apiURL = 'http://localhost:5000'

              echo "Prometheus URL: ${prometheusURL}"
              echo "Grafana URL: ${grafanaURL}"
              echo "Redis URL: ${redisURL}"
              echo "API URL: ${apiURL}"
          }
      }
  }
  

  }

Grafana Dashboard:

Conclusion:

In conclusion, the provided code snippets and configurations showcase an end-to-end deployment process of a monitoring and data visualization system using Jenkins, Docker, Prometheus, Grafana, and FastAPI. The pipeline defined in the Jenkinsfile automates the build and deployment process, ensuring a smooth and streamlined workflow for the project. This setup allows developers to efficiently monitor and visualize Bitcoin to USD exchange rate data, making it easier to identify trends and understand the data’s behavior over time. The use of FastAPI, Redis, and the provided API ensures a robust and efficient architecture for the system, while Docker and Jenkins enable seamless deployment and automation. Overall, this project demonstrates a practical application of modern technologies to create an effective and reliable monitoring and data visualization system.

Code Source Github