DEV Community

megared05
megared05

Posted on • Originally published at chainsight.network

Chainsight Hands-on: Generate original indicators from data in indexers

#blockchain #tutorial #oracle #rust

We have released three hands-on sessions so far, all of which involved creating components to "collect data.” By using the Snapshot Indexer in the Chainsight Platform, we were able to retrieve all kinds of data from various locations. In this hands-on session, we will build a component that generates a new index/indicator by performing a unique calculation using the data stored by the Snapshot Indexer as input.

This project uses Ethereum price data collected by multiple Snapshot Indexers and calculates their average value to calculate a reliable Ethereum price. The component used as the calculator is Algorithm Lens. In this case, we are collecting Ethereum price data from three Snapshot Indexers. By using different locations for each of these data sources, we aim to improve the reliability of the calculation results.

Completed Component Definition

Algorithm Lens, which performs the calculation, obtains the latest Ethereum price data from the three Snapshot Indexers and performs the calculation using those data as input. The configuration diagram is shown below.

chainsight_deepdive_into_showcase-handson-algorithm_lens.png

Prerequisite

The Chainsight CLI requires several tools and a csx installation. Please refer to the following articles for this preliminary preparation.

https://dev.to/hide_yoshi/step-by-step-creating-an-evm-price-oracle-with-chainsight-469g

Create a project

As before, build from an empty project; use csx new --no-samples to create a template.

% csx new --no-samples lens-project
Feb 06 02:00:18.697 INFO Start creating new project 'lens-project'...
Feb 06 02:00:18.698 INFO Project 'lens-project' created successfully
Feb 06 02:00:18.698 INFO You can add components with:

  cd lens-project && csx add
Enter fullscreen mode Exit fullscreen mode

Add Snapshot Indexers

This time, we will build three Snapshot Indexers, two of which are Snapshot Indexer EVMs. Two of them are Snapshot Indexer EVMs that use Mainnet's chainlink contract and Arbitrum One's API3 contract as data sources. The last one is Snapshot Indexer HTTPS, which uses coingecko's API as its data source. Please add the following manifest and update project.yaml.

# components/chainlink_eth_indexer.yaml
...
datasource:
  location:
        # Chainlink: ETH/USD Price Feed
    id: 5f4eC3Df9cbd43714FE2740f5E3616155c5b8419
    args:
      network_id: 1
      rpc_url: https://eth-mainnet.g.alchemy.com/v2/${ALCHEMY_MAINNET_KEY}
  method:
    identifier: latestRoundData():(uint80,int256,uint256,uint256,uint80)
    interface: EACAggregator.json
    args: []
interval: ${INTERVAL}
cycles: null
Enter fullscreen mode Exit fullscreen mode 
# components/api3_eth_indexer.yaml
...
datasource:
  location:
        # Api3ServerV1
    id: 709944a48caf83535e43471680fda4905fb3920a
    args:
      network_id: 42161
      rpc_url: https://arb-mainnet.g.alchemy.com/v2/${ALCHEMY_ARBITRUM_ONE_KEY}
  method:
    identifier: dataFeeds(bytes32):(int224,uint32)
    interface: Api3ServerV1.json
    args: [0x4385954e058fbe6b6a744f32a4f89d67aad099f8fb8b23e7ea8dd366ae88151d]
interval: ${INTERVAL}
cycles: null
Enter fullscreen mode Exit fullscreen mode 
# components/web_eth_indexer.yaml
...
datasource:
  url: https://api.coingecko.com/api/v3/simple/price
  headers:
    Content-Type: application/json
  queries:
    type: static
    value:
      ids: ethereum
      vs_currencies: usd
      include_market_cap: true
      include_24hr_vol: true
      include_24hr_change: true
      include_last_updated_at: true
      precision: 18
interval: ${INTERVAL}
cycles: null
Enter fullscreen mode Exit fullscreen mode 
# project.yaml
version: v1
label: eth-lens-project
components:
- component_path: components/chainlink_eth_indexer.yaml
- component_path: components/api3_eth_indexer.yaml
- component_path: components/web_eth_indexer.yaml
Enter fullscreen mode Exit fullscreen mode

I will add a supplement to the manifest you added.

In chainlink_eth_indexer, the Price Feed Contract for Mainnet's Chainlink is specified. Since the ABI used in the interface field declared in the manifest is required, copy the ABI from the link below, create interfaces/EACAggregator.json, and paste the copied contents directly into the file to create the ABI file.

https://etherscan.io/address/0x5f4eC3Df9cbd43714FE2740f5E3616155c5b8419#code

The api3_eth_indexer specifies the Contract for API3 of Arbitrum One. The ABI file must be placed here as well. Copy the ABI from the link below and create an ABI file interfaces/Api3ServerV1.json.

https://arbiscan.io/address/0x709944a48caf83535e43471680fda4905fb3920a#code

web_eth_indexer specifies Coingecko's API.

The Snapshot Indexer EVM manifest also sets up an RPC URL to communicate with the blockchain, and in most cases when using RPC, the private RPC endpoint is generated using the Platform associated with Web3 and used. This is accompanied by a special key, which cannot be made public. To enable such RPCs without problems, environment variables are available in the Chainsight manifest. For the above manifest, generate the following .env file.

INTERVAL=60
ALCHEMY_MAINNET_KEY=(YOUR_KEY)
ALCHEMY_ARBITRUM_ONE_KEY=(YOUR_KEY)
Enter fullscreen mode Exit fullscreen mode

Variables declared in this .env file are expanded when the manifest is read.

You have now completed creating the manifest for Snapshot Indexer! Make sure your folder structure looks like this

% ls
artifacts   components  interfaces  project.yaml    src
% ls -1 components
api3_eth_indexer.yaml
chainlink_eth_indexer.yaml
web_eth_indexer.yaml
% ls -1 interfaces 
Api3ServerV1.json
EACAggregator.json
Enter fullscreen mode Exit fullscreen mode

Once you have done this, you can build it by running csx build.

% csx build
Feb 11 01:38:07.827 INFO Start building project 'lens-project'
Feb 11 01:38:07.828 INFO Load env file: "./.env"
Feb 11 01:38:07.828 INFO Start code generation for project 'lens-project'
Feb 11 01:38:07.828 INFO Load env file: "./.env"
Feb 11 01:38:07.832 INFO [chainlink_eth_indexer] Start processing...
Feb 11 01:38:07.833 INFO [chainlink_eth_indexer] Interface file 'EACAggregator.json' copied from user's interface
Feb 11 01:38:08.387 INFO [chainlink_eth_indexer] Generate interfaces (.did files) ...
Feb 11 01:38:09.684 INFO [chainlink_eth_indexer] Succeeded: Generate interfaces (.did files)
Feb 11 01:38:09.684 INFO [api3_eth_indexer] Start processing...
Feb 11 01:38:09.685 INFO [api3_eth_indexer] Interface file 'Api3ServerV1.json' copied from user's interface
Feb 11 01:38:09.778 INFO [api3_eth_indexer] Generate interfaces (.did files) ...
Feb 11 01:38:10.533 INFO [api3_eth_indexer] Succeeded: Generate interfaces (.did files)
Feb 11 01:38:10.533 INFO [web_eth_indexer] Start processing...
Feb 11 01:38:11.203 INFO [web_eth_indexer] Generate interfaces (.did files) ...
Feb 11 01:38:12.087 INFO [web_eth_indexer] Succeeded: Generate interfaces (.did files)
Feb 11 01:38:12.087 INFO Project 'lens-project' codes/resources generated successfully
Feb 11 01:38:12.087 INFO Start building...
Feb 11 01:38:12.089 INFO Compile canisters...
Feb 11 01:38:15.098 INFO Succeeded: Compile canisters
Feb 11 01:38:15.098 INFO Shrink/Optimize modules...
Feb 11 01:38:15.389 INFO Succeeded: Shrink/Optimize modules
Feb 11 01:38:15.389 INFO Add metadata to modules...
Feb 11 01:38:15.640 INFO Succeeded: Add metadata to modules
Feb 11 01:38:15.640 INFO Project 'lens-project' built successfully
Enter fullscreen mode Exit fullscreen mode

When "built successfully" is displayed as shown above, it is OK. The implementation of Snapshot Indexer is now complete.

Algorithm Lens: Add a manifest

Here's the heart of this hands-on: let's build a component in Algorithm Lens that serves as a calculator to calculate the price of Ethereum by averaging multiple data sources.

Add the component with csx add, but this time select the algorithm_lens type.

% csx add
✔ Please input Component Name to add · eth_average_price_calculator
✔ Please select Component Type to add · algorithm_lens
Feb 11 06:16:01.723 INFO Start creating new component 'eth_average_price_calculator'...
Feb 11 06:16:01.727 INFO AlgorithmLens component 'eth_average_price_calculator' added successfully
Enter fullscreen mode Exit fullscreen mode

If this command completes without problems, you will see the following manifest in your project.

# components/eth_average_price_calculator.yaml
...
datasource:
  methods:
  - id: sample_snapshot_indexer_icp
    identifier: 'get_last_snapshot : () -> (record { value : text; timestamp : nat64 })'
    candid_file_path: null
...
Enter fullscreen mode Exit fullscreen mode

Edit this manifest to create a component that can retrieve data from the three Snapshot Indexers built in the previous section. Change the datasource field as follows

datasource:
  methods:
  - id: chainlink_eth_indexer
    identifier: 'get_last_snapshot_value : () -> (record { nat; text; text; text; nat })'
    func_name_alias: chainlink_eth_indexer
  - id: api3_eth_indexer
    identifier: 'get_last_snapshot_value : () -> (record { text; nat32 })'
    func_name_alias: api3_eth_indexer
  - id: web_eth_indexer
    identifier: 'get_last_snapshot_value : () -> (SnapshotValue)'
    func_name_alias: web_eth_indexer
Enter fullscreen mode Exit fullscreen mode

Create a record for each Snapshot Indexer.

In the id field, enter the id of the Snapshot Indexer. (The id is determined from the filename of the YAML manifest that defines the component.)

The identifier is a Candidate for the interface of the function when calling the component. get_last_snapshot_value is one of several fixed interfaces built into Snapshot Indexer. The type of the return value depends on the data structure managed by the component. For further understanding, please refer to the following articles.

Behind Chainsight: A Look Into Snapshot Indexer Component | by Chainsight | Feb, 2024 | Medium

func_name_alias is an optional parameter, but is set for code clarity in the logic to be implemented later. The value specified here will be used as the automatically generated function name.

Algorithm Lens: Generate a template code & make it into your logic

Let's generate code with csx generate.

% csx generate
Feb 11 14:02:32.560 INFO Start code generation for project 'lens-project'
Feb 11 14:02:32.560 INFO Load env file: "./.env"
Feb 11 14:02:32.562 INFO [chainlink_eth_indexer] Start processing...
...
Feb 11 14:02:34.298 INFO [eth_average_price_calculator] Start processing...
Feb 11 14:02:34.380 INFO [eth_average_price_calculator] Generate interfaces (.did files) ...
Feb 11 14:02:36.470 INFO [eth_average_price_calculator] Succeeded: Generate interfaces (.did files)
Feb 11 14:02:36.470 INFO Project 'lens-project' codes/resources generated successfully
Enter fullscreen mode Exit fullscreen mode

When the above generation is complete, the following template code is generated in the eth_average_price_calculator logic code file.

// src/logics/eth_average_price_calculator/src/lib.rs
use eth_average_price_calculator_accessors::*;
#[derive(Clone, Debug, Default, candid :: CandidType, serde :: Deserialize, serde :: Serialize)]
pub struct LensValue {
    pub dummy: u64,
}
pub async fn calculate(targets: Vec<String>) -> LensValue {
    let _result = get_chainlink_eth_indexer(targets.get(0usize).unwrap().clone()).await;
    let _result = get_api3_eth_indexer(targets.get(1usize).unwrap().clone()).await;
    let _result = get_web_eth_indexer(targets.get(2usize).unwrap().clone()).await;
    todo!()
}
Enter fullscreen mode Exit fullscreen mode

There are several module imports, one structure (struct LensValue) and one function (fn calculate). You can customize these structs and functions to turn them into your own logic. It may be easier to understand if you think of it as a fixed interface whose contents can be customized.

The structure LensValue is the output type of the logic used by the Algortihm Lens. In this case, a scalar value, Float, will be fine since it returns the result of calculating the average value. Modify as follows

pub struct LensValue {
-    pub dummy: u64,
+    pub value: f64,
}
Enter fullscreen mode Exit fullscreen mode

We will then touch on the main calculation logic. As you can imagine from the function name, the function declared from the beginning is the function to retrieve data from the data source.

...
    let _result = get_chainlink_eth_indexer(targets.get(0usize).unwrap().clone()).await;
    let _result = get_api3_eth_indexer(targets.get(1usize).unwrap().clone()).await;
    let _result = get_web_eth_indexer(targets.get(2usize).unwrap().clone()).await;
...
Enter fullscreen mode Exit fullscreen mode

Let's use the values obtained here as inputs to the calculation to compute the average. After processing each input to f64 (64-bits float), adjust the number of digits in each data and calculate the average value.

use std::str::FromStr;
...
pub async fn calculate(targets: Vec<String>) -> LensValue {
    let result = get_chainlink_eth_indexer(targets.get(0usize).unwrap().clone()).await;
    let price_from_chainlink = result.unwrap().1;

    let result = get_api3_eth_indexer(targets.get(1usize).unwrap().clone()).await;
    let price_from_api3 = result.unwrap().0;

    let result = get_web_eth_indexer(targets.get(2usize).unwrap().clone()).await;
    let price_from_web = result.unwrap().ethereum.usd;

    let prices = vec![
        f64::from_str(&price_from_chainlink).unwrap() / 10f64.powi(8),
        f64::from_str(&price_from_api3).unwrap() / 10f64.powi(18),
        price_from_web,
    ];
    let average = prices.iter().sum::<f64>() / prices.len() as f64;

    LensValue { value: average }
}
Enter fullscreen mode Exit fullscreen mode

Since the chainlink price data is adjusted to 8 digits and the API3 data is adjusted to 18 digits, we have set this back to an unscaled value. In order to see the actual values of the input data, you can use explorer for blockchain data or make API calls for web data. If you have already deployed Snapshot Indexer, you can also check the data by calling the implemented reference functions.

chainsight-handson-lens-1.png

Source: https://etherscan.io/address/0x5f4eC3Df9cbd43714FE2740f5E3616155c5b8419#readContract

chainsight-handson-lens-2.png

Source: https://arbiscan.io/address/0x709944a48caf83535e43471680fda4905fb3920a#readContract

This completes the logic construction. The final code should look like this If this is verified locally to you, development is complete!

// src/logics/eth_average_price_calculator/src/lib.rs
use eth_average_price_calculator_accessors::*;
use std::str::FromStr;
#[derive(Clone, Debug, Default, candid :: CandidType, serde :: Deserialize, serde :: Serialize)]
pub struct LensValue {
    pub value: f64,
}
pub async fn calculate(targets: Vec<String>) -> LensValue {
    let result = get_chainlink_eth_indexer(targets.get(0usize).unwrap().clone()).await;
    let price_from_chainlink = result.unwrap().1;

    let result = get_api3_eth_indexer(targets.get(1usize).unwrap().clone()).await;
    let price_from_api3 = result.unwrap().0;

    let result = get_web_eth_indexer(targets.get(2usize).unwrap().clone()).await;
    let price_from_web = result.unwrap().ethereum.usd;

    let prices = vec![
        f64::from_str(&price_from_chainlink).unwrap() / 10f64.powi(8),
        f64::from_str(&price_from_api3).unwrap() / 10f64.powi(18),
        price_from_web,
    ];
    ic_cdk::println!("{:?}", &prices);
    let average = prices.iter().sum::<f64>() / prices.len() as f64;

    LensValue { value: average }
}
Enter fullscreen mode Exit fullscreen mode

Algorithm Lens: Build & Deploy

Now let's build, generate modules, and deploy as before. This time, be careful about the amount of Cycles remaining since there are many canisters to deploy. By default, about 6T Cycles are required per component.

csx build
csx deploy --network ic
csx exec --network ic
Enter fullscreen mode Exit fullscreen mode

Execute the above command and if the following log is obtained, execution is complete!

% csx build
Feb 11 23:13:55.148 INFO Start building project 'lens-project'
Feb 11 23:13:55.148 INFO Load env file: "./.env"
Feb 11 23:13:55.148 INFO Start code generation for project 'lens-project'
Feb 11 23:13:55.148 INFO Load env file: "./.env"
Feb 11 23:13:55.150 INFO [chainlink_eth_indexer] Start processing...
...
Feb 11 23:13:55.758 INFO [api3_eth_indexer] Start processing...
...
Feb 11 23:13:56.376 INFO [web_eth_indexer] Start processing...
...
Feb 11 23:13:56.973 INFO [eth_average_price_calculator] Start processing...
...
Feb 11 23:13:58.084 INFO Project 'lens-project' codes/resources generated successfully
Feb 11 23:13:58.084 INFO Start building...
...
Feb 11 23:14:05.212 INFO Project 'lens-project' built successfully

% csx deploy --network ic
Feb 11 23:14:09.194 INFO Checking environments...
...
Feb 11 23:14:53.726 INFO Current deployed status:
Canister Name: api3_eth_indexer
  Canister Id: {"local": "br5f7-7uaaa-aaaaa-qaaca-cai"}
  Controllers: 7fpuj-hqaaa-aaaal-acg7q-cai be2us-64aaa-aaaaa-qaabq-cai k245a-2cvyi-rcnzj-cpq6h-dcy7o-qrg4b-325bv-5c3om-jfceo-ikmjz-aqe
  Module Hash: 0xf11c7d0d3f0cfec06ea1d7d7d6ac2ddc0219f964f65666461d758a9cce46e85f

Canister Name: chainlink_eth_indexer
  Canister Id: {"local": "bw4dl-smaaa-aaaaa-qaacq-cai"}
  Controllers: 7fpuj-hqaaa-aaaal-acg7q-cai be2us-64aaa-aaaaa-qaabq-cai k245a-2cvyi-rcnzj-cpq6h-dcy7o-qrg4b-325bv-5c3om-jfceo-ikmjz-aqe
  Module Hash: 0x1832aa2b92faa97065146047f87c503bd84a638ebd6073cb1c88606c663526f9

Canister Name: eth_average_price_calculator
  Canister Id: {"local": "b77ix-eeaaa-aaaaa-qaada-cai"}
  Controllers: 7fpuj-hqaaa-aaaal-acg7q-cai be2us-64aaa-aaaaa-qaabq-cai k245a-2cvyi-rcnzj-cpq6h-dcy7o-qrg4b-325bv-5c3om-jfceo-ikmjz-aqe
  Module Hash: 0xbbd87ce2d215bcc65b130cf13ca21d99dfbb2d9827dd5c8c2f8d10b0f8e8f337

Canister Name: web_eth_indexer
  Canister Id: {"local": "by6od-j4aaa-aaaaa-qaadq-cai"}
  Controllers: 7fpuj-hqaaa-aaaal-acg7q-cai be2us-64aaa-aaaaa-qaabq-cai k245a-2cvyi-rcnzj-cpq6h-dcy7o-qrg4b-325bv-5c3om-jfceo-ikmjz-aqe
  Module Hash: 0x6feb60c3a47517392b08b06f42a32ed544d00660137a35290ef43de5d589d14d

% csx exec --network ic
Feb 11 23:14:53.778 INFO Execute canister processing...
Feb 11 23:14:53.778 INFO Load env file: "./.env"
Feb 11 23:14:53.778 INFO Start processing for commands generation...
Feb 11 23:14:53.779 INFO Script for Component "chainlink_eth_indexer" generated successfully
Feb 11 23:14:53.779 INFO Script for Component "api3_eth_indexer" generated successfully
Feb 11 23:14:53.779 INFO Script for Component "web_eth_indexer" generated successfully
Feb 11 23:14:53.779 INFO Script for Component "eth_average_price_calculator" generated successfully
Feb 11 23:14:53.780 INFO Entrypoint Script generated successfully
...
Feb 11 23:16:47.932 INFO Project "lens-project" canisters executed successfully
Enter fullscreen mode Exit fullscreen mode

Calculations in Algotirhm Lens are performed at each call. It is not executed on a regular basis. Therefore, you can check the calculation results with the following command. (You can also use IC Dashboard)

% cd artifacts
% dfx canister call eth_average_price_calculator get_result "vec {\"$(dfx canister id chainlink_eth_indexer)\"; \"$(dfx canister id api3_eth_indexer)\"; \"$(dfx canister id web_eth_indexer)\"}"
(record { value = 2502.2185809524067 : float64 })
Enter fullscreen mode Exit fullscreen mode

If you see this command executed, you have deployed an Algorithm Lens that correctly reflects the original logic you have implemented! Thank you for your patience up to this point.

This hands-on session was more difficult than previous ones due to the large number of components to be built. We hope that through this hands-on, you will be able to experience how Chainsight can reflect the user's customization. This is the end of the hands-on on Algorithm Lens. We will continue to create and publish new hands-on sessions in the future, so please look forward to them!

Written by Megared@Chainsight

Top comments (0)

Read next

mattlewandowski93 profile image

How to Replay any request in Chrome 📤 🔄 📤

Matt Lewandowski -

devopspass profile image

Workshop: make your first AI app in a few clicks with Python+Ollama+llama3

DevOps Pass AI -

ramkrishnajha5 profile image

Important Array Method in JavaScript

Ramkrishna -

rrajat profile image

Fine-tune your first large language model (LLM) with LoRA, llama.cpp, and KitOps in 5 easy steps

Rajat Raina -