Every day is an API day: Share your Dinosaurs

WELCOME to Jurassic World! Do your kids love dinosaurs or do you love dinosaurs? This idea is inspired by my teammate's kid who is very into dinosaurs. With the love of a father, we make an API that can make drawing dinosaurs more interesting.
We are using the latest AI technology to recognize the features of the picture and match it with the dinosaurs in our database.

Our application has two major functions.

1. Match the drawing with dinosaurs in our database and analyze the ability of the drawing
2. See the drawings of others and find friends with similar interest

The application will show the dinosaur that is most alike to the drawing and analysis its ability and strengths like power, speed, aggressivity, defensive, and intelligence.

When the users access the page, we may request to access their location and mark it on Google map. The users can check others' drawing and know where the drawing is from.

So What we need for this project:

1. Anypoint Platform
2. Salesforce Developer Org
3. Amazon S3

STEP 1: Set up Salesforce Developer Org

First of all, you need to set up the Einstein Vision api for image classification. For more information about Einstein Vision api

You need to set up an account for Einstein Vision Api and get the Access Token. Please follow the step here
After we finished the steps above, we need to create a dataset and train it.

Creating a dataset

You will need to collect at least 10 photos for each category of dinosaurs and save it under the folder with name of dinosaur.

When your dataset is ready, zip the file and create dataset using the following CURL:

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "data=@C:\Data\mountainvsbeach.zip"  https://api.einstein.ai/v1/vision/datasets/upload/sync

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "path=http://metamind.io/images/mountainvsbeach.zip"  https://api.einstein.ai/v1/vision/datasets/upload/sync

You will get the result with Id and then used that Id to train the dataset using the following CURL:

curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=Beach Mountain Model" -F "datasetId=57" https://api.einstein.ai/v1/vision/train

Here you will get the result with modelId. Please save it and we will use this in our Api Later.

STEP 2: Create API

Now it's time to create API. What our API do is
1.Accept the image file Upload
2.Predict the image
3.Store it in S3 and get the presigned URI
4.Get the presigned URI of the photos with the same prediction result

So there is only one endpoint in our API which is /fileUpload and the RAML is as below.

#%RAML 1.0
title: Dinosaur Api

/fileUpload:
  post:
    queryParameters:
      name:
        description: children name
        type: string
        required: true
      age:
        description: children age
        type: string
        required: false
      gender:
        type: string
        required: false
      Long:
        description: Location
        type: string
        required: false
      Lad:
        description: Location
        type: string
        required: false
    body:
      multipart/form-data:
        description: The file(s) to be uploaded
    responses:
      200:
        body:
          application/json:
            example:
              output: !include /example.raml

The following is the example of responses which include the information of dinosaur, user and friends.

#%RAML 1.0 NamedExample
example:
  {
  "dinosaurs_info": {
    "uploadPhotoURL": "url1",
    "samplePhotoURL": "url2",
    "dinosaurName" : "Tyrannosaurus",
    "sampleDinosaurInfo": {
        "power": 80,
        "speed": 67,
        "attack": 76,
        "defence": 8,
        "Intelligence": 4
      },
    "power": 36,
    "speed": 89,
    "attack": 8,
    "defence": 11,
    "Intelligence": 29
  },
  "user_info": {
    "name": "Yamada Taro",
    "age": "6",
    "location": "lad_long"
  },
  "friend_list": [
    {
    "name": "Taro1",
    "age": "6",
    "location": "lad_long",
    "url":"url1" 
    },
    {
    "name": "Taro2",
    "age": "6",
    "location": "lad_long",
    "url":"url2" 
    }
  ]
}

The implementation of the flow is like this:

In this flow, instead of using transform message component, we do the transformation directly inside the target component. For example, in the HTTP request component(Predict), we wrote the dataweave directly in the body like below:

As you can see, reducing the component make the flow look more simple.

Details of Implementation

1. Predict This is a HTTP request component to make a request to Einstein vision API(/Prediction with Image Base64 String). Because of mime type of the input is Multipart/form-data, I got the content of the input data with payload.parts[0].content and transform it to another Multipart/form-data format data for Einstien Vision API prediction as the capture shown above.For more information about Multipart/form-data in Mulesoft

2.Set myVar
I set all the information of user in file name concatenate with underscore and as an object user_info to used in the output.

3.Create object and Create object presigned uri
Uploading the photo to S3 and get the presigned URI, I use the bucket name, I used the property value.

As the input data is base64 string, I need to decode it in order to upload it to S3. I also do this directly inside the target component instead of using transform message component.

Here I stored the result payload in variables by setting up the target variable on Advanced tab.

1. Init friend list and for loop I create an array List(friendList) and adding value of it by using forEach component with the result of ListObject. When uploading photo I stored all the information of the user in the file name by concatenating with underscore and here I want to set the information as an object so I used SplitBy call.

%dw 2.0
import * from dw::core::Strings
output application/json
var friendInfo = substringBeforeLast(payload, ".") splitBy("_")
---
{
    "user_name": friendInfo[1],
    "age" : friendInfo[2],
    "gender" : friendInfo[3],
    "location_Long": friendInfo[4],
    "location_Lad": friendInfo[5],
    "url" : vars.friendPhotoURL
}

1. Set up the response value Final Step is to set up the response value

%dw 2.0
output application/json
---
{
  "dinosaurs_info": { 
      "uploadPhotoURL" : vars.uploadPhotoURL,
      "samplePhotoURL" : vars.samplePhotoURL,
      "dinosaurName" : vars.response.probabilities[0].label,
      "sampleDinosaurInfo": {
        "power": round(random() * 100),
        "speed": round(random() * 100),
        "attack": round(random() * 100),
        "defence": round(random() * 100),
        "Intelligence": round(random() * 100)
      },
      "power": round(vars.response.probabilities[0].probability * 100),
      "speed": round(random() * 100),
      "attack": round(random() * 100),
      "defence": round(random() * 100),
      "Intelligence": round(random() * 100)
      },
     "user_info": vars.myVar.user_info,
     "friend_list": vars.friendlist
  }

Step 3: Set up UI on Salesforce Community Cloud（Experience Cloud）

We used Lightning Web Component for our pages and you can get the code from here: salesforceCode

About Our team

Keitaro K
Kazutaka U
Takamasa　H
Yun F
Holiday J

We are salesforce developers in Japan. We started learning Mulesoft this year. We decided to take part in this activity to learn more about this field. After discussing among the team, we decided the above topic because one of our member's children is currently crazy about dinosaurs and we think there are many kids with this interest around the world. By joining this competition, we not only feel accomplished for creating something fun for children, but also gain a bunch of knowledge during the implementation.

Reference
Dinosaur Api: https://github.com/HninPwintP/mulesoftHackathon2020/tree/master/dinosaur-api2
Salesforce code: https://github.com/HninPwintP/MuleHackathon