⛵ Simple way to upload data in format JSON to Datastore from GCP.
🔗Related content
You can find repo related in:
🐱🏍GitHub
You can connect with me in:
Resume 🧾
All this blog is using shell.
We will upload data to Datastore we can made this with Cloud SDK or directly from shell in GCP.
We will create a script in json_to_datastore.py to upload data from restaurants.json to Datastore.
If you want test this script, you can get example data in my repo that is above.
For more information and potential of this, click here.
1st - Create a program python ✍️
I use following command:
touch json_to_datastore.py
2nd - Write code to do transforms 🧵
Open IDE(I use vi) and put the following script
# Only import libraries
from google.cloud import datastore
import sys
import json
import os
def to_unicode(string):
return unicode(string, "utf-8")
def get_values_json(jsondata):
# Getting values of object root
restaurant_id = jsondata.get("restaurant_id")
name = jsondata.get("name")
cuisine = jsondata.get("cuisine")
borough = jsondata.get("borough")
address = jsondata.get("address", {})
# Getting values of object address
zipcode = address.get("zipcode", '')
street = address.get("street", '')
building = address.get("building", '')
coord = address.get("coord", [0, 0] )
longitude = coord[0]
latitude = coord[1]
return restaurant_id, name, cuisine,borough, zipcode, street, building, longitude, latitude
def set_values_entity(restaurants, restaurant_id, name, cuisine,borough, zipcode, street, building, longitude, latitude):
# Assign encoded values to key
restaurants['id'] = to_unicode(restaurant_id)
restaurants['name'] = to_unicode(name)
restaurants['cuisine'] = to_unicode(cuisine)
restaurants['borough'] = to_unicode(borough)
restaurants['zipcode'] = to_unicode(zipcode)
restaurants['street'] = to_unicode(street)
restaurants['building'] = to_unicode(building)
restaurants['longitude'] = longitude
restaurants['latitude'] = latitude
return restaurants
def convert():
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
readFile = os.path.join(BASE_DIR, "restaurants.json")
# Instantiates a client
datastore_client = datastore.Client()
# The kind for the new entity
kind = 'restaurants'
print ('\n' + 'Convert to Datastore - Please Wait' + '\n')
# Set lines where will be row data
lines = []
# Open file restaurants.json
with open(readFile, "r") as f:
lines = f.readlines()
# By each line of file we transform data and put it in entity in Datastore
for line in lines:
jsondata = json.loads(line)
# Get values from json
restaurant_id, name, cuisine,borough, zipcode, street, building, longitude, latitude = get_values_json(jsondata)
# The Cloud Datastore key for the new entity
rest_key = datastore_client.key(kind)
# Prepares the new entity
restaurants = datastore.Entity(key=rest_key)
# Set values to entity
restaurants = set_values_entity(restaurants, restaurant_id, name, cuisine,borough, zipcode, street, building, longitude, latitude)
# Saves the entity
datastore_client.put(restaurants)
print ('\n' + 'Saved Rows to csv' + '\n')
if __name__ == '__main__':
convert()
3rd - Run script 🙈
We need restaurants.json is in same path of script.
We run script with following command:
python3 json_to_datastore.py
UALAAAAA! 🎩🪄
You now have data in your instance of Datastore...
Top comments (0)