Data Processing
Welcome back to my adventures with DO hackathon.
Probably here comes the craziest part of the development process.
Seems interesting right. I got this data from the business team.
Here is a little expectation v/s reality check
Expectation:
Each company/division is a separate sheet in the excel file. I need to do a little processing and then save em all in the database (firestore).
Reality:
As you saw in the screenshot everything is in a single file. I had to use my python skills, extracted them and saved them in the firestore.
I want my code to look really clean and beautiful. So I decided to sprinkle a tad bit of clean code architecture on the python script application.
Not trying to beautify things much, just showing how the thinking process went. Let's put in this way, handwriting and ideas scribbling don't probably go hand in hand with each other.
Python part of the github repository
This project structure uses poetry as the package manager.
If want to learn more about python package managers please do check this
Package Managers in python 🚀
Rohith Gilla ・ Jun 13 '20 ・ 2 min read
Modules used for excel processing
- pandas
- xlrd
jsonpickle module to obtain serializability.
Other misc modules can be found in the pyproject.toml
which helped to achieve code quality.
google-cloud-firestore
for firebase connectivity.
Finally, the data in the firestore looks like this
In the next post, I will be describing how the web application was built.
Peace ✌🏻
Rohith Gilla
Top comments (0)