Github Repository: Streaming_API
In the project I developed, we return a great response message to the API calls of the customers. It takes a long time to process large data. At this stage, the memory swells and the server CPU increases. To prevent this, we started using streaming method.
I've done an analysis to explain what this process has brought us. I have compared a normal response to a call request and streamed a call request.
- PostgreSql (SQLAlchemy)
- 500000 rows of data set
(Referenced from the github readme file.)
git clone firstname.lastname@example.org:tolgahanuzun/Streaming_API.git virtualenv -p python3 venv source venv/bin/activate cd Streaming_API pip install -r requirement.txt
vim settings.py and you need to edit the settings file yourself.
- JWT needs it. You can remove it if you want.
curl -H 'Accept: text/plain' -H "Authorization: JWT eyJ0eX....." -v http://0.0.0.0:8080/standard
curl -H 'Accept: text/plain' -H "Authorization: JWT eyJ0eX....." -v http://0.0.0.0:8080/chunked
Chunked data comes to you piece by piece.
In the standard, the data comes at once.
Incoming data sample.
All data is captured, processed and sent at one time. Blocks each other. In this example, it corresponds to 500000 row.
- Up to 0.16 seconds no data flow occurs. The data comes after that second.
The dataset size is 500000 rows. Data from the database is taken as 1000 row. Each group of data is processed and streamed without waiting.
Up to 0.04 seconds no data flow occurs. The data comes after that second.
Look out! The Streaming API finished before the Standard API started sending data. It didn't block itself because it sent and received data in pieces.