Over this past month, I have had the pleasure to work along side a team of talented web developers and data scientists to polish and add features to an existing application that will be used by a non-profit organization called Human Rights First. The application will be used by asylum lawyers and human rights activists to determine biases that judges may have against certain groups of people who are seeking asylum to the United States.
The organization that this application is going to be used for is doing some admirable work. The application itself is very cool as well. But the thing that has peaked my interest this past month has been working on a cross-functional team to solve problems that actually have an impact to the project as a whole.
Discovering The Problem
In the first week of working on this project, my entire team was working on understanding the codebase that we had inherited. After all, if you don't understand how the application works, then you'll have a pretty tough time making meaningful contributions to it. In this process of discovery, one thing that the team realized was that PDF uploads were not functioning correctly within the application.
PDF uploads are a crucial part of the application; A user needs to have the ability to upload a case document to the app, the app should then upload that PDF to an AWS s3 bucket, after the PDF has been placed in the bucket, the backend of the application sends a request to our data science API with the UUID of the document that had just been uploaded, from there, the data science side runs a scraper on the PDF and pulls out useful information such as: judges, case outcome, case origin city/state, etc. The data science API then sends the scraped data back to the application's backend and it is stored in our database. PDF uploads are the lifeline of the application, they are the only way that the application collects data for visualization and table display.
Gathering Information and Diagnosing
After the team had realized that there was an issue with the document upload system, I decided to take on this problem. I quickly realized that I was a bit out of my depth. I had never worked with document uploads in an application and I had almost zero understanding of how the system worked.
The benefit of working in a large cross-functional team is that every team member has different skillsets and different experience under their belts.
I knew that one of my teammates was good with backend programming so I went to them to pair up and gain a better understanding of the issue at hand. After gaining understanding of how the document upload system is supposed to work, we began diagnosing the problem.
The first thing that we had noticed was that upon request to the data science API, the application returned a 500 error code. This led us to investigate the data science API. We got together with some of the data scientists on our team and gave them a run down of the information we had gathered so far. With the help from a few people from a few areas of the team, we had gathered that the data science application had a memory leak. The data science team came up with a fix for this within a few days, but there were more issues to come.
The biggest issue that we had ended up running into had to do with request timeouts and the time it takes for the data science PDF scraper to run. The application's backend is hosted on Heroku, which limits the time that a request can be open to thirty seconds, but the PDF scraper, which we requested to run from the data science API, took at least forty seconds to run. There is an obvious issue here, but I'll point it out anyways; the data science PDF scraper took longer than the set request timeout. This was causing the application to run into a 503 error every time a user tried to upload a case. Heroku has a few articles about H12 errors that closely relate to this issue.
Creating a Solution
Fixing this issue came down to two solid options. One of my teammates had suggested that we implement a queue that would serve as an intermediary between the backend of the application and the data science scraper. Essentially, the queue would prevent the data science application from running too many operations at once. The second option was to not wait for the data science endpoint to resolve/return anything, so that way the request cannot time out.
If we were to not wait for the API call to resolve, then we would need to store the scraped data on the data science side of the application. If we decided to implement the queue, it would take more than a few days, of which we did not have.
What we ended up deciding on was the temporary solution of not waiting for the API to resolve, and having the data science application store the information into the database directly.
As I mentioned, the solution we decided upon was a temporary one; we would need to leave detailed documentation and a solid plan for the developers that inherit this project to follow to implement the long term solution: the queue.
The Aftermath
Even though the issue at hand was only partially solved, I still learned many valuable lessons along the way. I spent hours learning about document uploads, HTTP errors, and most important of all: teamwork.
Working on a large team sometimes has its challenges; some team members might not be the best at speaking in large groups, team members from the data science team might have a tough time understanding concepts presented from the web team and vice versa, but I think that these challenges can be looked at as benefits.
Large teams bring together many different personalities from many different backgrounds, so everyone has something meaningful to contribute to the team. If I had not been working on a team to solve the issue I described, I definitely would not have been able to debug it, and I for sure would not have been able to come up with the solutions on my own.
This past month has taught me that being a developer is more than just sitting down at a computer and typing away for hours on end. Most applications are built on teams, and in order to be an effective contributor to a team, you need to know more than just how to program. The biggest thing you need to be successful in a large team is communication skills. Without communication, your team is dead in the water.
Working in a team also allows for many opportunities for personal growth. Every member of the team brings a new perspective to the table when it comes to every issue, so there is a lot of information that you can learn from. Teamwork also brings a lot of opportunity for many individuals to give you some direct feedback that you can try and implement into your professional or even personal life.
To wrap things up; working on an application that is already pretty established can be a daunting task at first, but if you have a great team to help you through it and teach you things along the way, some of the frustration can be mitigated. So if you ever have the chance to work on a large development team, do it!
Top comments (0)