At G Adventures, we’ve been building a publicly accessible API for the past three years. We’ve been pushing hard to make the G Adventures API a standard within the Adventure Travel industry, and we believe we’re doing so by providing superb documentation & support, as well as enabling our customers with a standards-compliant JSON/XML API, powered by a suite of services and tools.
When we first tackled the construction of our API, we wanted to keep things simple. We have many systems at G Adventures which own data, and we wanted to ensure that the API would not be an owner of data, simply focused on delivering data.
Thus, we looked towards frameworks that kept things simple. In the Python ecosystem, one of the most popular frameworks for keeping things simple is Flask.
Flask is a micro-framework that truly does not get in your way. It ensures you can write plain Python, with additional batteries included. This was great for our needs, as our mandate was:
- Build a system that can capture a single request, and transform that into potentially multiple requests to source systems. Effectively, an API Gateway
- Ensure we don’t re-invent the wheel when it comes to HTTP Standards
- Keep it lean and fast. Speed is important. Both from development ramp-up and data delivery perspectives
Thus we began iterating on our API prototype. Flask truly did not get in our way. In a couple of days, we had a prototype that would proxy a single client request into multiple requests behind-the-scenes. We iterated on this prototype until we came closer to a reasonable product. Building out our resource definitions in pure Python, we were able to define common patterns, and store them as simple dictionaries. We’ll touch a bit more on what this looks like today shortly.
Eventually, we launched our API. It was a success! We had many internal, and external clients consuming G Adventures data. We began expanding our system by offering more resources, tools like OAuth, payload validation, and so on. As we began to add these tools, we introduced concepts like an ORM (Via SQLAlchemy) and migrations (Alembic) to our infrastructure.
These aren’t foreign concepts, and the tools mentioned are superb, but we’d began to slide away from our core fundamentals. Our API was becoming larger in scope, we began re-inventing the wheel, and our simple approach of dictionaries for resource definition worked, but didn’t. Essentially, we’ve hit a point where we could identify that if we continue to build upon this micro-framework, we’d build upon our technical debt. This was an opportunity to rethink
We began to think of how we move forward. We want to keep the API layer simple, and allow other microservices to provide the various features of the G Adventures API, but we wanted to have a consistent, well-structured set of ideas we could leverage and build from.
And thus, we waved hello to Django Rest Framework. Django Rest Framework is a perfect companion to a system where you have Django Models, and you want to expose an API to them. It’s so simple that most of our internal systems that expose an API to our API Gateway use it. However, we were in a different situation than a typical Django project.
We were actively running an API service which was receiving millions of hits per week, we wanted to leverage Django Rest Framework, but would need to pivot it to fit our needs of multiple HTTP backend services. Essentially, we had to make Django Rest Framework work in the ways we’ve structured our API. Let’s take a look at what that looked like.
Migrating while the machine is running
On the API, we support 90 resources, each with a different degree of complexity. A read-only resource on one backend is easier to test than a read-write resource using many backends. At the end of the day, migrating more than 5 simple resources at once is asking for trouble. Our decision was to migrate a few at a time and proxy the requests to the appropriate process (Flask or DRF). This allowed us to migrate slowly, communicate with our clients in case of errors and easily route back to Flask if needed.
Being able to revert to Flask served us well. You can migrate resources with precise respect to the documentation, but that does not guarantee success. Django Rest Framework could reject a request while the Flask framework wouldn’t. For example, Flask will happily process:
POST to /bookings POST to /bookings/resource
While Django Rest Framework will only accept:
POST to /bookings
Our documentation always said to use bookings. bookings/resource was not prohibited in our documentation, it simply wasn’t mentioned. One of our clients used bookings/resource, it went through and they continued to use it. When we migrated bookings to DRF, the client messaged us about it. Flask overlooked this error but DRF did not. Well, we were surprised bookings/resource was a valid request on our own API. I told the client to remove resource for all our resources. Now imagine we migrated all 90 resources at once — all integrations using <resource>/resource would be sad.
Customizing our endpoints
With Django Rest Framework, a resource is served by binding an endpoint to a view. For example, to serve bookings on the API, we simply had to register it:
router.register(bookings, BookingView)
Using DRF generic router, this will support requests on:
bookings/\<booking\_id\>
Our Flask supported other URL variations, and one of our challenges was to properly route all existing requests. Here is what we needed to serve:
\<resource\>/\<id\>/ \<resource\>/\<id\>/\<resource\_2\> \<resource\>/\<id\>/\<variation\_id\> \<resource\>/\<id\>/\<variation\_id\>/\<resource\_2\>
First, we found how DRF parses URL: https://github.com/encode/django-rest-framework/blob/1c53fd32125b4742cdb95246523f1cd0c41c497c/rest_framework/routers.py#L241.
This means our bookings/<id> will be valid through DRF’s default regex:
base\_regex=’(?P\<{lookup\_prefix}{looking\_url\_kwarg}\>{lookup\_value})’
URLs are constructed in get_urls, we can easily overwrite this function:
def get\_urls(self): urls = super().get\_urls() return urls + \<our custom urls\>
Our custom URLs will take care of:
\<resource\>/\<id\>/\<resource\_2\> \<resource\>/\<id\>/\<variation\_id\> \<resource\>/\<id\>/\<variation\_id\>/\<resource\_2\>
Let’s say we had an itineraries resource with ID 2222, a possible variation ID of 3333 and also a nested resource of itinerary_maps. These are all valid requests:
Variation 1: itineraries/2222/3333 Variation 2: itineraries/2222/itinerary\_maps Variation 3: itineraries/2222/3333/itinerary\_maps
First, we overwrote this method https://github.com/encode/django-rest-framework/blob/1c53fd32125b4742cdb95246523f1cd0c41c497c/rest_framework/routers.py#L230 to add extra regexes, such as:
\<base\> + "/\*(?P\<variation\_id\>[^/]\*)"
Notice in our get_urls method, we call super() before we create our nested URLs. super().get_urls() calls self.get_lookup_regex(viewset) which constructs our variation_ids before constructing our nested URLs (i.e. those ending with <resource_2>).
The call to super() creates:
itineraries/2222 itineraries/2222/3333
While making our nested URLs, we append these urls, also creating:
itineraries/2222/itinerary\_maps itineraries/2222/3333/itinerary\_maps
By constructing our regex orderly, we are able to create these custom URLs and bind them to the appropriate views.
Routing our backends
I mentioned backends a few times in this article, but never formally defined them. A backend is what we call one of our internal systems that provides data into the API. Our reservation system provides bookings, our Salesforce piece provides contact data, our operations tool provides tour data and so on. This means our API must know which backend(s) to use for which resource. We did this by defining a set of backends and binding resources (example):
class Backend: config\_variable = 'B1'
def request: \<initializes a request with the system\> def get\_object: \<uses request to get an object\> def update\_object: \<uses request to update an object\> def create\_object: \<uses request to create an object\>
class Resource: object\_backends = [(READ\_WRITE, 'B1'), (READ\_ONLY, 'B2')]
The above declares Resource to read and write from B1 and then use B2. Notice the order, B2 will actually override data from B2 if fields collide. Now we just need to patch DRF to use this structure, and this is easy. Here is just the function to override: https://github.com/encode/django-rest-framework/blob/7078afa42c1916823227287f6a6f60c104ebe3cd/rest_framework/generics.py#L77 . We did the same for update_object and create_object, allowing us to select backends for resources. Our Flask application operated the same way, but the code for this functionality was much more complex.
Now and future of the API
Our DRF framework has been serving all resources for over a year now and we could not be happier. While Flask served us well initially, we eventually found ourselves reinventing the wheel. With Flask, all the API basics had to be done from scratch. With DRF, we get basic functionalities and an easy way to add our own. A structured framework allowed us to focus on the actual API while allowing us to slightly customize the framework. DRF is just flexible enough to give us endless possibilities. We are currently customizing the API to route different requests to different reservation systems, stay tuned!
Keep following us on Medium and Twitter for more snippets of the Tech world according to G!
Want to help G Adventures change people’s lives and travel the world? Check out all our jobs and apply today.
Top comments (0)