Yesterday I learned a very valuable lesson when iterating a while loop using the Google Places API - Web Service that cost me $646. That's right, I pinged the Places API 17,000 times in one hour 😳 🤦🏻♂️. So I decided to write an article today documenting my mistake in the hopes that this prevents someone else from running into this same issue. Let's dive in.
What Happened?
Well...I'm currently building a Web and Mobile Application that uses the Google Places API and I needed to return results using the nearby search. The Google API only returns 20 results with a max of 60 which you can get by using their version of pagination. So let's go through the code and explain what went wrong.
// This is the initial request that we send to get the initial 20 results.
$response = Http::get('https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=' . $request->lat .'%2C'. $request->lng.'&radius='.$request->radius.'&type=restaurant&key=' . env('GOOGLE_MAP_KEY'))->json();
// We push those results to our google_places collection
$this->google_places->push(...$response['results']);
// Here we check the status of the response, if it's ok we continue.
if ($response['status'] == 'OK') {
// If the response has a next page token we loop through until it doesn't, getting the next 20 results.
while ($response['next_page_token']) {
// The google places api has a delay from when the next_page_token is created and those results are actually available.
// So we need to create a delay before we try and grab the next results otherwise we will get an INVALID_REQUEST response.
sleep(2);
// Now we get the next 20 results.
$new_results = Http::get('https://maps.googleapis.com/maps/api/place/nearbysearch/json?pagetoken=' . $response['next_page_token'] . '&key=' . env('GOOGLE_MAP_KEY'))->json();
// Then we push those results to our collection.
$this->google_places->push(...$new_results['results']);
}
}
Were you able to figure out where I went wrong? A while loop in PHP will run until it gets a false back, the value that I put in the while loop always returns true and therefore sends the while loop into an infinite loop. I was using the $response['next_page_token'] from the initial request which will always return true.
How did I fix this issue?
$response = Http::get('https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=' . $request->lat .'%2C'. $request->lng.'&radius='.$request->radius.'&type=restaurant&key=' . env('GOOGLE_MAP_KEY'))->json();
$this->google_places->push(...$response['results']);
// We check to see if the above response includes the key next_page_token, if it does we set the next_page_token property to true
// This is super important, this is how we're going to ensure our loop doesn't run infinitely.
if (array_key_exists('next_page_token', $response)) {
$this->next_page_token = true;
}
if ($response['status'] == 'OK') {
// For the while loop we check to see if the next_page_token is true, it will end when it returns false.
while ($this->next_page_token) {
sleep(2);
$new_results = Http::get('https://maps.googleapis.com/maps/api/place/nearbysearch/json?pagetoken=' . $response['next_page_token'] . '&key=' . env('GOOGLE_MAP_KEY'))->json();
$this->google_places->push(...$new_results['results']);
// This is also super important, here we check to see if next_page_token does not exist or if we're getting an invalid request response.
// If we do we want to set the next_page_token to false so that the loop ends.
if (!array_key_exists('next_page_token', $new_results) || $new_results['status'] == 'INVALID_REQUEST') {
$this->next_page_token = false;
}
}
}
// Being that the Google API only returns 60 results that means the loop should only have to run a max of 2 times.
// So you could create a $count variable and add that like so while($this->next_page_token || $count <= 2) and increment the count each time.
// This will act as a safety in case you accidentally throw a forever truthy statement in there like I did.
Conclusion
Be cautious when working with APIs that charge and running them inside a loop. As a precaution you can always create a $count variable and stop the loop at a set count in case you make a mistake. This is what I will do for now on just to ensure I don't do this again. Hopefully this has helped you or someone else from making the same mistake. ✌🏻
Top comments (21)
I think the mistake here is more, always write unit tests when you can. A test running this function could easily find that a while loop is never completing.
Equally never hurts to put a limit on a loop for an API so it never iterates more than say 50 times etc.
Good idea, and since I see some Laravel code up there, just mock the API return so that you never really consume credits (and you still find out infinite loops, but for free)
I did a similar thing but with the lookup taking place in a background job using SQS. A bug in the code meant the job always failed and got retried.
£20,000 later...
Luckily Google were cool about it.
Lesson: set decent retry counts on your Laravel worker jobs.
Thanks for sharing this, and capturing the lessons you learned along the way! I believe this example underscores the importance of targeted testing to confirm expected behavior.
Assuming that you have a mock setup for the test, some examples of tests that come to mind are:
Could you not have changed this one line
$new_results Http::get('maps.googleapis.com ...
to
$response = Http::get('maps.googleapis.com ...
so the loop would work correctly by referencing the latest result set?
Ooof. 😅 Really happy that you can see the humor in this, and thanks for sharing!
Maybe you can pass on this story and they'll give ya some sorta credits for the pain ya went through here. 🤞
It would be a good publicity move, Google. We're watching... 👀
Full disclosure, I am a software engineer on Microsoft Azure Maps who built a feature to help address this exact problem because API calls == $.
My other suggestions if you consider another product like Azure maps is to leverage entry level Skus which have upper limits resulting in throttling on requests. Second option is look into the authentication support; SAS token authentication allows you to configure an upper bound how many requests per second to limit charges.
I have no affiliation with Google maps or other competitors.
Looks like you also need to use the correct next_key_token. You are using the initial next_key_token from the first request in your URL.
Good catch! I'm actually not using the Google Web Service anymore so I won't be using this code but, still good to know.
Thanks a lot for sharing your valuable eperience about this! I see this resonated on some other folks experiences as well, including mine.
I definitely advice, even if you have a great code, to put an hard coded limit on the maximum calls, wether it matches the plan limit or something lower.
We ran into the same issue at my job, where one of the plans we subscribed for a given service was mistakenly showing us limit reached, but in small characters we could actually go further by... Paying an extra per additional calls (knowing we already paid for the initial bucket of credit). Now that we have the hard limit, we get an error and we can decide to up the limit or just give up on this service until next month.
You might want to cache the results from Google Places API for at least a day.
And use the same cached result for the same and 5-10 meters around the given latitude and radius.
As much as I would love to, Google's TOS don't allow for this.
But how would Google verify that?
In both cases, the server(s) of Google Places API see only the IP-address of your application server.
But less frequently when you cache the results.
I did similar thing, causing 500$ for Google timezone api.
I just sent them a email explaining them it was a mistake, they gave me waiver off on everything lol
It happened in my company as well, after a 2k€ bill we set the max limit equal to the free tier usage. This way we only pay if there's a need to increase the limit in any given month.
Another error:
.status == "OK"should be checked first before saving the results.The pay as you go is a big trap for people. be careful about these services. you are making the rich richer and yourselves more poor.
Thanks for sharing. Lots of good advice going on.
$7000 was my mistake. 😅
Not a bug in my case just overlooked the number of API calls that were attached to the external library I was using and Google had just started charging at the time.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.