Idempotency in a REST API design. And why do we need it

Surely while designing some API or talking about API resiliency or in your graduation mathematics it is a very likely scenario that you had come across the term 'idempotency'. But ever wondered what it means for an API design before implementation? So here we will start with the basic definition and idempotency of HTTP methods and then discuss on designing an idempotent API and the role of this term. And by the end of this article, we will develop an understanding of this in the API context.

What is Idempotency?

The word 'idempotency' in mathematics is defined as the property of an operation that means when applied multiple times, it will yield the same result as it was expected to yield at the first operation. Some common examples are the addition of zero to any number, multiplication of number with one or zero and the multiplication of the Unit matrix with another matrix etc. Here in the context of REST API, it is closely related to effect which means a request if applied N times will have no additional effect on the state of the server after a request has been completed. It will have the same impact on the state of the server as one could expect for a single request.

Idempotency of Http Methods:

In HTTP request methods semantics POST AND PATCH methods are not idempotent while GET, PUT, DELETE, HEAD, OPTIONS and TRACE are idempotent.
The following are the reasons for the above classification.

1. POST method by virtue is not idempotent as it is meant for the creation of resources. N requests will create N resource
2. GET, HEAD, OPTIONS and TRACE are of read-only nature which means they can't change the state of the server. Hence they all are idempotent.
3. The DELETE method is used to delete a targeted resource that is requested for example /resources/resource-id. the resource associated with deletion will get deleted on the server giving a (20x status code ) response. Once the resource got deleted it will give a 404 not found status code as a response for any subsequent request but the state of the server will be the same after 1st request completion. It will delete only the targeted resource.
4. PUT method is used to update the target resource or it will create one if the resource is not present. On completion of the first PUT request, the resource will get updated or created accordingly after that subsequent requests will have the net same effect as of first. They will just overwrite the state of the resource.
5. PATCH is a tricky case as PATCH is being used to update the partial fields of a resource. The patch is different from PUT as in the put method enclosed entity is considered as the final version of the resource after modification while PATCH can also contain operations like move, add, remove etc. we will understand this in detail with the help of an example. Let's suppose there is a resource having a resource endpoint "categories/{categoryId}" which is nothing just a collection of items in a category

**for simplicity here we are considering 2 key-value categoryName and items.

Resource before request
{
  "categoryName" : "games",
  "items" :["ping-pong","super-mario"]
}

Consider JSON Patch request as follows:
[
  {
    "op": "add",
    "path": "/categoryName",
    "value": "retro-games"
  },
  {
    "op": "add",
    "path": "/items/-",
    "value": "contra"
  }
]   
And Put Request as Following 
{
  "categoryName" : "retro-games",
  "items" :["ping-pong","super-mario","contra"]
}

Here in both scenarios after a successful request
the final state of resources will be the same 
as shown below.
{
  "categoryName" : "retro-games",
  "items" :["ping-pong","super-mario","contra"]
}

• In the above example if the request is applied N times there will be N "contra" item in items as Patch contains a set of operations while PUT even after N times request will remain at the same final state. That is why PATCH is not an idempotent method by virtue.

** For Http methods notion of calling method safe is used to describe their nature . IETF documentation clearly states read-only methods are called safe (i.e. GET, HEAD, OPTIONS, and TRACE )
** reference

Designing an Idempotent REST API :

As we know that if HTTP semantics are followed the POST and PATCH method API will still get left from being idempotent what if we want to make it idempotent? Let's consider a scenario for better understanding.
Suppose Service A is sending a request to Service B due to some reason Service A received a failed response or it received no response from the server. There can be plenty of reasons for this like:

1. The request took more time to get processed on the server and even was a success but still got connection time-out.
2. Response packet getting dropped due to network issue.
3. The request to the server was heavily dependent on others and got failed on the first attempt.
4. There was a deadlock condition in DB operation and the request can't be processed for a while.

For all the above reasons it will be very reasonable to consider that a Service A wants to make a retry. But here we had to ensure that the operation is getting processed once for an intended request by Server A. A temporary solution to this is to hash the incoming request on server B and match if this request was processed recently which is not a great choice in itself. Since there can be a case that the Service wants to process a request twice intentionally.
There is a need for a mechanism to distinguish between a retry and a repeated request. This problem can be easily handled by the use of the Idempotence Key. Here a client generates a unique Id (eg. UUID) for each request which will be used to identify the request. There is an understanding between a client and server that the client will generate a unique key for different requests and use the same key for a retry. Now the server will register the request with the given key and process it. If the server finds the given key in lookup then it will not reprocess the request, it will send the response according to implementation. In case the previous request was a failure it will process the given request normally and update the status in look up to success. If the previous request was a success but a response was not received on the client side. It will reply with the cached result and guarantee that a request is processed exactly once.

But still, why?

Idempotency as a principle plays a key role in designing a robust API by making them more fault-tolerant and establishing consistency in distributed systems. Client using the API is assured with predictable results. Most of the time APIs that we make are for serving business purposes one such example of a use case is payment service and order placement where following exactly one semantics is a necessity. We can't tolerate inconsistency in this case like a customer deducted twice from their bank account or a customer placed an order once but didn't get success on the first attempt, and now he finds his order is placed twice. It will be a terrible experience if the idempotence principle is not applied in design. In modern-day design architecture message brokers like Kafka provide exactly one semantics by using the Idempotence principle (ref).APIs that can handle failure responsibly and consistently result in better development and a stable distributed system.