When we are designing software with performance and scalability in mind, we are always looking reduce the network activity and machine resources. We implement and design patterns, strategies and architectures to have the best possible performance in any circumstance.
Partial Response
A partial response is like a SQL select statement where you can specify only the fields you want back. Only the attributes of interest are requested. Previously all fields for objects were returned, even if the client didn't need them. So the goal is reduce payload sizes by being more selective about what data is returned.
A couple of years ago Google introduced us the idea With the simple goal of make the web faster.
The benefits of this strategy are pretty obvious. If the API behaves like a database, it adds the possibility to the user of choose what fields want to receive. Reducing like that, the number of bits that our API deliver.
Let's say that a client request some users from an API.
GET /users
response:
{
"results": [
{
"gender": "male",
"name": {
"title": "mr",
"first": "andy",
"last": "pawlik"
},
"location": {
"street": "meisenweg 164",
"city": "halle (westf.)",
"state": "schleswig-holstein",
"postcode": 81023,
"coordinates": {
"latitude": "-37.4061",
"longitude": "-95.1859"
},
"timezone": {
"offset": "+4:00",
"description": "Abu Dhabi, Muscat, Baku, Tbilisi"
}
},
"email": "andy.pawlik@example.com",
"login": {
"uuid": "0aaaa5ec-ab09-4720-b092-81610a218d55",
"username": "orangecat573",
"password": "111111",
"salt": "OUdLDkdm",
"md5": "64b62e0595cff0e112ed8d08364acc55",
"sha1": "84523e164a58b81f379b7cc86330dcaeeeee47cc",
"sha256": "1d5e441f6d2b5cb98c88741efe4993afe48327f18b6097010ca37f8c9eda3088"
},
"dob": {
"date": "1950-05-19T13:38:56Z",
"age": 69
},
"registered": {
"date": "2009-01-05T22:06:17Z",
"age": 10
},
"phone": "0061-0583330",
"cell": "0171-3132822",
"id": {
"name": "",
"value": null
},
"picture": {
"large": "https://randomuser.me/api/portraits/men/11.jpg",
"medium": "https://randomuser.me/api/portraits/med/men/11.jpg",
"thumbnail": "https://randomuser.me/api/portraits/thumb/men/11.jpg"
},
"nat": "DE"
},
{
"gender": "female",
"name": {
"title": "miss",
"first": "inés",
"last": "van herk"
},
"location": {
"street": "1201 voorstraat",
"city": "ridderkerk",
"state": "noord-brabant",
"postcode": 13096,
"coordinates": {
"latitude": "-34.5827",
"longitude": "-162.2972"
},
"timezone": {
"offset": "-6:00",
"description": "Central Time (US & Canada), Mexico City"
}
},
"email": "inés.vanherk@example.com",
"login": {
"uuid": "36164c34-dcf6-4d13-9eb8-0681227d7648",
"username": "crazylion993",
"password": "genesis1",
"salt": "xGhkkDZB",
"md5": "1b662495a9619402ff22219cb13b31a8",
"sha1": "39fcb31864451020f63275bddbd53c05ff353eeb",
"sha256": "63152369be81139d6c0e457c319416b4b457dc3badf2e6ad702d594fa1fac6cb"
},
"dob": {
"date": "1980-07-15T13:38:56Z",
"age": 38
},
"registered": {
"date": "2008-05-14T16:59:07Z",
"age": 11
},
"phone": "(141)-015-6780",
"cell": "(726)-722-0668",
"id": {
"name": "BSN",
"value": "49767252"
},
"picture": {
"large": "https://randomuser.me/api/portraits/women/13.jpg",
"medium": "https://randomuser.me/api/portraits/med/women/13.jpg",
"thumbnail": "https://randomuser.me/api/portraits/thumb/women/13.jpg"
},
"nat": "NL"
}
]
}
It receives pretty complex information (name, address, credentials, profile pic, etc.)
But what if the user only needs gender and names? The rest of the information will be delivered in vain, because the client doesn't care about address, email or DOB.
With a partial response we can let the user choose which fields want to get, so, nobody wastes resources and we all win.
Example. Requesting gender and name
GET /users?fields=results(gender,name)
response:
{
"results": [
{
"gender": "male",
"name": {
"title": "mr",
"first": "andy",
"last": "pawlik"
}
},
{
"gender": "female",
"name": {
"title": "miss",
"first": "inés",
"last": "van herk"
}
}
]
}
The response has been reduced in almost 90% in comparison with the previous!
In that way we can deliver a more compact response and is just what the user need.
The benefit is not only better performance, is also better syntax. The syntax is pretty clear, and personally it's very comfortable and specific. Only with read the endpoint you know what is that you going to get as response.
Examples
Exist a JavaScript library pretty simple to use json-mask it's a little engine (4KB) used to select specific parts of a JavaScript object.
Also exist an express middleware: express-partial-response (It uses json-mask under the hood.)
I'll be using json-mask for the next examples.
// server.js
let http = require('http');
let url = require('url');
let fs = require('fs');
let mask = require('json-mask');
let server;
server = http.createServer(function(req, res) {
let fields = url.parse(req.url, true).query.fields;
let data = fs.readFileSync('data.txt'); // read the JSON response example
data = JSON.parse(data);
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify(mask(data, fields)));
});
server.listen(3000);
console.log('listening in 3000');
Let's execute our server and start to make some requests.
$ node server.js
$ curl localhost:3000
{"results":[{"gender":"male","name":{"title":"mr","first":"andy","last":"pawlik"},"location":{"street":"meisenweg 164","city":"halle (westf.)","state":"schleswig-holstein","postcode":81023,"coordinates":{"latitude":"-37.4061","longitude":"-95.1859"},"timezone":{"offset":"+4:00","description":"Abu Dhabi, Muscat, Baku, Tbilisi"}},"email":"andy.pawlik@example.com","login":{"uuid":"0aaaa5ec-ab09-4720-b092-81610a218d55","username":"orangecat573","password":"111111","salt":"OUdLDkdm","md5":"64b62e0595cff0e112ed8d08364acc55","sha1":"84523e164a58b81f379b7cc86330dcaeeeee47cc","sha256":"1d5e441f6d2b5cb98c88741efe4993afe48327f18b6097010ca37f8c9eda3088"},"dob":{"date":"1950-05-19T13:38:56Z","age":69},"registered":{"date":"2009-01-05T22:06:17Z","age":10},"phone":"0061-0583330","cell":"0171-3132822","id":{"name":"","value":null},"picture":{"large":"https://randomuser.me/api/portraits/men/11.jpg","medium":"https://randomuser.me/api/portraits/med/men/11.jpg","thumbnail":"https://randomuser.me/api/portraits/thumb/men/11.jpg"},"nat":"DE"},{"gender":"female","name":{"title":"miss","first":"inés","last":"van herk"},"location":{"street":"1201 voorstraat","city":"ridderkerk","state":"noord-brabant","postcode":13096,"coordinates":{"latitude":"-34.5827","longitude":"-162.2972"},"timezone":{"offset":"-6:00","description":"Central Time (US & Canada), Mexico City"}},"email":"inés.vanherk@example.com","login":{"uuid":"36164c34-dcf6-4d13-9eb8-0681227d7648","username":"crazylion993","password":"genesis1","salt":"xGhkkDZB","md5":"1b662495a9619402ff22219cb13b31a8","sha1":"39fcb31864451020f63275bddbd53c05ff353eeb","sha256":"63152369be81139d6c0e457c319416b4b457dc3badf2e6ad702d594fa1fac6cb"},"dob":{"date":"1980-07-15T13:38:56Z","age":38},"registered":{"date":"2008-05-14T16:59:07Z","age":11},"phone":"(141)-015-6780","cell":"(726)-722-0668","id":{"name":"BSN","value":"49767252"},"picture":{"large":"https://randomuser.me/api/portraits/women/13.jpg","medium":"https://randomuser.me/api/portraits/med/women/13.jpg","thumbnail":"https://randomuser.me/api/portraits/thumb/women/13.jpg"},"nat":"NL"},{"gender":"female","name":{"title":"ms","first":"nicoline","last":"brekka"},"location":{"street":"ørakerstien 1564","city":"krossneset","state":"rogaland","postcode":"5042","coordinates":{"latitude":"88.7205","longitude":"64.5762"},"timezone":{"offset":"+7:00","description":"Bangkok, Hanoi, Jakarta"}},"email":"nicoline.brekka@example.com","login":{"uuid":"62172038-acd0-4c39-b440-aad7fcbb484c","username":"orangeduck796","password":"windows","salt":"4ccvZvuQ","md5":"b8e39475a2ed9f8ad68915177c3ced5c","sha1":"f01c9bfdb776df443cf302e674df259dafbe7be0","sha256":"c514c25ff3923483ce139b8baf1932bc1f03e82aed7b6c99e8472e36a9577070"},"dob":{"date":"1949-10-29T14:40:11Z","age":69},"registered":{"date":"2002-03-30T18:05:22Z","age":17},"phone":"58178313","cell":"97485634","id":{"name":"FN","value":"29104900508"},"picture":{"large":"https://randomuser.me/api/portraits/women/74.jpg","medium":"https://randomuser.me/api/portraits/med/women/74.jpg","thumbnail":"https://randomuser.me/api/portraits/thumb/women/74.jpg"},"nat":"NO"},{"gender":"female","name":{"title":"mademoiselle","first":"tiffany","last":"michel"},"location":{"street":"8012 rue des ecrivains","city":"tujetsch","state":"valais","postcode":8047,"coordinates":{"latitude":"17.7426","longitude":"84.7045"},"timezone":{"offset":"+11:00","description":"Magadan, Solomon Islands, New Caledonia"}},"email":"tiffany.michel@example.com","login":{"uuid":"1bdff6a3-e005-4d55-af83-c10c6e2bd524","username":"blackbird202","password":"devildog","salt":"DBjwqCNp","md5":"1dd3d19403c1af2d8fc99bcf88df3892","sha1":"8a38aab0fbd7bd2e5229c7589922fb87c4101aee","sha256":"ee3b4f24d7686f019cfbea1f1d08e8b7926277853a3b84be9ce1f41a1ced250d"},"dob":{"date":"1968-12-09T13:44:54Z","age":50},"registered":{"date":"2005-06-11T16:03:32Z","age":13},"phone":"(904)-994-7298","cell":"(707)-614-1326","id":{"name":"AVS","value":"756.6921.2803.01"},"picture":{"large":"https://randomuser.me/api/portraits/women/96.jpg","medium":"https://randomuser.me/api/portraits/med/women/96.jpg","thumbnail":"https://randomuser.me/api/portraits/thumb/women/96.jpg"},"nat":"CH"},{"gender":"female","name":{"title":"ms","first":"dorota","last":"schiffmann"},"location":{"street":"kiefernweg 166","city":"erwitte","state":"brandenburg","postcode":40265,"coordinates":{"latitude":"1.5847","longitude":"96.2037"},"timezone":{"offset":"+3:00","description":"Baghdad, Riyadh, Moscow, St. Petersburg"}},"email":"dorota.schiffmann@example.com","login":{"uuid":"6d9a566f-283a-4fbc-bcbe-e6772547a342","username":"lazyzebra922","password":"goodtime","salt":"pmhtDQad","md5":"857d0b65744d965497f46e2020cf1b19","sha1":"10c469122851d2c38273f97ec00ca648b3de7a34","sha256":"322a589cf7053a1274fd1b2c304c022503a64d536fa23ded1554a61b97776fbc"},"dob":{"date":"1961-12-16T18:41:26Z","age":57},"registered":{"date":"2004-10-28T15:06:26Z","age":14},"phone":"0881-0401075","cell":"0171-4104339","id":{"name":"","value":null},"picture":{"large":"https://randomuser.me/api/portraits/women/37.jpg","medium":"https://randomuser.me/api/portraits/med/women/37.jpg","thumbnail":"https://randomuser.me/api/portraits/thumb/women/37.jpg"},"nat":"DE"}]}
$ curl http://localhost:3000?fields=results(gender,name)
{"results":[{"gender":"male","name":{"title":"mr","first":"andy","last":"pawlik"}},{"gender":"female","name":{"title":"miss","first":"inés","last":"van herk"}},{"gender":"female","name":{"title":"ms","first":"nicoline","last":"brekka"}},{"gender":"female","name":{"title":"mademoiselle","first":"tiffany","last":"michel"}},{"gender":"female","name":{"title":"ms","first":"dorota","last":"schiffmann"}}]}
$ curl http://localhost:3000?fields=results(login(username,password))
{"results":[{"login":{"username":"orangecat573","password":"111111"}},{"login":{"username":"crazylion993","password":"genesis1"}},{"login":{"username":"orangeduck796","password":"windows"}},{"login":{"username":"blackbird202","password":"devildog"}},{"login":{"username":"lazyzebra922","password":"goodtime"}}]}
json-mask it describes to itself like a language, so it has it own syntax:
- a,b,c comma-separated list will select multiple fields
- a/b/c path will select a field from its parent
- a(b,c) sub-selection will select many fields from a parent
- a/*/c the star * wildcard will select all items in a field
You can start to play with diferent requests and experiment with the results.
Conclusion
The partial response its a flexible way to give the users the information that they need.
It may not be so easy to implement once you have a REST API with some time in production, the implementation can be expensive in matter of source code modification. But it really worth have in mind this technique the next time that you design an API.
Top comments (3)
For REST API, just masking the response fields will not add much value as long as we limit resource loading. For ex - if an API returns response of 50+ fields sourced from 5 data sources (can be DB, other downstream serv) then without controlling the data load from only required sources it does add any value to the API owner.
Any design approach to solve this problem ?
Isn't this what GraphQL is all about ? - ask for the only data you need. :)
Yeah! Partial responses in GraphQL are build in the tool. However for developers building RESTful APIs, this implementation would be useful.