I have to deal with a third-party backend with the following properties:
- It requires my request payload to be gzipped.
- It does not like a "chunked" transfer encoding, so I have to provide the actual content length as a header.
- It doesn't understand how to parse headers; it expects exact matches with its requirements.
- It sometimes responds with a gzipped payload, and other times does not. It depends on which physical server is hit.
- It sometimes responds with a JSON array wrapping the main body (
[{ ... }]) and other times with just the main body ({ ... }). It depends on which physical server is hit.
What follows is a true story. Some class names have been changed to protect the innocent.
Dictionary.com gets it.
Gzipping a request body and providing the real content length
Gzipping your request body is actually straightforward, and the official docs provide an exact recipe for doing so.
final class GzipRequestInterceptor implements Interceptor {
@Override public Response intercept(Interceptor.Chain chain) throws IOException {
Request originalRequest = chain.request();
if (originalRequest.body() == null || originalRequest.header("Content-Encoding") != null) {
return chain.proceed(originalRequest);
}
Request compressedRequest = originalRequest.newBuilder()
.header("Content-Encoding", "gzip")
.method(originalRequest.method(), gzip(originalRequest.body()))
.build();
return chain.proceed(compressedRequest);
}
private RequestBody gzip(final RequestBody body) {
return new RequestBody() {
@Override public MediaType contentType() {
return body.contentType();
}
@Override public long contentLength() {
return -1; // We don't know the compressed length in advance!
}
@Override public void writeTo(BufferedSink sink) throws IOException {
BufferedSink gzipSink = Okio.buffer(new GzipSink(sink));
body.writeTo(gzipSink);
gzipSink.close();
}
};
}
}
If you return -1 as the content length (as above), then OkHttp will automatically add the Transfer-Encoding: chunked header to your request. As mentioned earlier, my janky backend does not like such a transfer-encoding.
It is not immediately obvious how to provide a non--1 content length, and unfortunately there is no public recipe. However, a bit of searching online uncovered this solution, which happens to live on OkHttp's Github issue tracker (which has been upvoted enough it probably warrants being an official recipe, but I digress):
class GzipRequestInterceptor implements Interceptor {
@Override public Response intercept(Chain chain) throws IOException {
...
Request compressedRequest = originalRequest.newBuilder()
.header("Content-Encoding", "gzip")
.method(originalRequest.method(), forceContentLength(gzip(originalRequest.body())))
.build();
return chain.proceed(compressedRequest);
}
private RequestBody forceContentLength(final RequestBody requestBody) throws IOException {
final Buffer buffer = new Buffer();
requestBody.writeTo(buffer);
return new RequestBody() {
@Override
public MediaType contentType() {
return requestBody.contentType();
}
@Override
public long contentLength() {
return buffer.size();
}
@Override
public void writeTo(BufferedSink sink) throws IOException {
sink.write(buffer.snapshot());
}
};
}
private RequestBody gzip(final RequestBody body) {
...
}
}
where you can see that the gzipped RequestBody has been wrapped in yet another RequestBody that writes out to a buffer so we can get the number of bytes via buffer.size().
So, this solves my first two problems, which is that my request must be (1) gzipped and (2) provide a real content-length (not be chunked). 🎉
Understanding HTTP header semantics, or, The quest for HTTP 200
This is the story of how I learned that Retrofit will automatically add/override a Content-Type header to be "application/json; charset=UTF-8".
Despite resolving the first two problems and (seemingly) meeting the spec, I was met with an HTTP 400 at every turn. The response gave no indication what was causing the 400; all I knew was that the server did not like my request. I scrutinized every header, I double-checked the content length, I tweaked the payload in countless ways — always an opaque 400.
I learned that another team at a different company had found a solution using Volley. I confess I thought that library dead long ago. Trying to be a team player, I made an attempt at this, but despair set in. Surely there was another way!
That's when I decided to use what I called "raw OkHttp." Rather than use Retrofit as a fancy wrapper for creating my requests, I built the request myself out of JSONObjects. When I finally launched the app with this approach, I watched in excited bewilderment as it actually returned a 200! The body completely failed to deserialize, of course,1 but I was a step closer to a solution.
Here's what that "raw" request looked like:
class RawApi(private val client: OkHttpClient) {
fun makeRequest(theRequest: TheRequest): TheResponse? {
client.newCall(newRequest(theRequest)).execute().use { response ->
if (!response.isSuccessful) {
throw IOException("Unexpected code $response")
}
return bodyFromResponse(response)
}
}
private fun bodyFromResponse(response: Response): TheResponse {
val body = if (response.header("Content-Encoding", "")!!.contains("gzip")) {
GzipSource(response.body!!.byteStream().source())
.use { source ->
source.buffer().use {
it.readUtf8()
}
}
} else {
response.body!!.string()
}
return fromJson(body)
}
private fun fromJson(body: String): TheResponse = when {
body.startsWith("[") -> {
// The form should be "[{ ... }]". I.e., a json object wrapped in a json array.
val type = object : TypeToken<List<TheResponse>>() {}.type
Gson().fromJson<List<TheResponse>>(body, type).first()
}
body.startsWith("{") -> {
// The form should be "{ ... }". I.e., a json object, without an array wrapper.
Gson().fromJson<TheResponse>(body, TheResponse::class.java)
}
else -> {
throw IOException("Cannot deserialize this body. It should start with { or [. Was $body")
}
}
private fun newRequest(theRequest: TheRequest): Request = Request.Builder()
.url("https://some/url")
.post(newRequestBody(theRequest))
.build()
private fun newRequestBody(theRequest: TheRequest): RequestBody {
val request = JSONArray()
val body = JSONObject()
body.put("param-1", theRequest.param1)
// ...etc...
request.put(body)
return object : RequestBody() {
override fun contentType(): MediaType? {
return "application/json".toMediaType()
}
override fun writeTo(sink: BufferedSink) {
sink.writeUtf8(request.toString())
}
}
}
}
Here you can see how I optionally decompress from gzip, and also differentially deserialize from either a JSON array or a JSON object. I'll return to this topic in just a moment.
So, how does this solution differ from the one using Retrofit? I compared the two requests (their respective headers and bodies), and saw that the Retrofit request had this header:
Content-Type: application/json; charset=UTF-8
whereas the raw OkHttp request had
Content-Type: application/json
And this is how I learned that this janky backend doesn't do proper header parsing, because when I modified my Retrofit request (using a network interceptor) to remove the charset from the header, my request worked — I got a 200.
ðŸ˜
Returning to Retrofit + OkHttp
Optionally decompressing the response
First, you need an interceptor:
class GunzipInterceptor : Interceptor {
override fun intercept(chain: Interceptor.Chain): Response {
// Handle the response, which may or may not be gzipped
val response = chain.proceed(gzipped)
val contentEncoding: String? = response.header("Content-Encoding")
if (contentEncoding == null || !contentEncoding.contains("gzip")) {
// Not gzipped, so just proceed with the original response
return response
}
// Gzipped, so decompress it and return new, uncompressed response body
val originalResponseBody = response.body!!
val gzipSource = GzipSource(originalResponseBody.byteStream().source()).buffer()
val responseBody = decompressed(gzipSource, originalResponseBody.contentType())
return response.newBuilder()
.body(responseBody)
.build()
}
}
private fun decompressed(source: BufferedSource, contentType: MediaType?): ResponseBody {
return object : ResponseBody() {
override fun contentLength(): Long = source.buffer.size
override fun contentType(): MediaType? = contentType
override fun source(): BufferedSource = source
}
}
}
Here we trust that, if the server sends the Content-Encoding: gzip header, then the body truly is gzipped.2
Differentially deserializing the response
Here we leverage a Gson JsonDeserializer, alongside a Retrofit Converter.Factory.
class ResponseDeserializer : JsonDeserializer<TheResponse> {
override fun deserialize(
json: JsonElement, typeOfT: Type?, context: JsonDeserializationContext?
): TheResponse {
val jsonObject = when {
json.isJsonObject -> json.asJsonObject
json.isJsonArray -> json.asJsonArray.get(0).asJsonObject
else -> error("json neither an array nor an object")
}
return Gson().fromJson<TheResponse>(jsonObject, TheResponse::class.java)
}
}
"Fixing" the content type
To be clear, there is nothing inherently wrong with Retrofit's behavior of appending "; charset=UTF-8" to the Content-Type header. It's part of the standard. However, as indicated, I have a janky server, so I need to remove the charset. Here's how you set the exact header you want your backend to see. Note that it must be a network interceptor.
okHttpClientBuilder.
// ...etc...
.addNetworkInterceptor { chain ->
val original = chain.request()
val simplified = original.newBuilder()
.header("Content-Type", "application/json")
.build()
return chain.proceed(simplified)
}
.build()
(Please see the excellent documentation for a discussion of how a regular interceptor differs from a network interceptor.)
Tying it all together
Build your Retrofit instance like so
Gson gson = new GsonBuilder()
.registerTypeAdapter(TheResponse.class, new ResponseDeserializer())
.create();
GsonConverterFactory factory = GsonConverterFactory.create(gson);
return new Retrofit.Builder()
.addConverterFactory(factory)
.callFactory(okHttpClient)
// ...etc...
.build()
And now, finally, it cough just works.
Special thanks
Special thanks to Jesse Wilson and Jake Wharton for discussing this issue with me and helping me to resolve it. Only lost a bit of sanity in the process!
Endnotes
1 Because the server is janky. up
2 We shouldn't trust this, because the server is janky. up
Top comments (4)
As a backend developer, wtf
That makes me feel better, thank you :D
To expand a bit on this, and I feel I should write this in a longer form.
The backend (wtv you call it) should be built as a service, it serves the end user through frontend (wtv be it mobile/web). If the developers working with the backend have to jump through hoops, their work suffers, the users also suffer.
This feels like gatekeeping that just makes everyone want to quit.
PS:
depends on which physical server is hit.this makes me think that these backends aren't running the same version of the code, which is bad in several levels.Totally agree with you on all points. It has definitely made me want to quit several times :)