Adventures in Tracking Upload Progress With OkHttp and Retrofit

This article tells the story of how Stream’s Android team refined our progress tracking process during file uploads in the Stream Chat Android SDK.

Our original implementation to track file upload progress worked, but it had some in-code usability and UX issues that we wanted to clean up.

The following account gives an up-close look into the process we had, the problems we encountered, and what we did to improve.

Warning: As this is a story of mistakes we made and then corrected over time, do not use any of the initial or intermediate forms of code here in your own projects, only the final fixed version. 😉

Uploading Files With Retrofit

First things first, let's see how you can upload a file to an API using Retrofit. Most APIs will expect a multipart form to contain the file data. You can declare a method inside a Retrofit interface like the one below to support that operation:

@Multipart
@POST("/channels/{type}/{id}/file")
fun sendFile(
    @Path("type") channelType: String,
    @Path("id") channelId: String,
    @Part file: MultipartBody.Part,
): RetrofitCall<UploadFileResponse>

To invoke this method, you can create a Part by using the createFormData function, like so:

fun sendFile(file: File) {
    val part = MultipartBody.Part.createFormData("file", file.name,
                   file.asRequestBody(file.getMediaType()))
    retrofitCdnApi
        .sendFile(channelType, channelId, part)
        .enqueue(...)
}

Then you just enqueue the Retrofit Call to run it, and done! That's a basic, working file upload.

Counting in the Request Body

Time to add progress tracking. For our initial implementation, we used this callback interface:

public interface ProgressCallback {
    public fun onProgress(progress: Long)
    public fun onSuccess(file: String)
    public fun onError(error: ChatError)
}

We then created a ProgressRequestBody class, most likely based on this StackOverflow answer.

This wraps a file and a callback, and implements the OkHttp RequestBody class.

internal class ProgressRequestBody(
    private val file: File,
    private val callback: ProgressCallback
) : RequestBody() {
    override fun contentType(): MediaType = file.getMediaType()

    override fun contentLength(): Long = file.length()

    override fun writeTo(sink: BufferedSink) {
        val total = file.length()
        val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
        var uploaded = 0L
        FileInputStream(file).use { fis ->
            var read: Int
            val handler = Handler(Looper.getMainLooper())
            while (fis.read(buffer).also { read = it } != -1) {
                handler.post {
                    callback.onProgress((100 * uploaded / total))
                }
                uploaded += read.toLong()
                sink.write(buffer, 0, read)
            }
        }
    }

    companion object {
        private const val DEFAULT_BUFFER_SIZE = 2048
    }
}

Whenever writeTo is invoked to write this RequestBody to the network, we loop through the contents of the file manually, write the bytes, and invoke the callback, calculating the percentage of the upload completed so far.

This requires a modification to our API invocation, as we now have to create a ProgressRequestBody that we then embed in our Part, which will — remember this — contain the ProgressCallback instance that was passed in.

fun sendFile(file: File, callback: ProgressCallback) {
    val progressBody = ProgressRequestBody(file, callback)
    val part = MultipartBody.Part.createFormData("file", file.name, progressBody)

    retrofitCdnApi
        .sendFile(channelType, channelId, part)
        .enqueue(RetroProgressCallback(callback))
}

We also pass the callback to the enqueue call via a wrapper to adapt it to a Retrofit Callback, which will run the onFailure and onSuccess methods of our original callback as needed.

internal class RetroProgressCallback(
    private val callback: ProgressCallback
) : Callback<UploadFileResponse> {
    override fun onFailure(call: Call<UploadFileResponse>, t: Throwable) {
        callback.onError(ChatError(cause = t))
    }

    override fun onResponse(
        call: Call<UploadFileResponse>,
        response: retrofit2.Response<UploadFileResponse>
    ) {
        val body = response.body()
        if (body == null) {
            onFailure(call, RuntimeException("File response is null"))
        } else {
            callback.onSuccess(body.file)
        }
    }
}

If you want to explore this implementation, you can find the old version of our code here in the GitHub history.

Counting With a Custom Sink Implementation

This is okay, but we can make it a lot nicer. For this change, we essentially took the implementation from Paulina Sadowska's post about the topic.

The improvement is to (instead of handling a FileInputStream manually) create a custom OkHttp Sink implementation (based on the handy ForwardingSink), which will perform the counting and corresponding progress callbacks for us.

We'll also make ProgressRequestBody wrap a RequestBody and delegate to it whenever it needs to behave like a RequestBody.

internal class ProgressRequestBody(
    private val delegate: RequestBody,
    private val callback: ProgressCallback,
) : RequestBody() {
    override fun contentType(): MediaType? = delegate.contentType()
    override fun contentLength(): Long = delegate.contentLength()

    override fun writeTo(sink: BufferedSink) {
        val countingSink = CountingSink(sink).buffer()
        delegate.writeTo(countingSink)
        countingSink.flush()
    }

    private inner class CountingSink(delegate: Sink) : ForwardingSink(delegate) {
        private val handler = Handler(Looper.getMainLooper())
        private val total = contentLength()
        private var uploaded = 0L

        override fun write(source: Buffer, byteCount: Long) {
            super.write(source, byteCount)
            uploaded += byteCount

                        handler.post { callback.onProgress(uploaded, total) }
        }
    }
}

For this improvement, we've updated our ProgressCallback interface to take the more meaningful bytesUploaded and totalBytes values during progress updates instead of a percentage.

(We also added KDoc at this point to make the interface more obvious, which you can check out in the GitHub repo.)

public interface ProgressCallback {
    public fun onProgress(bytesUploaded: Long, totalBytes: Long)
    public fun onSuccess(url: String?)
    public fun onError(error: ChatError)
}

On the call site, we can now create a simple RequestBody first, and then wrap it with the ProgressRequestBody implementation:

val body = file.asRequestBody(file.getMediaType())
val progressBody = ProgressRequestBody(body, callback)
val part = MultipartBody.Part.createFormData("file", file.name, progressBody)

The Logging Problem

A major problem we encountered was that we had several interceptors added to the OkHttpClient that we used for these uploads. The configuration looked something like this:

baseClientBuilder()
    .connectTimeout(timeout, TimeUnit.MILLISECONDS)
    .writeTimeout(timeout, TimeUnit.MILLISECONDS)
    .readTimeout(timeout, TimeUnit.MILLISECONDS)
    .addInterceptor(ApiKeyInterceptor(...))
    .addInterceptor(HeadersInterceptor(...))
    .addInterceptor(TokenAuthInterceptor(...))
    .addInterceptor(HttpLoggingInterceptor())
    .addInterceptor(
         CurlInterceptor { message ->
             logger().logI("CURL", message)
         }
    )

This is natural, and lots of apps have setups like this. What's notable here is that HttpLoggingInterceptor and CurlInterceptor will both log the request and call requestBody.writeTo() internally to do that.

In the case of our file upload calls, this method is what contains our progress tracking implementation.

The end result is that whenever we make an upload call, we'll run the progress callback three times in a row: twice for the logging, and once when it's actually written to the network.

This results in an... interesting experience for the user looking at the UI, where progress goes something like this:

0 15 45 50 100 0 25 37 57 87 100 0 20 67 100

This only happened in debug builds, as we had the logging disabled in release builds, but it was still a problem.

Fixing this wasn't simple, as we wanted to keep these logging interceptors around. After some elaboration, we begrudgingly added an ugly, ugly workaround, like this:

internal class ProgressRequestBody(
    private val delegate: RequestBody, ...
) : RequestBody() {

    var writeCount = 0

    override fun writeTo(sink: BufferedSink) {
        if (writeCount >= progressUpdatesToSkip) {
            val countingSink = CountingSink(sink).buffer()
            delegate.writeTo(countingSink)
            countingSink.flush()
        } else {
            delegate.writeTo(sink)
        }

        writeCount++
    }

    companion object {
        private val progressUpdatesToSkip = 2
    }
}

As you can see, we simply hardcoded that the first two times when the ProgressRequestBody is written, it shouldn't invoke callbacks.

What made this worse is that this value wasn't actually 2 like I assumed, because as mentioned above, we only log with these interceptors in debug builds.

This meant that we had to make this a var and set it dynamically. In release builds it'd be 0, and in debug builds, we'd increment it to 2. Ouch.

That's bad enough, but then we also had the requirement to allow our users to set their own OkHttpClient instances for the SDK to use, where they could also add more interceptors of their own, which may or may not invoke writeTo in the request body (one or more times!).

We could've somehow provided an additional API where they can increment progressUpdatesToSkip to account for this, but then they could also have interceptors that will sometimes read the body but not at other times, based on some dynamic condition... There's clearly no winning with this approach, and it's an awful rabbit hole to go down.

So, to quote myself from the workaround PR linked above:

A real solution would be intercepting the call with an OkHttp network interceptor, but we can't pass in individual callbacks per different Retrofit upload calls with that approach (at least not easily).

So the problem was that we had no way to get the callback value from the call site that invokes the Retrofit method down to the interceptor attached to the underlying OkHttp client.

Turns out, thankfully, that I was wrong about that!

OkHttp Tags

Enter OkHttp's tags API (many thanks to Jesse Wilson for showing me this!). With this API, the library allows you to add arbitrary tags (objects) to your requests when building them.

As the docs say, you can:

Use this API to attach timing, debugging, or other application data to a request so that you may read it in interceptors, event listeners, or callbacks.

When building the Request, you can pass in a Class as a key and then an object of that type as the associated value:

val request = Request.Builder()
    .post(requestBody)
    .tag(ProgressCallback::class.java, callback)
    .build()

And then later you can read these values from the Request:

val cb: ProgressCallback? = request.tag(ProgressCallback::class.java)

One small problem is that in our file upload code we create a RequestBody manually, but not the actual Request object, as that's created under the hood by Retrofit.

Thankfully, the tagging API is also exposed through Retrofit, so you can add tags to methods using the @Tag annotation on parameters. We'll use this in the next section!

The last-interceptor-swaparoo

The strategy then is the following:

1. Add the ProgressCallback instance as a tag when making the Retrofit call.
2. Create an interceptor at the end of the interceptor chain that will check each outgoing request and wrap its RequestBody into a ProgressRequestBody if the callback is present on it.

First, we'll update the Retrofit interface to accept a progressCallback parameter that will be used as a tag:

@Multipart
@POST("/channels/{type}/{id}/file")
fun sendFile(
    @Path("type") channelType: String,
    @Path("id") channelId: String,
    @Part file: MultipartBody.Part,
    @Tag progressCallback: ProgressCallback?,
): RetrofitCall<UploadFileResponse>

Then, we'll implement the interceptor that reads this tag and performs the wrapping of the RequestBody if needed:

internal class ProgressInterceptor : Interceptor {
    override fun intercept(chain: Interceptor.Chain): Response {
        val request = chain.request()

        val progressCallback = request.tag(ProgressCallback::class.java)
        if (progressCallback != null) {
            return chain.proceed(wrapRequest(request, progressCallback))
        }

        return chain.proceed(request)
    }

    private fun wrapRequest(request: Request, progressCallback: ProgressCallback): Request {
        return request.newBuilder()
            // Assume that any request tagged with a ProgressCallback is a POST
            // request and has a non-null body            
            .post(ProgressRequestBody(request.body!!, progressCallback))
            .build()
    }
}

Finally, we'll add this interceptor to our OkHttpClient, making sure it's the last one added.

We'll add it as a network interceptor, as we want it to be as close to the actual upload as possible, and it doesn't need to be involved in requests that don't go out to the network.

return baseClientBuilder()
    .addInterceptor(...)
    .addInterceptor(...)
    .addNetworkInterceptor(ProgressInterceptor())

This way nothing that previous interceptors do with the request will interfere with the progress reporting, as they'll still have the original RequestBody to work with, and the special progress-tracking wrapper is only added at the very last moment before it actually goes out to the network.

If you want to see this last change in detail, check out the corresponding PR on GitHub.

Wrap-Up

So that's the implementation we ended up with for now! You can find all of this code in the Chat Android SDK's GitHub repository if you want to look at it in a real project.

There is one remaining issue with this progress tracking: Whenever the body gets written, it's only written into local buffers, and what we track is how fast we're writing to that buffer. This then still needs to make it over the network, which can take some time.

So the upload progress tends to get to 100% relatively quickly and then the request will remain pending for a while as the network call completes.

This is as close to the socket (so to say) as we can get with these APIs. For a bit more info and references, see this GitHub discussion.

If you want to learn more about how Okio (and OkHttp, Retrofit, and Moshi) work super efficiently with data, watch A Few Ok Libraries by Jake Wharton. For an introduction to Moshi, check out Say Hi to Moshi.