Scrapfly for Scrapfly

Posted on Dec 5 • Originally published at scrapfly.io on Dec 5

Comprehensive Guide to OkHttp for Java and Kotlin

#http #tools #java

When building applications that communicate over the web, a reliable and efficient HTTP client is essential. OkHttp, one of the most popular HTTP clients for Java and Android development, stands out as a powerful tool for handling network operations.

This article will explore OkHttp’s key features, setup instructions, basic usage, advanced functionalities, and common issues you might encounter.

What is OkHttp?

OkHttp is an HTTP client designed for both Java and Kotlin, widely recognized for its reliability and performance. It simplifies network communication by providing a consistent interface for making HTTP requests, handling responses, and optimizing network traffic.

Whether you're developing a mobile app or a backend service, OkHttp offers a robust solution for dealing with HTTP requests and responses efficiently.

In Android and Java development, OkHttp acts as a modern alternative to other HTTP clients, thanks to its lightweight design and support for features like connection pooling and timeouts. It’s a go-to choice for developers aiming to implement complex network operations with minimal configuration.

Main OkHttp Features

OkHttp offers an impressive range of features designed to enhance network performance and simplify development. Below is a feature table showcasing some of its capabilities:

Feature	Description
HTTP/2 Support	Enables multiplexed requests for better connection pooling.
Async Requests	Facilitates non-blocking, parallel network requests.
Interceptors	Customizes requests and responses (logging, modifying headers).
Built-in Caching	Reduces network usage by caching responses.
Timeouts	Configurable connection, read, and write timeouts.
HTTPS Support	Out-of-the-box HTTPS support with certificate pinning.
Compression	GZIP and deflate for response compression.
Redirect Handling	Automatic handling of 3xx HTTP redirects.
WebSocket Support	Full support for WebSocket connections.
Authentication Support	Basic, Digest, and custom authentication mechanisms.

While OkHttp covers a wide array of features, there are a few limitations to be aware of. Notably, it lacks built-in retry strategies, and modern conveniences like auto .json() methods are not natively included. However, these shortcomings can often be mitigated with custom implementations or third-party libraries.

Setting Up OkHttp

Setting up OkHttp is straightforward, whether you’re developing a Java application or an Android project. For Java, you can include OkHttp by adding it to your Maven or Gradle build configuration. Below are installation instructions:

To install okhttp using Maven :

<dependency>
  <groupId>com.squareup.okhttp3</groupId>
  <artifactId>okhttp</artifactId>
  <version>4.12.0</version>
</dependency>

To install okhttp using Java Gradle :

implementation 'com.squareup.okhttp3:okhttp:4.12.0'

To install okhttp using Kotlin Gradle :

implementation("com.squareup.okhttp3:okhttp:4.12.0")

Basic Usage

To get started with OkHttp, let’s explore some basic usage patterns.

GET Requests and Response Details

Making a simple GET request with OkHttp is simple. Here’s how you can retrieve the response body and status code:

OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
    .url("https://web-scraping.dev/api/products")
    .build();

try (Response response = client.newCall(request).execute()) {
    if (response.isSuccessful()) {
        System.out.println("Response Body: " + response.body().string());
        System.out.println("Status Code: " + response.code());
    }
}

Handling Errors

Error handling in OkHttp is critical to ensure smooth user experiences. Common issues such as connection timeouts or 4xx/5xx HTTP errors can be caught and managed using try-catch blocks. You can also inspect the status code to handle specific HTTP errors.

try (Response response = client.newCall(request).execute()) {
    if (!response.isSuccessful()) {
        throw new IOException("Unexpected code " + response);
    }
}

Synchronous and Asynchronous Requests

OkHttp supports both synchronous and asynchronous requests, providing flexibility based on your application’s needs. The synchronous request waits for the response before proceeding, while the asynchronous request runs in the background.

// Synchronous
Response response = client.newCall(request).execute();

// Asynchronous
client.newCall(request).enqueue(new Callback() {
    // Add callback for success:
    @Override
    public void onResponse(Call call, Response response) throws IOException {
        System.out.println(response.body().string());
    }
    // Add callback for failure:
    @Override
    public void onFailure(Call call, IOException e) {
        e.printStackTrace();
    }
});

Advanced Usage

OkHttp also supports a variety of advanced use cases, including different HTTP methods, handling headers and cookies, and configuring timeouts.

POST, PUT, HEAD, PATCH Requests

OkHttp allows you to make POST, PUT, HEAD, and PATCH requests with ease. Here’s are basic examples of each:

POST

RequestBody body = RequestBody.create(MediaType.parse("application/json; charset=utf-8"), "{\"key\":\"value\"}");
Request request = new Request.Builder()
    .url("https://httpbin.dev/post")
    .post(body)
    .build();

HEAD

Request request = new Request.Builder()
    .url("https://httpbin.dev/head")
    .head() // HEAD request does not require a body
    .build();

PUT

RequestBody body = RequestBody.create(MediaType.parse("application/json; charset=utf-8"), "{\"key\":\"updatedValue\"}");
Request request = new Request.Builder()
    .url("https://httpbin.dev/put")
    .put(body) // PUT request with a body
    .build();

PATCH

RequestBody body = RequestBody.create(MediaType.parse("application/json; charset=utf-8"), "{\"key\":\"patchedValue\"}");
Request request = new Request.Builder()
    .url("https://httpbin.dev/patch")
    .patch(body) // PATCH request with a body
    .build();

Adding Query Parameters

OkHttp provides a straightforward way to add query parameters to your requests without manually building the URL string. Using the HttpUrl.Builder class, you can easily add key-value pairs as query parameters to a URL before making the request.

OkHttpClient client = new OkHttpClient();

HttpUrl.Builder urlBuilder = HttpUrl.parse("https://web-scraping.dev/api/products")
        .newBuilder();
urlBuilder.addQueryParameter("page", "2");
urlBuilder.addQueryParameter("order", "desc");
urlBuilder.addQueryParameter("category", "apparel");

String url = urlBuilder.build().toString();

// Building the request with the URL containing query parameters
Request request = new Request.Builder()
        .url(url)
        .build();

Adding Headers

You can add custom headers to your requests, like authentication tokens or user-agent strings:

Request request = new Request.Builder()
    .url("https://web-scraping.dev/api/products")
    .addHeader("Authorization", "Bearer token")
    .build();

Adding Cookies

OkHttp also allows you to manage cookies by using its CookieJar interface. This enables you to persist cookies between requests or manage them manually.

OkHttpClient client = new OkHttpClient.Builder()
        .cookieJar(new CookieJar() {
            @Override
            public void saveFromResponse(HttpUrl url, List<Cookie> cookies) {
                for (Cookie cookie : cookies) {
                    System.out.println("Saving cookie: " + cookie);
                }
            }

            @Override
            public List<Cookie> loadForRequest(HttpUrl url) {
                return List.of(); // Empty for this example
            }
        })
        .build();

Request request = new Request.Builder()
        .url("https://web-scraping.dev/blocked")
        .build();

Adding Timeout

OkHttp gives you the ability to configure connection, read, and write timeouts for each HTTP request:

OkHttpClient client = new OkHttpClient.Builder()
        .connectTimeout(10, TimeUnit.SECONDS) // Set connection timeout
        .writeTimeout(10, TimeUnit.SECONDS) // Set write timeout
        .readTimeout(30, TimeUnit.SECONDS) // Set read timeout
        .build();

Request request = new Request.Builder()
        .url("https://web-scraping.dev/api/products")
        .build();

Adding Retries

Although OkHttp doesn’t have built-in retry strategies, you can easily add them yourself by using interceptors or implementing custom logic for retries.

To implement it using interceptors, we can create a new okhttp interceptor that has the retry logic.

public class RetryInterceptor implements Interceptor {

    private int maxRetries;

    public RetryInterceptor(int maxRetries) {
        this.maxRetries = maxRetries;
    }

    @Override
    public Response intercept(Chain chain) throws IOException {
        Request request = chain.request();
        Response response = null;
        IOException exception = null;

        int attempt = 0;

        // Attempt the request up to maxRetries times
        while (attempt < maxRetries) {
            try {
                response = chain.proceed(request);

                // If response is successful, return it
                if (response.isSuccessful()) {
                    return response;
                }
            } catch (IOException e) {
                exception = e;
            } finally {
                attempt++;
            }

            // Wait before retrying the request (optional delay)
            if (attempt < maxRetries) {
                try {
                    Thread.sleep(2000); // 2-second delay between retries
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }
        }

        // If all attempts failed, throw the exception
        if (exception != null) {
            throw exception;
        }

        // If the response is null and maxRetries are exhausted, throw an IOException
        throw new IOException("Maximum retry attempts (" + maxRetries + ") exhausted");
    }
}

Now we can use this interceptor within our okhttp client builder.

OkHttpClient client = new OkHttpClient.Builder()
                .addInterceptor(new RetryInterceptor(3)) // Add custom retry interceptor
                .build();

Request request = new Request.Builder()
        .url("https://web-scraping.dev/api/products")
        .build();

Disabling HTTPS

In certain scenarios, you might need to disable HTTPS validation. While this is generally not recommended, it can be done for testing purposes:

OkHttpClient client = new OkHttpClient.Builder()
    .sslSocketFactory(createUnsafeSslSocketFactory(), new X509TrustManager() {
        // Implementation here
    })
    .build();

WebSocket

OkHttp makes it simple to work with WebSockets, allowing you to easily connect, send, and receive messages in real-time. In the example below, you will see how to establish a WebSocket connection using OkHttp, send a message to the server, and handle incoming messages asynchronously using WebSocketListener.

OkHttpClient client = new OkHttpClient();

Request request = new Request.Builder()
        .url("wss://example.com/socket")
        .build();

WebSocketListener listener = new WebSocketListener() {
    @Override
    public void onOpen(WebSocket webSocket, Response response) {
        System.out.println("WebSocket Opened");
        webSocket.send("Hello Server!");
    }

    @Override
    public void onMessage(WebSocket webSocket, String text) {
        System.out.println("Message from server: " + text);
    }

    @Override
    public void onFailure(WebSocket webSocket, Throwable t, Response response) {
        t.printStackTrace();
    }
};

client.newWebSocket(request, listener);
client.dispatcher().executorService().shutdown();

Common OkHttp Errors and Issues

While OkHttp is highly reliable, you may encounter errors from time to time. Some common issues include:

Issue/Exception	Description	Solution
java.net.SocketTimeoutException	Occurs when the connection, read, or write operation exceeds the set timeout.	Increase the timeout duration with `.connectTimeout()`, `.readTimeout()`, or `.writeTimeout()` on `OkHttpClient`.
java.net.UnknownHostException	Indicates that the domain name cannot be resolved by DNS.	Check network connection, ensure DNS is reachable, or configure a custom DNS with OkHttp.
java.io.IOException: unexpected end of stream	Often caused by network disconnection or improper server response.	Retry the request or ensure server responses are properly formatted.
SSLHandshakeException	SSL/TLS handshake fails, commonly due to incorrect certificates or unsupported protocols.	Ensure correct SSL certificates, or set `hostnameVerifier` and `certificatePinner` if needed.
java.net.ProtocolException	Indicates a protocol error, such as an invalid HTTP response or request method.	Verify request setup (e.g., method type, headers) or handle specific protocols.
java.io.EOFException	Happens when OkHttp reads an incomplete response body from the server.	Use interceptors to handle partial responses or consider increasing the read timeout.
java.net.ConnectException	Failed to connect to the server, often due to unreachable server or network issues.	Check server availability, network connection, or retry the connection.
HTTP 413 Payload Too Large	Server rejects the request due to a large payload.	Reduce payload size or consult the server for acceptable payload limits.
HTTP 429 Too Many Requests	Server rate-limiting response due to too many requests in a short period.	Implement retry logic with backoff or adhere to rate limits.
ResponseBody is null	Occurs when a request is successful but the server returns no content.	Check if the endpoint is expected to return data, or handle null `ResponseBody` safely.
TimeoutException: A connection timeout occurred	Appears when establishing the connection takes too long.	Increase connection timeout or use exponential backoff for retries.
IOException: Cleartext HTTP traffic not permitted	Occurs on Android when accessing a plain HTTP URL instead of HTTPS.	Add `cleartextTrafficPermitted` in the network security config or switch to HTTPS if possible.
IllegalArgumentException: unexpected url	Raised when an invalid or malformed URL is used in the request.	Ensure the URL is correctly formatted and valid.

Power Up With Scrapfly

While OkHTTP can be used for web scraping purposes, you will often find yourself facing blocks and rate-limits.

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

Here is a simple example of how you can use okhttp with Scrapfly's Scraping API.

import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;

import java.io.IOException;

public class OkHttpExample {
    public static void main(String[] args) {
        OkHttpClient client = new OkHttpClient();
        HttpUrl.Builder urlBuilder = HttpUrl.parse("https://api.scrapfly.io/scrape")
                .newBuilder();
        // Required parameters: your API key and URL to scrape
        urlBuilder.addQueryParameter("key", "YOUR_API_KEY");
        urlBuilder.addQueryParameter("url", "https://web-scraping.dev/product/1");
        // Optional parameters:
        // enable anti scraping protection bypass
        urlBuilder.addQueryParameter("asp", "true");
        // use proxies of a specific countries
        urlBuilder.addQueryParameter("country", "US,CA,DE");
        // enable headless browser
        urlBuilder.addQueryParameter("render_js", "true");
        // see more on scrapfly docs: https://scrapfly.io/docs/scrape-api/getting-started#spec

        // Building and send request
        String url = urlBuilder.build().toString();
        Request request = new Request.Builder()
                .url(url)
                .build();
        try (Response response = client.newCall(request).execute()) {
            if (response.isSuccessful()) {
                System.out.println("Response Body: " + response.body().string());
                System.out.println("Status Code: " + response.code());
            } else {
                System.out.println("Request Failed: " + response.code());
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

FAQ

To wrap our intro to okhttp let's take a look at some frequently asked questions that we might have missed in our guide:

Can OkHttp be used for Crawling?

Yes, OkHttp can be used for web scraping and crawling tasks. It handles multiple concurrent HTTP requests, making it a solid choice for crawlers. However, for specialized crawling purposes, you may want to look into dedicated crawl APIs for better efficiency.

Can OkHttp handle large file uploads or downloads?

Yes, OkHttp is capable of handling large file uploads and downloads efficiently. It allows you to stream data via RequestBody for uploads and ResponseBody for downloads, ensuring that large files can be processed in chunks without consuming excessive memory. You can implement custom logic to track progress, manage buffers, and avoid memory issues during these operations.

Is OkHttp thread-safe?

Yes, OkHttp is designed to be thread-safe. The OkHttpClient instance itself can be shared across multiple threads, making it ideal for concurrent requests. However, individual Request and Response objects are not thread-safe, so they should be used in a single thread or context. To maximize efficiency, it’s recommended to reuse the same OkHttpClient instance across your application.

Summary

OkHttp is a versatile and powerful HTTP client that provides developers with the tools they need for smooth, efficient network communication in Java and Android development. Here's a quick recap of the key points we covered:

Supports HTTP/2 , asynchronous requests , interceptors , built-in caching , timeouts , and HTTPS.
Offers features like compression , redirect handling , WebSocket , and authentication support.
Simple to set up using Maven or Gradle.
Provides an easy way to handle GET , POST , and other HTTP methods, both synchronously and asynchronously.
Add custom headers , cookies , and configure timeouts easily.
Handle advanced cases like retry logic and HTTPS disabling.
Typical errors include timeouts , connection issues , and SSL/TLS errors , all of which can be managed with proper handling and configuration.

OkHttp is a reliable choice for developers seeking control, efficiency, and a rich set of features for network communication. Its strengths lie in its simplicity combined with the flexibility needed for more advanced use cases.

DEV Community