User agents are an important part of web scraping. In order to gather accurate and relevant information from the web, user agent strings have to be set right. User agents can define what information a target website sends to the user and how the content is displayed.
Data gathering for SEO, marketing, competitor monitoring, and other business cases requires careful preparation. Getting proxies, setting user agents, and bypassing blocks are essential for successful web scraping.
Find out what a user agent is and why it’s so important for web scraping. Learn about the most common user agents and their types depending on different devices. After reading this article, you’ll be able to set your user agents and get the most accurate and relevant data that you need for your business.
What Is a User Agent?
Every single browser has a user agent. It represents the user on the internet by providing information about the user, such as its browser, operating system, device type, and software. Providing this information manually every single time when connecting to the internet would be highly inefficient. That’s why every browser connected to the internet has a user agent.
For example, a user agent can look like this:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
User agents help websites adapt their content to different web browsers and operating systems.
Why Are User Agents Important?
User agents are very important because they set browsers apart from each other and ensure that each user gets the content displayed correctly.
User agent strings are included in the website’s HTTP header when they connect to a website. Identifying users based on their user agents allows websites to provide different versions of content through the same URL address.
For example, once you enter a URL, the web server checks your user agent and provides you with the appropriate website. If you want to access the same site via your mobile device, you don’t have to enter a different URL. The same URL gives you different versions of a website on both mobile and computer browsers.
To give you another example of why user agents are important, think about different image formats. A website can provide images in PNG and GIF formats and display those depending on the user agents. The GIF version will be displayed to users with MS Internet Explorer versions that cannot show PNG images. At the same time, the PNG image version will be displayed in more recent browser versions.
That’s why user agents are important because, without them, users wouldn’t get the content they expect.
What Are the Different Types of User Agents?
User agent strings enable website servers to identify the devices (among other things) requesting online content. The user agent tells the website what device is visiting the site, and this information is then used to determine what content should be returned.
Here’s a user agent list for different device types:
Android User Agents
Android mobile user agents depend on the mobile phone device. Android devices can be Samsung, Sony, Nexus, and other phones using Android OS for mobile devices. Since Android is based on the Linux kernel, the user agent will always contain Linux. For example, a user agent for a Samsung Galaxy S22 phone will look like this:
Mozilla/5.0 (Linux; Android 12; SM-S906N Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.119 Mobile Safari/537.36
iPhone User Agents
Apple passes different information through the user agents. Contrary to Android devices, Apple doesn’t use version numbers that would allow differentiating between different iPhone models. Here’s an example of an iPhone 13 Pro Max user agent:
Mozilla/5.0 (iPhone14,3; U; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/19A346 Safari/602.1
MS Windows User Agents
Microsoft Windows mobile devices also have their own user agents. For example, this could be a user agent of Microsoft Lumia 650:
Mozilla/5.0 (Windows Phone 10.0; Android 6.0.1; Microsoft; RM-1152) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Mobile Safari/537.36 Edge/15.15254
Tablet User Agents
Tablet user agents depend on the OS your device is using and the model of your tablet. For instance, the user agent for a Sony Xperia Z4 tablet could look like this:
Mozilla/5.0 (Linux; Android 6.0.1; SGP771 Build/32.2.A.0.253; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/52.0.2743.98 Safari/537.36
Desktop User Agents
Desktop user agents can have a myriad of different combinations. These depend on the device, OS, browser, etc. Here’s what a user agent for a MacOS X-based computer connecting via a Safari browser could appear like:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9
List of Most Common User Agents
For web scraping, you need to use the most common user agents to gather various information. With one user agent you may get different information than with another user agent string. Here are the latest and most common user agents for different web browsers and operating systems:
Chrome on Windows 10 User Agent
This is currently the most popular user agent:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
Chrome on macOS User Agent
The most common user agent on macOS:
Mozilla/5.0 (Macintosh; Intel Mac OS X 12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
Chrome on Linux User Agent
The latest and most popular Linux user agent:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
Chrome on Android User Agents
The latest Android user agent with Chrome browser is as follows:
Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5249.79 Mobile Safari/537.36
These are currently the most common user agents, but the list may change. You can find a dynamic list of the most popular user agents online.
How To Change the User Agent?
You can change your user agent string and appear as sending requests to your target website from a different browser or device than you actually are. How to change the user agent can depend on your browser.
We’ll tell you how to change your user agent string on Chrome, the most commonly used web browser.
- Click on the menu on your Chrome browser
- Go to “More tools” and then “Developer tools”
- In the Console tab, click on the menu (if you can’t see the
- Console, click on the menu and select “Show console”)
- Pick “Network conditions” and look for “User agent” option
- Uncheck the “Select automatically” box and choose the user agent from the list
If you’re not happy with the list, you can set a custom user agent. However, your custom user agent string will only apply as long as the “Developer tools” is open and only to your current tab.
Conclusion
Every internet user with a device and a browser has a user agent. This string helps the target website to identify the user and return adapted content, which displays content in the correct way. This can mean displaying certain image formats, language, etc.
When web scraping, you often need to collect information from various locations and for different devices. This information can be used for SEO, e-commerce purposes, or competitor monitoring. Customizing your user agent can help you get the data you need.
You can change your user agent on Chrome by editing your “Developer tools” settings and either choosing a suggested user agent string or customizing it.
User agents depend on device types and can vary greatly depending on your browser, language settings, operating system, software, etc. When gathering information from the web, you may want to use the most common user agents to make sure you get accurate and relevant information.
Top comments (0)