Rotating Proxy Servers in Python

Recently, I have used ProxyMesh proxy server for a project. ProxyMesh offers 15 proxy servers, each denoting a specific location (such as us-dc) with 10 IP addresses rotating twice per day.

However, the free trial allows you to have one proxy server at a time. This means that if you are working with a server or CDN that strictly throttles IPs, you have to change the proxy server manually to keep rotating between the 15 proxy servers.

Fortunately, ProxyMesh provides an API that can be used to add and delete the proxy servers in user dashboard. Once a proxy server is assigned to the user, it can be fetched and used. All we need is a script that rotates between the proxy servers, deleting and adding them as required.

Based on this, I have written a script that would rotate through ProxyMesh proxy servers using their API.

Prerequisites

An account with ProxyMesh, either free trial or paid. Set the user name and password in rotatingproxy.py

self.user = ""
self.password = "" 

Usage

Setting the Proxy Server

from rotatingproxy import RotatingProxy

rproxy = RotatingProxy()

The proxy server can either be set randomly or selected from an available list of proxy servers.
The active proxy server is saved in a text file which can be accessed as required.

rproxy.set_proxy(israndom="r")  # select a random proxy server

rproxy.set_proxy(proxy_num=1)   # select proxy server with index=1 from the list of proxy servers.

Accessing the Proxy Server

def get_proxy_from_file():
    # fetches proxy from proxy.txt
    with open("proxy.txt", "r") as f:
        return loads(f.read())

proxy = get_proxy_from_file()        

The proxy can now be used with requests:

import requests
response = requests.get("url-to-fetch", proxies=proxy)

Blog post: http://www.kashifaziz.me/proxy-server-rotation-python.html/

GitHub: https://github.com/kashaziz/rotating-proxy-python