Creating another proxy service

There are a lot of services out there that provide proxies in many different ways. Some provide them in bulk, others provide them via an API, others just provide them for free on their website. The problem is that many of these services serve dead or sluggish proxies making it very difficult to use without having to verify that they are working yourself.

For a few projects I have planned, I need an API that would provide working proxies every time I requested them. I needed a service that was extremely aggressive about checking its proxies so that no proxy that was dead would last in the pool long enough to be served.

From this Proxy Orbit was born.

Proxy Orbit is a service that aims to do one set of tasks very well:

Scan the web for open proxies, save them, check them constantly, return a random proxy when an API request is made, and allow for filtering of the results.

Basic API Usage

A basic API request currently looks something like this:

https://api.proxyorbit.com/v1/?token=Iut1LQeCvN7bxHRkplubawI75qGiWxiXrKR1ARflDKA

Which returns an object that looks like the following

{
    anonymous: true,
    cookies: true,
    curl: "http://182.96.147.126:8118",
    get: true,
    ip: "182.96.147.126",
    isp: "No.31,Jin-rong Street",
    lastChecked: 1564551731.418154,
    location: "CN",
    port: 8118,
    post: true,
    protocol: "http",
    rtt: 1.008148431777954,
    ssl: false,
    websites: {
       amazon: false,
       facebook: false,
       google: false,
       instagram: false,
       netflix: false,
       nike: false,
       shopify: false,
       stubhub: false,
       supreme: false,
       ticketmaster: false,
       twitch: false,
       twitter: false,
       youtube: false
    }
}

All proxies returned have been checked within the last hour.

Proxies can also be returned in bulk using the count query argument.

https://api.proxyorbit.com/v1/?token=Iut1LQeCvN7bxHRkplubawI75qGiWxiXrKR1ARflDKA&count=5

The above will return five proxy objects in a JSON array.

Proxies can also be filtered by all parameters in the object. For example

https://api.proxyorbit.com/v1/?token=Iut1LQeCvN7bxHRkplubawI75qGiWxiXrKR1ARflDKA&amazon=true&protocol=http&ssl=true

will return HTTP proxies that can load Amazon.com and support HTTPS.

Examples in Python and NodeJS

Since everything is provided over a basic REST API, integration into existing code should be trivial. Here are two examples of using Proxy Orbit in Python and NodeJS. The scripts request a new proxy and check the proxy's IP.

Python Code

import requests

resp = requests.get("https://api.proxyorbit.com/v1/?token=Iut1LQeCvN7bxHRkplubawI75qGiWxiXrKR1ARflDKA&ssl=true&protocol=http")
if resp.status_code != 200:
    raise Exception("Token Expired!")

json = resp.json()
curl = json['curl']

resp2 = requests.get("https://api.proxyorbit.com/ip", proxies={"https":curl, "http":curl})
print("Proxy IP is:", resp2.content)

NodeJS

const request = require("request");

request.get("https://api.proxyorbit.com/v1/?token=Iut1LQeCvN7bxHRkplubawI75qGiWxiXrKR1ARflDKA&ssl=true&protocol=http", (err, resp, body) => {
    if(err) {
        console.log(err)    
    } else if(resp.statusCode != 200) {
        console.log("Token has expired");
    } else {
        json = JSON.parse(body);
        request({
            url:"https://api.proxyorbit.com/ip",
            method:"GET",
            proxy:json.curl
        }, (err, resp, body) => {
            console.log("Proxy IP is:", body);
        })
    }
});

Checking pool quality

To test the quality of the pool I have been using the following script

import requests

total = 100
failed = 0
url = input("URL: ")
for x in range(total):
    resp = requests.get("https://api.proxyorbit.com/v1/?token=Iut1LQeCvN7bxHRkplubawI75qGiWxiXrKR1ARflDKAA&protocols=http&ssl=true").json()
    curl = resp['curl']
    try:
        r = requests.get(url, proxies={"http":curl, "https":curl}, timeout=10)
    except Exception as e:
        print(e)
        print(curl, "failed")
        failed += 1
    else:
        print(r.status_code)
        #print(r.content)
        print(curl)
        if r.status_code != 200:
            failed += 1

error_rate = (failed / total) * 100

print(f"{error_rate}% failed")

The script makes 100 requests to a specific URL and tracks the number of failures. It then takes the # failures / 100 * 100 to calculate the error percentage. So far my tests to https://proxyorbit.com have shown roughly 3% of the proxies failing (3 / 100 do not work). Though proxyorbit.com isn't very strict about blocking proxy IPs.

Proxies are bound to fail because they are open. Others are using them as well and they get banned much quicker than private proxy servers. Though it is difficult to attain 100% working proxies in the pool at any given time, the proxy checking algorithm ensures that broken proxies are removed as soon as they are noticed to be dead. This is normally within minutes of the proxy no longer working.

Conclusion

Proxy Orbit will be used as a foundation for a couple of other products that are currently in the works. It's definitely been a lot of fun creating this one and I hope that some other people could find use in it as well.

As for the tech stack Proxy Orbit uses Python 3, Flask, and MongoDB.

If you are interested you can check the site out at https://proxyorbit.com. If you aren't interested in paying for the service, we do provide 2000 API requests per month for free when you sign up.

Feedback is always greatly appreciated!

Thanks!