Python Requests: How to Use & Rotate Proxies

To use and rotate proxies with the Python Requests library, you can follow these steps:

Install the requests library if you haven't already. You can do this using pip:

pip install requests

Import the necessary modules:

import requests

Prepare a list of proxies that you want to rotate. Each proxy should be in the format http://ip:port or https://ip:port. Here's an example list of proxies:

proxies = [

    'http://proxy1.example.com:8080',

    'http://proxy2.example.com:8080',

    'http://proxy3.example.com:8080',

]

Create a session object that will handle the requests and rotate the proxies:

session = requests.Session()

Define a function to rotate the proxies:

def get_proxy():

    proxy = next(proxy_pool)

    return {'http': proxy, 'https': proxy}

Create a proxy pool using an iterator:

proxy_pool = iter(proxies)

Make requests using the session object and the get_proxy() function to fetch a new proxy for each request:

for i in range(10):  # Make 10 requests

    proxy = get_proxy()

    try:

        response = session.get('http://example.com', proxies=proxy, timeout=5)

        print(response.text)

    except requests.exceptions.RequestException as e:

        print('Error:', e)

In this example, the get_proxy() function is responsible for retrieving the next proxy from the proxy pool. The proxies argument in the session.get() method specifies the proxy to be used for each request.

Note that not all proxies may be reliable or available at all times. You may need to handle exceptions and retries accordingly, and ensure that the proxies you use are valid and authorized for scraping purposes.

Additionally, keep in mind that rotating proxies does not guarantee complete anonymity or foolproof bypassing of restrictions. Be aware of the legal and ethical considerations discussed earlier when scraping websites or using proxies.

Copy Rights Digi Sphere Hub