SeleniumBase is a tool designed to run Selenium in a test environment and supports authenticated proxies. This feature helps you overcome restrictions and increase your success rate in web scraping.
Problems with Proxy Usage in Selenium
Selenium is limited in its proxy support. In particular, it does not support authenticated proxies. The SeleniumWire tool, previously widely used for working with proxies, has not been updated for over a year and is no longer recommended.
Getting Started
First, we need to install SeleniumBase and write a test case. The test case checks Selenium and runs our webdriver instance.
Installation
You can install SeleniumBase with the following command:
pip install seleniumbaseLet’s test
Below is a simple test result. This code sends a request to the IPinfo API and parses the received JSON response, printing its contents to the console.
from seleniumbase import BaseCase
from selenium.webdriver.common.by import By
import json
class ProxyTest(BaseCase):
def test_proxy(self):
# Siteye git
self.driver.get("https://ipinfo.io/json")
# JSON yanıtını yükle
location_info = json.loads(self.driver.find_element(By.TAG_NAME, "body").text)
# Sözlük içeriğini yazdır
for k, v in location_info.items():
print(f"{k}: {v}")
Let’s Start the Test
We will use pytest to run our code. If you want to run your test without using a proxy, you can use the following command:
Run without proxy
pytest proxy_test.py -s
This command will print your IP address and location information to the console.
Running with a proxy:
You can use the following command to run the test using a proxy:
pytest proxy_test.py --username:poassword@proxy.proxynet.io:8000 -s
This command will route your traffic through the specified proxy and you can see in the console that your IP address has changed.
=================================================== test session starts ===================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/nultinator/clients/bright-data/seleniumbase-proxies
plugins: html-4.0.2, metadata-3.1.1, anyio-4.0.0, seleniumbase-4.34.6, ordering-0.6, rerunfailures-15.0, xdist-3.6.1
collected 1 item
proxy_test.py ip: 144.202.4.246
hostname: 144-202-4-246.lum-int.io
city: Piscataway
region: New Jersey
country: US
loc: 40.4993,-74.3990
org: AS20473 The Constant Company, LLC
postal: 08854
timezone: America/New_York
readme: https://ipinfo.io/missingauth
.
==================================================== 1 passed in 3.25s ====================================================This command routes your traffic through a proxy and you can see in the console that your IP address is in a different location.
Proxy Rotation
Using Python’s basic features, you can create an efficient proxy return system. The following code returns proxies using specific country codes:
import subprocess
# Ülke kodları listesi
countries = [
"us", # Amerika Birleşik Devletleri
"es", # İspanya
"il", # İsrail
"gb", # Büyük Britanya
]
# Ülkeler arasında döngü yap ve her biri için bir komut oluştur
for country in countries:
command = f"pytest proxy_test.py --proxy=brd-customer-<KULLANICI_ADINIZ>-<BÖLGE_ADINIZ>-country-{country}:<ŞİFRENİZ>@brd.superproxy.io:33335 -s"
# Komutu çalıştır
subprocess.run(command, shell=True)
You can also take a look at our previous article on Proxy Rotation with Pyhton here.
This script runs the test sequentially using proxies in the specified countries and allows you to use a different IP address each time.
Conclusion
In this tutorial, you learned how to use proxies, check location and return proxies with SeleniumBase. These tools will help you overcome any obstacles you may encounter in your web scraping operations.
Unlock the full potential of your Selenium-based scraping with Bright Data’s industry-leading proxy services. Start your free trial today and enjoy smooth, reliable data collection!
Which of our main proxy server products can you use in SeleniumBase?
