Python Hacking Tools - Web Scraper

Preparation:

Python Libray in the following programming:

1. Requests Document: https://2.python-requests.org//en/master/

2. Beautiful Soup Documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

Install the lib on Kali Linux:

apt-get install python-requests

apt-get install python-bs4

Proxy Domain:

https://free-proxy-list.net/

https://free-proxy-list.com/

 

 

 Python Scraper Code:

import requests
from bs4 import BeautifulSoup

proxyDomain = "https://free-proxy-list.com/"

r = requests.get(proxyDomain,headers={"User-Agent": "XY"})

soup = BeautifulSoup(r.content, 'html.parser')

table = soup.find('table', {"class" : "table table-striped proxy-list"})

for row in table.find_all('tr'):
    columns = row.find_all('td')
    try:
        print "%s:%s\t%-20s\t%-10s" %(columns[0].get_text(),columns[1].get_text().strip(),columns[2].get_text().strip(),columns[3].get_text().strip())
    except:
        pass

 

Execute Result: