Problem HTTP error 403 in Python 3 Web Scraping
Asked 07 September, 2021
Viewed 2.4K times
  • 54
Votes

I was trying to scrape a website for practice, but I kept on getting the HTTP Error 403 (does it think I'm a bot)?

Here is my code:

#import requests
import urllib.request
from bs4 import BeautifulSoup
#from urllib import urlopen
import re

webpage = urllib.request.urlopen('http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1').read
findrows = re.compile('<tr class="- banding(?:On|Off)>(.*?)</tr>')
findlink = re.compile('<a href =">(.*)</a>')

row_array = re.findall(findrows, webpage)
links = re.finall(findlink, webpate)

print(len(row_array))

iterator = []

The error I get is:

 File "C:Python33liburllib
equest.py", line 160, in urlopen
    return opener.open(url, data, timeout)
  File "C:Python33liburllib
equest.py", line 479, in open
    response = meth(req, response)
  File "C:Python33liburllib
equest.py", line 591, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:Python33liburllib
equest.py", line 517, in error
    return self._call_chain(*args)
  File "C:Python33liburllib
equest.py", line 451, in _call_chain
    result = func(*args)
  File "C:Python33liburllib
equest.py", line 599, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

7 Answer