Web scraping with python's request module requires submit button

See also questions close to this topic

• How do I change the Intervals on the axis when graphing in python MatPlotLib?

  I have Used this webpage and other webpages to try and find a answer that is clear and accurate but nothing so far. I am plotting a graph that has the x-axis being years and the intervals between the years are going up by 2.5 years. I would like that to be 1 and not 2.5. Any Idea how to fix this??
• Python Pandas 'head' attribute not working

  I am doing a quick script playing around with python and pandas to work with some data, however, I am trying to print the 'head' but when I do I get an error. Any ideas what might be the problem?

  AttributeError: 'collections.OrderedDict' object has no attribute 'head'

  import pandas as pd
  my_file_location = "whatever.xlsx"
  dfs = pd.read_excel(my_file_location, sheet_name=None)
  
  print(dfs.head())
  
• Trimming empty lines when using OCR(pytesseract) in Python

  image = Image.open('C:/Users/#######/Pictures/Saved Pictures/image0.png')
  text = pytesseract.image_to_string(image, lang='eng')
  text = os.linesep.join([s for s in text.splitlines() if s])
  

  I looked around stackoverflow and this was the fix most people were saying to use, but it hasn't removed the empty lines.
• Why do i get the error 'NoneType' object is not subscriptable when looping URL's with beautiful soup?

  I have written some code to scrape a JSON string, parse it and send the results to a SQL database. There are about 1000 pages I need to parse so I have imported the URL's from a CSV file to loop through them.

  I have written the code without the loop and everything works fine.

  url = 'https://www.nrl.com/draw/nrl-premiership/2018/round-25/Broncos-vs-Sea-Eagles/'
  
  response = requests.get(url)
  soup = BeautifulSoup(response.text, 'lxml')
  text = soup.find(class_='l-content pre-quench')['q-data']
  unescaped_text = html.unescape(text)  
  

  However when I import the data from a CSV and add a loop to it

  for game in url:
      response = requests.get(game)
      soup = BeautifulSoup(response.text, 'lxml')
      text = soup.find(class_='l-content pre-quench')['q-data']
      unescaped_text = html.unescape(text)
  

  i get the error

    File "<ipython-input-71-1b8888dd0ee0>", line 5, in <module>
      text = soup.find(class_='l-content pre-quench')['q-data']
  
  TypeError: 'NoneType' object is not subscriptable
  

  I have tried using a single entry (take works in the non-looped version) in the CSV file but i still get the error.

  from reading i have done the error normally means that the element you are trying to get is not in the page but this is not the case in this example.

  Any ideas why this is working without the loop but fails in side the loop?
• Scrape using XMLHTTP throws error at specific class name

  I am trying to scrape a site with this code to extract names and contacts ...

  Sub Test()
  Dim htmlDoc         As Object
  Dim htmlDoc2        As Object
  Dim elem            As Variant
  Dim tag             As Variant
  Dim dns             As String
  Dim pageSource      As String
  Dim pageSource2     As String
  Dim url             As String
  Dim row             As Long
  
  row = 2
  dns = "https://www.zillow.com/detroit-mi/real-estate-agent-reviews/"
  
  With CreateObject("MSXML2.XMLHTTP")
      .Open "GET", dns, True
      .send
  
      While .readyState <> 4: DoEvents: Wend
  
      If .statusText <> "OK" Then
          MsgBox "ERROR" & .Status & " - " & .statusText, vbExclamation
          Exit Sub
      End If
  
      pageSource = .responseText
  End With
  
  Set htmlDoc = CreateObject("htmlfile")
  htmlDoc.body.innerHTML = pageSource
  

  Dim xx 'Got error here Set xx = htmlDoc.getElementsByClassName("ldb-contact-summary")

  Set htmlDoc = Nothing
  Set htmlDoc2 = Nothing
  End Sub
  

  When trying to use this line

  Set xx = htmlDoc.getElementsByClassName("ldb-contact-summary")
  

  I got an error 'Object doesn't support that property or method' (438) Can you help me please as I am not so good at scraping issues?
• BeautifulSoup: Merge all p elements into one string?

  I currently use following Python code excerpt to get all

  elements of a webpage:

  def scraping(url, html):
      data = {}
      soup = BeautifulSoup(html,"lxml")
  
      data["news"] = []
  
      page = soup.find("div", {"class":"container_news"}).findAll('p')
      page_text = ''
  
      for p in page:
          page_text += ''.join(p.findAll(text = True))
          data["news"].append(page_text)
      print(page_text)
  
      return data
  

  However, the output of page_text looks like:

  "['New news on the internet. ', 'Here is some text. ', ""Here is some other."", ""And then there are other variations \n\nLooks like there are some non-text elements. \n\xa0""]" ...
  

  Is it possible to get the content cleaner and merge the lists into one string? BeautifulSoup solutions would be preferred over regex variants.

  Thank you!
• How to re-write O(n²) function in Python using Flask

  In the process of using Eventbrite's API to power a website with their events, however, I am stuck on how to implement project.

  Currently, I am wanting to filter events via dates and show them on the front page, however, Eventbrite API limits to only 50 items per page. The API request pushes out 86 pages with 50 items each.

  How can I go about presenting those events at once while lowering latency? What I currently have takes 104 seconds to respond and render results to page. I'm scratching the surface on Big O Notation theory, however, I think I'm using O(n²).

  here's the code:

  routes.py:

  from flask import Flask, render_template
  from eventbrite import Eventbrite
  from datetime import datetime
  
  import requests, json
  
  app = Flask(__name__)
  
  
  # TODO: Remove outh key in file and register in sys
  oKey = ‘<oauth-key>’
  
  o_token_url = '&token={0}'.format(oKey)
  
  @app.route("/")
  def index():
  
      base_url = 'https://www.eventbriteapi.com/v3/'
      regions_url = base_url + 'system/regions/?token={0}'.format(oKey)
      regions_result = requests.get(regions_url)
      regions_obj = regions_result.json()
  
      e_url = base_url + 'events/search/?token={0}&location.address=<city>&start_date.range_start=2018-07-02T12%3A00%3A00Z&page=2'.format(oKey)
      e_result = requests.get(e_url)
      e_obj = e_result.json()
  
     list_events = []
  
      for pg in range(1, e_obj['pagination']['page_count']):
          print(pg)
          startTime = datetime.now()
  
          e_url = base_url + 'events/search/?token={0}&location.address=<city>&start_date.range_start=2018-07-02T12%3A00%3A00Z&page={1}'.format(oKey, pg)
          e_result = requests.get(e_url)
          e_obj = e_result.json()
  
          for i in e_obj['events']:
              an_event = {}
              an_event['name'] = i['name']['text']
              list_events.append(htx_event)
  
      print(datetime.now() - startTime)
      return render_template('index.html', events=list_events)
  
  if __name__ == "__main__":
      app.run(host='0.0.0.0')
  

  index.html:

  {% for event in events %}
  {{event['name']}}<br>
  
  {% endfor %}
  

  I am aware of EB API batched requests endpoint however, I can't wrap my head around on how to send the list of url dictionaries with requests.

  Any tips on how to optimize the code above would be appreciated. I figured I can limit the amount of events that are shown at any given time, however, interested in how others would approach this issue.
• Python3 - Web Scraper w/ BeautifulSoup, Email Price Alert

  I'm attempting to setup a script to email me when a phone meets my price and condition threshold on swappa.com. I've gotten to the point when I can email out the latest post that meets my criteria, but I've hit a bit of wall. I need it to only email me when a new post meets my criteria. Currently, it will email me each time the script is ran, whether or not the listing is new.

  I'm very new to Python and spent the better part of yesterday and today hobbling together this script, but any insight or information that would help accomplish my goal would be much appreciated!

  from bs4 import BeautifulSoup
  import re
  import requests
  import smtplib
  from email.mime.multipart import MIMEMultipart
  from email.mime.text import MIMEText
  
  # define the url in a variable and get the content
  page_url = 'https://swappa.com/buy/google-pixel-2-xl-unlocked?sort=date_min'
  page_content = requests.get(page_url)
  
  # Put the content into a variable using BeautifulSoup
  swappa = BeautifulSoup(page_content.text, 'html.parser')
  
  # Pull text from the h1 tag
  phone_name = swappa.h1
  
  # Pull the device details into a list
  condition = swappa.find(class_='condition_label')
  color = swappa.find(class_='color_label')
  storage = swappa.find(class_='storage_label')
  price = swappa.find(class_='price')
  
  links = []
  
  for link in swappa.findAll('a', attrs={'href': re.compile("^/listing")}):
      links.append(link.get('href'))
  
  
  listing_detail = [
      phone_name.contents[0], condition.contents[0], color.contents[0], storage.contents[0],
      '$' + price.contents[1], 'https://swappa.com' + links[0]
      ]
  
  
  def listing_result():
      if (price.contents[1] <= '420') and (condition.contents[0] == 'Good' or condition.contents[0] == 'Mint'):
          return listing_detail
  
  
  with open("result.txt", "w+") as result_file:
      result_file.write(str(listing_result()))
  
  
  with open('result.txt', 'r') as result_file_read:
      result_data = result_file_read.read().replace('\n', '')
  
  if result_data != result_file:
      #  setup sending an email via SMTP
      server = smtplib.SMTP_SSL('smtp.gmail.com', 465)
      server.login("redacted", "redacted")
  
      # use mime to send email
      msg = MIMEMultipart()
  
      # setup parameters of the email
      msg['From'] = 'redacted'
      msg['To'] = 'redacted'
      msg['Subject'] = 'Swappa - New Pixel 2 XL Listing'
  
      # add in the message body
      body = str(listing_result())
      body = MIMEText(body) # convert the body to a MIME compatible string
      msg.attach(body) # attach it to your main message
  
      # send the email
      server.send_message(msg)
      del msg
  

  My idea was to have the latest listing be sent to a text file, then have the contents of that text file saved to a variable so that I could compare it to the text being created. However, I'm at a loss for how to accomplish that.
• Python TKinter redirect terminal output to GUI window

  I’m new in python and trying to create a program which should, among other things, to print a terminal output to a Tkinter GUI.

  For example the output of a ping command or any other shell command or a variable print.

  I’ve searched the web for a solution but did not find any simple way to do so.

  Maybe I’m missing something or there is other way to print the terminal output in python

  Thanks in advance,