Web scraping with python's request module requires submit button

See also questions close to this topic

• 5 fold cross validation of time series data

  I'm doing some electricity load forecasting in which I need to split the dataset into custom training and testing sets. For this I've got 3 years of electricity load data starting from 01-01-2015 00:30:00 to 01-01-2018 00:00:00. The load is recorded every half hour. Now I want to split the training and evaluation periods into 312 sets of 84 consecutive hours (36+48) and use them to define the 5-fold validation splitting.

  

  One can notice that odd numbered sets starts at 00:00 and even numbered starts at 12:00. This also happens with the missing spots in the evaluation period. As for the 5-fold splitting, it has to be done like this:

  • Validation folds: Fold 1 contained sets 1, 6, 11 and so on as validation set and the remaining as training set. Fold 2, contained 2, 7, 12, ..., as validation, and analogous splitting were done for folds 3, 4 and 5.
  • Test fold: Contained all available filled data and was used to build models to predict missing data.

  I'm struck with the splitting of dataset to form the validation folds. I tried using k-fold and timeseries splitting from sklearn but didn't get exactly what I wanted.

  import pandas as pd
  import numpy as np
  import matplotlib.pyplot as plt
  
  #Data pre-processing
  state = {0: 'NSW', 1: 'QLD', 2: 'SA', 3: 'TAS', 4: 'VIC'}
  year = {0: '2015', 1: '2016', 2: '2017'}
  #year = {0: '2017'}
  
  df_nsw = pd.DataFrame()
  df_qld = pd.DataFrame()
  df_sa = pd.DataFrame()
  df_tas = pd.DataFrame()
  df_vic = pd.DataFrame()
  
  df_nsw_test = pd.DataFrame()
  df_qld_test = pd.DataFrame()
  df_sa_test = pd.DataFrame()
  df_tas_test = pd.DataFrame()
  df_vic_test = pd.DataFrame()
  
  df = {'NSW': df_nsw, 'QLD': df_qld, 'SA': df_sa, 'TAS': df_tas, 'VIC': df_vic}
  df_test = {'NSW': df_nsw_test, 'QLD': df_qld_test, 'SA': df_sa_test, 'TAS': df_tas_test, 'VIC': df_vic_test}
  
  for st in state.values():
      for ye in year.values():
          for mn in range(1,13):
              if mn < 10:            
                  dataset = pd.read_csv('./datasets/train/' + st + '/PRICE_AND_DEMAND_' + ye + '0' + str(mn) +'_' + st + '1.csv')
              else:
                  dataset = pd.read_csv('./datasets/train/' + st + '/PRICE_AND_DEMAND_' + ye + str(mn) +'_' + st + '1.csv')
              df[st] = df[st].append(dataset.iloc[:,1:3])
      df[st] = df[st].set_index('SETTLEMENTDATE')
  
  
  for st in state.values():
      dataset = pd.read_csv('./datasets/test/' + st + '/PRICE_AND_DEMAND_201801_' + st + '1.csv')
      df_test[st] = df_test[st].append(dataset.iloc[:,1:3])
      df_test[st] = df_test[st].set_index('SETTLEMENTDATE')
  
  
  plt.plot(df['NSW'].iloc[:,0].values)
  plt.show()
  plt.plot(df['QLD'].iloc[:,0].values)
  plt.show()
  plt.plot(df['SA'].iloc[:,0].values)
  plt.show()
  plt.plot(df['TAS'].iloc[:,0].values)
  plt.show()
  plt.plot(df['VIC'].iloc[:,0].values)
  plt.show()
  

  The dataset is uploaded here. Any suggestion ?
• Including ssl module in py2app

  I am trying to add the ssl module in my plugin when using py2app. However, when the plugin is used I get the following error:

  pymongo.errors.ConfigurationError: The ssl module is not available.
  

  There are 2 parts that I don't understand. First, since the ssl module is included with the python package, the module should be added automatically, which doesn't happen. Second, when I'm trying to add ssl to the install_requires variable, the module won't be found.

  Why is this module so hard to include, and how can I include it?
• IOError in python for meanwhile slide a query result

  I've got an issue with this part of python script. When I run it, it throws this error:

  IOError: [Errno 0] Error
  

  in a RANDOM ELEMENT of dbCur (result of query).

  Here is my code:

  dbCur.execute("SELECT parent_name FROM crowd.cwd_membership WHERE child_name = \'"+source+"\'")
  for group in dbCur:
      code = 999
  
      # Check sul nome visualizzato
      if ('-' in group):
          print "Aggiungo il gruppo: ", group[0], " all' utente di destinazione..."
          code = giveGroup(jira,target,group[0])
      if (code != 400 or code != 404):
          # Se andata a buon fine 
          print " ...Gruppo aggiunto con successo \n"
      if (code == 400 or code == 404):
          # Se utente o gruppo non trovati, manda in review manuale
          requireReview = True
      if (code==999):
          # Se il nome non è corretto O non esiste, non fa partire la giveGroup
          print "...Nome gruppo non valido, "
          autogrant_crowd_clog.logger.info("Errore nel nome del gruppo: "+group[0]+ " l'inserimento è stato saltato.")
  
• Fail Scrapy - It provides an empty output

  I'm scrapping a website and I use the correct xpath of the objects but I get an empty result. I use the following code:

  import scrapy
  from scrapy.http.request import Request
  from indicators.ESGIndicators import ESGIndicators
  from scrapy.spiders import CrawlSpider,Rule
  from scrapy.linkextractors import LinkExtractor
  from lxml import html
  
  class mySpider(scrapy.Spider):
      name = "YALE"
      allowed_domains = ["epi.envirocenter.yale.edu"]
      start_urls = (
          'https://epi.envirocenter.yale.edu/epi-indicator-report/WWT',
          )
  
      def parse(self, response):
          return Request(
              url='https://epi.envirocenter.yale.edu/epi-indicator-report/WWT',
              callback=self.parse_table
          )
  
      def parse_table(self,response):
          for tr in response.xpath('//*[@id="block-system-main"]/div/div/div/div[3]/table[2]/tr'):
              item = ESGIndicators()
              item['country'] = tr.xpath('td[1]/a/text()').extract_first()
              item['data1'] = tr.xpath('td[2]/text()').extract()
              item['data2'] = tr.xpath('td[3]/text()').extract()
              item['data3'] = tr.xpath('td[4]/text()').extract()
              item['data4'] = tr.xpath('td[5]/text()').extract()
              print(item)
              yield item
  

  I don't get any error but it doesn't scrap anything. I tried with tdoby in the xpath and without it, but it doesn't work.

  Has anyone any idea about the problem?

  Thanks in advance!
• What database the best for real-time web-scraping?

  I have a web scraper which every 5 minutes check web site and save data to database. And remove old data (based on timestamp)

  Also I have Restful API wrapper for this database.

  It's a lot request to DB what DB is most appropriate for this case?
• Iterate the for Loop in selenium for list with different size

  from selenium import webdriver
  from selenium import webdriver
  from selenium.common.exceptions import StaleElementReferenceException
  from selenium.common.exceptions import NoSuchElementException
  import itertools
  import time
  from time import gmtime, strftime
  print("*" * 60)
  print("MIPYTHON .com PYTHON PROXY SCRAPE and GO SCRIPT in SELENIUM")
  print("*" * 60)
  browser = webdriver.Chrome(executable_path='chromedriver.exe')
  proxy_website = "https://free-proxy-list.net/"
  proxy1 = []
  proxy_port1= []
  sachin = []
  newlist = []
  browser.get(proxy_website)
  browser.find_element_by_xpath('/html/body/header/div[1]/div/div[1]/div/div/a[2]').click()
  proxy = browser.find_elements_by_xpath("//tr[@role='row']/td[1]")
  proxy_port = browser.find_elements_by_xpath("//tr[@role='row']/td[2]")
  for element in proxy:
      proxy1.append(element.text)
  for element in proxy_port:
      proxy_port1.append(element.text)
  
  for i in range(len(proxy1)):
      safale = proxy1[i]+":"+proxy_port1[i]
      sachin.append(safale)
  content = []
  content = ['https://www.facebook.com/pg/virsaamusical/events/','https://www.facebook.com/pg/TheInnerUniverse.withVirsaaIndia/events/','https://www.facebook.com/pg/sadhguru/events/','https://www.facebook.com/pg/after8events/events/','https://www.facebook.com/pg/dovlinent/events/']
  for ip,link in zip(itertools.cycle(sachin), content):
      try:
          path_to_chromedriver = 'C:/Users/Admin/AppData/Local/Programs/Python/Python37-32/chromedriver.exe'
          chrome_options = webdriver.ChromeOptions()
          prefs = {"profile.default_content_setting_values.notifications": 2}
          chrome_options.add_experimental_option("prefs", prefs)
          PROXY = ip
          chrome_options.add_argument("start-maximized")
          chrome_options.add_argument('--proxy-server=%s' % PROXY)
          browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=path_to_chromedriver)
          browser.get('https://www.facebook.com/')
          browser.find_element_by_id('email').send_keys('USERID')
          browser.find_element_by_id('pass').send_keys('PASSWORD')
          browser.find_element_by_id('loginbutton').click()
          browser.get(link)
          mainclass = browser.find_elements_by_css_selector('._4dmd._4eok.uiGrid._51mz')
          Month1 = []
          Date1 = []
          Event_link1 = []
          Event_Name1 = []
          Event_Time1 = []
          No_of_Guests1 = []
          Organizer_HomepageLink1 = []
          Location1 = []
          SCROLL_PAUSE_TIME = 2
          # Get scroll height
          last_height = browser.execute_script("return document.body.scrollHeight")
          while True:
              # Scroll down to bottom
              browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
              # Wait to load page
              time.sleep(SCROLL_PAUSE_TIME)
              # Calculate new scroll height and compare with last scroll height
              new_height = browser.execute_script("return document.body.scrollHeight")
              if new_height == last_height:
                  break
              last_height = new_height
          browser.implicitly_wait(20)
          Month = browser.find_elements_by_xpath("//tr[@class='_51mx']/td[1]/span[1]/span[1]")
          Date = browser.find_elements_by_xpath("//tr[@class='_51mx']/td[1]/span[1]/span[2]")
          Event_link = browser.find_elements_by_xpath("//tr[@class='_51mx']/td[2]/div[1]/div[1]/a[1]")
          Event_Name = browser.find_elements_by_xpath("//tr[@class='_51mx']/td[2]/div[1]/div[1]/a[1]/span[1]")
          Event_Time = browser.find_elements_by_xpath("//tr[@class='_51mx']/td[2]/div[1]/div[2]/span[1]")
          No_of_Guests = browser.find_elements_by_xpath("//tr[@class='_51mx']/td[2]/div[1]/div[2]")
          Organizer_HomepageLink = browser.find_elements_by_xpath("//tr[@class='_51mx']/td[3]/div[1]/div[1]/a[1]")
          Location = browser.find_elements_by_xpath("//tr[@class='_51mx']/td[3]/div[1]/div[2]")
          try:
              for element in Month:
                  Month1.append(element.text)
          except StaleElementReferenceException:
              pass
          except NoSuchElementException:
              pass
          try:
              for element in Date:
                  Date1.append(element.text)
          except StaleElementReferenceException:
              pass
          except NoSuchElementException:
              pass
          try:
              for element in Event_link:
                  sep = '?ref_page_id'
                  text = element.get_attribute('href')
                  rest = text.split(sep, 1)[0]
                  Event_link1.append(rest)
          except StaleElementReferenceException:
              pass
          except NoSuchElementException:
              pass
          try:
              for element in Event_Name:
                  Lco = element.text
                  lcoa = Lco.replace(",", "|")
                  Event_Name1.append(lcoa)
          except StaleElementReferenceException:
              pass
          except NoSuchElementException:
              pass
          try:
              for element in Event_Time:
                  Event_Time1.append(element.text)
          except StaleElementReferenceException:
              pass
          except NoSuchElementException:
              pass
          try:
              for element in No_of_Guests:
                  Lco = element.text
                  lcoa = Lco.replace(",", "")
                  Lcoas = lcoa.replace(" . ", ",")
                  No_of_Guests1.append(Lcoas)
          except StaleElementReferenceException:
              pass
          except NoSuchElementException:
              pass
          try:
              for element in Organizer_HomepageLink:
                  sep = '?ref_page_id'
                  text = element.get_attribute('href')
                  rest = text.split(sep, 1)[0]
                  Organizer_HomepageLink1.append(rest)
          except StaleElementReferenceException:
              pass
          except NoSuchElementException:
              pass
          try:
              for element in Location:
                  Lco = element.text
                  lcoa = Lco.replace(",", "|")
                  Location1.append(lcoa)
          except StaleElementReferenceException:
              pass
          except NoSuchElementException:
              pass
          with open('E:\Events.csv', 'a') as f:
              headers = ("Source Link,Month_Of_Event,Date-Of-Event,Event_Link,Event-Name,Time & No-of-Guests,Organizer-Link,Location, IP Used, Time Scraped")
              f.write(headers)
              f.write("\n")
              try:
                  current = strftime("%Y-%m-%d %H:%M:%S", gmtime())
                  seperator = ':'
                  for i in range(len(Month1)):
                      f.write(link + "," + Month1[i] + "," + Date1[i] + "," + Event_link1[i] + "," + Event_Name1[i] + "," + No_of_Guests1[
                          i] + "," + Organizer_HomepageLink1[i] + "," + Location1[i] + "," + PROXY.split(seperator, 1)[0] + "," + current +"\n")
              except IndexError:
                  pass
      except NoSuchElementException:
          pass
  browser.close()
  

  The above code does what i watn but it needs only a part being modified which would be for ip,link in zip(itertools.cycle(sachin), content): in this loops it takes the IP from sachin list and Link from content all works fine if the proxy ip is good but if the there is no response from the proxy it throws NosuchElementException which i have handled using pass, but what i actually want to implement is if there is no response for the ip it should move to the next ip in the sachin list and restart with the same link where it stopped.
• How to check URL status code in bulk without flooding requests

  I have a requirement to check, on an ongoing basis, the status code returned by many URLs. This can be up to 50,000 on the same domain.

  I am concerned that by doing this, I risk flooding the site(s) with requests, and impacting performance, which cannot happen.

  My current approach to this is to use requests.head in order to minimise the size of the response, however I'm unsure as to what additional context I need to be aware of:

  • Is there a best practice for rate limiting requests?
  • How is this best balanced against execution time?
  • Is this enough to ensure that there is no negative impact as a result of running this across a specific domain?

  I've looked for similar topics but can't seem to find this, so apologies if this is solved elsewhere.

  Thanks
• Using requests.post to submit a form

  Building a webscraper to query the USGA golf course database for a list of all courses and ratings. A form submission generates a table of links to different courses, which can then be scraped for the data I'm curious about.

  How do I use requests.post to submit all possible queries into the form. Currently, I have:

  import csv
  from BeautifulSoup import BeautifulSoup
  from requests import Session
  import requests
  
  
  url = "https://ncrdb.usga.org/NCRDB/NCRListing.aspx"
  data = {'txtClubCity': r"Newton"}
  r = requests.post(url, data=data)
  
  print r.content
  

  I've tried this a bunch of different ways, but the page doesn't update after posting. Does anyone have a sense of what's wrong here?
• python requests SSLError with Heroku set up

  I running on Heroku and am getting this error SSLError with requests when sending a request out to an API any Ideas? I beielve this is a issue with my cert or the way I make a connection with requests

  Here is my code and error, obviously code is just the just the calls used, and shorted up.

  Here is URL used in settings.py

  //settings.py
  KONNEKTIVE_BASE_URL = 'https://api.konnektive.com/'
  

  Here is api integrations class NEWKonnektive that is used in to call GET request from requests

  //konnektive.py
  from wrapper import Request
  import logging
  logger = logging.getLogger(__name__)
  
  class NEWKonnektive(Request):
      params = {}
  
      def __init__(self):
  
          self.auth_params = {
              'loginId': os.getenv("KONNEKTIVE_USERNAME"),
              'password': os.getenv("KONNEKTIVE_PASSWORD")
          }
  
      def import_click(self, *args, **kwargs):
  
          function_name = currentframe().f_back.f_code.co_name
  
          self.url = settings.KONNEKTIVE_BASE_URL + 'landers/clicks/import'
          self.params = {}
  
          page_type = self.set_param_or_none(kwargs, 'pageType', 'page_type', list_var=PAGE_TYPE_LIST)
          if page_type is None: return False
          ip_address = self.set_param_or_none(kwargs, 'ipAddress', 'ip_address')
          user_agent = self.set_param_or_none(kwargs, 'userAgent', 'user_agent')
          campaign_id = self.set_param_or_none(kwargs, 'campaignId', 'campaign_id')
          request_uri = self.set_param_or_none(kwargs, 'requestUri', 'request_uri')
          session_id = self.set_param_or_none(kwargs, 'sessionId', 'session_id')
  
          if not session_id and (not (ip_address and campaign_id and request_uri)):
          logger.error("Konnektive {}: session_id={}, ip_address={}, campaign_id={}, request_uri={}".format(function_name, session_id, ip_address, campaign_id, request_uri))
          return False
          response = self.get(params=self.params)
          logger.info("Konnektive Import Click Response: {}".format(response))
          return response
  

  Here is api integrations class NEWKonnektive that is used in to call GET request from requests

  // request wrapper.py
  import json
  import requests
  import logging
  logger = logging.getLogger(__name__)
  
  HEADERS = {'Content-type': 'application/json'}  # Default headers
  
  
  class Request:
      def get(self, params=None):
          if not self.auth_params or not self.url:
              return
          if not params:
              params = {}
          params.update(self.auth_params)
          response = requests.get(self.url, params=params, headers=HEADERS)
          logger.info('response url: {}'.format(self.url))
          logger.info('response: {}'.format(response))
          logger.info('response.ok: {}'.format(response.ok))
          logger.info('response.content: {}'.format(response.content))
          logger.info('response.header: {}'.format(response.headers))
          logger.info('response.url: {}'.format(response.url))
          return response.content
  

  current error appearing in New Relic:

  // Error logs
  Error message
  requests.exceptions:SSLError: HTTPSConnectionPool(host='api.konnektive.com', port=443): Max retries exceeded with url: /landers/clicks/import?pageType=leadPage&ipAddress=73.160.94.162&campaignId=1&requestUri=https%3A%2F%2Fwww.workoutwarriors.com%2Fcontact%2F&loginId=**********&password=********** (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
  
  
  Sample stack trace
  Traceback (most recent call last):
  File "/app/.heroku/python/bin/gunicorn", line 11, in <module>
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 203, in run
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 72, in run
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 202, in run
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 544, in manage_workers
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 611, in spawn_workers
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 131, in init_process
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 124, in run
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 68, in run_for_one
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 30, in accept
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
  File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/api/web_transaction.py", line 1280, in _nr_wsgi_application_wrapper_
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/api/web_transaction.py", line 807, in __init__
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/api/web_transaction.py", line 1169, in _nr_wsgi_application_wrapper_
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/handlers/wsgi.py", line 177, in __call__
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 423, in _nr_wrapper_BaseHandler_get_response_
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/handlers/base.py", line 123, in get_response
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/hooks/framework_django.py", line 220, in wrapper
  File "/app/wow/middleware/tracking.py", line 35, in process_request
  File "/app/wow/middleware/tracking.py", line 64, in set_url_parameters_in_session
  File "/app/integrations/konnektive.py", line 1100, in konnektive_import_click
  File "/app/integrations/konnektive.py", line 298, in import_click
  File "/app/integrations/wrapper.py", line 17, in get
  File "/app/.heroku/python/lib/python3.6/site-packages/requests/api.py", line 72, in get
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/api/external_trace.py", line 61, in dynamic_wrapper
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/api/coroutine_trace.py", line 171, in return_value
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/api/external_trace.py", line 61, in <lambda>
  File "/app/.heroku/python/lib/python3.6/site-packages/requests/api.py", line 58, in request
  File "/app/.heroku/python/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/api/external_trace.py", line 61, in dynamic_wrapper
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/api/coroutine_trace.py", line 171, in return_value
  File "/app/.heroku/python/lib/python3.6/site-packages/newrelic/api/external_trace.py", line 61, in <lambda>
  File "/app/.heroku/python/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
  File "/app/.heroku/python/lib/python3.6/site-packages/requests/adapters.py", line 506, in send
  

  My cert isn't invalid right? I am using the default ACM cert Heroku gives out if you are using a custom domain.

  SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)")
  

  this is how i checked:

  openssl s_client -connect “workoutwarriors.com:443” -showcerts -servername “workoutwarriors.com”