Having trouble extracting dynamic div list while scrolling down using Webdriver (selenium & python)

See also questions close to this topic

• how to add non .py files into python egg

  I have a flask app which looks like

  my-app 
  │   └── src
  │       └── python
  │             └── config
  │             └── app
  │── MANIFEST.in
  └── setup.py
  

  The config folder is full of *.yaml files, I want to add all the static config files into my python egg after using

  python setup.py install
  

  My setup.py looks like

  import os
  from setuptools import setup, find_packages
  path = os.path.dirname(os.path.abspath(__file__))
  setup(
      name="app",
      version="1.0.0",
      author="Anna",
      description="",
      keywords=[],
      packages=find_packages(path + '/src/python'),
      package_dir={'': path + '/src/python'},
      include_package_data=True
  )
  

  I am trying the use the MANIFEST.in to add the config file However, it always give error

  error: Error: setup script specifies an absolute path:
  
  /Users/Anna/Desktop/my-app/src/python/app
  
  setup() arguments must *always* be /-separated paths relative to the
  setup.py directory, *never* absolute paths.
  

  I have not used any absolute paths in my code, I've seen other posts trying to bypass this error, by removing

  include_package_data=True
  

  However, in my case, if i do this to avoid this error, all my yamls won't be added.

  I was wondering if there are ways to fix this problem. Thanks
• How to extract all functions and API calls used in a Python source code?

  Let us consider the following Python source code;

  def package_data(pkg, roots):
      data = []
      for root in roots:
          for dirname, _, files in os.walk(os.path.join(pkg, root)):
              for fname in files:
                  data.append(os.path.relpath(os.path.join(dirname, fname), pkg))
  
      return {pkg: data}
  

  From this source code, I want to extract all the functions and API calls. I found a similar question and solution. I ran the solution given here and it generates the output [os.walk, data.append]. But I am looking for the following output [os.walk, os.path.join, data.append, os.path.relpath, os.path.join].

  What I understood after analyzing the following solution code, this can visit the every node before the first bracket and drop rest of the things.

  import ast
  
  class CallCollector(ast.NodeVisitor):
      def __init__(self):
          self.calls = []
          self.current = None
  
      def visit_Call(self, node):
          # new call, trace the function expression
          self.current = ''
          self.visit(node.func)
          self.calls.append(self.current)
          self.current = None
  
      def generic_visit(self, node):
          if self.current is not None:
              print("warning: {} node in function expression not supported".format(
                    node.__class__.__name__))
          super(CallCollector, self).generic_visit(node)
  
      # record the func expression 
      def visit_Name(self, node):
          if self.current is None:
              return
          self.current += node.id
  
      def visit_Attribute(self, node):
          if self.current is None:
              self.generic_visit(node)
          self.visit(node.value)  
          self.current += '.' + node.attr
  
  tree = ast.parse(yoursource)
  cc = CallCollector()
  cc.visit(tree)
  print(cc.calls)
  

  Can anyone please help me to modified this code so that this code can traverse the API calls inside the bracket?

  N.B: This can be done using regex in python. But it requires a lot of manual labors to find out the appropriate API calls. So, I am looking something with help of Abstract Syntax Tree.
• use correct version of 'pip' installed for your Python interpreter

  I am using pycharm , i am having this error by adding any package , Click Here

  i have tried lot of methods , but didn't succeed yet.

  Info :

  1. python 3.6.2
  2. pip 10.0.1
  3. VirtualEnv
• Locating elements through "begins with" method of CssSelector using selenium and python

  I'm using selenium to locate and select a HTML element using find_element_by_id. I have had no issues with this until yesterday, when the web developers changed the ID of the element to a dynamically string.

  Here is an example of what the ID looks like: input_205.

  Before, I used browser.find_elements_by_xpath('//*[@id="input_205"]').send_keys(viewId) to locate the element and this worked perfectly, but now the number value in that string dynamically changes.

  The only pattern I've identified is that it is a three digit number every time the page loads.

  I'm wondering if there's a way I can set my xpath to some sort of "begins with" or "contains" so that I can still target the ID regardless of the three digit numbers after input_. Thank you in advance!
• How to get an 'nth-of-type' in Selenium

  I'm using Selenium Webdriver to check the text of this particular paragraph (the one highlighted in blue here):

  

  But how do I 'query' that paragraph?

  This is what I'm trying (not working):

      def test_intro_text(self):
         """Test that intro text is expected text"""
         container = self.browser.find_element_by_id('visual-17')
         hed_dek_wrapper = container.find_element_by_class_name('hed-dek-wrapper')
         intro_text = hed_dek_wrapper.findElement('p:nth-of-type(2)')
         self.assertIn(
          'Over the past 20 years, we have seen an evolution.',
          intro_text.text
         )
  

  I'm getting this error: AttributeError: 'WebElement' object has no attribute 'findElement'
• selenium vba extract all <span class="" title="">

  Medium

  i am looking to extract all titles(there are many in website) in class="text" in excel with seleniumbasic
• Clicking on links to open on tabs producing stale element reference in chrome, whereas in firefox it does not produce any tab

  I am coding to open the links as tabs and get all the link title printed; as I am using keys and then using window concept, it should open links on tabs and then can print each link's page title. However, in chrome it is throwing stale element reference exception, whereas in firefox it is not even producing any tab.
  Please see the following code and correct me if I am doing wrong somewhere. Thank you.

  package com.practice.one;
  
  import java.util.Iterator;
  import java.util.List;
  import java.util.Set;
  import java.util.concurrent.TimeUnit;
  
  import org.openqa.selenium.By;
  import org.openqa.selenium.Keys;
  import org.openqa.selenium.WebDriver;
  import org.openqa.selenium.WebElement;
  import org.openqa.selenium.chrome.ChromeDriver;
  
  public class ClickingAllTheLinks {
  
  public static void main(String[] args) throws InterruptedException {
  System.setProperty("webdriver.chrome.driver", 
   "/Users/owner/desktop/chromedriver");
      WebDriver driver = new ChromeDriver();
      driver.manage().window().maximize();
      driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
  
      driver.get("http://www.qaclickacademy.com/practice.php");
      List <WebElement> links=driver.findElements(By.tagName("a"));
      for (int i=0;i<links.size();i++) {
          System.out.println(links.get(i).getText());
      }
      WebElement footer_links= driver.findElement(By.cssSelector("div#gf- 
     BIG"));
  
  List<WebElement>sub_links=footer_links.findElements(By.tagName("a"));
      for (int i=0;i<sub_links.size();i++) {
          System.out.println(sub_links.get(i).getText());
      }
      WebElement 
    left_footer_Links=driver.findElement(By.xpath("//table[@class='gf- 
     t']/tbody/tr/td[1]/ul"));
  
  
  List<WebElement>leftfoot_Links=left_footer_Links.findElements
  (By.tagName("a"));
      for (int i=0;i<leftfoot_Links.size();i++) {
          String clickandopenlinksonTabs=Keys.chord(Keys.CONTROL, 
   Keys.RETURN);
          leftfoot_Links.get(i).sendKeys(clickandopenlinksonTabs);
          Thread.sleep(3000L);
  
      Set<String> wins=driver.getWindowHandles();
      Iterator<String> it=wins.iterator();
      while(it.hasNext()) {
          driver.switchTo().window(it.next());
          System.out.println(driver.getTitle());
      }   
  
      }
  
  }
  
  }       
  
• Python selenium Error: no such element: Unable to locate element

  I am getting this error while trying to fill up form.

  Error:

  "no such element: Unable to locate element:

  {"method":"id","selector":"formId:codEmpresa"}".

  Any help would be appreciated.

  Here is my code:

  from selenium import webdriver
  chrome = webdriver.Chrome()
  chrome.get("https://www3.honda.com.br/newihs")
  
  chrome.find_element_by_id("formId:codEmpresa").send_keys("Text1")
  chrome.find_element_by_id("formId:codUsuario").send_keys("Text2")
  chrome.find_element_by_id("formId:senha").send_keys("Text2")
  chrome.find_element_by_id("formId:j_id68").click
  
• Web Scraping with Java/Jsoup

  I am trying to extract average salary from GlassDoor. This is the HTML code where it is:

  <span class="OccMedianBasePayStyle__payNumber" data-test="AveragePay">$118,034</span>
  

  Here is what I have so far.. This code outputs the line I want but I don't know how to just pull out the salary from data-test="AveragePay"

  public class Trans {
  
      public static void main(String[] args) {
          String url = "https://www.glassdoor.com/Salaries/seattle-software-engineer-salary-SRCH_IL.0,7_IM781_KO8,25.htm";
          Document document = null;
          try {
              document = Jsoup.connect(url).get();
          } catch (IOException e) {
              e.printStackTrace();
          }
  
          //a with href
          Elements links = document.select("span");
  
          for (Element link : links) {
  
              System.out.println("Text: " + link.getElementsByAttributeValueContaining("data-test", "Average"));
  
              //System.out.println("Text: " + link.text()); 
          }
  
• Scraping with Rvest - table has only headers

  I have been trying to scrape a website for tables with Rvest. Currently I can get to a table, but it only pulls the headers of the table but not the content. I haven't figured out how to complete resolve this, but I've tried a few similar solutions. (How to get table using rvest())

  I had to figure out how to submit to a single button without appropriate unique identifiers. Done through: How to submit login form in Rvest package w/o button argument, https://github.com/hadley/rvest/issues/156

  Here's my operating script:

  library(rvest)
  library(httr)
  library(R.utils)
      url<-'https://itmdapps.milwaukee.gov/publicApplication_QD/zipcode.jsp'
  ##############################
  #fix for package available online. https://github.com/hadley/rvest/issues/156
  custom.submit_request <-
    function (form, submit = NULL)
    {
      submits <- Filter(function(x) {
        identical(tolower(x$type), "submit")
      }, form$fields)
  
      nsubmits <- Filter(function(x) {
        !identical(tolower(x$type), "submit")
      }, form$fields)
  
      # if list take name and vakue as inputs
      if (is.list(submit)) {
        submits[[1]]$name  <- names(submit)[1]
        submits[[1]]$value <- submit[[1]]
        submit <- submits[[1]]
      }
  
      # if character filter by name
      if (is.character(submit)){
        submit <- Filter(function(x){x$name==submit},submits)[[1]]
      }
  
      # if null choose first
      if (is.null(submit)) {
        submit <- submits[[1]]
        message("Submitting with '", submit$name, "'")
      }
  
      # handle method
      method <- form$method
      if (!(method %in% c("POST", "GET"))) {
        warning("Invalid method (", method, "), defaulting to GET",
                call. = FALSE)
        method <- "GET"
      }
  
      # url
      url <- form$url
  
      # fields
      fields <- nsubmits
      fields[submit$name] <- list(submit)
      fields <- Filter(function(x) length(x$value) > 0, fields)
      values <- rvest::pluck(fields, "value")
      names(values) <- names(fields)
  
      # return
      list(
        method = method, 
        encode = form$enctype, 
        url = url,
        values = values
      )
    }
  reassignInPackage('submit_request', 'rvest', custom.submit_request)
  #####################################
  
  target_zip_code_position<-2
  webpage.session <- html_session(url) #start website
  form<-html_form(webpage.session) #here's the form.
  
  #log in!
  form #let's look at it. We have to log in!
  filled_form<-form #create a copy to fill so we don't ruin the original.
  filled_form[[2]]<-set_values(filled_form[[2]], 
                               username = "address",
                               password = "user") #fill forms
  filled_form #how does it look?
  #filled_form[[2]]$url<-""  #URL needs to be cleared to prevent error message when submitting.
  logged_in.session<-submit_form(session = webpage.session,
                                 form = filled_form[[2]]) # defaults to first submission button with message.
  #We have successfully logged in.
  zip_search.session<-jump_to(logged_in.session,url) #navigate to the page with the query we want.
  zip_search.form<-html_form(zip_search.session)
  zip_search.form_filled<-zip_search.form
  zip_search.form_filled[[2]]<-set_values(zip_search.form_filled[[2]],
                                         zipcode = target_zip_code_position,
                                         format = 1,
                                         startDate = "01/01/2005",
                                         endDate = "01/01/2006"
                                         )
  list_submit<-list('WIBR Detailed')
  names(list_submit)<-c('submit')  #Very bizzare submit approach.
  output.session<- submit_form(session = zip_search.session,
                              form = zip_search.form_filled[[2]],
                              submit = list_submit
                              ) #how does it know which one? Requires fancy submit technique here. Now works.
  ### we have sent a query
  #output.read_html<-read_html(output.session)
  #output_table<-html_table(output.session, fill = TRUE)[[1]]  #no rows.
  
  table_node<-html_node(output.session,'div.main div.content:nth-child(5) table.bordered:nth-child(1)')
  html_text(table_node) #no rows still. Only selects proper table.
  html_table(table_node)
  
• Can't fetch titles from some box-like containers

  I've written a script in python in association with selenium to click on some dots available on a map in a web page. When a dot is clicked, a small box containing relevant information pops up.

  Link to that site

  I would like to parse the title of each box. When I execute my script, it throws an error while clicking on a dot. How can I make a go successfully?

  This is the script:

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  link = "replace with above link"
  
  driver = webdriver.Chrome()
  driver.get(link)
  wait = WebDriverWait(driver, 10)
  wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,"#mapDiv_zoom_slider .esriSimpleSliderIncrementButton"))).click()
  for item in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"#NWQMC_VM_directory_June2016_3915_0_layer circle"))):
      item.click()
  

  Error I'm having:

  line 242, in check_response
      raise exception_class(message, screen, stacktrace)
  selenium.common.exceptions.WebDriverException: Message: unknown error: Element <circle fill="rgb(237, 81, 81)" fill-opacity="1" stroke="rgb(153, 153, 153)" stroke-opacity="1" stroke-width="1" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="4" cx="720" cy="430" r="4" transform="matrix(1.00000000,0.00000000,0.00000000,1.00000000,0.00000000,0.00000000)" fill-rule="evenodd" stroke-dasharray="none" dojoGfxStrokeStyle="solid"></circle> is not clickable at point (720, 430). Other element would receive the click: <circle fill="rgb(20, 158, 206)" fill-opacity="1" stroke="rgb(153, 153, 153)" stroke-opacity="1" stroke-width="1" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="4" cx="720" cy="430" r="4" transform="matrix(1.00000000,0.00000000,0.00000000,1.00000000,0.00000000,0.00000000)" fill-rule="evenodd" stroke-dasharray="none" dojoGfxStrokeStyle="solid"></circle>
    (Session info: chrome=67.0.3396.99)
    (Driver info: chromedriver=2.36.540470 (e522d04694c7ebea4ba8821272dbef4f9b818c91),platform=Windows NT 6.1.7601 SP1 x86)
  

  This is how a box gets popped up:

  
• Part of HTML instead of full using Soup/Selenium

  I've been struggling with the following problem:

  I am trying to retrieve the full HTML of a certain page. I've managed to scrape a few other sites, but this one just won't cooperate.

  from urllib.request import urlopen as uReq
  from bs4 import BeautifulSoup as soup
  import requests
  from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  ##########################################
  
  url = "https://fd.nl/laatste-nieuws"
  
  driver = webdriver.Firefox()
  driver.get(url)
  
  html = driver.page_source
  
  page_soup1 = soup(html, "html5lib")
  
  page_soup1
  

  The output is just a subpart of the HTML. When I inspect the page through Chrome there are many more elements.

  I've tried just using Soup with multiple parsers (html.parser, html5lib and lxml), as well as using Selenium before Soup, both to no avail.

  I'm fairly new to all of this so any tips/guides are welcome!

  Cheers!
• Extract string within script tags from HTML

  I'm trying to make a web scraper to get data from the following website (I would later like to do it for several airlines on the same website): https://www.flightradar24.com/data/airlines/kl-klm/routes

  I currently have the following code:

  from bs4 import BeautifulSoup
  import requests
  
  airlines = ['kl-klm']
  
  for a in airlines:
      url = 'https://www.flightradar24.com/data/airlines/' + a + '/routes'
      page = requests.get(url)
      soup = BeautifulSoup(page.text, 'html.parser')
      print(soup)
  

  This gives me the source code for the whole page, but I would like to extract a specific chunk of text within script tags, which is

  var arrRoutes=[{"airport1":{"country":"Denmark","iata":"AAL","icao":"EKYT","lat":57.092781,"lon":9.849164,"name":"Aalborg Airport"},"airport2":{"country":"Netherlands","iata":"AMS","icao":"EHAM","lat":52.308609,"lon":4.763889,"name":"Amsterdam Schiphol Airport"}},{"airport1":{"country":"United Kingdom","iata":"ABZ","icao":"EGPD","lat":57.201939,"lon":-2.19777,"name":"Aberdeen International Airport"},"airport2":{"country":"Netherlands","iata":"AMS","icao":"EHAM","lat":52.308609,"lon":4.763889,"name":"Amsterdam Schiphol Airport"}}...
  

  ...etc. all the way until the end of the list.

  How can I extract this in such a way that I can find the total number of inbound and outbound flights for each airport? For example, the total number of times Amsterdam Schiphol Airport appears as airport 1 or 2?

  Is there a way to first extract the string from the HTML and then convert it into a Python list with dictionaries? Or would it make more sense to just directly count each element in the string?
• Can't grab some content using regex

  I've written some code in python in combination with BeautifulSoup to get some addresses within br tag. If it is about parsing the required text using BeautifulSoup alone, I could have done this using .next_sibling which I've shown below. My intention is to scrape the content within br in combination with BeautifulSoup and re.

  This is my try so far:

  import re
  from bs4 import BeautifulSoup
  
  content = """
  <div class="store"">
  <b>address</b><br>BLOCK ANG MO KIO AVE<br>
  <b>address_one</b><br>BLOCK 407 ANG MO KIO AVE 10 #01-741<br>
  <b>address_two</b><br>NO. 53 ANG MO KIO AVE 3 AMK HUB#B1-82<br>
  </div>
  """
  # soup = BeautifulSoup(content,"lxml")
  # for addr in soup.find_all("b"):
  #     print(addr.next_sibling.next_sibling)
  
  soup = BeautifulSoup(content,"lxml")
  for addr in soup.find_all(text=re.compile(r"<br>(.*?)</br>")):
      print(addr)  #It prints nothing, no error either
  

  FYI, the commented out portion is just working flawlessly if uncommented. As I'm trying to figure out what would be the usage of re in such cases, I would be very happy if someone provided me with a helping hand.