Having trouble extracting dynamic div list while scrolling down using Webdriver (selenium & python)

See also questions close to this topic

• Python Kafka Streaming API - Binning

  I am using python kafka stream binning example given in this, Python Kafka Streaming API

  I am able to generate the data using generator.py file given under winton-kafka-streams/examples/binning/, whereas when i run the binning.py file from the same folder, i got the below issue. Could someone help me, to resolve this?

  

  
• Change color of missing values in Seaborn heatmap

  Consider the example of missing values in the Seaborn documentation:

  corr = np.corrcoef(np.random.randn(10, 200))
  mask = np.zeros_like(corr)
  mask[np.triu_indices_from(mask)] = True
  sns.heatmap(corr, mask=mask, vmax=.3, square=True)
  

  

  How do I change the color of the missing values to, for example, black? The color of the missing values should be specified independent of the color scheme of the heatmap, it may not be present in the color scheme.

  I tried adding facecolor = 'black' but that didn't work. The color can be affected by e.g. sns.axes_style("white") but it isn't clear to me how that can be used to set an arbitrary color.
• Xpath + Scrapy + Python : data point couldn't be scraped

  This is the XML structure:

  <tr>
     <td>
         <font size="3">
             <strong>Location:</strong> 
             Hiranandani Gardens, Powai
         </font>
     </td> 
  </tr>  
  

  I want to extract : Hiranandani Gardens, Powai

  I tried with these:

  response.xpath('//font[@size="3"]/strong[text()="Location:"]/text()').extract()
  
  response.xpath('//font[@size="3"]/strong[text()="Location:"]').extract()
  

  Both returned an empty list.

  Note: we must have to use the text of tag, i.e., "Location:". Otherwise, there are many other places on the site where the same XML structure is used. So, it'll fetch many more unnecessary things apart from the desired value if the text of strong tag is not used.
• How to test existing Angularjs overall performance (DOM rendering)?

  I got an existing angularjs project for optimization. How can I find gaps like (memory leakage, performance on browser , Data rendering , screen got stuck kind of issues). I tried Jmeter but its not showing javascript performance just shows page is navigated in few Milli seconds. I tried to find with chrome profiling , I got some specific scenarios but not overall image.

  How can I get overall performance results?
• Python Selenium Webdriver NoSuchElementException error

  I'm getting this error with Selenium:

  ~\Anaconda3\lib\site-packages\selenium\webdriver\support\wait.py in >until(self, method, message) 78 if time.time() > end_time: 79 break ---> 80 raise TimeoutException(message, screen, stacktrace) 81 82 def until_not(self, method, message=''):

  TimeoutException: Message:

  Below is the code i'm working with. Could anyone kindly point me in the correct direction? I want the code to visit the next item in the list and eventually try to execute that try block.

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.common.exceptions import NoSuchElementException
  items =['page1', 'page2']
  for item in test_list:
      driver.get("https://www.example.com/" + item)
  
      try:
          WebDriverWait(driver, 150).until(EC.visibility_of_element_located((By.ID, 'windowConfirmAutomatch')))
          save_suggested_matches_button = driver.find_element_by_link_text('Save Suggested Matches')
          driver.execute_script("arguments[0].click();", save_suggested_matches_button)
          WebDriverWait(driver, 150).until(EC.invisibility_of_element_located((By.ID, 'windowConfirmAutomatch')))
          print("AutoMatch Complete for " + item)
      except NoSuchElementException:
          continue
  
• Finding an element using placeholder in Protractor?

  <input _ngcontent-c6="" formcontrolname="password" 
      name="password" onblur="this.placeholder = 'Password'" 
      onfocus="this.placeholder = ''" 
      placeholder="Password" 
      required="" type="password" 
      ng-reflect-required="" ng-reflect-name="password" 
      class="ng-pristine ng-invalid ng-touched"> 
  

  What is the best way to find an element: placeholder = "Enter Username"?

  Scenario: find an element using Snippet above User clicks on the "Enter Name" box, another window pops-up for entering a name.
• Getting this Selenium webdriver http error in angular 4 project

  ERROR in

  ./node_modules/selenium-webdriver/http/index.js 
  Module not found: Error: Can't resolve 'http' in 'C:\Users\aprajita.singh\Documents\Angular 4\Auto-Trender-Project\node_modules\selenium-webdriver\http' 
  ERROR in ./node_modules/selenium-webdriver/http/index.js 
  Module not found: Error: Can't resolve 'https' in 'C:\Users\aprajita.singh\Documents\Angular 4\Auto-Trender-Project\node_modules\selenium-webdriver\http'.  
  

  Following are my imports:

  import { Injectable } from '@angular/core'; 
  import { Http, Headers, Response, RequestOptions } from '@angular/http'; 
  import { HttpClient } from '@angular/common/http';
  import { Observable } from 'rxjs/internal/Observable'; 
  import { FormGroup, FormBuilder, Validators } from '@angular/forms'; 
  import { HttpClientModule } from '@angular/common/http';
  

  How do I resolve this?
• Getting "java.lang.NoClassDefFoundError" for Testng Dataprovider with Excel

  I am not able to get past the error "java.lang.NoClassDefFoundError"

  Tried: 1- Cleaning the project 2- Updating testng xml 3- Verified the package/class structure

  Nothing seems to work:

  Note: I am not using Maven and all jar files are latest (Testng, Apache, Selenium)

  Test Class code :

  @DataProvider(name ="calldata")
  public Object[][] getData() {
  
      ExcelUtility util = new ExcelUtility("excel path");
  
      int rows = util.getRowCount(0);
  
      Object data [][] = new Object[rows] [6];
  
      for(int i=1; i<rows; i++)
      {
          data[i][0]=util.getData(0, i, 0);
          data[i][1]=util.getData(0, i, 1);
          data[i][1]=util.getData(0, i, 2);
          data[i][1]=util.getData(0, i, 3);
          data[i][1]=util.getData(0, i, 4);
          data[i][1]=util.getData(0, i, 5);
      }
  
      return data;
  }
  
  @Test(dataProvider ="calldata")
   public void AddaDependent(String url, String client, String eliggrp, String empnumber, String firstname, String SSN) 
      {
  

  Excel Utility code:

  package AllExcel;

  import java.io.File; import java.io.FileInputStream;

  import org.apache.poi.xssf.usermodel.XSSFSheet; import org.apache.poi.xssf.usermodel.XSSFWorkbook;

  public class ExcelUtility {

  XSSFWorkbook wb;
  XSSFSheet sheet1;
  
  public ExcelUtility(String excelPath)
  {
      try
      {
          File src = new File(excelPath);
  
          FileInputStream fis= new FileInputStream(src);
          wb = new XSSFWorkbook(fis);
  
      }
      catch (Exception e) {
          System.out.println(e.getMessage());
  
      }
  }
  
  public String getData(int sheetNumber, int row, int column)
  {
      sheet1 = wb.getSheetAt(0);
  
      String data = sheet1.getRow(row).getCell(column).getStringCellValue();
  
      return data;
  }
  
  public int getRowCount(int sheetIndex)
  {
      int row = wb.getSheetAt(sheetIndex).getLastRowNum();
  
      row = row+1;
  
      return row;
  }
  

  }

  Error:

  [RemoteTestNG] detected TestNG version 6.14.3 [Utils] [ERROR] [Error] java.lang.NoClassDefFoundError: org/apache/commons/compress/archivers/zip/ZipFile at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:298) at org.apache.poi.ooxml.util.PackageHelper.open(PackageHelper.java:37) at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:307) at AllExcel.ExcelUtility.<init>(ExcelUtility.java:22) at AutFramework.AddDependent.getData(AddDependent.java:36) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source)
• Problem with using Python library - Pyotodom

  I wanted to use this library to scrape data from otodom. I've read the docs however I'm stuck on a very basic level, trying to do basic import. The code I use is taken directly from the docs. On top of that I've tried to launch example.py from Github but I'm getting the same error.

  Here's the excerpt from the code:

  import otodom, os, logging
  from otodom.category import get_category
  from otodom.offer import get_offer_information
  

  Error message I get:

  Traceback (most recent call last):

  File ".\otodom_import.py", line 2, in

  from otodom.category import get_category

  File "C:\Users\Dom\Anaconda3\lib\site-packages\otodom\category.py", line 9, in

  from otodom.utils import get_response_for_url, get_url

  File "C:\Users\Dom\Anaconda3\lib\site-packages\otodom\utils.py", line 14, in

  from scrapper_helpers.utils import caching, normalize_text, key_sha1, get_random_user_agent

  File "C:\Users\Dom\Anaconda3\lib\site-packages\scrapper_helpers\utils.py", line 22, in

  MAX_FILENAME_LENGTH = subprocess.check_output("getconf NAME_MAX /", shell=True).strip()

  File "C:\Users\Dom\Anaconda3\lib\subprocess.py", line 336, in check_output **kwargs).stdout

  File "C:\Users\Dom\Anaconda3\lib\subprocess.py", line 418, in run output=stdout, stderr=stderr)

  subprocess.CalledProcessError: Command 'getconf NAME_MAX /' returned non-zero exit status 1.

  Thanks in advance!

  EDIT: Don't understand downvotes, I realize that this is a niche module but I did my homework. I've read the docs and tried to find help in other places. If there is a chance that as a noobie I'll find at least one person that can help me, why wouldn't I use this opportunity? Even info that module is buggy can help.
• Cannot scrape website using Selenium in Python

  Trying to scrape off some data from footlocker.com for an academic project but I get an error when I try to open the page using Selenium:

  from selenium import webdriver
  
  
  driver = webdriver.Chrome("/Users/rushabhnahar/Downloads/chromedriver")
  
  driver.get("https://www.footlocker.com/adidas-Originals/Shoes/_-_/N- 
  zrZrj?cm_REF=Shoes&crumbs=991")
  

  It opens up the browser and the respective page but gives an error saying 'We are sorry'.

  Any help will be appreciated.
• How to Check Parent Element of find_element_by_partial_link_text

  I'm using the find_element_by_partial_link_text selector to find the "next" button, so that I can click it and continue crawling.

  However, the issue I'm having is that sometimes the word "next" is also in other random links on the page, which breaks the script.

  I've tried addressing this with the code below, which obviously doesn't work.

  pagination_ul = ff.find_element_by_xpath('//*[contains(@id, "pagination-both-")]')
  next_button = pagination_ul.find_element_by_partial_link_text('Next')
  

  So my question is...

  How can I retrieve and compare the parent element of a find_element_by_partial_link_text element?

  In other words, how can I make sure that the parent of the find_element_by_partial_link_text is, say, a li element?
• Python sort to avoid "KeyError: 'href'"

  I use code as below to sort links:

  links = set(soup.findAll('a'))
  links = sorted(links, key=lambda x: x['href'], reverse=True)
  

  sometime I got error as this:
• How can I use Beautiful Soup to parse the batter's names?

  I have had no issues grabbing three stats: hits, runs and rbi's. Here is the code I have been working with so far:

  #import modules
  
  from bs4 import BeautifulSoup
  import requests, os
  from selenium import webdriver
  
  #start webdriver 
  
  os.chdir('C:\webdrivers')
  header = {'User-agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) 
  AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}
  options = webdriver.ChromeOptions(); options.add_argument("--start- 
  maximized")
  driver = webdriver.Chrome(chrome_options=options)
  driver.get('https://www.baseball-reference.com/leagues/MLB/2018-standard- 
  batting.shtml')
  
  #grab html
  
  soup = BeautifulSoup(driver.page_source, 'html.parser')
  driver.quit()
  
  #parse three stats: rbi's, runs and hits
  
  hits = [i.text for i in soup.find_all('td', {'data-stat': 'H'})]
  runs = [i.text for i in soup.find_all('td', {'data-stat': 'R'})]
  rbi = [i.text for i in soup.find_all('td', {'data-stat': 'RBI'})]
  
  #print data
  
  print(hits, runs, rbi)
  

  The code above works great. When I try to grab the batter's names, however, I run into some problems. The batter's names are not parsed correctly. I would like just their first and last name if possible.

  Here is what I tried:

  print(soup.find_all('td', {'data-stat': 'player'}))
  

  The batter's names are in the code but there is a lot of extra data. Also, my computer slowed down a lot when I tried this line of code. Any suggestions? Thanks in advance for any help you may offer!
• How to extract different types of bold text and the text in between them using BeautifulSoup?

  I have to parse html documents having bold text as section identifiers. But bold text is in different forms , some examples shown below .

  Using Beautiful soup I am able to parse them but have to write lot of if else to handle different types of bold. Is there an optimal way to find such bold text and the text in between with using so many if else.

  

  <div style="line-height:120%;padding-bottom:12px;font-size:10pt;">
    <font style="font-family:inherit;font-size:10pt;font-weight:bold;">List 1.&nbsp;&nbsp;&nbsp;&nbsp;Work</font>
  </div>
  <td style="vertical-align:top;padding-left:2px;padding-top:2px;padding-bottom:2px;padding-right:2px;">
    <div style="text-align:left;font-size:10pt;">
      <font style="font-family:inherit;font-size:10pt;font-weight:bold;">List 1.</font>
    </div>
  </td>
  <td style="vertical-align:top;padding-left:2px;padding-top:2px;padding-bottom:2px;padding-right:2px;">
    <div style="text-align:left;font-size:10pt;">
      <font style="font-family:inherit;font-size:10pt;font-weight:bold;">Work.</font>
    </div>
  </td>
  <p style="font-family:times;text-align:justify">
    <font size="2">
      <a name="de42901_List_1._Work"> </a>
      <a name="toc_de42901_2"> </a>
    </font>
    <font size="2"><b>  List&nbsp;1.&nbsp;&nbsp;&nbsp;&nbsp;Work    <br>    </b></font>
  </p>
  <p style="font-family:times;text-align:justify">
    <font size="2">
      <a name="da18101_List_1._Work"> </a>
      <a name="toc_da18101_3"> </a>
    </font>
    <font size="2"><b>  List&nbsp;1.&nbsp;&nbsp;&nbsp;&nbsp;</b></font>
    <font size="2"><b><i>Work    <br>    </i></b></font>
  </p>