Trying to web scrape a merged Html table with r

See also questions close to this topic

• How to display questions and option of mcq quiz in javascript

  I am building an app which conduct online exam. I have to select question and option from JS files. In these, four options are mentioned and answer will be calculated automatically. I am facing error in loading of questions from js file.I have tried with giving directly question in input. That part is working fine. I am very thankful if anyone can help me in this code.

  var currentQuestion = 0;
  var score = 0;
  var totQuestions = questions.length;
  
  var container = document.getElementById('quizContainer');
  var questionEl = document.getElementById('question');
  var opt1 = document.getElementById('opt1');
  var opt2 = document.getElementById('opt2');
  var opt3 = document.getElementById('opt3');
  var opt4 = document.getElementById('opt4');
  var nextButton = document.getElementById('nextButton');
  var resultCont = document.getElementById('result');
  
  function loadQuestion(questionIndex)
  {
  	var q = questions[questionIndex];
  	questionEl.textContent = (questionIndex + 1) + '. ' + q.question;
  	opt1.textContent = q.option1;
  	opt2.textContent = q.option2;
  	opt3.textContent = q.option3;
  	opt4.textContent = q.option4;
  };
  
  function loadNextQuestion()
  {
  	var selectedOption = document.querySelector('input[type=radio].checked');
  	if(!selectedOption)
  	{
  		alert('please select your answer!');
  		return;
  	}
  	var answer = selectedOption.value;
  	if(questions[currentQuestion].answer == answer)
  	{
  		score += 1;
  	}
  	selectedOption.checked = false;
  	currentQuestion++;
  	if(currentQuestion == totQuestions -1)
  	{
  		nextButton.textContent = 'Finish';	
  	}
  	if(currentQuestion == totQuestions)
  	{
  		container.style.display ='none';
  		resultCont.style.display ='';
  		resultCont.textContent = 'Your Score:' + score;
  		return;
  	}
  	loadQuestion(currentQuestion);
  }
  
  var questions = [
  {
  	"question": "What would be the behaviour if this() and super() used in a method?",
  	"option1" : "Runtime error",
  	"option2" : "Throws exception",
  	"option3" : "compile time error",
  	"option4" : "Runs successfully",
  	"answer": "3"
  },
  {
  	"question": "What would be behaviour if constructor has a return type?",
  	"option1" : "Compilation error",
  	"option2" : "Runtime error",
  	"option3" : "Compilation and runs successfully",
  	"option4" : "Only String return type is allowed",
  	"answer": "1"
  },
  {
  	"question": "Which of these keywords can be used to prevent Method overriding?",
  	"option1" : "static",
  	"option2" : "constant",
  	"option3" : "protected",
  	"option4" : "final",
  	"answer": "4"
  },
  {
  	"question": "Which of these is supported by method overriding in Java?",
  	"option1" : "bstraction",
  	"option2" : "Encapsulation",
  	"option3" : "Polymorphism",
  	"option4" : "None of the mentioned",
  	"answer": "3"
  },
  {
  	"question": "Which of these keyword must be used to inherit a class?",
  	"option1" : "super",
  	"option2" : "this",
  	"option3" : "extent",
  	"option4" : "extends",
  	"answer": "4"
  },
  {
  	"question": "Which of these methods is an alternative to getChars() that stores the characters in an array of bytes",
  	"option1" : "getchars()",
  	"option2" : "GetChars()",
  	"option3" : "Getchars()",
  	"option4" : "getChars()",
  	"answer": "4"
  },
  {
  	"question": "Which of these method of class StringBuffer is used to extract a substring from a String object?",
  	"option1" : "substring()",
  	"option2" : "Substring()",
  	"option3" : "SubString()",
  	"option4" : "None of the mentioned",
  	"answer": "1"
  },
  {
  	"question": "Which of these method of class StringBuffer is used to get the length of sequence of characters",
  	"option1" : "length()",
  	"option2" : "capacity()",
  	"option3" : "Length()",
  	"option4" : "Capacity()",
  	"answer": "1"
  },
  {
  	"question": "Which of the following are incorrect form of StringBuffer class constructor?",
  	"option1" : "StringBuffer()",
  	"option2" : "StringBuffer(int size)",
  	"option3" : "StringBuffer(String str)",
  	"option4" : "StringBuffer(int size , String str)",
  	"answer": "4"
  },
  {
  	"question": "Which of these process occur automatically by java run time system?",
  	"option1" : "Serialization",
  	"option2" :  "Memory allocation",
  	"option3" : "File Filtering",
  	"option4" : "Deserialization",
  	"answer": "4"
  },
  {
  	"question": "Which of these interface extends DataInput interface?",
  	"option1" : "Serializable"
  	"option2" : "Externalization"
  	"option3" : "ObjectOutput"
  	"option4" : "ObjectInput"
  	"answer": "4"
  },
  {
  	"question": "Which of these class extend InputStream class?",
  	"option1" : "ObjectStream",
  	"option2" : "ObjectInputStream",
  	"option3" : "ObjectOutput",
  	"option4" : "ObjectInput",
  	"answer": "2"
  }]

  	body{
  	font-family: cambria;
  	background: #F0FFFF;
  	
  }
  .container
  {
  	height: 330px;
  	width:1000px;
  	position: absolute;
  	top:50%;
  	left:50%;
  	transform: translateX(-50%) translateY(-50%);
  	background: rgba(225, 225, 225, 0.5);
  	padding: 20px;
  	border: 1px solid #08038C;
  	box-shadow: 0 0 8px 3px #fff;
  	border-radius: 10px;
  
  }
  .title
  {
  	padding-top: 0px;
  	text-align: center;
  	font-size: 36px;
  	text-transform: uppercase;
  	color: #08038C
  }
  .question
  {
  	padding: 20px;
  	font-size: 22px;
  	background: #5F4BBE;
  	border-radius: 20px;
  	margin:10px 0 10px 0;
  	color: #f6f6f6;
  }
  .option
  {
  	width: 470px;
  	display: inline-block;
  	padding: 10px 0 10px 15px;
  	vertical-align: middle;
  	background: #C37DDA;
  	margin: 10px 0 10px 10px;
  	color: #000000;
  	border-radius: 20px;
  }
  .option:hover
  {
  	background: #5F4BBE;
  	color: #f6f6f6;
  }
  .next-btn
  {
  	border:none;
  	outline: none;
  	background:#C37DDA;
  	width: 100px;
  	height: 35px;
  	border-radius: 20px;
  	cursor: pointer;
  	float:right;
  	margin: 10px;
  }
  .next-btn:hover
  {
   background: #5F4BBE;
   color: #f6f6f6;
  }
  .result
  {
  	height: 100px;
  	text-align: center;
  	font-size: 75px;	
  }
  
  .option input:checked.option
  {
  	background: #0803BC;
  	color: #000;
  }

  <!DOCTYPE HTML>
  <html>
  <head>
  <link rel="stylesheet" type="text/css" href="login.css">
  	<title>Test</title>
  </head>
  <body>
  	<div id="quizContainer" class="container">
  		<div class="title">Java Exam</div>
  		<div id="question" class="question"></div>
  		<label class="option"><input type="radio" name="option" value="1" /><span id ="opt1"></span></label> 
  
  		<label class="option"><input type="radio" name="option" value="2" /><span id ="opt2"></span></label> 
  
  		<label class="option"><input type="radio" name="option" value="3" /><span id ="opt3"></span></label> 
  
  		<label class="option"><input type="radio" name="option" value="4" /><span id ="opt4"></span></label>
  		
  		<button id="nextButton" class="next-btn" onclick="loadNextQuestion();">Next</button>
  	</div> 
  	<div id="result" class="conatainer result" style="display:none;">
  	</div>
  	<script src="question_script.js"></script>
  </body>
  </html>
• How to DOM target in jQuery?

  So, in JS I'd do something like.

  var subItems = document.querySelectorAll(".dropdown");
  

  Then target by "dropItem[0]" or a forEach loop. Now I'm also trying to do this in jQuery. By doing something like:

  $("p").click(function() {
    var dropwDown = $(this).find(".dropdown-contain");
    $(dropDown).show();
  });

  <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
  <div class="menu-dropdown">
    <div class="whisky">
      <p>Whisky</p>
      <div class="dropdown-contain">
        <ul class="dropdown">
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
        </ul>
      </div>
    </div>
    <div class="wine">
      <p>Wine</p>
      <div class="dropdown-contain">
        <ul class="dropdown">
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
        </ul>
      </div>
    </div>
    <div class="beer">
      <p>Beer</p>
      <div class="dropdown-contain">
        <ul class="dropdown">
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
        </ul>
      </div>
    </div>
    <div class="cocktail">
      <p>Cocktail</p>
      <div class="dropdown-contain">
        <ul class="dropdown">
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
          <li><a href="#">test</a></li>
        </ul>
      </div>
    </div>
  </div>

  However, that doesn't work. This is the HTML I'm trying to target. Then especially the dropdown class.
• How to fill up a textbox with javascript?

  I am very new to javascript and I was trying to populate a textbox from html with javascript but for some reason isn't working. This is what I have:

  <!DOCTYPE html>
  <html>
  <head>
    <meta charset="UTF-8">
    <title>Relations</title>
    <link rel="stylesheet" href="kandi.css" type="text/css" media="screen">
    
    <script src="myscript.js"></script>
    
    <body>
  	<h1 align="center" style="font-family:Arial;" >RELATION TAGGING </h1>
  	<textarea id="your_textarea" rows="15" cols="160">
  	</textarea>
    
    </body>
    </head>
  </html>

  Javascript:

  var textarea = document.getElementById("your_textarea");
  textarea.innerHTML = "testing123";
  
• Group several columns then aggregate a set of columns in Pandas (It crashes badly compared to R's data.table)

  I am relatively new to the world of Python and trying to use it as a back-up platform to do data analysis. I generally use data.table for my data analysis needs.

  The issue is that when I run group-aggregate operation on big CSV file (randomized, zipped, uploaded at http://www.filedropper.com/ddataredact_1), Python throws:

  grouping pandas return getattr(obj, method)(*args, **kwds) ValueError: negative dimensions are not allowed

  OR (I have even encountered...)

  File "C:\Anaconda3\lib\site-packages\pandas\core\reshape\util.py", line 65, in cartesian_product for i, x in enumerate(X)] File "C:\Anaconda3\lib\site-packages\pandas\core\reshape\util.py", line 65, in for i, x in enumerate(X)] File "C:\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py", line 445, in repeat return _wrapfunc(a, 'repeat', repeats, axis=axis) File "C:\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py", line 51, in _wrapfunc return getattr(obj, method)(*args, **kwds) MemoryError

  I have spent three days trying to reduce the file size (I was able to reduce the size by 89%), adding breakpoints, debugging it, but I was not able to make any progress.

  Surprisingly, I thought of running the same group/aggregate operation in data.table in R, and it hardly took 1 second. Moreover, I didn't have to do any data type conversion etc., suggested at https://www.dataquest.io/blog/pandas-big-data/.

  I also researched other threads: Avoiding Memory Issues For GroupBy on Large Pandas DataFrame, Pandas: df.groupby() is too slow for big data set. Any alternatives methods?, and pandas groupby with sum() on large csv file?. It seems these threads are more about matrix multiplication. I'd appreciate if you wouldn't tag this as duplicate.

  Here's my Python code:

  finaldatapath = "..\Data_R"
  ddata = pd.read_csv(finaldatapath +"\\"+"ddata_redact.csv", low_memory=False,encoding ="ISO-8859-1")
  
  #before optimization: 353MB
  ddata.info(memory_usage="deep")
  
  #optimize file: Object-types are the biggest culprit.
  ddata_obj = ddata.select_dtypes(include=['object']).copy()
  
  #Now convert this to category type:
  #Float type didn't help much, so I am excluding it here.
  for col in ddata_obj:
      del ddata[col]
      ddata.loc[:, col] = ddata_obj[col].astype('category')
  
  #release memory
  del ddata_obj
  
  #after optimization: 39MB
  ddata.info(memory_usage="deep")
  
  
  #Create a list of grouping variables:
  group_column_list = [
                   "Business",
                   "Device_Family",
                   "Geo",
                   "Segment",
                   "Cust_Name",
                   "GID",
                   "Device ID",
                   "Seller",
                  "C9Phone_Margins_Flag",
                   "C9Phone_Cust_Y_N",
                   "ANDroid_Lic_Type",
                   "Type",
                   "Term",
                   'Cust_ANDroid_Margin_Bucket',
                   'Cust_Mobile_Margin_Bucket',
  # #                'Cust_Android_App_Bucket',
                   'ANDroind_App_Cust_Y_N'
  ]
  
  print("Analyzing data now...")
  
  def ddata_agg(x):
      names = {
          'ANDroid_Margin': x['ANDroid_Margin'].sum(),
          'Margins': x['Margins'].sum(),
          'ANDroid_App_Qty': x['ANDroid_App_Qty'].sum(),
          'Apple_Margin':x['Apple_Margin'].sum(),
         'P_Lic':x['P_Lic'].sum(),
         'Cust_ANDroid_Margins':x['Cust_ANDroid_Margins'].mean(),
         'Cust_Mobile_Margins':x['Cust_Mobile_Margins'].mean(),
         'Cust_ANDroid_App_Qty':x['Cust_ANDroid_App_Qty'].mean()
      }
      return pd.Series(names)
  
  ddata=ddata.reset_index(drop=True)
  
  ddata = ddata.groupby(group_column_list).apply(ddata_agg) 
  

  The code crashes in above .groupby operation.

  Can someone please help me? Compared to my other posts, I have probably spent the most amount of time on this StackOverflow post, trying to fix it and learn new stuff about Python. However, I have reached saturation--it even more frustrates me because R's data.table package processes this file in <2 seconds. This post is not about pros and cons of R and Python, but about using Python to be more productive.

  I am completely lost, and I'd appreciate any help.

  ---

  Here's my data.table R code:

  path_r = "../ddata_redact.csv"
  ddata<-data.table::fread(path_r,stringsAsFactors=FALSE,data.table = TRUE, header = TRUE)
  
  group_column_list <-c(
    "Business",
    "Device_Family",
    "Geo",
    "Segment",
    "Cust_Name",
    "GID",
    "Device ID",
    "Seller",
    "C9Phone_Margins_Flag",
    "C9Phone_Cust_Y_N",
    "ANDroid_Lic_Type",
    "Type",
    "Term",
    'Cust_ANDroid_Margin_Bucket',
    'Cust_Mobile_Margin_Bucket',
    # #                'Cust_Android_App_Bucket',
    'ANDroind_App_Cust_Y_N'
    )
  
      ddata<-ddata[, .(ANDroid_Margin = sum(ANDroid_Margin,na.rm = TRUE), 
  Margins=sum(Margins,na.rm = TRUE), 
  Apple_Margin=sum(Apple_Margin,na.rm=TRUE),  
  Cust_ANDroid_Margins = mean(Cust_ANDroid_Margins,na.rm = TRUE),  
  Cust_Mobile_Margins = mean(Cust_Mobile_Margins,na.rm = TRUE),
  Cust_ANDroid_App_Qty = mean(Cust_ANDroid_App_Qty,na.rm = TRUE),
  ANDroid_App_Qty=sum(ANDroid_App_Qty,na.rm = TRUE)
  ), 
  by=group_column_list]
  

  ---

  I have a 4-core 16GB RAM Win10x64 machine. I can provide any details needed by experts.
• Installing RODBC in R

  I am trying to install RODBC and when I do so i get this error:

  Installing package into ‘\\lakesh/Documents/R/win-library/3.5’
  (as ‘lib’ is unspecified)
  trying URL 'https://cran.rstudio.com/bin/windows/contrib/3.5/RODBC_1.3-15.zip'
  Content type 'application/zip' length 879575 bytes (858 KB)
  downloaded 858 KB
  
  package ‘RODBC’ successfully unpacked and MD5 sums checked
  Warning in install.packages :
    cannot remove prior installation of package ‘RODBC’
  

  Then I tried to remove the package:

  > remove.packages("RODBC", lib = NULL)
  Removing package from ‘\\lakesh/Documents/R/win-library/3.5’
  (as ‘lib’ is unspecified)
  Error in remove.packages : there is no package called ‘RODBC’
  

  Need some guidance on installing RODBC package properly.
• Rearranging columns in R

  I have this table:

  ID Gene1 Gene2 Gene3
  a.  1a.   2a.   3a.
  b.  1b.   2b.   3b.
  c.  1c.   2c.   3c.
  

  I want two new columns "Gene" and "effect" to generate this table:

  ID  Gene  effect
  a.   1.    1a.
  a.   2.    2a.
  a.   3.    3a.
  b.   1.    1b.
  b.   2.    2b.
  b.   3.    3b.
  c.   1.    1c.
  c.   2.    2c.
  c.   3.    3c.
  
• How to remove tag in lxml?

  Firstly I followed this question, but I still have issues with the remove method.

  tag.getparent().remove(tag)
  

  I used this piece of code for removing anchor tag in question with attributes name="2" and id = "2" in this webpage

  and when the line is executed I was still able to see the tag and its properties and when I iterate through all children I was still able to see the element which i deleted

  What exactly does remove method does and why the tag which is deleted still persists?

  This is the screenshot of the debugger after the line is executed.

  
• (Flexible) Python Web Scrapers for a Search Engine (with content reformatting and posting)

  I need some help relating to building new web scrapers for a search engine. To give concrete details, I am building a new search engine for heavy trucks and equipment export targeted to countries in Africa, South America, and parts of Southeast Asia (it is like IronPlanet.com but global and trucks will be sent from Europe and North America to the rest of the world). My startup already has funding and basically has been greenlighted from all of the business people. This website will send business to all of the source websites to export their machinery and has been cleared for all terms of use (it's like free advertising, sort of).

  I basically need to build the entire search engine from scratch and will need to scrape thousands of different types of websites (eBay, Craigslist, Autotrader, Scout24, etc.) and webpages in different formats and languages (English, German, Spanish, etc.), take the data, and then re-post the data in a standard template that is country-specific to the user. I have built many scrapers in the past using Selenium and Beautiful Soup to scrape all kinds of data out of HTML, but I am struggling for how to build 1) a scraper flexible enough to handle many different kinds of HTML pages, 2) a second scraper that can decode JavaScript pages (visually if needed), and 3) how to quickly save the data into a common re-post-able format. I am less familiar with Scrapy and some other tools, do not want to rely on natural language processing, and I know that this is possible.

  The brute-force methodology (which a bunch of websites have done) is to literally build thousands of site-specific scrapers and language-specific conditions in order to search for the x-paths or CSS sections and maintain these scrapers as websites change; I would like to avoid this if possible. If anyone has experience building 1) search engine back-end systems for collecting data from webpages (main pages or subpages), 2) complex, flexible scrapers that can handle diverse page formatting (almost a complement to a web spider), or 3) a good means to reliably reformat scraped text data for reposting (like a truck's details), please kindly send me a note or post below. I wanted to build this all in Python if possible using Pandas, Numpy, Selenium, and Beautiful Soup, but if there are better systems for Python, I am definitely open to learning.

  Thank you everyone! I'm a new guy.
• Why do i get the error 'NoneType' object is not subscriptable when looping URL's with beautiful soup?

  I have written some code to scrape a JSON string, parse it and send the results to a SQL database. There are about 1000 pages I need to parse so I have imported the URL's from a CSV file to loop through them.

  I have written the code without the loop and everything works fine.

  url = 'https://www.nrl.com/draw/nrl-premiership/2018/round-25/Broncos-vs-Sea-Eagles/'
  
  response = requests.get(url)
  soup = BeautifulSoup(response.text, 'lxml')
  text = soup.find(class_='l-content pre-quench')['q-data']
  unescaped_text = html.unescape(text)  
  

  However when I import the data from a CSV and add a loop to it

  for game in url:
      response = requests.get(game)
      soup = BeautifulSoup(response.text, 'lxml')
      text = soup.find(class_='l-content pre-quench')['q-data']
      unescaped_text = html.unescape(text)
  

  i get the error

    File "<ipython-input-71-1b8888dd0ee0>", line 5, in <module>
      text = soup.find(class_='l-content pre-quench')['q-data']
  
  TypeError: 'NoneType' object is not subscriptable
  

  I have tried using a single entry (take works in the non-looped version) in the CSV file but i still get the error.

  from reading i have done the error normally means that the element you are trying to get is not in the page but this is not the case in this example.

  Any ideas why this is working without the loop but fails in side the loop?
• Error in html_table.xml_node(., header = FALSE) : html_name(x) == "table" is not TRUE

  this is the code I have written to extract the table present on this webpage given below but i am not able to extract the table .

  problem code line -

  html_node("#embed") %>% #embed is I have selected using selectorgadget.
  

  snippet:

  install.packages("rvest")
  library(rvest)
  install.packages("dplyr")
  library(dplyr)
  google <-read_html("https://data.gov.in/major-indicator/gross-enrolment-ratio-higher-education")
  google %>%  
  html_node("#embed") %>%
  html_table(header=FALSE)
  

  please help me .
• Scraping map markers from ajax call

  How can I scrape the coordinates of the bus stops which are shown on this site:

  When I go to Network I can see the endpoint of the GET request and I can see the result I desire.

  However when I try to httr::GET("www.eway.bg/ajax/bg/sofia/routeInfo/19") I receive a 404.

  ---

  I guess it is connected with cookies and referer, however, how should I proceed to scrape it ideally in R?

  P.S. if not in R, a pure curl request which works would also be appreciated

  Thanks
• Page limit using rvest

  I'm having an issue when using rvest to scrape 466 pages from a wiki. Each page represents a metric that I need further information about. I have the following code which loops through each link (loaded from a csv file) and extracts the information I need from a html table on each page.

  Metrics <- read.csv("C:\\Users\\me\\Documents\\WebScraping\\LONMetrics.csv")
  
      Metrics$Theme <- as.character(paste0(Metrics$Theme))
      Metrics$Metric <- as.character(paste0(Metrics$Metric))
      Metrics$URL <- as.character(paste0(Metrics$URL))
  
      n = nrow(Metrics)
  
      i = 1
  
      while (i <= n) {
        webPage <- read_html(Metrics$URL[i])
        pageTable <- html_table(webpage)
        Metrics$Definition[i] <- pageTable[[1]]$X2[1]
        Metrics$Category[i] <- pageTable[[1]]$X2[2]
        Metrics$Calculation[i] <- pageTable[[1]]$X2[3]
        Metrics$UOM[i] <- pageTable[[1]]$X2[4]
        Metrics$ExpectedTrend[i] <- pageTable[[1]]$X2[6]
        Metrics$MinTech[i] <- pageTable[[1]]$X2[7]
        i = i+1
      }
  

  The problem I'm having is that it stops returning data after 32 pages giving an error as:

  Error in read_connection_(x, n) :
  Evaluation error: Failure when receiving data from the peer

  I'm wondering what the cause may be and how to get around this seeming limitation?

  Thanks.

  Rob