Why is CSV getting blank row after headers?

I am using BeautifulSoup to scrape a table from a website, but am confused as to why this is printing a blank row after my header and how I can fix it.

My code is:

page = requests.get('http://racing-reference.info/loopdata/2018-18/W')
page.encoding = 'utf-8'
soup = BeautifulSoup(page.text, 'html.parser')

table = soup.find_all(class_ = 'tb')
headers = [th.text for th in table[2].select("tr th")]

with open("out.csv", "w", newline='') as f:
    wr = csv.writer(f)
                   for td in row.find_all("td")] 
                   for row in table[2].select("tr + tr")])

The HTML this is reading from looks like:

<TR><TD COLSPAN=19 HEIGHT=20 ALIGN=CENTER class=newhead>Loop data for this race:</TD></TR>
<TR><TH class=col><A HREF=/loopdata?s=1&series=W&id=2018-18>Driver</A></TH>
<TH class=col><A HREF=/loopdata?s=2&series=W&id=2018-18>Start</A></TH>
<TH class=col><A HREF=/loopdata?s=3&series=W&id=2018-18>Mid Race</A></TH>
<TH class=col><A HREF=/loopdata?s=4&series=W&id=2018-18>Finish</A></TH>
<TH class=col><A HREF=/loopdata?s=5&series=W&id=2018-18>High Pos.</A></TH>
<TH class=col><A HREF=/loopdata?s=6&series=W&id=2018-18>Low Pos.</A></TH>
<TH class=col><A HREF=/loopdata?s=7&series=W&id=2018-18>Avg. Pos.</A></TH>
<TH class=col><A HREF=/loopdata?s=8&series=W&id=2018-18>Pass Diff.</A></TH>
<TH class=col><A HREF=/loopdata?s=9&series=W&id=2018-18>Green Flag Passes</A></TH>
<TH class=col><A HREF=/loopdata?s=10&series=W&id=2018-18>Green Flag Times Passed</A></TH>
<TH class=col><A HREF=/loopdata?s=11&series=W&id=2018-18>Quality Passes</A></TH>
<TH class=col><A HREF=/loopdata?s=12&series=W&id=2018-18>Pct. Quality Passes</A></TH>
<TH class=col><A HREF=/loopdata?s=13&series=W&id=2018-18>Fastest Lap</A></TH>
<TH class=col><A HREF=/loopdata?s=14&series=W&id=2018-18>Top 15 Laps</A></TH>
<TH class=col><A HREF=/loopdata?s=15&series=W&id=2018-18>Pct. Top 15 Laps</A></TH>
<TH class=col><A HREF=/loopdata?s=16&series=W&id=2018-18>Laps Led</A></TH>
<TH class=col><A HREF=/loopdata?s=17&series=W&id=2018-18>Pct. Laps Led</A></TH>
<TH class=col><A HREF=/loopdata?s=18&series=W&id=2018-18>Total Laps</A></TH>
<TH class=col><A HREF=/loopdata?s=19&series=W&id=2018-18>DRIVER RATING</A></TH>
<TR CLASS=odd><TD class=col NOWRAP><A HREF=/driverlog/joneser02/W/2018 title="View this driver's loop data for all races">Erik Jones</A></TD><TD class=col ALIGN=RIGHT>29</TD><TD class=col ALIGN=RIGHT>26</TD><TD class=col ALIGN=RIGHT>1</TD><TD class=col ALIGN=RIGHT>1</TD><TD class=col ALIGN=RIGHT>31</TD><TD class=col ALIGN=RIGHT>18</TD><TD class=col ALIGN=RIGHT>31</TD><TD class=col ALIGN=RIGHT>153</TD><TD class=col ALIGN=RIGHT>122</TD><TD class=col ALIGN=RIGHT>46</TD><TD class=col ALIGN=RIGHT>30.1</TD><TD class=col ALIGN=RIGHT>9</TD><TD class=col ALIGN=RIGHT>49</TD><TD class=col ALIGN=RIGHT>29.2</TD><TD class=col ALIGN=RIGHT>1</TD><TD class=col ALIGN=RIGHT>0.6</TD><TD class=col ALIGN=RIGHT>168</TD><TD class=col ALIGN=RIGHT>84.6</TD></TR>

The headers and data are printing out fine, but there is a blank row between the headers and data that I can't seem to get rid of. I've tried using any() function on rows, but this didn't work.


1 answer

  • answered 2018-07-11 03:34 Bailey Parker

    It looks to me like this is coming from:

    [[td.text for td in row.find_all("td")] for row in table[2].select("tr + tr") if row]

    If we look at your HTML, it has the general form:

      <!-- note this tr has no tds -->
      <tr><th></th><!-- ... --></tr>
      <tr><td></td><!-- ... --></tr>

    Note that the row that you obtain the headings from is a <tr> preceded immediately by a <tr> (this matches tr + tr). But since it doesn't have any <td>s inside it, this list comprehension is empty: [td.text for td in row.find_all("td")].

    Instead of this approach, I'd recommend the following:

    1. Gather up all <tr>s in the table: rows = table.select('tr')
    2. Remove rows from the top of that list until you find the header (one with multiple <th>s)
    3. Then, now that the header has been stripped, you can use your list comprehension on the remaining rows to extract the data: [[td.text for td in row] for row in rows]

    Alternatively, if you don't need to do any processing to the data (and just want to convert it directly to a CSV), you can just do one list comprehension and extract either <th>s or <td>s.

    [[x.text for x in row.find_all('td, th')] for row in table.select('tr')]