How to web scrape sports team betting lines?

Is it possible to web scrape this site using Selenium and BeautifulSoup?

I am brand new to web scraping and have no previous experience coding. Here is my attempt at scraping the nba teams:

#import modules

from bs4 import BeautifulSoup
import requests, os
import pandas as pd
from selenium import webdriver


#enter your user-agent
header = {'User-agent' : 'your user-agent'}
options = webdriver.ChromeOptions(); options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=options)

#scrape data

soup = BeautifulSoup(driver.page_source, 'html.parser')

#parse data (Teams)

team = [i.text for i in soup.find_all('td', {'class': 'col_teamname bdevtt'})]


The code above successfully scraped the teams like I wanted.

I would now like to scrape the betting info: a) spread, b) money line, c) total points and d) start time.

a) spread:

<td class="hdcp bdevtt ">-9½</td>

info wanted: -9.5

b) money line:

<td class="odds bdevtt moneylineodds displayOdds" cfg="{'a':-500,'d':1.2}">-500</td>

info wanted: -500

c) total points:

<td class="hdcp bdevtt ">212½</td>

info wanted: 212.5

d) start time:

<td rowspan="3" class="col_time bdevtt">09:05 PM</td>

info wanted: 09:05 PM

I noticed that the data of these 4 items is located within td classes. I tried a similar code like the one above but was unsuccessful. Not sure if it's because the code is in JavaScript. If so, is there anyway to pull the code using BS4 and Selenium?

Any advice would be greatly appreciated. Thanks for your time!

1 answer

  • answered 2018-11-08 06:36 Andersson

    Try below code to get required values:

    for tr in soup.find_all('tr', {'class': 'firstline'}):
        print(tr.find('td', {'class': 'col_teamname'}).text)
        print(tr.find('td', {'class': 'hdcp'}).text)
        print(tr.find('td', {'class': 'moneylineodds'}).text)
        print(tr.find('td', {'class': 'col_time'}).text)