Changing User-Agent String for Every Get

I'm using the following code to change the user-agent string, but I'm wondering whether or not this will change the user-agent string for each and every browser.get request?

ua_strings = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36',
    ...
]

def parse(self, response):
    profile = webdriver.FirefoxProfile()
    profile.set_preference('general.useragent.override', random.choice(ua_string))
    options = Options()
    options.add_argument('-headless')
    browser = webdriver.Firefox(profile, firefox_options=options)
    browser.get(self.start_urls[0])

    hrefs = WebDriverWait(browser, 60).until(
        EC.visibility_of_all_elements_located((By.XPATH, '//div[@class="discoverableCard"]/a'))
    )

    pages = []

    for href in hrefs:
        pages.append(href.get_attribute('href'))

    for page in pages:
        browser.get(page)

        """ scrape page """

    browser.close()

Or will I have to browser.close() and then create new instances of browser in order to use new user-agent strings for each request?

    for page in pages:
        browser = webdriver.Firefox(profile, firefox_options=options)
        browser.get(page)

        """ scrape page """

        browser.close()

1 answer

  • answered 2018-07-11 03:41 Erisan Olasheni

    Since random.choice() has been called initially, the user-agent string remains the same of all browser.get() requests. To ensure a constantly random user-agent, you can create a set_preference() function, which you call on every loop.

    def set_prefrences(self):
        user_agent_string = random.choice(ua_string)
    
        #print out user-agent on each loop
        print(user_agent_string)
        profile = webdriver.FirefoxProfile()
        profile.set_preference('general.useragent.override', user_agent_string)
        options = Options()
        options.add_argument('-headless')
        browser = webdriver.Firefox(profile, firefox_options=options)
        return browser
    

    Then in your loop can be something like this:

    for page in pages:
        browser = set_preferences()
        browser.get(page)
    
        """ scrape page """
    
        browser.close()
    

    Hope this helps!