Related questions

Selenium Chromedriver not working from Google Cloud Server

2020-11-25 06:51 Rehan Mahmood imported from Stackoverflow

I am trying to program a web scraping python script from a basic Google cloud server. I have succesfully downloaded selenium and chromedriver onto the VM. When I run my code i return this error

My python code is:

from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'/Users/owner/Desktop/chromedriver')
URL = 'https://www.forexfactory.com/news'

driver.get(URL)

driver.implicitly_wait(5) # wait for seconds

uiOuter = driver.find_element_by_id('ui-outer')

aHref = driver.find_elements_by_css_selector('div.flexposts__story-title a')

span = driver.find_elements_by_css_selector('div.flexposts__storydisplay-info')

headline = open("headlines.txt","w+")

for x in aHref:
    for y in span:
        headline.write(x.text + "\n" + y.text + "\n" + "\n")

The Error message is:

selenium.common.exceptions.WebDriverException: Message: unknown error: DevToolsActivePort file doesn't exist

Im not sure exactly what this means and of course not sure what to do to fix it. All help is greatly appreciated. Thank you.

See also questions close to this topic

  • Python2.7 works but Python3.x throws error 
    def calculate(self):
        normals = []
        for f in self.sides:
            sideN= A(0.0, 0.0, 0.0)
            for i in range(0, len(f)):
                p1 = self.ver[f[(i - 1) % len(f)]]
                p2 = self.ver[f[i]]
                p3 = self.ver[f[(i + 1) % len(f)]]
                v1 = p2 - p1
                v2 = p2 - p3
                v33 = v2.cross(v1)
                v33 = v33.normalize()
                sideN += v33
            sideN = sideN.normalize()
            normals.append(sideN)
        return normals

    this runs OK on python 2.7 but when I run in on python 3.8 it throws

     p1 = self.ver[f[(i - 1) % len(f)]]
TypeError: list indices must be integers or slices, not float

    when I edit that part;

    p1 = self.ver[f[(i - 1) % len(f)]]
p2 = self.ver[f[i]]
p3 = self.ver[f[(i + 1) % len(f)]]

    to that

    p1 = self.ver[int(f[(i - 1) % len(f)])]
p2 = self.ver[int(f[i])]
p3 = self.ver[int(f[(i + 1) % len(f)])]

    then it throws;

    p3 = self.ver[int(f[(i + 1) % len(f)])]
IndexError: list index out of range

    I tried 2to3 but nothing is changed. What is the problem here?

  • How can I find full path for a file?

    Let's say I have this path D:\something\something1\from_here_I_now\stuff\stuff2.

    So, I know that \from_here_I_now\stuff\stuff2 is a permanent path, but the beginning is different, like I know that D:\something\something1\ may be different for someone else. How can I find the D:\something\something1\ knowing only \from_here_I_now\stuff\stuff2?

  • Why is this PyGTK menu empty?

    I am working on a PyGTK application with a Gtk.Toolbar. One of the items in taht toolbar is a Gtk.MenuToolButton. However, clicking on the arrow once the program runs results in an empty menu (a tiny white sliver). Here is the code that sets up the menu:

                    self.compileMenu = Gtk.Menu()
                self.defaultCompileOption = Gtk.MenuItem(label = "Compile with defaults")
                self.configCompileOption = Gtk.MenuItem(label = "Change compile options...")
                self.compileWithFiles = Gtk.MenuItem(label = "Bundle files...")
                
                self.compileMenu.append(self.defaultCompileOption)
                self.compileMenu.append(self.configCompileOption)
                self.compileMenu.append(self.compileWithFiles)
                
                self.compileButton = Gtk.MenuToolButton(icon_name = "x-package-repository", label = "Compile...", menu = self.compileMenu)

    There is also a call to self.toolbar.add at the bottom, to add the MenuToolButton to the toolbar.

  • Building softlink inside a file in Linux : Not a directory

    I am running the script tedlium/s5/run.sh of Kaldi repository https://github.com/kaldi-asr/kaldi/tree/master/egs, and encountered the following error :

    run.sh: line 30: utils/parse_options.sh: Not a directory

    Therefore it must be that for some reason in running run.sh, the utils dir goes wrong, which led me to debug what actually when wrong. After some study of the code structure, in tedlium/s5/utils, I found a path ../../wsj/s5/utils is directing to a real directory utils. However, as I expect there should be a command building such softlink using ln -s, I did not find such in the compilation procedure.

    Also one bash command before the error s5/path.sh seems to be related, but it only adds utils file to the path.

    Of course I could build it manually using ln, but which should not occur as I expect (since not mentioned at all in the repo readme). I then wonder how should this softlink be built ? Is there other way to build softlink besides ln ?

  • dlsym crash when called from assembler

    I have a small program in assembler that loads an .so file using dlopen, and then tries to load a function pointer using dlsym. Calling dlopen seems to be fine but it crashes when I call dlsym.

    SECTION .text

;default rel

EXTERN dlopen ; loads a dynamic library
EXTERN dlsym ; retrieves the address for a symbol in the dynamic library

; inputs:
;   rdi: rdi the pointer to print
printHex:
    sub rsp, 19 ; allocate space for the string 0x0123456789ABCDEF\n
    mov BYTE [rsp + 0], '0'
    mov BYTE [rsp + 1], 'x'
    xor rcx, rcx ; int loop variable to 0
    .LOOP1:
        lea rsi, [rsp + rcx] ; rsi will we the offset where we will store the next hex charcter
        mov rax, rdi
        and rax, 0xf
        sar rdi, 4 ; shift right 4 bits (divide by 16)
        lea rdx, [hexLookUp + rax]
        mov bl, [rdx]
        mov BYTE [rsi +18], bl
        dec rcx ; rcx--
        cmp rcx, -16 ; while rcx > -16
        jne .LOOP1
    mov BYTE [rsp + 18], 10

    ; print
    mov rax, 1 ; syscall: write
    mov rdi, 1 ; stdout
    mov rsi, rsp
    mov rdx, 19
    syscall
    
    ; release stack memory
    add rsp, 19
    ret

global _start ; "global" means that the symbol can be accessed in other modules. In order to refer to a global symbol from another module, you must use the "extern" keyboard
_start:

    ; load the library
    mov rdi, str_libX11so
    mov rsi, 2; RTLD_NOW=2
    call dlopen wrt ..plt
        ; PLT stands for Procedure Linkage Table:
        ; used to call external library functions whose address is not know at link time,
        ; so it must be resolved by the dynamic linker at run time
        ; more info: https://reverseengineering.stackexchange.com/questions/1992/what-is-plt-got
    mov [ptr_libX11so], rax ; the previous function call returned the value in rax
    mov rdi, rax 
    call printHex

    ; load the function
    mov rdi, [str_libX11so]
    mov rsi, fstr_XOpenDisplay
    call dlsym wrt ..plt
    mov [fptr_XOpenDisplay], rax
    mov rdi, rax
    call printHex


    mov rax, 60 ; syscal: exit
    mov rdi, 0 ; return code
    syscall

hexLookUp: db "0123456789ABCDEF"
str_libX11so: db "libX11.so", 0

; X11 function names
fstr_XOpenDisplay: db "XOpenDisplay", 0


SECTION .data
ptr_libX11so: dq 0 ; ptr to the X11 library

; X11 function ptrs
fptr_XOpenDisplay: dq 0

    I have tried to make the same program in C and it seems to work. So I must be doing something wrong.

    extern void* dlopen(const char* name, int);
extern void* dlsym(void* restrict handle, const char* restrict name);

int main()
{
    void* libX11so = dlopen("libX11.so", 2);
    void (*XOpenDisplay)() = dlsym(libX11so, "XOpenDisplay");
}

    I tried to disassemble the C version and compare, but I can't still figure out what is the problem.

    An interesting thing I noticed is that the pointer returned by dlopen (which is different in each execution), in the asm version is quite small compared to the C version (e.g 0x0000000001A932D vs 0x5555555592d0). But maybe that could be because I'm using the -no-pie flag for linking:

    nasm -f elf64 -g -F dwarf minimal.asm && gcc -nostartfiles -no-pie minimal.o -ldl -o minimal && ./minimal
  • Different WebDriverException Erros when using Flask Apache2

    i am trying to use a flask script which is working in debug mode fine.

    When I set app.wsgi to the file i get a lot of erros in context with selenium.

    When I set the user-data-dir i get the error: failed to write prefs file

    I tried to copy the folder to a temporary location because i found out that selenium cant be used parallel.

    By the way i dont use selenium parallel so i dont know where the error message is from.

    But that doesnt work.

    Do you have an idea why selenium script is working in flask debug mode but not wsgi apache2 server???

  • SwiftUI macOS view starts lagging when displaying multiple charts

    I am displaying multiple charts in a view.
    The charts are just paths in a frame and data comes from an csv file (not bigger than 400mb).
    Those charts are displayed inside of a LazyVGrid and the whole view is inside a scroll view.

    Loading the view takes about 10 - 20 seconds, but when I start scrolling through the charts, it becomes very laggy.
    To solve this I tried to attach .drawingGroup() to the chart views, but there where nearly no performance change.

    the code looks something like this: 

    import SwiftUI


struct GroupView: View {

@State var chartData = [CGFloat]()

  let columns = [
        GridItem(.adaptive(minimum: 720))
 
       ]
    

  var body: some View {

VStack{

 ScrollView {
                
   LazyVGrid(columns: columns){

     CSVChart(data: chartData, chartName: String, x-yaxisName: [String](), lineColor: CGColor)
        .frame(width: 700, height: 300, alignment: .center)
        .drawingGroup() // there is as well no performance change when .drawingGroup()  is inside the chart view

        }

     
    }

}.onAppear{
   
 chartData = // get csv data logic

   }



}
}

    I know that the csv file is quite large, and that this will cause the slow behaviour, but I thought that other apps aren't that laggy when working with large files so this might be fixable.
    It would be great if someone could help me with this problem.

  • Simulator not showing in Spotlight search - Big Sur

    Hopefully a straight forward one here, but my Simulator app (Xcode IOS Simulator) is not being found through Spotlight search on Mac. I recently upgraded to Big Sur and I am currently using 11.1.

    Just wondering if anyone is having the same problem. I am able to access Simulator through other methods but it just doesn't seem to be indexed in the Spotlight search. Cheers

  • Encoding problem on Mac with MySQL ODBC driver 8.0

    I have an Excel file that is on a Google Drive. I use a Windows PC and my colleague uses a macOS 11 PC. We use Excel 365.

    The file is connected to a Google MySQL database through ODBC MySQL Unicode driver 8.0. It's a MySQL table in French encoded in UTF-8 (with accent: à é ê è).

    When they update the data the data is all messed up, but when I do it on Windows the data is fine.

    For example, they get "FrÈdÈrique Brassard" but I get "Frédérique Brassard" which is correct. I would like to know if you know a way to fix this?

  • Handling with random button with Robot Framework

    I'm trying to use Robot Framework with SeleniumLibrary to click on a random button generator, the HTML was:

    <div class="btn-number-password noselect">
<p class="btn btn-secondary btn-userpassword" data-keyboard="0">6 or 7</p>
<p class="btn btn-secondary btn-userpassword" data-keyboard="1">2 or 0</p>
<p class="btn btn-secondary btn-userpassword" data-keyboard="2">3 or 8</p>
<p class="btn btn-secondary btn-userpassword" data-keyboard="3">9 or 4</p>
<p class="btn btn-secondary btn-userpassword" data-keyboard="4">5 or 1</p></div>

    It will generete buttons to a field like:

    [6 or 7] [2 or 0] [3 or 8] [9 or 4] [5 or 1]

    These numbers shuffle everytime, so I'm trying to click using text but nothing what I tried works.

    Is there any way to click with Robot using text in these buttons?

  • NoSuchWindowException when using element_to_be_clickable()

    I'm trying to use EC.element_to_be_clickable to select an element and to click on it, but I'm getting the following error message:

    NoSuchWindowException: Browsing context has been discarded

    This element is inside an iframe ("contentAreaFrame") that is inside another iframe ("isolatedWorkArea").

    The frame I'm in before I try to switch to "contentAreaFrame" and "isolatedWorkArea" to run the command below, also happens to contain iframes named "contentAreaFrame" and "isolatedWorkArea".

    WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//*[contains(text(),'7003388855')]")))

    However, these "frame1" iframes differ in that they contain a different number of iframe elements. I tried using the code below to make sure that I'd switch to the correct "contentAreaFrame" before trying to find the element I desire:

    test_v = 2

while test_v == 2:
    
    driver.switch_to_default_content()
    
    WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"contentAreaFrame")))
    
    test_v = len(get_ids("iframe"))

print(get_ids("iframe"))

time.sleep(5)
    
driver.switch_to_default_content()
    
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"contentAreaFrame")))
        
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"isolatedWorkArea")))

    Despite these efforts to work around this issue, I'm still getting the NoSuchWindowExcepetion error message.

    I have used time.sleep() to make sure the page is fully loaded before running the element_is_clickable() command and it worked just fine, but I would like to be able to click it as soon as possible, considering there is a wild variation in the time the page takes to be loaded for some reason.

    Any idea on what could be causing this problem? Thanks in advance.

    (Dont worry - I'm fully aware I'm pretty bad at coding)

  • Xpath doesn't work with string, which have quotes. Chrome

    I have the text: var1 = 'ТСЖ "ВОЛГОГРАДСКИЙ ПР-Т 50-1, 52-1, 52-2, 56-1, 56-2, 58-1, 58-3, 60-2, 64-1, 64-2, 66-2, ВОЛЖСКИЙ 21"'

    And I have xpath: driver.find_element_by_xpath("//a[text()='"+var1+"']/ancestor::div[contains(@class,'chakra-stack') and contains(@class,'css-1oap1wr')]/following-sibling::div/descendant::button[text()='Отклонить']").click() - It does'nt work

    //a[text()='ТСЖ "ВОЛГОГРАДСКИЙ ПР-Т 50-1, 52-1, 52-2, 56-1, 56-2, 58-1, 58-3, 60-2, 64-1, 64-2, 66-2, ВОЛЖСКИЙ 21"']/ancestor::div[contains(@class,'chakra-stack') and contains(@class,'css-1oap1wr')]/following-sibling::div/descendant::button[text()='Отклонить'] - But this work!

    Python writes me :selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[text()='ТСЖ "ВОЛГОГРАДСКИЙ ПР-Т 50-1, 52-1, 52-2, 56-1, 56-2, 58-1, 58-3, 60-2, 64-1, 64-2, 66-2, ВОЛЖСКИЙ 21"']/ancestor::div[contains(@class,'chakra-stack') and contains(@class,'css-1oap1wr')]/following-sibling::div/descendant::button[text()='Отклонить']"}

    But if I Cut xpath from error messeage and put it like xpath in console of Chrome - element would finded.

    Element absoulutle visible and already loaded. Sory for my English.

  • handshake failed; returned -1, SSL error code 1, net_error -101 in selenium webdriver

    I have facing this error even though code runs and gives output, this error part is given especially when it is on .get() or is it?.I have tried solutions of other SSL error code but doesn't seem to work sometimes, gives 44 links as required but sometimes it gives only 10 or 20, it seems inconsistent and what is SSL error code 1, net_error -101 mean?

    Also, tried increasing delay in time.sleep() but doesn't seem to help?
    In headless mode, it is giving: has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource, did get any solution for this, also? for this tried CORS extension and it also didn't work.
    Console output: 
    [2664:25084:0119/211237.647:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211238.068:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211238.842:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -113
[2664:25084:0119/211238.844:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -113
[2664:25084:0119/211253.591:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211253.879:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211253.959:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211253.963:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -100
[2664:25084:0119/211254.251:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211255.211:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -100
[2664:25084:0119/211255.788:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -100
[2664:25084:0119/211319.752:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211319.973:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211320.001:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211320.228:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211320.285:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211320.507:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
I am here
[2664:25084:0119/211350.324:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211350.536:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211350.589:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211350.795:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
I am here
Going Down
Going Down
[2664:25084:0119/211420.337:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211420.562:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211420.594:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211420.823:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
Going Down
Going Down
Going Down
[2664:25084:0119/211450.376:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211450.607:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211450.647:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
[2664:25084:0119/211450.861:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -101
44

    CODE:

     from selenium import webdriver
    from webdriver_manager.chrome import ChromeDriverManager
    from bs4 import BeautifulSoup
    import pandas as pd
    import time
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    import re
    from dateutil.parser import parse
    import scrapy
    #To avoid bot detection
    options = webdriver.ChromeOptions()
    #options.add_argument('--headless')
    options.add_argument("--disable-blink-features=AutomationControlled")
    options.add_argument("--ignore-certificate-errors")
    options.add_argument('--ignore-ssl-errors')
    #options.add_argument("start-maximized")
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)
    browser = webdriver.Chrome(ChromeDriverManager().install(),options=options)
    #browser.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
    #browser.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
    #print(browser.execute_script("return navigator.userAgent;"))
    ##################################################
    #options=options)
    wait = WebDriverWait(browser, 20)
    browser.set_window_size(1280, 800)
    
    
    #time.sleep(20)
    base_url='https://www.legacy.com/obituaries/bostonglobe/obituary-search.aspx?daterange=88888&startdate=20190101&enddate=20210119&townname=Abington&countryid=1&stateid=22&affiliateid=1232'
    print('Gttibg url')
    browser.get(base_url)
    time.sleep(50)
    print("I am here")
    last_height = browser.execute_script("return document.body.scrollHeight")
    time.sleep(5)
    print("I am here")
    while True:
                
            # Scroll down to the bottom.
                browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    
            # Wait to load the page.
                time.sleep(10)
    
            # Calculate new scroll height and compare with last scroll height.
                new_height = browser.execute_script("return document.body.scrollHeight")
                time.sleep(5)#***************************
                print('Going Down')
                if new_height == last_height:
    
                    break
    
                
                last_height = new_height
    time.sleep(20)
    links=browser.find_elements_by_xpath('/html/body/div[3]/div[2]/div[2]/form/div[3]/div[1]/div[1]/div[3]/div/div[2]/div/div/div[2]/div[1]/div[1]/a')
    print(len(links))
    for i in links:
        
        print(i.get_attribute('href'))
        print(i.text)


    How to avoid this error?