Selenium - ERR_TOO_MANY_REDIRECTS

2021-04-08 03:21 braindead imported from Stackoverflow

Hi I am trying to create a script that loops through a list of urls/links of products on a website. If the product is not available it will continue to the next product. So far I am able to loop through the list of product links but I am getting an error

This page isn’t working
www.bestbuy.ca redirected you too many times.
Try clearing your cookies.
ERR_TOO_MANY_REDIRECTS

I am using driver.get(url) to go to each url and I have included driver.delete_all_cookies() in my code aswell.

See also questions close to this topic

Sparse Matrix Creation : KeyError: 579 for text datasets
I am trying to use the make_sparse_matrix function to create a sparse matrix for my text dataset, and I face KeyError: 579. Does anyone has any leads on the root of the error.
```
def make_sparse_matrix(df, indexed_words, labels):
"""
Returns sparse matrix as dataframe.

df: A dataframe with words in the columns with a document id as an index (X_train or X_test)
indexed_words: index of words ordered by word id
labels: category as a series (y_train or y_test)
"""

nr_rows = df.shape[0]
nr_cols = df.shape[1]
word_set = set(indexed_words)
dict_list = []

for i in range(nr_rows):
    for j in range(nr_cols):
        
        word = df.iat[i, j]
        if word in word_set:
            doc_id = df.index[i]
            word_id = indexed_words.get_loc(word)
            category = labels.at[doc_id]
            
            item = {'LABEL': category, 'DOC_ID': doc_id,
                   'OCCURENCE': 1, 'WORD_ID': word_id}
            
            dict_list.append(item)

return pd.DataFrame(dict_list)



make_sparse_matrix( X_train, word_index,  y_test ) 
```
X_train is a DF that contains one single word in each cell, word_index contains all the index of words and y_test stores all labels.

The Key Error I am facing is:

--------------------------------------------------------------------------- KeyError Traceback (most recent call last) ~\New folder\envs\geo_env\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance) 3079 try: -> 3080 return self._engine.get_loc(casted_key) 3081 except KeyError as err:

pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 579

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last) in

in make_sparse_matrix(df, indexed_words, labels) 20 doc_id = df.index[i] 21 word_id = indexed_words.get_loc(word) ---> 22 category = labels.at[doc_id] 23 24 item = {'LABEL': category, 'DOC_ID': doc_id,

~\New folder\envs\geo_env\lib\site-packages\pandas\core\indexing.py in getitem(self, key) 2154 return self.obj.loc[key] 2155 -> 2156 return super().getitem(key) 2157 2158 def setitem(self, key, value):

~\New folder\envs\geo_env\lib\site-packages\pandas\core\indexing.py in getitem(self, key) 2101 2102 key = self._convert_key(key) -> 2103 return self.obj._get_value(*key, takeable=self._takeable) 2104 2105 def setitem(self, key, value):

~\New folder\envs\geo_env\lib\site-packages\pandas\core\series.py in _get_value(self, label, takeable) 959 960 # Similar to Index.get_value, but we do not fall back to positional --> 961 loc = self.index.get_loc(label) 962 return self.index._get_values_for_loc(self, loc, label) 963

~\New folder\envs\geo_env\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance) 3080 return self._engine.get_loc(casted_key) 3081 except KeyError as err: -> 3082 raise KeyError(key) from err 3083 3084 if tolerance is not None:

KeyError: 579
Finding part of string in list of strings
```
GCM = ([519,520,521,522,533],[534,525],[526,527,530,531], [4404])

slice = int(str(df["CGM"][row_count])[:3])
```
I am looking through a row in a csv file and taking out the number I want. i want the number that starts with the number I have in GCM. since they represent info I want in other columns. this has working fine with the slice function because all the number i wanted started with 3 digits. now that i need to look for any number that starts with 4404 and later on going to probably need to look for 57052 the slice function no longer work.

is there a way I can, instead of slicing and comparing to list, can take 5 digit number and see if part of it is in list. preferably look for it starting 3 or more same digits. the real point of that part of code is finding out which list in GCM list the number is. it need to be able to have the number 44042 and know that the part of it a care about is in GCM[3], but on the other side do not want it to say that 32519 is in DCM[0] since I only care about number that start with 519 not ends with it.

ps. I am norwegian and have been learning programming by myself. been some long nights. so something here can be lost in translation.
How to forecast a time series out-of-sample using an ARIMA model in Python?
I have seen similar questions at Stackoverflow. But, either the questions were different enough or if similar, they actually have not been answered. I gather it is something that modelers run into often, and have a challenge solving.

In my case I am using two variables, one Y and one X with 50 time series sequential observations. They are both random numbers representing % changes (they could be anything you want, their true value does not matter. This is just to set up an example of my coding problem). Here are my basic codes to build this ARIMAX(1,0,0) model.
```
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

df = pd.read_excel('/Users/gaetanlion/Google Drive/Python/Arima/df.xlsx', sheet_name = 'final')

from statsmodels.tsa.arima_model import ARIMA
endo = df['y']
exo = df['x']
```
Next, I build the ARIMA model, using the first 41 observations
```
modelho = sm.tsa.arima.ARIMA(endo.loc[0:40], exo.loc[0:40], order =(1,0,0)).fit()
print(modelho.summary())
```
So far everything works just fine.

Next, I attempt to forecast or predict the next 9 observations out-of-sample. Here I want to use the X values over these 9 observations to predict Y. And, I just can't do it. I am showing below just the one code, that I think gets me the closest to where I need to go.
```
modelho.predict(exo.loc[41:49], start = 41, end = 49, dynamic = False)
TypeError: predict() got multiple values for argument 'start'
```

How to set "value" to input using Selenium when "element not interactable"

my code look like this:

Integer unitOrder = Integer.valueOf(RandomStringUtils.randomNumeric(2));
purchaseOrderData.setOrderUnit(unitOrder);
typeOn(ORDER_FIELD_INGREDIENT, String.valueOf(unitOrder));

I have tried this code:

getDriver().findElement(ORDER_FIELD_INGREDIENT).sendKeys("value", "16");

but still got this error: "element not interactable"

Gradle not finding tests to run

I have a parent project "EqcareAutomationFramework" and a child project "EqcareDcbtTests". When I run tests as "Run as Gradle test", I am getting the following error but runs fine when i run as TestNg.

***Tests: com.eqcare.integration.tests.CloseSessionTest***

    > Task :EqcareAutomationFramework:compileJava UP-TO-DATE
    > Task :EqcareAutomationFramework:processResources NO-SOURCE
    > Task :EqcareAutomationFramework:classes UP-TO-DATE
    > Task :EqcareAutomationFramework:jar UP-TO-DATE
    > Task :compileJava NO-SOURCE
    > Task :processResources UP-TO-DATE
    > Task :classes UP-TO-DATE
    > Task :compileTestJava NO-SOURCE
    > Task :processTestResources UP-TO-DATE
    > Task :testClasses UP-TO-DATE
    > Task :runSuite NO-SOURCE
    > Task :test NO-SOURCE
    > Task :EqcareAutomationFramework:compileTestJava UP-TO-DATE
    > Task :EqcareAutomationFramework:processTestResources NO-SOURCE
    > Task :EqcareAutomationFramework:testClasses UP-TO-DATE
    > Task :EqcareAutomationFramework:test

FAILURE: Build failed with an exception.

* What went wrong:
No matching tests found in any candidate test task.
    Requested tests:
        Test class com.eqcare.integration.tests.CloseSessionTest

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 803ms
6 actionable tasks: 1 executed, 5 up-to-date

Following is the build.gradle file for parent Project "EqcareAutomationFramework"

> /*  * This file was generated by the Gradle 'init' task.  */  apply
> plugin: 'java'  apply plugin: 'eclipse'
> 
>  sourceSets {
>     main {
>         java {
>            srcDir './src'
>         }
>     } 
>     test {
>         java {
>            srcDirs = ["./src"]
>         }
>     }
>      }
> 
> repositories {        
>     mavenCentral()   }
> 
> dependencies {
>       implementation 'org.testng:testng:7.1.0'
>     implementation 'org.seleniumhq.selenium:selenium-server:3.141.59'
>     implementation 'log4j:log4j:1.2.17'
>     implementation 'org.apache.poi:poi:4.1.2'
>     implementation 'org.apache.httpcomponents:httpclient:4.5.13'
>     implementation 'commons-io:commons-io:2.8.0'
>     implementation 'activation:activation:1.0.2'
>     implementation 'javax.mail:mail:1.4.7'
>     implementation 'com.aventstack:extentreports:3.1.5'
>     implementation 'io.appium:java-client:7.4.1'
>     implementation 'com.android.tools.build:gradle:2.3.0' 
>     implementation 'software.amazon.awssdk:aws-sdk-java:2.15.32' 
>     implementation 'jfree:jfreechart:1.0.13' }
> 
> test{     
       useTestNG() 
      }

Following is the build.gradle file for child Project "EqcareDcbtTests"

apply plugin: 'java'


sourceSets {    
           main {           
               resources.srcDirs = ['clinic', 'patient',
 'integration', 'commonUtils']      
             }  
            test{           
              resources.srcDirs =['clinic', 'patient', 
                  'integration','commonUtils']      
                } 
    }
dependencies {  
           implementation project(':EqcareAutomationFramework') 
   }
project repositories {
    mavenCentral() 
   }
 task runSuite (type: Test) {   
     test.testLogging.showStandardStreams = true    
     useTestNG() {
         suites 'testsuite/'+ System.getProperty('suiteName')
    } }

Folder Structure for Parent Project: EqcareAutomationFramework

Folder Structure for Child Project: EqcareDcbtProject

How to get text with Selenium WebDriver in a For Loop Python

I'm trying to run a for loop in a website table with Selenium. I'm using Xpath to get the data.

When I run this solo, I get the proper output:

driver.find_element_by_xpath('//*[@id="top-team-stats-summary-content"]/tr[1]/td[2]').text

But when I put in a for loop to extract all values, it runs an error

vals = (list(np.arange(0,20,1)))
for i in vals :
    print(driver.find_element_by_xpath('//*[@id="top-team-stats-summary-content"]/tr[i]/td[2]').text)

Error:

NoSuchElementException: Message: no such element: Unable to locate element:
 {"method":"xpath","selector":"//*[@id="top-team-stats-summary-content"]/tr[i]/td[2]"}

This is the HTML

<td class="goal   ">59</td>

Any thoughts?

EDIT: Before coming to a post, I looked at this post, that helped me a lot.

How to fill in a web form with python

I am trying to fill this form via a python script: https://reassurez-moi.fr/devis-assurance-pret-immobilier/60773413ea18bd60703427df9c39cbd2c5c1c8d46fda062f92513e1c266bbfb5

But unfortunately I can't find the code that allows me to select the values from my drop-down lists.

You can notice that in the html we do not have the different options from the drop-down list.

Can you tell me how to make a python script that allows me to fill out this form?

Thank you in advance.

The code below show what I have done so far but it does not allow me to choose one of the values from my drop-down list

from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.common.by import By
web1 = webdriver.Chrome(ChromeDriverManager().install())
web1.get('https://reassurez-moi.fr/simulation-tarifaire-assurance-habitation?homeType=1')
#time.sleep(2)
topics_xpath = '//*[@id="home-comparator-base"]/div[2]/div/div[2]/div/form/div[1]/div[1]/div/div[1]/div[2]/div/div[2]/button'
WebDriverWait(web1,2).until(expected_conditions.visibility_of_element_located((By.XPATH, topics_xpath)))
Rue_Adresse = "Rue de la Paix"
last = web1.find_element_by_xpath('//*[@id="home-comparator-base"]/div[2]/div/div[2]/div/form/div[3]/div[1]/div[2]/div/input')
last.send_keys(Rue_Adresse)
ast = web1.find_element_by_xpath('//*[@id="home-comparator-base"]/div[2]/div/div[2]/div/form/div[1]/div[1]/div/div[1]/div[2]/div/div[2]/button')
ast.click()

Using Selenium, loop click on a link till new page appears

I recently started using the Selenium module and have faced a hurdle that I was hoping would get cleared.

I made automation that clicks a link in a loop. The link has a 70% chance of doing nothing and a 30% chance of opening a new page. I wanted to make it so that the automation stops the looping process once the new page appears and clicks another link on that new page.

What I have so far:

# Searching for the link.
    link = EC.presence_of_element_located((By.CSS_SELECTOR, 'css_selector'))
    WebDriverWait(driver, 30).until(link)
    # Looping the click.
    for i in range(500):
        while True:
            try:
                linkclick = driver.find_element_by_css_selector('css_selector')
                linkclick.click()
            except
                newlink = EC.presence_of_element_located(
                    (By.CSS_SELECTOR, 'new_css_selector'))
                WebDriverWait(driver, 30).until(newlink)
                newlinkclick = driver.find_element_by_css_selector('new_css_selector')
                newlinkclick.click()

Problem with screen of image when webdriver is in headless option
I have problem with screenshot of image, when the webdriver is on headless mode.
Without option.headless = True
```
time.sleep(2)
img = driver.find_element_by_xpath('//*[@id="img"]')
time.sleep(3)
img.screenshot('C:/foo.png')
time.sleep(3)
```
When i used it:
```
option = webdriver.ChromeOptions()
option.headless = True

time.sleep(2)
img = driver.find_element_by_xpath('//*[@id="img"]')
time.sleep(3)
img.screenshot('C:/foo.png')
time.sleep(3)
```
I also try that options:
```
option = webdriver.ChromeOptions()
option.headless = True
option.add_argument("--window-size=1920,1080")
option.add_argument("--start-maximized")
```
For all pictures sizes are okay (172,62). The second picture looks pretty good, but is covered by a strip at the bottom that reminds me of moving the page to the side(?) (that is not end of site, on the bottom also have some buttons and ads). I tried to set the browser to normal size without the GUI, but then it crops the image (img 3). Some solution?

Related questions

Selenium - ERR_TOO_MANY_REDIRECTS

See also questions close to this topic