R Studio keeps crashing when I'm trying to merge multiple csv files into a data frame. How do I fix this?

2022-05-06 22:15 LordPuggo imported from Stackoverflow

I have 12 csv files that I need to merge for analysis project and their size ranges from 20mb to 120mb per file.

I attempted cutting down to only using the necessary columns by using fread() so it reads 6 columns instead of the total 11.

I've assigned each of them into a data frame as shown below.

However, at some point doing these manually, especially for using View() of the data frame that contains the 12 csv data, I keep getting crashed from R Studio probably due to the memory usage and the whole environment just resets and I have to do everything over again.

Is there a shorting and less ugly way to do this without crashing?

Packages <- c("dplyr", "janitor", "skimr", "readr", "lubridate","tidyverse","tidyr")
lapply(Packages, library, character.only = TRUE)
library("data.table")

td2105 <- fread("/cloud/project/Capstone Cyclistic Project/202105-divvy-tripdata.csv",
                  select = c("rideable_type", "started_at", "ended_at",
                             "start_station_name","end_station_name","member_casual"))

td2106 <- fread("/cloud/project/Capstone Cyclistic Project/202106-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2107 <- fread("/cloud/project/Capstone Cyclistic Project/202107-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2108 <- fread("/cloud/project/Capstone Cyclistic Project/202108-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2109 <- fread("/cloud/project/Capstone Cyclistic Project/202109-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2110 <- fread("/cloud/project/Capstone Cyclistic Project/202110-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2111 <- fread("/cloud/project/Capstone Cyclistic Project/202111-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2112 <- fread("/cloud/project/Capstone Cyclistic Project/202112-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2201 <- fread("/cloud/project/Capstone Cyclistic Project/202201-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2202 <- fread("/cloud/project/Capstone Cyclistic Project/202202-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2203 <- fread("/cloud/project/Capstone Cyclistic Project/202203-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td2204 <- fread("/cloud/project/Capstone Cyclistic Project/202204-divvy-tripdata.csv",
                select = c("rideable_type", "started_at", "ended_at",
                           "start_station_name","end_station_name","member_casual"))

td_2105_to_2204 <- rbind(td2105,td2106,td2107,td2108,td2109,td2110,td2111,td2112,td2201,td2202,td2203,td2204)

View(td_2105_to_2204)

See also questions close to this topic

pivot_wider does not keep all the variables

I would like to keep the variable cat (category) in the output of my function. However, I am not able to keep it. The idea is to apply a similar function to m <- 1 - (1 - se * p2)^df$n based on the category. But in order to perform that step, I need to keep the variable category.

Here's the code:

#script3

suppressPackageStartupMessages({
  library(mc2d)
  library(tidyverse)
})

sim_one <- function() {
df<-data.frame(id=c(1:30),cat=c(rep("a",12),rep("b",18)),month=c(1:6,1,6,4,1,5,2,3,2,5,4,6,3:6,4:6,1:5,5),n=rpois(30,5))
  nr <- nrow(df)
  df$n[df$n == "0"] <- 3
  se <- rbeta(nr, 96, 6)
  epi.a <- rpert(nr, min = 1.5, mode = 2, max = 3)
  p <- 0.2
  p2 <- epi.a*p
  m <- 1 - (1 - se * p2)^df$n
  results <- data.frame(month = df$month, m, df$cat)
  results %>%
    arrange(month) %>%
    group_by(month) %>%
    mutate(n = row_number(), .groups = "drop") %>%
    pivot_wider(
    id_cols = n,
    names_from = month,
    names_glue = "m_{.name}",
    values_from =m
    ) 
}

set.seed(99)

iters <- 1000
sim_list <- replicate(iters, sim_one(), simplify = FALSE)
sim_list[[1]]
#> # A tibble: 7 x 7
#>       n    m_1    m_2    m_3    m_4   m_5    m_6
#>   <int>  <dbl>  <dbl>  <dbl>  <dbl> <dbl>  <dbl>
#> 1     1  0.970  0.623  0.905  0.998 0.929  0.980
#> 2     2  0.912  0.892  0.736  0.830 0.890  0.862
#> 3     3  0.795  0.932  0.553  0.958 0.931  0.798
#> 4     4  0.950  0.892  0.732  0.649 0.777  0.743
#> 5     5 NA     NA     NA      0.657 0.980  0.945
#> 6     6 NA     NA     NA      0.976 0.836 NA    
#> 7     7 NA     NA     NA     NA     0.740 NA

^{Created on 2022-05-07 by the reprex package (v2.0.1)}

calculate weighted average over several columns with NA

I have a data frame like this one:

ID  duration1   duration2   total_duration  quantity1   quantity2
 1     5            2             7             3         1
 2     NA           4             4             3         4
 3     5            NA            5             2         NA

I would like to do a weighted mean for each subject like this:

df$weighted_mean<-  ((df$duration1*df$quantity1) + (df$duration2*df$quantity2) ) / (df$total_duration)

But as I have NA, this command does not work and it is not very nice....

The result would be this:

ID  duration1   duration2   total_duration  quantity1   quantity2   weighted_mean
 1     5            2             7             3         1          2.43
 2     NA           4             4             3         4          4
 3     5            NA            5             2         NA         2

Thanks in advance for the help

I am to extract data from netCDF file using R for specific loaction the code i've written as showen and I have an error at the end of the code

I need some help with extracting date from NetCDF files using R , I downloaded them from cordex (The Coordinated Regional climate Downscaling Experiment). In total I have some files. This files have dimensions of (longitude, latitude, time) and the variable is maximum temperature (tasmax). At specific location, I need to extract data of tasmax at different time. In total I have some files. This files have dimensions of (longitude, latitude, time) and variable maximum temperature (tasmax). At specific location, I need to extract data of tasmax at different time.I wrote the code using R but at the end of code, an error appeared. Error ( location subscript out of bounds)

getwd() setwd("C:/Users/20120/climate change/rcp4.5/tasmax")

dir()
library ("ncdf4")
libra,-ry(ncdf4.helpers)
library ("chron")
ncin <- nc_open("tasmax_AFR-44_ICHEC-EC-EARTH_rcp45_r1i1p1_KNMI-RACMO22T_v1_mon_200601-201012.nc")

lat <- ncvar_get(ncin, "lat")

lon <- ncvar_get(ncin, "lon")

tori <- ncvar_get(ncin, "time")
title <- ncatt_get(ncin,0,"title")
institution <- ncatt_get(ncin,0,"institution") 
datasource <- ncatt_get(ncin,0,"source") 
references <- ncatt_get(ncin,0,"references")
history <- ncatt_get(ncin,0,"history") 
Conventions <- ncatt_get(ncin,0,"Conventions") 
tustr <- strsplit(tunits$value,"")
ncin$dim$time$units
ncin$dim$time$calendar
tas_time <- nc.get.time.series(ncin, v = "tasmax",
                           time.dim.name = "time")
tas_time[c(1:3, length(tas_time) - 2:0)]
tmp.array <- ncvar_get(ncin,"tasmax")
dunits <- ncatt_get(ncin,"tasmax","units")
tmp.array <- tmp.array-273.15
tunits <- ncatt_get(ncin,"time","units")
nc_close(ncin)
which.min(abs(lat-28.9))
which.min(abs(lon-30.2))
tmp.slice <- tmp.array[126,32981,]
tmp.slice

Error in tmp.array[126, 32981, ] : subscript out of bounds

How to bring data frame into single column from multiple columns in python

I have data format in these multiple columns. So I want to bring all 4 columns of data into a single column.

YEAR Month pcp1 pcp2 pcp3 pcp4              
1984    1   0   0   0   0
1984    2   1.2 0   0   0
1984    3   0   0   0   0
1984    4   0   0   0   0
1984    5   0   0   0   0
1984    6   0   0   0   1.6
1984    7   3   3   9.2 3.2
1984    8   6.2 27.1    5.4 0
1984    9   0   0   0   0
1984    10  0   0   0   0
1984    11  0   0   0   0
1984    12  0   0   0   0

separate datetime column in R while keeping time accurate

4/12/2016 12:00:00 AM I have dates in the format above and have tried to use separate() to create two columns in the data frame where the data is present. When I do the columns are created but AM/PM so the times just become numbers or worse appear as "12H 0M 0S". Can anyone help me out, pretty new to data analysis as a whole and would be much appreciated!

How do I implement rank function for nearest values for a column in dataframe?

df.head():

                      run_time match_datetime         country                league             home_team                  away_team
0   2021-08-07 00:04:36.326391     2021-08-06          Russia       FNL 2 - Group 2             Yenisey 2          Lokomotiv-Kazanka
1   2021-08-07 00:04:36.326391     2021-08-07          Russia          Youth League              Ural U19  Krylya Sovetov Samara U19
2   2021-08-07 00:04:36.326391     2021-08-08           World         Club Friendly                Alaves                    Al Nasr
3   2021-08-07 00:04:36.326391     2021-08-09           China            Jia League     Chengdu Rongcheng          Shenyang Urban FC
4   2021-08-06 00:04:36.326391     2021-08-06           China          Super League              Wuhan FC       Tianjin Jinmen Tiger
5   2021-08-06 00:04:36.326391     2021-08-07  Czech Republic            U19 League     Sigma Olomouc U19                Karvina U19
6   2021-08-06 00:04:36.326391     2021-08-08          Russia          Youth League  Konoplev Academy U19            Rubin Kazan U19
7   2021-08-06 00:04:36.326391     2021-08-09           World         Club Friendly         Real Sociedad                      Eibar

desired df

                      run_time match_datetime         country                league             home_team                  away_team
0   2021-08-07 00:04:36.326391     2021-08-06          Russia       FNL 2 - Group 2             Yenisey 2          Lokomotiv-Kazanka
1   2021-08-07 00:04:36.326391     2021-08-07          Russia          Youth League              Ural U19  Krylya Sovetov Samara U19
4   2021-08-06 00:04:36.326391     2021-08-06           China          Super League              Wuhan FC       Tianjin Jinmen Tiger
5   2021-08-06 00:04:36.326391     2021-08-07  Czech Republic            U19 League     Sigma Olomouc U19                Karvina U19

How do i use rank function to filter only the 2 nearest match_datetime dates for every run_time value. i.e. desired dataframe will be a filtered dataframe that will have all the nearest 2 match_datetime values for every run_time

ValueError: All arrays must be of the same length when scraping

I try to input different zip codes and scrape information for Target products. However, results in this error:ValueError: All arrays must be of the same length and there is nothing in my CSV file. I guess because I did not successfully scrap e all the information. Can anyone give me some suggestions on how to improve the code? I appreciate any help. Thanks.

Following is my code:

#Target Url list
urlList = [
    'https://www.target.com/p/pataday-once-daily-relief-extra-strength-drops-0-085-fl-oz/-/A-83775159?preselect=81887758#lnk=sametab',
    'https://www.target.com/p/kleenex-ultra-soft-facial-tissue/-/A-84780536?preselect=12964744#lnk=sametab',
    'https://www.target.com/p/claritin-24-hour-non-drowsy-allergy-relief-tablets-loratadine/-/A-80354268?preselect=14351285#lnk=sametab',
    'https://www.target.com/p/opti-free-pure-moist-rewetting-drops-0-4-fl-oz/-/A-14358641#lnk=sametab',
    'https://www.target.com/p/allegra-24-hour-allergy-relief-tablets-fexofenadine-hydrochloride/-/A-15068699?preselect=14042732#lnk=sametab',
    'https://www.target.com/p/nasacort-allergy-relief-spray-triamcinolone-acetonide/-/A-15143450?preselect=15503329#lnk=sametab',
    'https://www.target.com/p/genexa-dextromethorphan-kids-39-cough-and-chest-congestion-suppressant-4-fl-oz/-/A-80130848#lnk=sametab',
    'https://www.target.com/p/zyrtec-24-hour-allergy-relief-tablets-cetirizine-hcl/-/A-15075280?preselect=79847258#lnk=sametab',
    'https://www.target.com/p/pataday-twice-daily-eye-allergy-itch-and-redness-relief-drops-0-17-fl-oz/-/A-78780978#lnk=sametab',
    'https://www.target.com/p/systane-gel-drops-lubricant-eye-gel-0-33-fl-oz/-/A-14523072#lnk=sametab']


zipCodeList = [3911,4075,4467,96970,96960,49220,49221,49224,48001,49227,48101,48002,48003,48004]

while(True):
    priceArray = []
    nameArray = []
    zipCodeArray =[]
    GMTArray = []
    TCIN = []
    UPC = []

    def ScrapingTarget(url):
        wait_imp = 10
        CO = webdriver.ChromeOptions()
        CO.add_experimental_option('useAutomationExtension', False)
        CO.add_argument('--ignore-certificate-errors')
        CO.add_argument('--start-maximized')
        wd = webdriver.Chrome(r'D:\chromedriver\chromedriver_win32new\chromedriver_win32 (2)\chromedriver.exe',options=CO)
        wd.get(url)
        wd.implicitly_wait(wait_imp)
        # needed to click onto the "Show more" to get the tcin and upc
        xpath = '//*[@id="tabContent-tab-Details"]/div/button'
        element_present = EC.presence_of_element_located((By.XPATH, xpath))
        WebDriverWait(wd, 5).until(element_present)
        showMore = wd.find_element(by=By.XPATH, value=xpath)
        sleep(3)
        showMore.click()
        # showMore = wd.find_element(by=By.XPATH, value="//*[@id='tabContent-tab-Details']/div/button")
        # sleep(2)
        #showMore.click()
        soup = BeautifulSoup(wd.page_source, 'html.parser')
        # gets a list of all elements under "Specifications"
        try:
            # gets a list of all elements under "Specifications"
            div = soup.find("div", {"class": "styles__StyledCol-sc-ct8kx6-0 iKGdHS h-padding-h-tight"})
            list = div.find_all("div")
            for a in range(len(list)):
                list[a] = list[a].text
            # locates the elements in the list
            tcin = [v for v in list if v.startswith("TCIN")]
            upc = [v for v in list if v.startswith("UPC")]
        except:
            tcin = "Error"
            upc = "Error"
        TCIN.append(tcin)
        UPC.append(upc)

        for zipcode in zipCodeList:
            try:
                #click the delivery address
                address = wd.find_element(by=By.XPATH, value="//*[@id='pageBodyContainer']/div[1]/div[2]/div[2]/div/div[4]/div/div[1]/button[2]")
                address.click()
                #click the Edit location
                editLocation = wd.find_element(by=By.XPATH, value="//*[@id='pageBodyContainer']/div[1]/div[2]/div[2]/div/div[4]/div/div[2]/button")
                editLocation.click()
            except:
                #directly click he Edit location
                editLocation = wd.find_element(by=By.XPATH, value="//*[@id='pageBodyContainer']/div[1]/div[2]/div[2]/div/div[4]/div[1]/div/div[1]/button")
                editLocation.click()

            #input ZipCode
            inputZipCode = wd.find_element(by=By.XPATH, value="//*[@id='enter-zip-or-city-state']")
            inputZipCode.clear()
            inputZipCode.send_keys(zipcode)

            #click submit
            clickSubmit = wd.find_element(by=By.XPATH, value="//*[@id='pageBodyContainer']/div[1]/div[2]/div[2]/div/div[4]/div/div[2]/div/div/div[3]/div/button[1]")
            clickSubmit.click()

            #start scraping
            name = wd.find_element(by=By.XPATH, value="//*[@id='pageBodyContainer']/div[1]/div[1]/h1/span").text
            nameArray.append(name)
            price = wd.find_element(by=By.XPATH, value="//*[@id='pageBodyContainer']/div[1]/div[2]/div[2]/div/div[1]/div[1]/span").text
            priceArray.append(price)
            currentZipCode = zipcode
            zipCodeArray.append(currentZipCode)
            tz = pytz.timezone('Europe/London')
            GMT = datetime.now(tz)
            GMTArray.append(GMT)


    with concurrent.futures.ThreadPoolExecutor() as executor:
         executor.map(ScrapingTarget, urlList)

    data = {'prod-name': nameArray,
            'Price': priceArray,
            'currentZipCode': zipCodeArray,
            "Tcin": TCIN,
            "UPC":UPC,
            "GMT": GMTArray
            }
    df = pd.DataFrame(data, columns= ['prod-name', 'Price','currentZipCode',"Tcin","UPC","GMT"])
    df.to_csv(r'C:\Users\12987\PycharmProjects\python\Network\priceingAlgoriCoding\export_Target_dataframe.csv', mode='a', index = False, header=True)
    sleep(20)

Python: I have a list of dictionaries but when I try to call a specific Key and Value it is throwing a error

def readParksFile(fileName="national_parks(1).csv"):
    nationalParks = open(fileName)
    headers = nationalParks.readline()
    keys = headers.split(",")
    numKeys = len(keys)
    parksList = []
    values = nationalParks.readlines()
    rowsList = []
    parksDictionary = {}
    for row in values:
        rowsList.append(row.split(","))
    for item in rowsList:
        parksDictionary = {}
        for i in range(numKeys):
            parksDictionary[keys[i]] = item[i]
        parksList.append(parksDictionary)
    for i in range(len(parksList)):
        return(parksList[i])

    nationalParks.close()

I created a list of dictionaries using the code above

def printParksInState(parksList):
    state = getStateAbbr()
    for parksDictionary in parksList:
        if state in parksDictionary["State"]:
            print(parksDictionary["Name"] + " (" + parksDictionary["Code"] + ")")
            print("\t" + "Location:" + parksDictionary["State"])
            print("\t" + "Area:" + parksDictionary["Acres"] + " acres")
            print("\t" + "Date Established:" + tasks.convertDate(parksDictionary["Date"]))
        else:
            print("There are no national parks in " + state + " or it is not a valid state")

I have functions that uses the list of dictionaries to print information

def main():
    print("National Parks")

    parksList = tasks.readParksFile()
    menuDict = interface.getMenuDict()

    choice = ("")
    while choice != "Q":
        print(interface.displayMenu(menuDict))
        choice = interface.getUserChoice(menuDict)
        if choice == "A":
            interface.printAllParks(parksList)
        elif choice == "B":
            interface.printParksInState(parksList)
        elif choice == "C":
            interface.printLargestPark(parksList)
        elif choice == "D":
            interface.printParksForSearch(parksList)
        else:
            print("This is not an option")

In my main function I call the other functions that uses the list of dictionaries

Error However no matter what function I call it throws the error that string indices must be integers and I am not sure what this error is or how to fix it. Please help!

Support with R software

I am an early career student researcher and carrying out a meta-analysis on the incidence and prevalence of ear disease (proportion meta-analysis). I have downloaded R and R studio. I was able to import the data. I need further help. Are there any videos on the basics of R in relation to proportion meta-analysis? OR any written text OR website OR anyone brave enough to show me live? any help or comment is highly appreciated.
RStudioServer - load/upgrade preferred set of packages to default sys path (/opt/R/4.2.0/lib/R/library) from R shell
Though an 'old guy' in unix world, I've little experience with RStudioServer and R, acting here as an 'admin' for a shared server for statisticians on a cancer research study. And my unix 'admin' experience is ... rusty !

I have a list of about 100 packages used by our senior Statistician on the study, and would prefer to load (AND UPDATE PERIODICALLY) them from an R shell directly (vs inside RStudioServer). I think using the default system location to add them ( /opt/R/4.2.0/lib/R/library) is the way to go. The RStudioServer setup, i think, still allows user to load their own (which is fine by me), but I thought having a default group loaded per our Statistician's ask... is wise.

The commands I was given which load these packages have the syntax below (only including a few lines of the 90 line .sh script) - they seem to install if the package doesnt already exist.

SO... I wonder if I set an environment (or ?) variable properly, then run R (as root ?), issue all these commands... would the libraries be placed appropriately in the library folder(for 4.2) that I set in the env variable ? (this example, it'd be /opt/R/4.2.0/lib/R/library ) ??

Since this is Linux, we would want sources when approprite, which i THINK is default loading for this os.

If there is a better/easier way to do this, I'm all ears !

Thanks in advance for any thoughts you have...

r
```
if (!requireNamespace("markdown", quietly = TRUE)) install.packages("markdown")
if (!requireNamespace("devtools", quietly = TRUE)) install.packages("devtools")
# graphical/table output
if (!requireNamespace("igraph", quietly = TRUE)) install.packages("igraph")
if (!requireNamespace("ggplot2", quietly = TRUE)) install.packages("ggplot2")
# advanced regression
if (!requireNamespace("glmnet", quietly = TRUE)) install.packages("glmnet")
if (!requireNamespace("sm", quietly = TRUE)) install.packages("sm")
```
RStudioServer packageStatus()

environment: CentOS 8 RStudio Server 1.4.1717-3 R 4.2 (4.1.3, 3.6.3)

Related questions

R Studio keeps crashing when I'm trying to merge multiple csv files into a data frame. How do I fix this?

See also questions close to this topic