Problems with bimodal variable: lmers give boundary is singular, inverse gaussian glmers?

See also questions close to this topic

• How to make gaussian distribution in pytorch?

  I want to make probability like

  torch.bernoulli() 
  

  this function. Has any solution?
• Pdf of product of two i.i.d random variable distributed as circular symmetric gaussian

  what is a distribution of the product of two circular symmetric Gaussian random variables?
• How to draw a plot with srveral Gaussuans without any data in Python?

  So, I wanna do a spectrum with 4 lines described as gaussian fuctions. I know only their x and y and know how it's supposed to look, but have no data. How do I do a dataset for this plot so that would be arrays for x and y axes? The plot has to look like that: enter image description here
• R blank/NA ouput of ANOVA when comapring 2 GLMER function

  I'm trying to compare 2 GLMER functions (negative binomial distribution) which differ only in an interaction term they have. However, the ANOVA results show blank space for all the parameters other than DF. when I checked the summary of the ANOVA it showed that all the parameters were filled with 'NA'. ANOVA of each function separately gives normal results. I am deeply puzzled about why this is happening/and how to solve this.

  mod_survival_base <- glmer(log(1+x)~y+z + (1|A/B/C),
                     family=nbinom2, data = mortality4, 
                     control = glmerControl(optimizer = "bobyqa", optCtrl = list(maxfun=2e5)))
  mod_survival_inter <- glmer(log(1+x)~y*z + (1|A/B/C),
                     family=nbinom2, data = mortality4, 
                     control = glmerControl(optimizer = "bobyqa", optCtrl =list(maxfun=2e5)))
  
  anova(mod_survival_inter, mod_survival_base)
  
• How to save plots in a loop to different pdfs in R (for lmer plots)?

  in each itteration of a loop I want to save my plots into a pdf. I have found some good code examples (e.g. here), but for some reason it doesn´t work in my case. What have I been doing wrong?

  I did the following:

  save_as = "xxxx.pdf"
  
  # dataframe 
  Var1 <- c(1.0, 1.4, 2.0, 0.7, 0.8, 1.3, 1.6, 0.9, 0.5)
  Var2 <- c(100, 99, 320, 219, 280, 299, 210, 200, 190)
  Var3 <- c(10, 1, 4, 8, 19, 20, 7, 8, 2)
  indep1 <- c(10, 11, 14, 25, 23, 21, 33, 11, 14)
  indep2 <- c("species1", "species1", "species2", "species3", "species1", "species3", "species2", "species2", "species2")
  
  df <- data.frame(Var1, Var2, Var3, indep1, indep2)
  
  # dependent variables
  dep_vars = c("Var1", "Var2", "Var3")
  
  for (i in dep_vars){  
    ## model specification
    model1 = lmer(eval(parse(text=paste("df$", i, sep = ""))) ~ df$indep1 + (1|df$indep2))
    model2 = lmer(eval(parse(text=paste("df$", i, sep = ""))) ~ (1|df$indep2))
  
    # open pdf
    new_save_as = gsub('xxxx', i, save_as)
    pdf(new_save_as)
  
    # put plots in 
    par(mfrow = c(1,2))
  
    plot(model1, type=c("p", "smooth"), col.line=1)
    plot(model2, type=c("p", "smooth"), col.line=1)
    
    # close file
    dev.off()
  }
  

  If I run this without the loop, the file is created as I want it. But, if I use the loop the pdf is created but stays empty. Does anyone have an idea why?

  Also, any other comments on my code are welcome, as I am new to R.

  Thanks!
• Dealing with confounded effects in interactions, in R

  I am doing a regression analysis of hunting and wind power turbine data.

  My response variable is hunting harvest density, and a set of wind turbine variables: height, number of turbines, distance to turbines and establishment phase (pre-construction, construction and operative.

  examplemodel <- lme(harvest ~ WP_distance + WP_height + phase + phase:WP_distance + phase:WP_turbines, weight = ~ I(1/Area), data = mydata, random =~ 1 | randomvar, method="ML")

  In pre-construction phase, height, turbines and distance is set as zero (not built yet).

  When conducting linear mixed effects, i try to include the turbine variables and interactions phase:distance, phase:height, phase:turbines. However, when doing this i get error:

  Error in MEEM(object, conLin, control$niterEM) : Singularity in backsolve at level 0, block 1

  I figure it's because the variables phase and the wp-variables get confounded in this way, but how could i possibly deal with it otherwise? I want to display that the height, number of turbines and distance to the wind power parks depend on the construction phase.

  Thank you!
• remove outliers from multiple columns in R

  I used below codes to identify outliers on different columns:

  outliers_x1 <- boxplot(mydata$x1, plot=FALSE)$out
  outliers_x4 <- boxplot(mydata$x4, plot=FALSE)$out
  outliers_x6 <- boxplot(mydata$x6, plot=FALSE)$out
  

  Now, how can I remove those outliers from the dataset by one code?
• Is it possible to filter out outliers that overlap based on time variable in R?

  How do I delete outliers by checking for an overlap between ID column and date/time columns withfilter()?

  For example, rows with ID =1 overlap in time as shown in the first 2 rows below,thus need to be deleted.

  ID	Time start	Time end
  1	2015-03-16 10:40:00	2015-03-16 11:10:00
  1	2015-03-16 10:50:00	2015-03-16 10:59:00
  2	2015-03-16 10:40:00	2015-03-16 10:45:00
  1	2015-03-16 11:20:00	2015-03-16 11:28:56
• ipython-input-9-69155119dc7e>:1: FutureWarning: Automatic reindexing on DataFrame

  ipython-input-9-69155119dc7e>:1: FutureWarning: Automatic reindexing on DataFrame vs Series comparisons is deprecated and will raise ValueError in a future version. Do left, right = left.align(right, axis=1, copy=False) before e.g. left == right

  i get this error trying to remove outliers by using quantiles.

  here's the code i wrote :

  df99 = df.price.quantile(0.99) df001 = df.price.quantile(0.01)

  max_thresold= df[df.price>df99] min_thresold= df[df.price<df001]

  df2 = df[(df.price>min_thresold)&(df.price<max_thresold)] df2.shape

  i don't get it ? thanks
• how to run a config file for boundary by hashicorp

  Tried this command boundary dev -config file name

  encountered the following error: flag provided but not defined: -config
• Unity Drag and Drop Canvas Boundary

  What I want is simple, I already have the drag and drop mechanic done. All I want to know is how I can set boundaries so that I can't drag and drop any UI images out of the canvas.
• How can I use an SVM to seperate two classes in a 2D map?

  I have 2d-numpy array of points defining a map with obstacles Image of the 2D Map of which I am trying to create an SVM with. The values of this 2d-array are either 0 for open space (grey) or 1,-1 for obstacles (black/white). The formatting of the map is shown below:

  [[ 1.  1.  1. ...  1.  1.  1.]
   [ 1.  0.  0. ...  0.  0.  1.]
   [ 1.  0.  0. ...  0.  0.  1.]
   ...
   [ 1.  0.  0. ...  0.  0.  1.]
   [ 1.  0.  0. ...  0.  0.  1.]
   [ 1.  1. -1. ...  1.  1.  1.]]
  

  My goal is to use a Support Vector Machine (SVM) to create a line/decision boundary between the obstacles. Since the obstacles are two separate classes, it should be possible to generate an SVM separating them. The ideal SVM boundary would look something like this.

  What I am unsure of us how to transform my data to be in the format shown in the sklearn SVM example. I need to be able to specify the features, X and the target, y, so I can train the SVM by calling something like svm.SVC(kernel='linear', C=C).fit(X, y) as shown in the example.

  In summary: how to convert my map (a 2-d array of either -1, 1, 0) into a set of features and targets?
• Invertibility / singularity of matrix resulting from a symmetric(0,1) matrix pointwise multiplicated with a vector (consisting of a,b)+identity matrix

  I am trying to find conditions then a certain matrix is invertible or not (which is problematic as the matrix is random). The matrix results from the following: $A=\Tilde{A}+diag(n)$. Furthermore $\Tilde{A}$ results from the pointwise multiplication of a random symmetric matrix (consisting of 0 and 1, but necessarily 0 on the diagonal) with a random vector which constsis of $\alpha$ and $\beta$ entries.

  Does anyone have any ideas how to deduce some criterions for the invertibility of matrix $A$?

  Thank you so much!

  I already tried thinking about LU decomoposition, but could not deduce any criterion. Obviosly, it fully depends on how the random matrices look and linear dependence between the rows are less likely if the dimension is higher...
• fixed-effect model matrix is rank deficient so dropping 27 columns / coefficients AND boundary (singular)

  Study background: I want to see if mean caterpillar abundance for a given year and within a given population, can explain differences in bird(blue tit) density. The blue tits breed in nest boxes, so I calculated density as the total number of occupied nest boxes/the number of unoccupied nest boxes within a given year and population.

  Below I show the structure of the data, my model and the error messages.

  Model:

  model1.1 <-glmer(cbind(data.density$number.nest.boxes.occupied.that.year,
            data.density$number_of_nest.boxes)~population*year*caterpillar+(1|site),
     data = data.density, family=binomial)
  

  The first error is:

  fixed-effect model matrix is rank deficient so dropping 27 columns / coefficients

  I think this this is due to not having enough combinations of caterpillars with population x year.

  The second error is

  boundary (singular) fit: see ?isSingular

  I'm just not sure how to go about fixing this. I also don't understand what the other error means and how to fix it.

  I appreciate any advice.

  #loading data density data
  
  data.density<-read.csv ("nest_box_caterpillar_density.csv") 
  View(data.density)
  
  
  str(data.density)
  #> 'data.frame':    63 obs. of  8 variables:
  #>  $ year                                : int  2011 2012 2013 2014 2015 2016 2017 2018 2019 2011 ...
  #>  $ number.nest.boxes.occupied.that.year: int  17 13 12 16 16 16 15 17 12 17 ...
  #>  $ number_of_nest.boxes                : int  20 20 20 20 20 20 20 20 20 30 ...
  #>  $ proportion_occupied_boxes           : num  0.85 0.65 0.6 0.8 0.8 0.8 0.75 0.85 0.6 0.57 ...
  #>  $ site                                : Factor w/ 7 levels "ari","ava","fel",..: 5 5 5 5 5 5 5 5 5 1 ...
  #>  $ population                          : Factor w/ 3 levels "D-Muro","E-Muro",..: 2 2 2 2 2 2 2 2 2 2 ...
  #>  $ mean_yearly_frass                   : num  295 231 437 263 426 ...
  #>  $ site_ID                             : Factor w/ 63 levels "2011_ari_","2011_ava_",..: 5 12 19 26 33 40 47 54 61 1 ...
  
  data.density$year<-factor (data.density$year)# making year a factor (categorical variable)
  
  str(data.density) # now we see year as a factor in the data. 
  #> 'data.frame':    63 obs. of  8 variables:
  #>  $ year                                : Factor w/ 9 levels "2011","2012",..: 1 2 3 4 5 6 7 8 9 1 ...
  #>  $ number.nest.boxes.occupied.that.year: int  17 13 12 16 16 16 15 17 12 17 ...
  #>  $ number_of_nest.boxes                : int  20 20 20 20 20 20 20 20 20 30 ...
  #>  $ proportion_occupied_boxes           : num  0.85 0.65 0.6 0.8 0.8 0.8 0.75 0.85 0.6 0.57 ...
  #>  $ site                                : Factor w/ 7 levels "ari","ava","fel",..: 5 5 5 5 5 5 5 5 5 1 ...
  #>  $ population                          : Factor w/ 3 levels "D-Muro","E-Muro",..: 2 2 2 2 2 2 2 2 2 2 ...
  #>  $ mean_yearly_frass                   : num  295 231 437 263 426 ...
  #>  $ site_ID                             : Factor w/ 63 levels "2011_ari_","2011_ava_",..: 5 12 19 26 33 40 47 54 61 1 ...
  
  
  
  density<-data.density$proportion_occupied_boxes # making a new object called density
  caterpillar<-data.density$mean_yearly_frass # making new object called caterpillar
  
  model1.1<-glmer(cbind(data.density$number.nest.boxes.occupied.that.year,data.density$number_of_nest.boxes)~population*year*caterpillar+(1|site),data = data.density, family=binomial)  
  #> fixed-effect model matrix is rank deficient so dropping 27 columns / coefficients
  #> boundary (singular) fit: see ?isSingular
  
  summary(model1.1)
  #> Generalized linear mixed model fit by maximum likelihood (Laplace
  #>   Approximation) [glmerMod]
  #>  Family: binomial  ( logit )
  #> Formula: 
  #> cbind(data.density$number.nest.boxes.occupied.that.year, data.density$number_of_nest.boxes) ~  
  #>     population * year * caterpillar + (1 | site)
  #>    Data: data.density
  #> 
  #>      AIC      BIC   logLik deviance df.resid 
  #>    343.7    403.7   -143.8    287.7       35 
  #> 
  #> Scaled residuals: 
  #>     Min      1Q  Median      3Q     Max 
  #> -1.1125 -0.1379  0.0000  0.2264  0.6778 
  #> 
  #> Random effects:
  #>  Groups Name        Variance Std.Dev.
  #>  site   (Intercept) 0        0       
  #> Number of obs: 63, groups:  site, 7
  #> 
  #> Fixed effects:
  #>                              Estimate Std. Error z value Pr(>|z|)  
  #> (Intercept)                -0.4054532  0.2454754  -1.652   0.0986 .
  #> populationE-Muro           -0.1158123  0.2301030  -0.503   0.6147  
  #> populationE-Pirio          -0.4945158  0.2932707  -1.686   0.0918 .
  #> year2012                    0.0905137  0.2109513   0.429   0.6679  
  #> year2013                   -0.1223076  0.2160367  -0.566   0.5713  
  #> year2014                   -0.0703760  0.2304236  -0.305   0.7600  
  #> year2015                   -0.0507882  0.2127083  -0.239   0.8113  
  #> year2016                   -0.0562139  0.2077616  -0.271   0.7867  
  #> year2017                   -0.0994962  0.2070464  -0.481   0.6308  
  #> year2018                    0.0977751  0.2192755   0.446   0.6557  
  #> year2019                   -0.2312869  0.2133430  -1.084   0.2783  
  #> caterpillar                 0.0004598  0.0005432   0.846   0.3973  
  #> populationE-Muro:year2012  -0.1217344  0.3294773  -0.369   0.7118  
  #> populationE-Pirio:year2012 -0.3121173  0.2912256  -1.072   0.2838  
  #> populationE-Muro:year2013  -0.0682892  0.3600992  -0.190   0.8496  
  #> populationE-Pirio:year2013 -0.3345701  0.3051039  -1.097   0.2728  
  #> populationE-Muro:year2014   0.1604636  0.3383121   0.474   0.6353  
  #> populationE-Pirio:year2014 -0.1074231  0.3171972  -0.339   0.7349  
  #> populationE-Muro:year2015   0.0838557  0.3491699   0.240   0.8102  
  #> populationE-Pirio:year2015 -0.0640988  0.2943189  -0.218   0.8276  
  #> populationE-Muro:year2016   0.0679017  0.3333771   0.204   0.8386  
  #> populationE-Pirio:year2016 -0.0899343  0.2919975  -0.308   0.7581  
  #> populationE-Muro:year2017   0.1643493  0.3300491   0.498   0.6185  
  #> populationE-Pirio:year2017  0.0338824  0.2730344   0.124   0.9012  
  #> populationE-Muro:year2018   0.0315607  0.3264224   0.097   0.9230  
  #> populationE-Pirio:year2018 -0.4196974  0.3180515  -1.320   0.1870  
  #> populationE-Muro:year2019  -0.0587593  0.3619408  -0.162   0.8710  
  #> ---
  #> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
  #> 
  #> Correlation matrix not shown by default, as p = 27 > 12.
  #> Use print(x, correlation=TRUE)  or
  #>     vcov(x)        if you need it
  #> fit warnings:
  #> fixed-effect model matrix is rank deficient so dropping 27 columns / coefficients
  #> optimizer (Nelder_Mead) convergence code: 0 (OK)
  #> boundary (singular) fit: see ?isSingular
  

  ^{Created on 2022-03-21 by the reprex package (v2.0.1)}

  I tried removing caterpillars from the model, and the error first error goes away. But the point of my model is to see how caterpillar effects density. I also still get the "boundary (singular) fit: see ?isSingular" error.
• Singular in MuMIn::dredge

  I'm about to run a glm model to test my hypothesis. Once I ran the model I got a singularity warning massage: glmakde_mdDredge <- lmer(log(akdearea) ~ 1 + age_binary + sex + trimester_number + Year + trackNightnum + mean_max_displacement.s + meanIIV_md.s + (1|Ring_ID), data = HRfullData, REML = F)

  boundary (singular) fit: see ?isSingular
  

  I ran MuMIn::dredge on this model

  options(na.action = 'na.fail')
  dreModelsScore <- MuMIn::dredge(glmakde_mdDredge)
  
  
  Fixed term is "(Intercept)"
  boundary (singular) fit: see ?isSingular
  boundary (singular) fit: see ?isSingular
  boundary (singular) fit: see ?isSingular
  

  Summary did give results.

  My question is whether despite the warning messages the model ran all the possible options (including the singularity ones) or only on the models where there is no singularity problem?